Predictive Video Coding Employing Virtual Reference Frames Generated by Direct MV Projection (DMVP)

Abstract

Techniques are disclosed for generating a virtual reference frame that can be used for prediction of an input video frame. The virtual reference frame may be derived from an already coded reference frame, thereby incurring reduced signaling overhead. Furthermore, signaling of the virtual reference frame may be avoided until the encoder selects the virtual reference frame as a prediction reference for the current frame. In this way, the techniques proposed herein contribute to improving coding efficiency.

Description

[Background technology]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of 35 U.S.C. 119(e) of U.S. patent application Ser. No. 63 / 305,111, filed January 31, 2022, and U.S. patent application Ser. No. 63 / 331,469, filed April 15, 2022, the contents of which are incorporated by reference herein in their entireties.

[0002] FIELD This disclosure relates to video coding, and in particular to video coding that employs predictive coding techniques.

[0003] In many video applications, strong temporal correlation may exist between successive frames. This is especially true when pictures in a video sequence are densely sampled (e.g., captured quickly) and the temporal distance between two successively sampled frames is small. To remove temporal redundancy and improve video compression efficiency, existing video coding systems and standards often utilize "motion compensated prediction," which codes the content of one frame (the "current" frame) by searching for matching content in previously coded frames. Many coding systems follow published coding standards. For complexity and implementation purposes, many standards utilize rectangular blocks of various sizes for prediction. Such blocks (conveniently referred to as "pixel blocks") may be organized using flexible and adaptive tree structures, such as binary, ternary, and quad-tree structures, or combinations thereof, for efficient coding.

[0004] Several strategies for prediction have been explored. Inter prediction (including unidirectional prediction, bidirectional prediction, bi-predictive prediction, and multiple hypothesis prediction), for example, exploits the temporal correlation between previously coded frames (commonly called "reference frames") and the current frame. The encoder may perform a motion search on available reference frames to find one or more matching blocks that can best predict a block in the current frame. Such blocks are commonly called inter prediction hypotheses. In most coding standards and coding systems, up to two such hypotheses from these reference frames may be mixed together to form a final prediction for the current block. Inter prediction can largely remove the temporal correlation and redundancy between the reference frames and the current frame, making the coding of the current frame less costly / more efficient compared to the use of intra prediction methods. However, these techniques often incur significant signaling overhead that may reduce coding efficiency. [Brief description of the drawings]

[0005] [Figure 1] FIG. 1 is a block diagram of a video coding system according to one embodiment of the present disclosure.

[0006] [Diagram 2] 1 is a functional diagram of a video coding system according to an embodiment of the present invention;

[0007] [Diagram 3] 1 illustrates an example virtual reference frame and its relationship to an exemplary reference frame according to one embodiment of the present invention.

[0008] [Figure 4] 1 illustrates a method for generating a virtual reference frame according to an embodiment of the present disclosure.

[0009] [Diagram 5]13(a) and 13(b) show an example application of the translational motion technique for motion projection according to an embodiment of the present disclosure.

[0010] [Figure 6] 1 illustrates a method for performing affine motion vector projection according to one embodiment of the present disclosure.

[0011] [Figure 7] 7 illustrates an exemplary set of pixel blocks on which the method of FIG. 6 may operate.

[0012] [Figure 8] 1 illustrates an example concept of derivation of a virtual reference frame according to an embodiment of the present disclosure.

[0013] [Figure 9] 1 illustrates an example of a hole in which embodiments of the present disclosure can operate.

[0014] [Figure 10] 1 illustrates a conceptual example of a warping process according to an embodiment of the present disclosure.

[0015] [Figure 11] 1 illustrates an application of motion vector prediction according to one embodiment of the present disclosure.

[0016] [Figure 12] 13 illustrates an application of motion vector prediction according to another embodiment of the present disclosure.

[0017] [Figure 13] 4 illustrates an application of motion vector prediction according to a further embodiment of the present disclosure.

[0018] [Figure 14] 13 illustrates an application of motion vector prediction according to another embodiment of the present disclosure.

[0019] [Figure 15] 4 illustrates an application of motion vector prediction according to a further embodiment of the present disclosure.

[0020] [Figure 16] 13 illustrates an application of motion vector prediction according to another embodiment of the present disclosure.

[0021] [Figure 17] FIG. 17 is a functional block diagram of a coding system 1700 according to one embodiment of the present disclosure.

[0022] [Figure 18] FIG. 18 is a functional block diagram of a decoding system 1800 according to one embodiment of the present disclosure.

[0023] [Figure 19] 1 illustrates a method for signaling coding information according to one embodiment of the present disclosure.

[0024] [Figure 20] 1 illustrates a method for generating a TIP frame according to one embodiment of the present disclosure.

[0025] [Figure 21] 1 illustrates an example coding scenario according to an embodiment of the present disclosure.

[0026] [Figure 22] 1 illustrates a method for coding video according to one embodiment of the present disclosure.

[0027] [Figure 23] 1 illustrates an example coding scenario according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] The embodiments of the present disclosure provide techniques for generating a virtual reference frame that can be used for prediction of an input video frame. The virtual reference frame may be derived from an already coded reference frame, thereby incurring reduced signaling overhead. Furthermore, signaling of the virtual reference frame may be avoided until the encoder selects the virtual reference frame as a prediction reference for the current frame. In this way, the techniques proposed herein contribute to improving coding efficiency.

[0029] FIG. 1 is a block diagram of a video coding system according to an embodiment of the present disclosure. The system 100 may include multiple terminals 110, 120 arranged in mutual communication over a network 130. The terminals 110, 120 may encode video data for transmission to a destination over the network 130. Thus, a first terminal 110 may capture video data locally, encode the video data, and transmit the encoded video data to a destination terminal 120 over the network 130. The receiving terminal 120 may receive the encoded video data, decode, and render it locally, e.g., on a display of the terminal 120. If the terminals are engaged in a bidirectional exchange of video data, the terminal 120 may capture video data locally, encode the video data, and transmit the encoded video data to the destination terminal 110 over the network 130. The receiving terminal 110 may receive the encoded video data transmitted from the terminal 120, decode, and render it locally, e.g., on its own display.

[0030] The video coding system 100 can be used in a variety of applications. In a first application, the terminals 110, 120 can support real-time two-way exchange of coded video and establish video conference sessions between them. In another application, the terminal 110 can encode pre-generated video (e.g., television or movie programs) and store the encoded video for distribution to one or, more likely, many downloading clients (e.g., the terminal 120). Thus, the encoded video can be live or pre-generated, and the terminal 110 can function as a media server that distributes the encoded video according to a one-to-one or one-to-many distribution model. For the purposes of this discussion, the type of video and video distribution scheme is not important unless otherwise specified.

[0031] 1, the terminals 110, 120 are depicted as smartphones, but the principles of the present disclosure are not limited thereto. Aspects of the present disclosure also apply to various types of computers (desktop, laptop, and tablet computers), computer servers, media players, gaming systems, dedicated video conferencing equipment, and / or dedicated video encoding equipment.

[0032] Network 130 represents any number of networks that convey encoded video data between terminals 110, 120, including, for example, wired and / or wireless communication networks. Communication network 130 may exchange data over circuit-switched or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. In some applications involving one-way video distribution, the communication channel may be provided by a storage device (not shown), such as an electrical, magnetic, and / or optical storage medium. For purposes of this description, the architecture and topology of the network is not important to the operation of the present disclosure unless otherwise specified.

[0033] FIG. 2 is a functional diagram of a video coding system 200 according to an embodiment of the present invention. The system 200 may be applied to the system 100 of FIG. 1. The system 200 may include a video encoder terminal 210 and a video decoder terminal 240. The video encoder terminal 210 may include an encoder 215, a decoder 220, a reference picture cache 225, a virtual reference picture generator 230, and a virtual reference picture cache 235. The encoder 215, as its name implies, may apply video coding techniques to a source video input to the video encoder terminal 210 and output coded video data that is provided to the video decoder terminal 240. The decoder 220 in the video encoder terminal 210 may reverse the coding operations applied by the encoder 215 for coded frames selected to serve as reference frames for coding. The reconstructed video output from the decoder 220 may be input to the reference picture cache 225. The virtual reference picture generator 230 may generate virtual reference frames from reference frames stored in the reference picture cache 225 according to techniques described herein. The virtual reference frames may be stored in the virtual reference picture cache 235. The reconstructed frames in the reference picture cache 225 and the virtual reference picture cache 235 may be used in prediction operations performed by the encoder 215 when processing the received source video at a later time.

[0034] Video decoder terminal 240 may include a decoder 245, a reference picture cache 250, a virtual reference picture generator 255, and a virtual reference picture cache 260. The decoder 245 may decode coded video data received from the video encoder terminal 210 to obtain reconstructed video data therefrom. The reconstructed video data may be output from the video decoder terminal 240 for use by other processes of the terminal in which the video decoder terminal 240 resides (FIG. 1).

[0035] In application, the video encoder terminal 210 and the video decoder terminal 240 may conform to a coding protocol that defines coding operations that may be applied by the encoder 215 and syntax elements that identify the coding operations to be applied. Typically, the encoder 215 selects some frames to be used as "reference frames", frames that may serve as prediction references for frames to be coded later. When a frame is designated as a reference frame, the decoders 220 and 245 may store a frame reconstructed therefrom in a reference picture cache 250. Because the decoder 220 of the video encoder terminal and the decoder 245 of the video decoder terminal both generate reconstructed video from the coded video output by the encoder 215, the contents of the reference picture caches 225 and 250 in both terminals should be synchronized in the absence of any type of communication error.

[0036] The virtual reference picture generator 255 may generate virtual reference frames from reference frames stored in the reference picture cache 260 according to techniques described herein. The virtual reference frames may be stored in the virtual reference picture cache 260. The reconstructed frames in the reference picture cache 250 and the virtual reference picture cache 260 may be used in prediction operations performed by the decoder 245 as determined by coding parameters provided in the coded video data.

[0037] Figure 3 shows a virtual reference frame F VREF 3 illustrates the concept of a virtual reference frame F generated from a pair of coded frames F0 and F2. In reality, a frame processed by an encoding system may have its own temporal location that identifies the frame's position in the video timeline. In the example of FIG. 3, frames F0 and F2 may each have their own temporal location. VREFis shown to be created at an intermediate temporal location between the location of frame F0 and the location of frame F2. In the illustrated example, the interpolated frame F is generated using motion vectors (also referred to as "MVs") that extend from frame F2 to frame F0 and are thus used to predict the content of frame F2. VREF In this way, motion projection is used to generate a virtual reference frame F VREF The content of may be derived from prediction data provided to develop another reconstructed frame (frame F2 in the example of FIG. 3).

[0038] In fact, the virtual reference frame F VREF may be located in the same time position as the frame F1 being coded by the coding system. VREF It may be convenient to code frame F1 using a virtual reference frame F VREF Depending on the motion projection model used for the creation of the virtual reference frame F VREF is the virtual reference frame F VREF It is likely that the motion is well-aligned and similar to other frames (e.g., frame F1 and other frames not shown) that have a temporal location close to that of the virtual reference frame F. This is because the motion is translational and the virtual reference frame F VREF This is especially true when the temporal distance between F and the frame F1 being coded is small. VREF If the collocated positions in have similar motion, the encoder 220 (FIG. 2) uses F to signal the corresponding predicted locations for the current block. VREF In some cases, only a single spatial displacement from F1 to F2 needs to be signaled. Thus, the proposed coding technique can potentially improve coding efficiency by reducing the bits of side information used to predict the current block. This can also be extended when the predictions are from the same direction, in which case the motion projections use extrapolation.

[0039] FIG. 4 illustrates a method 400 for generating a virtual reference frame according to an embodiment of the present disclosure. The method 400 may begin by selecting reference frames F0, F2 for derivation of the virtual reference frame (Box 410). The method 400 may project motion vectors extending between the reference frames onto the virtual reference frame (Box 420). The method 400 may then determine whether the virtual reference frame has a "hole" that the motion vector projection does not reach (Box 430). If so, the method may derive the content of the hole through an alternative technique (Box 440). The method 400 may warp the reference frame based on the created projected motion vector field (Box 450). Once the projection has been performed on the entire spatial area of ​​the virtual reference frame, the method 400 may blend the content of the reference frames F0, F2 (Box 460).

[0040] The reference frames F0, F2 for motion projection may be selected based on an evaluation of the temporal distance of the candidate reference frames from the temporal location of the virtual reference frame FVREF, based on whether interpolation or extrapolation is required, and based on an estimate of the quality of the candidate reference frames (which may be evaluated on a region-based or block pixel quality basis, or based on the coding parameters (e.g., QP) used in coding the candidate reference frames). Furthermore, the selection of the reference frames may evaluate the type of motion occurring between the pair of candidate reference frames (e.g., translation, zoom, rotation, affine, etc.), the magnitude of the motion, the difference in spatial resolution, spatial signal characteristics (spatial texture or smooth regions), etc.

[0041] In general, when the temporal distance between two candidate reference frames is small, there is a high probability that there is a strong temporal correlation between them. Selecting frames with a strong correlation can result in a better prediction. Therefore, the temporal distance can be an important factor for the method 400 when selecting a reference frame.

[0042] Furthermore, in inter prediction, a virtual reference frame F VREF If the reference frames are selected such that the temporal location of the coded frame is located between two reference frames (interpolation), the motion from these two reference frames will usually have higher accuracy than the motion from two reference frames located on the same side of the current frame (extrapolation). Furthermore, some frames may be coded with higher quality compared to other frames. The reconstruction quality may also vary spatially due to adaptive quantization methods or rate control schemes. The quality level may be controlled through the features / parameters of quantization, rate-distortion optimized quantization, coefficient thresholding, and quantization rounding, among others. Thus, the quality of the reference frames and prediction blocks that may be used may vary. In one embodiment, the reference frame selection is determined by the following two factors, in particular the two virtual reference frames F VREF A reference frame coded with a higher quality can be preferred over another reference frame if they have the same temporal distance to a virtual reference frame F VREF may be used to select a reference frame for use in the derivation of

[0043] As described, the type of motion may be evaluated as part of the reference frame selection process. Reference frames identified as having translational motion may be assigned a relatively higher priority than reference frames identified as having occlusion or disocclusion motion, and may be assigned a lower selection priority than frames.

[0044] Additionally, illumination gradients may be evaluated as part of the reference frame selection process. Frames identified as having relatively consistent illumination may be assigned a relatively higher selection priority than reference frames that are part of a video sequence that has highly varying illumination. Similarly, frames identified as being part of a video sequence that has a smooth illumination gradient may be assigned a relatively higher selection priority than reference frames that are part of a video sequence that has erratic varying illumination.

[0045] The method 400 may generate a score for each pair of candidate reference frames based on an evaluation of these factors, which are modeled as follows: Score=f{temporal_distance, interpolation / extrapolation, pixel_quality, light change, occlusion / disocclusion}(1) The score may be modeled as a function f of, but is not limited to, time distance, interpolation / extrapolation, pixel, quality, light change, and occlusion / disocclusion. The function f may be, but is not limited to, linear, non-linear models, decision trees, neural networks, SVM (spatial vector model), etc. For error resilience, once the priority is determined based on the score values, the encoder terminal may explicitly signal the order to the decoder terminal. The signaling may be done at the frame, tile, or slice level of the coding syntax.

[0046] Motion vector projection can be implemented in several ways, such as translational motion modeling, affine motion modeling, perspective motion modeling, etc. The following description presents exemplary motion projection using translational motion model and affine motion model, respectively. Illumination compensation, i.e., weighted prediction and its associated parameters, can be applied as well. Furthermore, multi-hypothesis prediction can be applied for motion vector projection, where prediction is achieved using more than two hypotheses.

[0047] 5(a) and 5(b) show an example application of the translational motion technique for motion projection. In both examples, the motion vectors between a pair of reference frames F2 and F4 are used to generate a motion vector for a virtual reference frame F3.

[0048] Figure 5(a) shows an example of direct motion vector projection. For a pixel block PB in a reference frame F4, if it is coded as inter prediction and its motion vector mv points to another selected reference frame F2, the motion vector mv can be projected onto a current frame F3 based on the temporal distance of the frames.

[0049] 5(b) shows an example of indirect motion vector projection. Indirect motion vector projection may be used for pixel blocks in reference frame F4 for which no motion information is associated, or for pixel blocks whose motion information points to a reference frame or frames different from the second reference frame F2 used for predicting the virtual reference frame F3. In a situation where both reference frames F2, F4 may have a common reference frame, e.g., frame F1, a motion vector mv1, mv2 for the virtual reference frame F3 from a chain of motion vectors mv1, mv2 extending between frames F4, F2, and F1 may be selected. v The motion vector mv1 between F1 and F4 and the motion vector mv2 between F1 and F2 are used to calculate the motion vector mv between frames F2 and F4. v The motion vector mv between F2 and F4 can be derived. v After deriving the motion vector mv v can be projected onto the virtual reference frame F3 based on the temporal distance between the current frame F3 and its reference frames F2 and F4.

[0050] Direct motion vectors between reference frames can potentially yield better quality for projected motion vectors because the reference blocks can be searched by performing a block matching search with the current block. For blocks that are not associated with direct motion vector information, indirect motion vectors can still provide reliable information to help derive the projected motion vector.

[0051] In one embodiment, video sequences that may have different types of motion, such as rotation, zoom in / out, scaling, and other non-translational motion, may be processed by affine motion projection techniques. If the entire frame follows a single affine model, a global affine model may be applied to motion compensate the entire frame. Different locally applicable affine models may be applied at the block level as needed to locally adaptively track more complex motion.

[0052] FIG. 6 illustrates a method 600 for performing motion vector projection according to an embodiment of the present disclosure. The method 600 may operate with reference to a current block of a reference frame, examples of which are illustrated in FIG. 7(a) and FIG. 7(b) herein. The method 700 may begin by identifying other pixel blocks or blocks of the reference frame having associated motion vectors (Box 610). The method 600 may derive transformation model parameters from the motion vectors of the identified pixel blocks (Box 620). The method 600 may derive motion vectors for a virtual reference frame based on the model parameters (Box 630). The method 600 may then derive the content of the virtual reference frame by motion compensation using the motion vectors derived in Box 630.

[0053] The selection of pixel blocks can be done in various ways. In one example shown in Figure 7(a), motion vectors from neighboring pixel blocks can be selected based on the reference frames they refer to.

[0054] 7(a) illustrates an example set of pixel blocks PB0-PB10 on which the method 600 of FIG. 6 may operate. As shown, a current pixel block PB0 being processed by the method 600 may have several neighboring pixel blocks PB1-PB10. Video coding systems often code video as pixel blocks of various sizes. Thus, the relative sizes of the pixel blocks PB0-PB10, as well as the number and placement of the pixel blocks PB1-PB10 neighboring the current pixel block PB0, may vary based on other coding decisions made by the encoder.

[0055] The affine transformation model parameters from the reference frame may be estimated by utilizing the motion vectors mv1-mv10 of the neighboring blocks PB1-PB10 in both the encoder and the decoder, or may be estimated in the encoder and signaled to the decoder in the bitstream. A hybrid approach may also be applied, in which initial estimates of the parameters are derived in both the encoder and the decoder, and refinements are signaled in the bitstream. The following description describes an embodiment for deriving block-based affine transformation model parameters for motion compensation of a virtual reference frame, with reference to FIG. 7(a).

[0056] For example, the method may consider neighboring pixel blocks PB1-PB10, such as the pixel blocks in the row above, the left column, the top right, and the top left of the current pixel block PB0, to find blocks that have the same reference frame as the current block. If such blocks are found, their motion vectors may be used to derive the affine model parameters.

[0057] The neighboring blocks PB1-PB10 can be scanned sequentially from left to right for the top row, then top to bottom for the left column, etc. Alternatively, the pixel blocks can be sampled in a pre-designed or signaled pattern without the need to scan all the blocks.

[0058] In a scenario where there is no block with the same reference frame as the current block PB0, the motion vectors of these neighboring blocks PB1-PB10 with reference frames different from the current frame can also be utilized by performing temporal scaling on the MVs of these neighboring blocks PB1-PB10 based on the temporal distance of the current frame F2 (FIG. 3) to the reference frame F1 of the current block and the temporal distance of the current frame F2 to the reference frame of the neighboring block (not shown in FIG. 3). Alternatively, neighbors using the N closest reference frames for prediction can be utilized while performing temporal scaling of their motion vectors again. In an alternative embodiment, the neighbors PB1-PB10 can also be scanned by first finding the neighbor using the closest reference and using it to determine the value of the spatial motion vector predictor (after temporal scaling). If necessary, the neighbors PB1-PB10 with the next order / distance reference frame can be added next until N neighbors have been added or until there are no neighbors using a reference frame within the distance X. Such information, i.e., how many neighbors and / or X distance, can be pre-determined or signaled in the bitstream.

[0059] Taking Figure 7(a) as an example, pixel blocks PB1, PB3, PB5, and PB8 are shown as using the same reference frame as that used by current block PB0, in which case motion vectors mv0, mv1, mv3, mv5, and mv8 of these blocks may be utilized for affine model parameter derivation.

[0060] In an embodiment, the derivation of the transformation model parameters may be performed as follows: When a neighboring block is identified as relevant in box 610, the sample offset of the neighboring block relative to the current block may be derived. k ,y k ) and its motion vector is mv kConsider a neighboring block k where k is the center position of the current block, (x0,y0), and its motion vector is mv0. Then, the sample position offset between the current block and its neighboring block can be derived as follows:

number

[0061] Motion vector mv k Use to find the reference block of the current block.

number

number

number

[0062] In box 630, the sample position offset between the reference block of the current block and the reference block of the neighboring block may be derived as follows:

number

[0063] The transformation model parameter A can be derived through this formula.

number

[0064] The motion vectors identified by box 610 are likely to be motion vectors from corresponding blocks in the reference frame relative to their own reference frame, and thus a motion vector from that reference frame to the current frame can be derived. Using all available MVs from the block's neighboring blocks, the current block's own motion vector, denoted as MV, and velocity and acceleration information, a motion vector from that reference frame to the current frame, denoted as MV', can be derived. Using the affine model parameters A derived in box 620, the motion vector of each small block as shown in FIG. 7(b) in the current block can be derived based on its center position, and the center position and motion vector of the current block.

[0065] The sample position offset from sub-block i within the current block to the center of the current block may be calculated as follows:

number

[0066] The sample position offset between the reference block of small block i and the reference block of the current block in the reference frame may be derived as follows:

number

[0067] The sample position offset between the reference block of small block i and the reference block of the current block in the reference frame can also be expressed as:

number

[0068] Then, the motion vector of the subblock i in the current block

number

number

number

number

[0069] Motion compensation can then be performed to interpolate the virtual reference frame.

[0070] In one embodiment, motion from a high priority reference frame may potentially result in a more accurate projected motion and should have the highest priority when multiple motion vectors are projected to the same position. Alternatively, all motion vectors may be considered and a weighted sum of all hypotheses may be calculated based on their priorities (or based on some "distortion" or probability / confidence criterion, e.g., based on the similarity of each sample in a first block in reference A to a second block in reference B). When multiple motion projections are present at the same position, motion vectors projected from higher priority reference frames are assigned a higher priority than motion vectors projected from lower priority reference frames. In certain embodiments, such motion vectors are not allowed to be overwritten by motion vectors from lower priority reference frames. In this way, a better motion projection may be provided and thus a high quality virtual reference frame may be interpolated. Priority may be determined based on some or all of the motion information corresponding to the same location, using techniques such as weighted average, majority voting, non-linear, e.g., median, or linear filtering, using machine learning methods, etc.

[0071] As explained, motion vector projection may not result in coverage of the entire area of ​​the virtual reference frame. Regions of a frame that remain uncovered after application of motion vector projection are termed "holes" for ease of reference. Blocks with no or only partial projected motion vector information are called holes in this context. Holes may appear due to occlusion / disocclusion, or source blocks in the reference frame do not have motion vectors, e.g., coded using intra prediction. To better generate an interpolated frame, holes may be filled by using different methods.

[0072] In one aspect, holes can be filled by applying available projected motion vectors in neighboring blocks to the block in which the hole appears. For example, FIG. 8(b) shows an example virtual reference frame FVREF in which a hole occurs after the other motion projection operations depicted in FIG. 8(a) are performed. Neighboring blocks often have high spatial correlation in a video sequence, and therefore the projected motion vector mv0 from the neighboring block is a reliable estimate mv for the current block occurring within the hole. v can be provided.

[0073] Holes can also be filled using more complex techniques. In another aspect, the encoder can perform motion search in reference frame F2 (FIG. 8) with another reference frame F1 when the prediction mode of the block in reference frame F2 is intra or there is no direct / indirect motion vector between the two reference frames F1 and F2. This can improve the accuracy of the motion vector projected for the block occurring in the hole.

[0074] In another aspect, when the projected position relative to the current block PB0 position is a hole, the method may analyze neighboring blocks for the current block to determine whether a motion vector is available and whether the motion vector has some sample similarity with the sample of the current block. If neighboring blocks with similar samples to the current block are available and are coded using bi-prediction, the motion vectors of these neighboring blocks can be utilized to fill the hole. If multiple motion vectors are available from the neighborhood, a weighting function on these motion vectors can be performed and then used to fill the hole. Since neighboring blocks with similar samples have a strong correlation with the current block, this on-the-fly update algorithm can help provide high quality projected motion for holes and can improve the interpolated virtual reference frame pixel quality, which can help reduce prediction residual and improve coding efficiency.

[0075] In one aspect, inpainting can also be utilized to fill holes, such as traditional patch-based and diffusion-based inpainting, whereas machine learning-based inpainting, such as convolutional neural network ("CNN")-based and / or generative adversarial network ("GAN")-based methods, can also be applied to fill holes, since they have already been successfully applied to restore images with missing parts of the image and achieve visually pleasing results.

[0076] The hole filling technique is an optional feature that need not be applied in all situations. Applying hole filling may lead to increased complexity and consumption of processing resources in the video coding system that the system designers may determine are not warranted for their applications. In such situations, the hole filling process may be skipped or replaced by a coarse interpolation technique such as directly copying pixels from a collocated block in the reference frame. The virtual reference frame is proposed for use as a prediction reference that does not need to be perfect. Other prediction mechanisms in the encoder, especially the cost-distortion estimation process, are likely to avoid poorly constructed regions of the virtual reference frame being selected as prediction references for other frames because, for example, those regions of the virtual reference frame would ultimately provide poor prediction references for other frames. In such situations, the prediction algorithm is likely to be a different reference frame for use in predictive coding.

[0077] If the motion vectors are block-based and the projected motion vectors in holes can be filled with the projected motion vectors of neighboring blocks, this can result in mismatch of motion vectors around block boundaries. In one embodiment, different filters can be applied to smooth the motion vector field to reduce the possibility of block-based artifacts in the interpolated frame and improve quality. For example, an averaging or median filter can be used to smooth the motion vectors, or a deblocking scheme similar to that used in AVC, HEVC, AV1, or VVC to smooth samples at block boundaries. Other filters such as bilateral filtering, non-local means denoising, etc. can also be used. Machine learning based algorithms such as CNN models can also be applied to filter the projected motion vectors, since they have already shown a competitive advantage of fetching context not only from local regions but also from global regions. Encoder signaling of filters can also be included to improve the properties of this reference frame. A similar concept is applied to illumination change parameters, and not only to motion vectors but also to other types of disparity information that may be associated with blocks.

[0078] Adaptive MV smoothing can also be applied based on the difference between the motion vector of the current block and the motion vector of its neighboring block. If the difference between the motion vector of the current block and the motion vector of its neighboring block is small, the current block and its neighboring block have very similar characteristics, so an averaging filter can be used. However, if the difference between the motion vector of the current block and the motion vector of its neighboring block is very large, the current block and its neighboring block may be around an object boundary or edge, and using an averaging filter to smooth the motion vector field may impair performance, in which case no smoothing or using a median filter may be a better choice.

[0079] Once the projected motion vector for the current frame is derived, the neighboring reference frames can be warped based on the projected motion vector to obtain a warped reference frame. Various interpolation methods can be employed to generate samples with sub-pixel accuracy, such as fixed tap filters, separable / non-separable filters, adaptive filters, or even send filter coefficients to the decoder, similar to how ALF is signaled in VVC. Warping is done by using a virtual reference frame F VREF A pair of frames, one unfolded from frame F1 and the other unfolded from frame F2, may be generated that represent the projection of each frame along the unfolded motion vector at the temporal location of F1. A conceptual example of the warping process is shown in Figure 10(a).

[0080] After the warped frames are generated, they can be blended together to generate the final interpolated frame. Various algorithms can be used to blend the warped frames, such as: · Average blending: equal weights can be assigned to the two warped frames; Pixel Difference Weighted Blending: The weighting coefficient for each pixel can be calculated based on the difference of related pixels in the two warped frames; spatial neighbor weighting can also be taken into account to further adjust the weighting for the current pixel. Distance-weighted mixing: Weighting factors may be derived based on the temporal distance of the two reference frames with the current frame. In general, a large temporal distance is assigned a small weighting factor since a large temporal distance may indicate a weak temporal correlation. Adaptive Blending: For the on-the-fly update process, the similarity between the current block and its neighboring blocks can be checked. If the samples of the neighboring blocks are similar to the samples in the current block, the blending algorithm of the neighboring blocks can be utilized to blend the warped block and then update the virtual reference frame.

[0081] To compute the mixture weights, a machine learning based model such as a CNN can be applied.

[0082] The projected motion vector of the virtual reference frame and the motion vectors from other frames that are predictively coded with reference to the virtual reference frame can use motion vector prediction to improve the compression efficiency of the motion vectors and reduce bit signaling overhead. In existing video coding standards, the motion vectors of neighboring reconstructed blocks are commonly used as predictors of the motion vectors of the current coding block. Using this technique, a motion vector differential (MVD) or motion vector prediction residual is coded and transmitted to the decoder. Signaling the MVD instead of the motion vector itself can significantly reduce the amount of overhead bits and improve compression efficiency. Beyond these techniques, further coding efficiency can be achieved to represent the motion vectors.

[0083] FIG. 11 illustrates the application of spatial motion vector prediction according to an embodiment of the present disclosure. FIG. 11(a) illustrates a currently coded pixel block C 1110 ("current block") spatially oriented with respect to two previously coded pixel blocks A 1120, B 1130 in its frame F1. FIG. 11(b) illustrates a current block C and pixel block A in frame F1. Also illustrated in FIG. 11(b) is a virtual reference frame F1' at the same time position as frame F1, and reference frames F0, F2 that serve as prediction references for virtual reference frame F1. In the example of FIG. 11, the current block C is coded using a motion vector projection mode (denoted as "DMVP"), and neighboring blocks A, C are coded as a conventional mixed inter prediction mode, but neighboring blocks A, C have the same reference frames (F0, F2) as virtual reference frame F1', as shown. Further, in this example, mv0 represents the motion vector of the current coding block C relative to its matching block in the virtual reference frame F1', mv1 and mv2 represent the motion vectors of the neighboring block A relative to its reference frames F0 and F2, respectively, d1 represents the temporal distance between the current frame F1 and its forward reference F0, and d2 represents the temporal distance between the two reference frames F0 and F2.

[0084] Through mv1 and mv2, the offset between pos_A1 and pos_A2 (the positions of block A in reference frames F0 and F2) can be derived as follows: dmvp_mv=mv2-mv1. (13)

[0085] With motion projection, the projected position (pos_A3) of block A in the virtual reference frame F1' can be derived as follows: pos_A3=pos_A1+d1 / d2 * dmvp_mv. (14)

[0086] The offset between the neighbor block A (pos_A) and its projection position in F1' can be calculated as follows:

number

[0087] FIG. 12 illustrates the application of spatial motion vector prediction according to another embodiment of the present disclosure. FIG. 12(a) illustrates a current block C 1210 currently being coded, spatially oriented with respect to two previously coded pixel blocks A 1220, B 1230 in frame F1. FIG. 12(b) illustrates a current block C and pixel block A in frame F1. FIG. 12(b) also illustrates a virtual reference frame F1' at the same time position as frame F1, and reference frames F0, F2 serving as prediction references for virtual reference frame F1. In the example of FIG. 12, the current block C is coded using DMVP motion vector projection mode, and neighboring blocks A, C are coded as conventional mixed inter prediction mode, and neighboring blocks A, C have the same reference frames (F0, F2) as virtual reference frame F1', as shown. Further, in this example, mv0 represents the motion vector of the current coding block C relative to its matching block in the reference frame F0; mv1 and mv2 represent the motion vectors of the neighboring block A in the virtual reference frame F1' and the reference frame F0, respectively; d1 represents the temporal distance between the current frame F1 and its forward reference F0, and d2 is the temporal distance between the two reference frames F0 and F2.

[0088] Using mv1 and the position pos_A of the neighboring block, the position of reference block A in the virtual reference frame F1' can be derived as follows: pos_A3=pos_A+mv1. (16)

[0089] Using the projected motion vector dmvp_mv extending from pos_A3, the projected motion vector into the forward reference frame F0 can be derived as follows: dmvp_mv1=-d1 / d2 * dmvp_mv (17)

[0090] Once dmvp_mv1 is derived, mv2 can be derived as the vector addition of dmvp_mv1 and mv1 as follows: mv2=mv1+dmvp_mv1 (18)

[0091] The motion vector mv2 may be employed as a predictor of the motion vector mv0 of the current block to reduce motion vector redundancy.

[0092] FIG. 13 illustrates the application of spatial motion vector prediction according to a further embodiment of the present disclosure. FIG. 13(a) illustrates a current block C 1310 spatially oriented with respect to two previously coded pixel blocks A 1320, B 1330 in frame F1. FIG. 13(b) illustrates a current block C and pixel block A in frame F1. FIG. 13(b) also illustrates a virtual reference frame F1' at the same time position as frame F1, and reference frames F0, F2 serving as prediction references for virtual reference frame F1. In the example of FIG. 13, the current block C is coded using DMVP motion vector projection mode, and neighboring blocks A, C are coded as conventional mixed inter prediction mode, and neighboring blocks A, C have the same reference frames (F0, F2) as virtual reference frame F1', as shown. Further, in this example, mv0 represents the motion vector of the current coding block C relative to the virtual reference frame; mv1 represents the motion vector of the neighboring block A relative to the virtual reference frame F1'; d1 represents the temporal distance between the current frame F1 and its forward reference F0, and d2 represents the temporal distance between the two reference frames F0 and F2.

[0093] In this example, the reference block of the current coding block and its neighboring blocks all point to the virtual reference frame F1', so that the motion vector mv1 of the neighboring block can be directly used as a prediction of the motion vector mv0 of the current coding block to reduce motion vector redundancy.

[0094] In other embodiments, motion vectors from blocks coded by direct motion vector prediction (again DMVP) may be used for prediction of motion vectors of blocks coded by other prediction modes. Figure 14(a) shows a coding example where a current block C 1410 is spatially oriented with respect to two previously coded pixel blocks A 1420, B 1430 in its frame F1. Figure 14(b) shows a current block C and pixel block A in frame F1. Also shown in Figure 14(b) is a virtual reference frame F1' at the same time position as frame F1, and reference frames F0, F2 that serve as prediction references for virtual reference frame F1. In the example of Figure 14, the current block C is coded using a composite motion vector projection mode, and neighboring blocks A, C are coded using DMVP mode, and neighboring blocks A, C have the same reference frames (F0, F2) as virtual reference frame F1', as shown. Further, in this example, mv0 and mv1 represent the motion vector of the current coding block C relative to reference frames F0 and F2, respectively, and mv2 represents the motion vector from block A (at pos_A) to its reference block in the hypothetical reference frame F1'.

[0095] Using mv2 and the position pos_A of the neighboring block, the reference block position in the virtual reference frame F1' can be derived as follows: pos_A3=pos_A+mv1. (19)

[0096] Using the projected motion vector dmvp_mv at pos_A3, the projected motion vectors for the forward reference frame F0 and the backward reference frame F2 can be derived as follows: dmvp_mv1=-d1 / d2 * dmvp_mv (20) dmvp_mv2=(1-d1 / d2) * dmvp_mv (21)

[0097] By vector addition of mv2 and the derived dmvp_mv1 and dmvp_mv2, mv2 and mv4 can be derived as follows. mv3=mv2+dmvp_mv1 (22) mv4=mv2+dmvp_mv2 (23)

[0098] Then, to reduce the motion vector redundancy, mv3 and mv4 can be used as predictions for the motion vectors mv0 and mv1 of the current coding block C, respectively.

[0099] In another embodiment, direct motion vector prediction (DMVP) may be used to derive motion vectors for later coded frames (referred to as "temporal motion vector prediction" or "TMVP") and frames that are later coded using DMVP. FIG. 15 shows a coding example in which TMVP for later coded frames (also referred to as "future" frames) may be implemented. In FIG. 15, a current block C 1510 may be coded using motion vector mv0 that references a block in a hypothetical reference frame F1'. Using mv0 and the current block's position pos_C, a reference block position pos_A in the hypothetical reference frame F1' may be derived as follows: pos_A=pos_C+mv0. (24)

[0100] From pos_A and using the projected motion vector dmvp_mv, the projected motion vectors to the forward and backward reference frames can be derived as follows: dmvp_mv0=-d1 / d2 * dmvp_mv (25) dmvp_mv1=(1-d1 / d2) * dmvp_mv (26)

[0101] By vector addition of mv0 and the derived dmvp_mv0 and dmvp_mv1, mv1 and mv2 can be derived as follows. mv1=mv0+dmvp_mv0 (27) mv2=mv0+dmvp_mv1 (28) Once they are derived, the motion vectors mv1 and mv2 can be saved from uni-prediction or bi-prediction for future frames. The motion vectors mv1 and mv2 can also be utilized for motion projection of future frames to generate corresponding DMVP frames. This technique helps to reduce motion vector redundancy and the number of holes after motion projection, which can improve the video quality of the interpolated virtual reference frame and reduce prediction residuals.

[0102] As mentioned above, holes in the virtual reference frame (FIG. 8(b)) can be filled on the fly with the motion information of its neighboring previously coded blocks. The same or similar process can be applied to refine the projected motion vector of the current block by using the motion vector of its neighboring previously coded blocks before coding the current block. Using the motion vector of the previously coded blocks in the neighborhood of the current block can help refine the projected motion vector of the current block and improve the interpolated DMVP frame quality. FIG. 16 shows a coding example in which the motion vectors of the previously coded blocks PB1, PB5 in the spatial neighborhood of the current block PB0 are used to refine the projected motion vector of the current block.

[0103] As shown in Figure 16, mv k represents the motion vector of the previous coding block k in the neighborhood of the current block PB0; and

number

number

[0104] If multiple blocks are available, their MVD between the motion vector and the projected motion vector may be calculated as follows:

number

[0105] Final motion vector difference

number

number

[0106] In another embodiment, an available neighboring block having the same reference frame as the current block can be used to derive the affine transformation model parameters as described with respect to Fig. 7(b) using its motion vector together with the refined projected motion vector of the current block. The derived affine transformation model parameters can then be used to perform motion compensation and interpolate the corresponding block in the virtual reference frame.

[0107] FIG. 17 is a functional block diagram of a coding system 1700 according to one aspect of the disclosure. The system 1700 may include a pixel block encoder 1710, a pixel block decoder 1720, a frame buffer 1730, an in-loop filter system 1740, a reference picture buffer 1750, a virtual reference picture generator 1760, a predictor 1770, a controller 1780, and a syntax unit 1790. The coding system 1700 may operate on predefined units of a video frame, conveniently referred to as "pixel blocks," and differentially code the pixel blocks according to prediction techniques. Thus, a frame of video to be coded may be parsed into pixel blocks, which the pixel block encoder 1710 processes on a pixel block by pixel block basis. The pixel block encoder 1710 may present the coded pixel block data to a syntax unit 1790, which formats the coded pixel block data into a transmission syntax conforming to a governing coding protocol.

[0108] The pixel block decoder 1720 may decode the coded pixel block data and generate decoded pixel block data therefrom. The frame buffer 1730 may generate reconstructed frame data from the decoded pixel block data. The in-loop filter 1740 may perform one or more filtering operations on the reconstructed frame. For example, the in-loop filter 1740 may perform deblocking filtering, sample adaptive offset (SAO) filtering, adaptive loop filtering (ALF), maximum likelihood (ML) based filtering schemes, de-ringing, de-banding, sharpening, resolution scaling, etc. The filtered frame may be stored in a reference picture buffer 1750, where it may be used as a source of prediction for later received pixel blocks.

[0109] The pixel block encoder 1710 may include a subtractor 1712, a transform unit 1714, a quantizer 1716, and an entropy encoder 1718. The pixel block encoder 1710 may accept a pixel block of input data at the subtractor 1712. The subtractor 1712 may receive a predicted pixel block from the predictor 1770 and generate therefrom an array of pixel residuals representing the difference between the input pixel block and the predicted pixel block. The transform unit 1714 may apply a transform to the sample data output from the subtractor 1712 to convert the data from the pixel domain to the domain of transform coefficients. In some scenarios (e.g., when operating with a high dynamic range), the input may be reshaped or an adaptation scheme may be applied to adjust the content transfer characteristics before the transform unit 1714 and / or the subtractor 1712. Such adaptation may be either a simple scaling based on a remapping function, or more sophisticated pixel manipulation techniques. The quantizer 1716 may perform quantization of the transform coefficients output by the transform unit 1714 according to a quantization parameter qp. The quantizer 1716 may apply either a uniform or a non-uniform quantization parameter. A non-uniform quantization parameter may vary over a given location of a block of coefficients output from the transform unit 1714. The entropy coder 1718 may reduce the bandwidth of the output of the coefficient quantizer by coding the output, for example, with variable length codewords or using a context-adaptive binary arithmetic coder.

[0110] The transform unit 1714 may operate in various transform modes as determined by the controller 1780. For example, the transform unit 1714 may apply a discrete cosine transform (DCT), a discrete sine transform (DST), a Walsh-Hadamard transform, a Haar transform, a Daubechies wavelet transform, etc. In one aspect, the controller 1780 may select the coding mode applied by the transform unit 1715, configure the transform unit 1715 accordingly, or explicitly or implicitly signal the coding mode in the coded video data.

[0111] The quantizer 1716 may operate according to a quantization parameter qp determined by a controller 1780. Techniques for generating the quantization parameter are described below. The controller 1780 may provide data representing its quantization parameter selection to a syntax unit 1790.

[0112] The entropy encoder 1718, as its name implies, may perform entropy coding of the data output from the quantizer 1716. For example, the entropy encoder 1718 may perform run-length coding, Huffman coding, Golomb coding, context-adaptive binary arithmetic coding, etc.

[0113] The pixel block decoder 1720 may reverse the coding operation of the pixel block encoder 1710. For example, the pixel block decoder 1720 may include an inverse quantizer 1722, an inverse transform unit 1724, and an adder 1726. In some scenarios (e.g., when operating in a high dynamic range), after the inverse transform unit 1724 and / or the adder 1726, the input may be inverse reshaped or remapped, typically according to a function applied in the encoder and content characteristics. The pixel block decoder 1720 may take its input data from the output of the quantizer 1716. Although allowed, the pixel block decoder 1720 is not required to perform entropy decoding of entropy coded data, since entropy coding is a lossless event. The inverse quantizer 1722 may reverse the operation of the quantizer 1716 of the pixel block encoder 1710. The inverse quantizer 1722 may perform uniform or non-uniform inverse quantization as specified by quantization parameter data qp. Similarly, the inverse transform unit 1724 may invert the operation of the transform unit 1714. The inverse quantizer 1722 and the inverse transform unit 1724 may use the same quantization parameter qp and transform mode as their counterparts in the pixel block encoder 1710. The quantization operation may truncate data at various points, and thus the data restored by the inverse quantizer 1722 may have coding errors when compared to the data presented to the quantizer 1716 in the pixel block encoder 1710.

[0114] The adder 1726 may invert the operation performed by the subtractor 1712. It may receive the same predicted pixel block from the predictor 1770 that the subtractor 1712 used to generate the residual signal. The adder 1726 may add the predicted pixel block to the reconstructed residual values ​​output by the inverse transform unit 1724 and may output reconstructed pixel block data.

[0115] As described, the frame buffer 1730 may assemble a reconstructed frame from the output of the pixel block decoder 1720. The in-loop filter 1740 may perform various filtering operations on the reconstructed pixel block data. For example, the in-loop filter 1740 may include a deblocking filter, a sample adaptive offset ("SAO") filter, and / or other types of in-loop filters (not shown). The reference picture buffer 1750 may store the filtered frame data output by the in-loop filter 1740 for use in subsequent prediction of other pixel blocks.

[0116] The virtual reference picture generator 1760 may generate a virtual reference frame. The virtual reference picture generator 1760 may include a virtual reference picture predictor 1762 and a virtual reference picture buffer 1764. The virtual reference picture predictor 1762 may generate a virtual reference frame according to the techniques described herein with respect to any of Figures 3-16. The virtual reference picture predictor 1762 may generate a virtual reference frame from a reference frame stored in the reference picture buffer 1750. The virtual reference picture buffer 1764 may store the virtual reference frame so generated for use in prediction by the predictor 1770.

[0117] Different kinds of prediction data for different prediction modes are made available to the predictor 1770. For example, for an input pixel block, intra prediction takes prediction reference from the decoded data of the same frame in which the input pixel block is located. Thus, the reference frame store 1750 can store the decoded pixel block data of each frame as it was coded. For the same input pixel block, inter prediction can take prediction reference from a previously coded and decoded frame(s) designated as a reference frame. Thus, the reference frame store 1750 can store these decoded reference frames.

[0118] The predictor 1770 may provide a prediction block to the pixel block encoder 1710 for use in generating the residual. The predictor 1770 may perform a prediction search operation according to intra-mode coding, as well as uni-predictive, bi-predictive, and / or multi-hypothesis inter-mode coding. For intra-mode coding, the predictor 1770 may search among pixel block data from the same frame as the pixel block being coded that provides the closest match for the input pixel block. For inter-mode coding, the predictor 1770 may search among pixel block data of other previously coded frames stored in the reference picture buffer 1750 that provide a match for the input pixel block. Among predictions generated according to various modes, the predictor 1770 may select the mode that achieves the lowest distortion when the video is decoded given a target bit rate. Exceptions may occur when a coding mode is selected to satisfy a particular channel behavior or to satisfy other policies to which the coding system 1700 adheres, such as supporting a random access or data refresh policy.

[0119] When performing a predictive search within the virtual reference frames stored in the virtual reference frame buffer 1764, the predictor 1770 may perform direct motion projection and temporal motion projection, as described above.

[0120] The controller 1780 may control the overall operation of the coding system 1700. The controller 1780 may select operating parameters for the pixel block encoder 1710 and the predictor 1770 based on an analysis of the input pixel blocks and also external constraints such as coding bit rate targets and other operating parameters. As relevant to this description, when it selects a quantization parameter, the use of a uniform or non-uniform quantizer, and / or a transform mode, it may provide those parameters to a syntax unit 1790, which may include data representing those parameters in a data stream of coded video data output by the system 1700. The controller 1780 may also select between different operating modes for which the system may generate reference pictures and may include metadata identifying the selected mode for each portion of the coded data.

[0121] In operation, the controller 1780 can modify the operating parameters of the quantizer 1716 and the transform unit 1715 at different granularities of the image data, either per pixel block or at a larger granularity (e.g., per frame, per slice, per largest coding unit (LCU) or coding tree unit (CTU), or another region). In one aspect, the quantization parameters can be modified for each pixel within a coded frame.

[0122] Additionally, as described, the controller 1780 can control the operation of the in-loop filter 1750 and the prediction unit 1770. Such control can include, for the prediction unit 1770, mode selection (lambda, modes to be tested, search window, distortion strategy, etc.), and for the in-loop filter 1750, selection of filter parameters, parameter reordering, weighted prediction, etc.

[0123] The above description presented techniques for deriving a virtual reference frame for use in predictive coding of a pixel block of an input video sequence. Often, the encoder 1700 generates a virtual reference frame that is not selected by the predictor 1770 for use as a prediction reference. In one embodiment, the controller 1780 may refrain from providing signaling representing the virtual reference frame until the predictor 1770 selects a virtual reference picture for use in predicting an input pixel block.

[0124] 18 is a functional block diagram of a decoding system 1800 according to one embodiment of the present disclosure. The decoding system 1800 may include a syntax unit 1810, a pixel block decoder 1820, a frame buffer 1830, an in-loop filter 1840, a reference picture buffer 1850, a virtual reference picture generator 1860, a predictor 1870, and a controller 1880.

[0125] The syntax unit 1810 may receive the coded video data stream and may parse the coded data into its component parts. Data representing the coding parameters may be provided to a controller 1880, and data representing the coded residual (data output by the pixel block encoder 1710 of FIG. 17) may be provided to a pixel block decoder 1820. The predictor 1870 may generate predictive blocks from reference frame data available in the reference picture buffer 1850 or the virtual reference picture generator 1860, as determined by coding parameter data provided in the coded video. The predictor 1870 may provide predictive blocks to the pixel block decoder 1820.

[0126] The pixel block decoder 1820 may reverse the coding operations applied by the pixel block encoder 1710 (of FIG. 17). The frame buffer 1830 may generate a reconstructed frame from the decoded pixel blocks output by the pixel block decoder 1820. The in-loop filter 1840 may filter the reconstructed frame data. The filtered frame may be output from the decoding system 1800. The filtered frames designated to serve as reference frames may also be stored in a reference picture buffer 1850.

[0127] The pixel block decoder 1820 may include an entropy decoder 1822, an inverse quantizer 1824, an inverse transform unit 1826, and an adder 1828. The entropy decoder 1822 performs entropy decoding and may reverse the process performed by the entropy encoder 1718 (of FIG. 17). The inverse quantizer 1824 may reverse the operation of the quantizer 1716 of the pixel block encoder 1710 (of FIG. 17). Similarly, the inverse transform unit 1826 may reverse the operation of the transform unit 1714 (of FIG. 17). They may use quantization parameters and transform modes identified in the coded video data stream. Because quantization may truncate data, the pixel blocks reconstructed by the inverse quantizer 1824 are likely to have coding errors when compared to the input pixel blocks s presented to the pixel block encoder 1710 (of FIG. 17) of the encoder.

[0128] The adder 1828 may invert the operation performed by the subtractor 1710 (of FIG. 17). It may receive a predicted pixel block from the predictor 1870, as determined by a prediction reference in the coded video data stream. The adder 1828 may add the predicted pixel block to the reconstructed residual values ​​output by the inverse transform unit 1826, and may output reconstructed pixel block data.

[0129] As described, the frame buffer 1830 may assemble a reconstructed frame from the output of the pixel block decoder 1820. The in-loop filter 1840 may perform various filtering operations on the reconstructed pixel block data as identified by the coded video data. For example, the in-loop filter 1840 may include a deblocking filter, a sample adaptive offset ("SAO") filter, and / or other types of in-loop filters. In this manner, the operation of the frame buffer 1830 and the in-loop filter 1840 mimics the operation of the corresponding frame buffer 1730 and in-loop filter 1740 of the encoder 1700 (of FIG. 17).

[0130] The reference picture buffer 1850 may store filtered frame data for use in later prediction of other pixel blocks. The reference picture buffer 1850 may store a decoded frame as it was coded, for use in intra prediction. The reference picture buffer 1850 may also store decoded reference frames.

[0131] The virtual reference picture generator 1860 may generate a virtual reference frame. The virtual reference picture generator 1860 may include a virtual reference picture predictor 1862 and a virtual reference picture buffer 1864. The virtual reference picture predictor 1862 may generate a virtual reference frame according to the techniques described herein with respect to any of Figures 3-16. The virtual reference picture predictor 1862 may generate a virtual reference frame from a reference frame stored in the reference picture buffer 1850. The virtual reference picture buffer 1864 may store the virtual reference frame so generated for use in prediction by the predictor 1870.

[0132] When performing a predictive search within the virtual reference frames stored in the virtual reference frame buffer 1864, the predictor 1870 may perform direct motion projection and temporal motion projection, as described above.

[0133] As described, the predictor 1870 may provide predictive blocks to the pixel block decoder 1820 according to a coding mode identified in the coded video data. The predictor 1870 may provide predicted pixel block data as determined by a prediction reference indicator provided in the coded video data stream.

[0134] A controller 1880 may control the overall operation of the coding system 1800. The controller 1880 may set operating parameters for the pixel block decoder 1820 and the predictor 1870 based on parameters received in the coded video data stream. As relevant to this description, these operating parameters may include a quantization parameter for the inverse quantizer 1824 and a transform mode for the inverse transform unit 1810. As described, the received parameters may be set at various granularities of image data, e.g., per pixel block, per frame, per slice, per LCU / CTU, or based on other types of regions defined for the input image.

[0135] As described, in many cases, the encoder 1700 (FIG. 17) may refrain from providing signaling representing a virtual reference frame until its predictor 1770 selects a virtual reference picture for use in predicting an input pixel block. Thus, the contents of the decoder's virtual reference picture buffer 1864 do not need to be synchronized with the contents of the encoder's virtual reference buffer 1764 (FIG. 17). In this way, the virtual reference picture buffer 1864 may be implemented using an "on-the-fly" principle, where a virtual reference frame is generated only when needed. This does not require a dedicated full frame of buffer. More specifically, the on-the-fly approach results in the generation of a corresponding DMVP block after parsing the bitstream syntax, based on an indication of whether the coded block uses a block from a DMVP frame as a reference. This can reduce resource requirements in practical decoder implementations, such as for high-definition video (e.g., 8k), by reducing the decoder's memory and complexity requirements.

[0136] In one embodiment, a temporal interpolation prediction mode signaled as "TIP_mode" may be created based on the DMVP interpolation technique. TIP_mode may indicate how the virtual reference frame is used by the decoder.

[0137] For example, TIP_mode equal to 0 may indicate that the virtual reference frame is not to be directly output for display. It may instead indicate that the virtual reference frame is to be used as an additional reference frame for another coded frame. An encoding / decoding procedure is performed to reconstruct / decode the coded frame. The coded frame is compressed and transmitted in the bitstream.

[0138] Continuing with this example, when TIP_mode is 1, it may indicate that the virtual reference frame will be output directly from the decoder for display. The virtual reference frame is used as a reconstructed / decoded frame of the current coding frame. This indication may also indicate that the coded video bitstream has no other information sent to reconstruct / decode the coding frame.

[0139] Further under this example, when TIP_mode is 2, it may indicate that some side information is sent in the bitstream to help the decoder improve the quality of the virtual reference frame before it is output for display. In practice, this side information is expected to be much less than that generated in the full encoding / decoding procedure when TIP_mode is 0. For example, residual between the virtual reference frame and the source frame, motion information, partitioning, transformation information, filtering information, noise information, film grain model information, etc., may be selected, compressed, and sent in the bitstream. With this side information, the coding frame does not need to go through the full normal encoding / decoding procedure as TIP_mode is 0.

[0140] In all these modes, additional information can be sent to the decoder to indicate the creation of a new TIP virtual reference frame / region or the reuse of an existing TIP virtual reference frame / region from a previous coding frame.

[0141] In all these modes, additional information can be sent to the decoder to indicate how to create or help create the virtual reference frame / region.

[0142] In certain embodiments, the loop filter may be applied or disabled on the virtual reference frame based on application and resources.

[0143] The above TIP technique utilizing the DMVP technique can be extended to the tile, slice, or region level. An entire tile, slice, or region can be signaled and then predicted using a different TIP mode. If a tile, slice, or region is coded using TIP_mode equal to 1, the samples of the entire tile or region can be directly copied from the interpolated virtual reference frame.

[0144] In one embodiment, region-based inter prediction, conveniently referred to as "advanced inter prediction", may be implemented using the DMVP technique. The region may be a frame, tile, slice, another predefined rectangle or any region (generally larger than a coding block such as a coding tree unit, macroblock, or superblock, such a region is usually a collection of such block units) specified by other syntax elements. In this advanced inter prediction mode, the encoder may use syntax elements in the header of a collection of frames, frames, slices, tiles, or regions to indicate what type of inter prediction may be utilized to predict this frame / slice / tile / region, which may vary in terms of its generation method and signaling overhead, as well as whether additional information such as residuals is associated with such a frame / slice / tile / region. In a particular embodiment, such a frame / slice / tile / region may be indicated to have three different modes, again for convenience: "full_skip", "semi_skip", and "no_skip" modes. Other modes may also be introduced as needed. Although the following description refers to this process operating on frames, the principles described herein may be applied to different granularities within a coding sequence (collections of frames at the slice level or tile level, etc.) These modes allow such regions to be coded while taking advantage of different tradeoffs between complexity and coding performance of different sub-units of a video encoder or decoder.

[0145] Consider a coding application where the above mentioned three modes are available. When the coded frame is in full_skip mode, prediction of the current frame can be performed using exactly the same as described above for the virtual reference frame. Instead of generating a virtual reference frame only for prediction as described above, the virtual reference frame can also be the output of the decoding process and can be stored in the decoded picture buffer. This frame can also be further tagged as being a hidden / display frame, as supported in various specifications. For this frame, no additional information needs to be sent in the bitstream, e.g. there is no residual, making it very efficient to decode both in terms of complexity (only motion prediction needs to be performed) and bitrate. Other operations such as loop filtering may be skipped and still be applied to such frames.

[0146] When a coded frame is signaled as being in no_skip mode, the coded frame may be coded using conventional coding methods and block-by-block motion information as well as residual data may also be signaled.

[0147] When a coded frame is signaled as being in semi_skip mode, not all motion information for the frame is directly derived and signaled, but some additional information may be sent in the bitstream that may aid in the reconstruction process of the current frame. For example, the residual between the DMVP frame and the source frame, as well as parameters that may control loop filtering and film grain model parameters, may also be included in the frame header and signaled in the bitstream.

[0148] In certain embodiments, a loop filter may be used with such highly predictive coded frames based on the current application and / or available resources.

[0149] The motion projection of TMVP mode or DMVP mode may be created using pixel blocks from different reference frames or discontinuous blocks. This may cause discontinuities around pixel block boundaries after motion compensation. To remove blocking artifacts and provide good prediction quality, a deblocking filter may be applied to the boundaries of each pixel block unit in the virtual reference frame. The filtering strength or method used may be determined based on the reference frame used and / or the differences in motion vectors and weighted prediction parameters (if considered) between adjacent blocks.

[0150] Similar considerations can be made for other loop filters, for example, SAO or CCSO can also be applied to reduce ringing artifacts and / or improve quality by classifying samples in a region of a virtual reference frame and then adding an offset to each sample based on its category.

[0151] When a virtual reference frame is used as a reference frame during the coding of an input pixel block, different in-loop filter strategies can be applied to the current frame to improve its quality. If the reference of one block is a virtual reference frame and a neighboring block does not use a virtual reference frame, the decoder can decide to use a specific "deblocking filter" strength or filtering method targeted at blocks that utilize different references for prediction, regardless of whether the virtual reference frame itself uses that same reference. If the neighboring blocks both utilize DMVP prediction, only the corresponding motion vectors and lighting parameters can be considered. This could possibly make it easier from memory management. In an alternative method, the virtual reference frame is "decomposed" into real references and the motion information is fully calculated. In this case, the deblocking process can evaluate the real references and motion information used for prediction when performing any in-loop filter operations. This can be much more accurate and can improve the loop filtering process, but at the cost of higher complexity.

[0152] Signaling data may be defined at the frame or tile level in the coding protocol to indicate which method(s) is selected.

[0153] As described, the above-mentioned DMVP technique is applied when the motion between reference frames is translational motion. However, some video sequences may contain complex motion such as affine, including rotation, zoom, and deformable motion. In the case of affine and deformable motion, the encoder may decide not to apply the DMVP technique. The encoder may signal the DMVP mode at the block level (e.g., using the field "enable_dmvp"). However, other embodiments provide techniques for signaling the DMVP mode in a flexible and adaptive manner that avoids the performance loss introduced by the DMVP mode in the case of complex motion.

[0154] In one embodiment, the enable_dmvp flag may be signaled at the slice, tile, or even video or frame level. For example, this information may be signaled in the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), picture, tile, or slice header of a standard or technology like HEVC or VVC. In this case, the decoder assumes that the enable_dmvp flag is on or off for the entire video, frame, or subframe being coded.

[0155] In another embodiment, the enable_dmvp flag may be signaled at a lower level unit such as a region / collection of blocks, or at the coding block / unit level.

[0156] In further embodiments, the enable_dmvp flag may be signaled separately for different frames / slices and blocks, for example, depending on the frame or slice characteristics, the enable_dmvp flag may be signaled as in paragraph

[0154] or paragraph

[0155] for that particular frame / slice / tile.

[0157] In another embodiment, the enable_dmvp flag may be signaled based on the selected coding mode and parameters in the neighboring block / slice / tile. For example, a coding block / slice / tile can use information from the neighboring block / tile / frame above or to the left and / or information about the current block / slice / tile / frame to infer the enable_dmvp flag without signaling it, or use this information to signal the enable_dmvp flag more efficiently by using a separate entropy coding context depending on the neighboring block information. In the case of slices and tiles, the last coding block of the neighboring slice / tile can be used.

[0158] In a further embodiment, when a tile, slice, or superblock contains multiple smaller coding blocks (e.g., four sub-coding blocks within a coding tree unit or slice / tile), the enable_dmvp flag may be signaled for only one of the coding blocks, e.g., the first sub-block, and other sub-blocks in the tile, slice, or superblock can infer the enable_dmvp flag value from neighboring coding blocks.

[0159] In another embodiment, in the case of separate coding trees for luma and chroma channels (e.g., similar to the dual trees in the H.266 / VVC coding standard), the enable_dmvp flag may be signaled for the luma coding block and inferred for the chroma block from the co-located luma block. In an alternative design, they may be signaled separately and the context derivation for entropy coding may be based on the color components.

[0160] In further embodiments, the enable_dmvp flag may be signaled at the block level to ensure block-level control of DMVP, and may be simultaneously controlled at higher levels such as video or frame / slice level, VPS (Video Parameter Set), SPS (Sequence Parameter Set), or PPS (Picture Parameter Set) layers. In this case, the condition for signaling the enable_dmvp flag at the block level will also depend on the higher level flag. If the higher level video or frame / slice level, VPS, SPS, or PPS flag is 0, all lower block level enable_dmvp flags are inferred as 0. If the upper level flag is 1, the encoder can determine the value of enable_dmvp on a block-by-block basis. The control at the higher level ensures that if for certain video content and / or some frames where DMVP is not beneficial, the signaling of enable_dmvp at the block can be turned off to minimize signaling overhead.

[0161] In another embodiment, for complex and more efficient algorithms such as on-the-fly updates, affine transformation models, acceleration, and multiple hypothesis prediction, enable_dmvp can be signaled as a parameter at the block level. This can help ensure that the block level control can adaptively switch between different modes based on the features and statistics of the current block, so that the corresponding block in the DMVP frame can be interpolated using the affine transformation model or multiple hypothesis prediction based on the content features of the block. This can improve the video quality of the DMVP frame and reduce prediction residuals. The enable_dmvp parameter of DMVP can also be controlled at higher levels simultaneously, such as the video or frame / slice level, VPS, SPS, or PPS layers. The highest level can allow the maximum number of possible values, while the lower levels can only allow a subset of the allowed possible values ​​from the higher levels. For example, N possible values ​​of the enable_dmvp parameter may be allowed in the SPS, which indicates the number of possible modes or mode groups that may be allowed in the lower layers, and then in the PPS it may further refine the number of possible modes (e.g., M possible options out of N) and move to the block level where the options may be, for example, flags. Such a hierarchical layer structure of signaling the enable_dmvp parameter may improve signaling efficiency.

[0162] As mentioned above, the virtual reference frame is used in video coding in addition to the coded input frame designated as the reference frame (e.g., the one stored in the reference picture buffer 1750 (FIG. 17) and 1850 (FIG. 18)). Also, the interpolated frame is temporally aligned with the coded input frame(s) to some extent since it is already mixed from one forward reference frame and one backward reference frame. Therefore, the virtual reference frame can be used for single prediction. If the reference block of the current coding block is from the virtual reference frame, this prediction mode is called DMVP mode as one alternative to the available inter prediction modes. A more complex and efficient prediction mode can be constructed by using one predictor from the DMVP frame and another predictor from another reconstructed reference frame. This then creates a bi-predictive frame using the DMVP frame and the other reference frame. In this case, since the DMVP frame is originally generated using two reference frames, three or more levels of hypothesis prediction can be utilized. This can provide better prediction and reduce prediction residuals.

[0163] The selection of the DMVP mode may be implemented in various ways at different levels of the coding syntax.

[0164] In one embodiment, the selection scheme may be selected based on the frame / slice / tile type, e.g., intra frame / slice / tile or inter frame / slice / tile. enable_dmvp may be turned off for intra frame / slice / tile.

[0165] In another embodiment, the scheme can be selected based on the number of available reference frames. The enable_dmvp flag can be turned off if there is only one available reference frame. However, the scheme can use more than two reference frames, which can use more previously decoded reference frames to form a better prediction.

[0166] In further embodiments, the scheme may be selected based on signal statistics of the contents of a frame, slice, tile, region, or superblock, for example, if the majority of motion types in a frame indicate affine and deformable motion, the enable_dmvp flag may be turned off.

[0167] In another embodiment, the scheme may be selected based on other than the signal statistics of the current coding block, for example on characteristics of the spatial or temporal neighboring blocks of the current block.

[0168] In further embodiments, the scheme may be selected separately for different frames / slices / tiles and blocks. For example, depending on the frame or slice characteristics, the enable_dmvp flag may be turned on for a particular frame / slice / tile. However, different slices / tiles / frames may have it turned off.

[0169] In another embodiment, if the enable_dmvp flag is not turned off in a higher level syntax, the scheme can be selected at the block level based on a rate-distortion optimization (RDO) decision. For example, as shown in Figure 14, the DMVP mode is signaled after the inter or intra mode flag and before the conventional single / mixed inter prediction. If a large amount of DMVP mode is selected, it can also be signaled before the skip flag based on statistical analysis so that some bits can be saved for signaling skip and inter-intra modes.

[0170] In further embodiments, in case of complex and more efficient algorithms such as on-the-fly update, affine transformation model, acceleration, multiple hypothesis prediction, etc., to ensure that the block-level control can adaptively switch between different modes based on the characteristics and statistics of the current block, the enable_dmvp parameter can be signaled as a non-binary parameter at the block level, so that the corresponding block in the DMVP frame can be interpolated using the affine transformation model or multiple hypothesis prediction based on the content characteristics of the block. This can improve the video quality of the DMVP frame and reduce the prediction residual.

[0171] As described above, a frame, tile, slice, or region may be coded using a temporal interpolation mode that utilizes the DMVP process. In one embodiment, the encoder may signal a TIP_mode value that indicates the coding decision made by the encoder.

[0172] In one embodiment, TIP_mode can indicate three states, for example as follows: [Table 1] Additional syntax elements may be included in the signaling protocol to provide other types of information, for example, a syntax element may also be defined to indicate whether a loop filter may be applied to the current frame, or a syntax element may also be defined to indicate whether to create a new TIP virtual reference frame / region or to reuse an existing TIP virtual reference frame / region from a previous coding frame.

[0173] The TIP mode indicator TIP_mode can be represented in the signaling protocol in various ways.

[0174] In one embodiment, the TIP_mode indicator may be signaled at the slice, tile, or even video or frame level. For example, this information may be signaled in the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), picture, tile, or slice header of a standard or technology such as HEVC or VVC. In this case, the decoder assumes that the TIP_mode flag is determined for the entire video, frame, or subframe being coded.

[0175] In another embodiment, the TIP_mode indicator may be signaled at a lower level unit in the signaling protocol, such as a region / collection of blocks, or at the coding block / unit level.

[0176] In further embodiments, the TIP_mode indicator may be signaled separately for different frames / slices and blocks. For example, depending on the frame or slice characteristics, the TIP_mode indicator may be signaled as in paragraph

[0174] or paragraph

[0175] for that particular frame / slice / tile.

[0177] In another embodiment, the TIP_mode indicator may be signaled based on the selected coding mode and parameters in spatial or temporal neighboring blocks / slices / tiles. For example, a coding block / slice / tile may use information from the neighboring blocks / tiles / frames above or to the left and / or information about the current block / slice / tile / frame to infer the TIP_mode indicator without signaling it, or use this information to signal the TIP_mode indicator more efficiently by using a separate entropy coding context depending on the neighboring block information. In the case of slices and tiles, the last coding block of the neighboring slice / tile may be used.

[0178] In further embodiments, in applications employing separate coding trees for the luma and chroma channels, the TIP_mode indicator can be signaled for the luma channel and can be coded or inferred for the chroma channels based on previous coded information. In alternative designs, they can be signaled separately and the context derivation for entropy coding can be based on each color component.

[0179] In another embodiment, the TIP_mode indicator may be signaled at the tile or region level to ensure fine-grained control of the TIP mode, and may be simultaneously controlled at higher levels, such as the video or frame / slice level, VPS, SPS, or PPS layer. In this case, the condition for signaling the TIP_mode indicator at the tile or region level also depends on the syntax of the higher level. For example, if the corresponding syntax of the higher level video or frame / slice level, VPS, SPS, or PPS is 0, the TIP_mode indicators of all lower tile or region levels will be inferred as 0. If the corresponding syntax of the higher level is 1, the encoder can determine the value of TIP_mode on a tile or region basis. The control at the higher level ensures it for a particular video content and / or a particular frame.

[0180] As described above, a frame, tile, slice, or region may be predictively coded using the DMVP process.

[0181] In one embodiment, the signaling protocol may include a syntax element in the current frame, slice, tile, and / or region header to indicate the coding mode to be applied. For example, the coding mode may indicate one of three modes, referred to as "no_skip", "semi_skip", or "full_skip" modes, respectively, as follows: [Table 2] Additionally, the signaling protocol may include other syntax elements to indicate whether a loop filter may be applied to the current frame.

[0182] For Semi_skip and Full_skip modes, additional parameters may also be signaled that can control how the DMVP process is performed. For example, a parameter may be used to indicate the process to be used to fill holes, another parameter may indicate that only a single motion vector should be generated per block or pixel, the granularity of the block, or whether the motion model used may use a constant velocity or an acceleration-based motion model.

[0183] Signaling the prediction mode type of a region, referred to herein as the "advanced_prediction_mode" indicator, can be done in a variety of ways.

[0184] In one embodiment, the advanced_prediction_mode indicator may be signaled at a slice, tile, or group of frames, video, or frame level. For example, this information may be signaled in a video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), picture, tile, or slice header of a standard or technology such as HEVC or VVC. In this case, the decoder assumes that the advanced_prediction_mode syntax element is determined for the entire video, frame, or subframe being coded.

[0185] In another embodiment, the advanced_prediction_mode indicator may be signaled at a lower level unit, such as a region / set of blocks, or at the coding block / unit level.

[0186] In further embodiments, the advanced_prediction_mode indicator may be signaled separately for different frames / slices and blocks. For example, depending on the frame or slice characteristics, the advanced_prediction_mode indicator may be signaled as in paragraph

[0184] or paragraph

[0185] for that particular frame / slice / tile.

[0187] In another embodiment, the advanced_prediction_mode indicator may be signaled based on the selected coding mode and parameters in spatial or temporal neighboring blocks / slices / tiles. For example, a coding block / slice / tile may use information from the neighboring blocks / tiles / frames above or to the left and / or information about the current block / slice / tile / frame to infer the advanced_prediction_mode indicator without signaling it, or use this information to signal the advance_prediction_mode indicator more efficiently by using a separate entropy coding context depending on the neighboring block information. In the case of slices and tiles, the last coding block of the neighboring slice / tile may be used.

[0188] In a further embodiment, when separate coding trees for luma and chroma channels are employed, the advanced_prediction_mode indicator can be signaled for the luma channel and can be coded or inferred for the chroma channels based on previous coded information. In an alternative design, they can be signaled separately and the context derivation for entropy coding can be based on the color components.

[0189] In another embodiment, the advanced_prediction_mode indicator can be signaled at the tile or region level to ensure fine-grained control of the advanced prediction mode, and can be simultaneously controlled at higher levels, such as video or frame / slice level, VPS, SPS, or PPS layers. In this case, the condition for signaling the advanced_prediction_mode indicator at the tile or region level also depends on the syntax of the higher level. For example, if the corresponding syntax of the higher level video or frame / slice level, VPS, SPS, or PPS is 0, the advanced_prediction_mode indicator of all lower tile or region levels will be inferred as 0. If the corresponding syntax of the higher level is 1, the encoder can determine the value of advanced_prediction_mode on a tile or region basis. The control at the higher level ensures optimal adaptation of this mode to a particular video content and / or a particular frame.

[0190] The DMVP prediction mode has been described so far as it applies to single prediction, however, it can be extended to unidirectional, bidirectional, inter-intra and skip modes to further reduce temporal redundancy and improve compression efficiency.

[0191] The DMVP prediction mode has been described so far as being applied when one forward (past) reference frame and one backward (future) reference frame are available, and motion vector projection is performed via interpolation. However, it can be extended to unidirectional inter prediction, where only one side of the reference frame is available. Motion vectors can be projected by extrapolation.

[0192] As described, by utilizing previously coded motion information to interpolate a virtual reference frame that is temporally close to the current frame, the interpolated frame can be used for prediction of the current input frame, reducing temporal redundancy to improve coding efficiency.

[0193] As described, by signaling the enable_dmvp parameter at different levels (e.g., tile / slice / frame / video and picture level, block level), the coding system may have more flexibility and be able to better adapt to different environments and applications. For example, in some applications, the majority of the motion types between frames may be translational motion, and signaling the enable_dmvp parameter with a translation model may provide better coding performance. However, in some of the sequences that may include many non-translational motions such as affine / zoom in / zoom out / scaling, etc., utilizing an affine transformation model to generate DMVP frames or blocks may provide better coding performance. Thus, the signaling of enable_dmvp may include an affine transformation model. In alternative embodiments, additional models, such as for handling and signaling the derivation of multi-hypothesis predictions using more than two references for prediction, may provide better coding performance and may be included in the enable_dmvp parameter that may be signaled. Furthermore, in another embodiment, if the entire frame or tile has the same transformation model, allowing signaling of the enable_dmvp parameter at the frame or tile level may help to reduce the signaling overhead at the block level. Similarly, if consecutive frames contain the same transformation model, the transformation model may be signaled at the SPS or PPS level to further reduce the signaling overhead. On the other hand, to provide more flexibility, an adaptive hierarchical solution may be provided by allowing the maximum number of possible values ​​at the high level, and then at the lower level, only a subset of possible values ​​from the level immediately above may be allowed. For example, if N possible values ​​of the enable_dmvp parameter are allowed in the SPS, which indicates the maximum number of possible modes or mode groups that may be allowed in the lower layer, then in the PPS, the number of possible modes may be further restricted (M possible options out of N).This can be continued further up to the block level where the option is a flag or can be inferred if a limit has already been reached to one possible mode at a higher level. Such a hierarchical structure of the signaling enable_dmvp parameter can improve signaling efficiency.

[0194] As described above, a TIP frame may be generated by utilizing existing motion vectors from a previously coded frame that are available at both the encoder and the decoder. It is expected that a TIP frame is often better temporally aligned with a current frame to be coded than a separated reference frame, and therefore may provide better prediction candidates for the current frame. Therefore, when an encoder provides a TIP frame as an additional reference frame (tip_frame_mode=1 mode) and uses the TIP frame as a prediction reference for pixel blocks of a current frame, the value of the motion vector of a pixel block generated from the TIP reference frame after motion estimation (ME) often tends to be 0 or close to 0. With this prior information, coding of pixel blocks may be performed more efficiently than using the motion vector differential (MVD)-based motion signaling described above.

[0195] FIG. 19 illustrates a method 1900 for signaling coding information according to an embodiment of the present disclosure. The method 1900 may begin when an encoder codes a current frame using a TIP frame as a reference frame. It is determined whether the coding produces a motion vector having a value that is approximately 0 (box 1910). If so, the method 1900 may communicate the coding of the current frame using signaling indicating that the frame is coded as a TIP frame with a 0-valued motion vector (box 1930). If not, the method 1900 may communicate the coding of the current frame using signaling indicating that the frame is coded using a TIP coding technique, providing additional data including values ​​of the motion vectors (e.g., by motion vector prediction indexes and motion vector delta values, as well as motion compensation interpolation filter identifiers).

[0196] As an example, the signaling provided in box 1930 may include a syntax element called zero_mv to indicate whether the MV of a block indicated as a TIP mode block is 0. At the encoder, the encoder sets and signals the syntax zero_mv element equal to 1 if the value of TIP MV is 0 and no other syntax elements related to MV signaling need to be signaled. If the value of TIP MV is non-zero, the existing syntax elements related to MV signaling need to be signaled. With this new signaling method of TIP MV, a significant amount of signaling overhead can be saved when TIP MV is 0.

[0197] The technique of FIG. 19 may be applied to coded data with motion vectors that are not exactly 0. In some coding applications, the encoder may truncate motion vectors with relatively low values ​​to 0. For example, when the magnitude of a motion vector in TIP mode is close to 0, e.g., if the motion vector is within ±1 integer sample, the encoder may decide to quantize it to an integer 0 value and then use the zero_mv flag to save signaling overhead.

[0198] In an embodiment, the coding operation may constrain the motion vector range to simplify the application in a hardware environment. FIG. 20 shows the method 2000 applied to an environment where TIP frames are generated on the fly, and FIG. 21 shows the application of the method 2000 to an exemplary coding environment. Given a position (pos_C) of a pixel block to be coded, the method may use a motion vector (mv_0) to derive a position (pos_A) of a reference frame that serves as a prediction reference (box 2010). The method may determine a motion vector (tip_mv) that generates the content of the reference block (pos_A) (box 2020). The method may derive forward and backward motion vectors (tip_mv0, tip_mv1) for the reference block (pos_A) (box 2030). The method may perform motion compensation of the reference block (pos_A) using the forward and backward motion vectors (box 2040), and then generate the reference block (box 2050).

[0199] Hardware implementations may incur additional processing costs when large range motion vectors (MV0) are used, as they may result in significant motion vector fetch and storage costs. In applications, the value of MV0 may be normatively limited to within a range around the current block. With this constraint, the decoder only needs to fetch a small local range (e.g., + / - 4 TMVP units) around the current block, which can significantly reduce cost and complexity.

[0200] In another aspect, the aforementioned prediction techniques can be improved in the case of non-translational motion in the image data. Video sequences generally include complex motions such as zooming, rotation, scaling, etc., and can include more complex motion modeling such as affine motion models, acceleration-based motion models, etc. A prediction model that uses linear motion modeling when generating TIP frames can create coded pixel residuals that can be very large when using tip_frame_mode=2 (directly output TIP frame as current frame, do not code current frame), and both coding pixel residuals and signaling overhead of motion vectors from coding blocks to TIP frames can be large with tip_frame_mode=1 (TIP frame is used as additional reference). In such cases, a generic motion refinement algorithm (e.g., using optical flow, affine, parametric, parabolic, acceleration-accounting high-order motion, etc.) can be applied to refine the TMVP projected motion field to generate better quality TIP frames.

[0201] FIG. 22 illustrates a method 2200 of coding video according to an embodiment of the present disclosure, and FIG. 23 illustrates application of the method 2200 to an example coding scenario. The method 2200 may operate on a TMVP motion vector (tip_mv) for a block to be coded. The method 2200 may temporally scale the TMVP motion vector (tip_mv) and derive TMVP motion vectors (tip_mv0 / tip_mv1) corresponding to both reference frames (box 2010). The method 2200 may generate a reference block (ref_block0 / ref_block1) by motion compensation using the forward and backward motion vectors (tip_mv0 / tip_mv1) (box 2220). The method 2200 may generate pixel statistics for the reference block (ref_block0 / ref_block1) using a motion refinement algorithm such as mean, variance, gradient, correlation, etc. (box 2230). The method 2200 may derive a refined motion vector (refined_mv) for each pixel block based on the calculated pixel statistics (box 2240). The method 2200 may temporally scale the refined motion vector (refined_mv) for each block to derive a corresponding refined motion vector (refined_mv0 / refined_mv1) for both of the reference frames (box 2250). The method 2200 may perform motion compensation using each of the refined motion vectors (tip_mv0' / tip_mv1') to obtain a respective reference block (ref_block0' / ref_block1') (box 2260). The method 2200 may generate a final block of the TIP frame from the reference blocks (ref_block0' / ref_block1') (box 2270).

[0202] In one example, an optical flow refinement algorithm can be used to derive the refined motion vector refined_mv. This algorithm assumes that the intensity values ​​of pixels along the motion trajectory in the video are constant, which is the so-called equal intensity constraint. This can be expressed as follows:

number

[0203] where I(x,y,t) is the current block luma pixel value at position (x,y) at time t. The equal intensity constraint can also be expressed by:

number

[0204] Assume that the two reference blocks at positions x, y are ref_block0 and ref_block1, respectively. For simplicity, the position (x, y) index is omitted below. The refined_mv can be derived from the pixel values ​​and statistics by solving equation (33).

number

number

[0205] When the optical flow refinement algorithm is adopted, the temporal scaling of the refined motion vector (refined_mv) for each block to derive the corresponding refined motion vector refined_mv0 / refined_mv1 may be performed as follows: refined_mv0 x =d1 * refined_mv x (39) refined_mv0 y =d1 * refined_mv y (40) refined_mv1 x =(d2-d1) * refined_mv x (41) refined_mv1 y =(d2-d1) * refined_mv y (42)

[0206] When the optical flow refinement algorithm is adopted, the revised motion vector tip_mv0' / tip_mv1' can be derived as follows:

number

[0207] The above description describes a process for generating blocks for a TIP frame. In the above described examples, the blocks are assumed to be pixel blocks of regular size (e.g., N×N), but the principles of this disclosure apply to pixel blocks that are not of regular shape. The principles of this disclosure apply to pixel blocks that are not square (e.g., N×M pixel blocks), pixel blocks with adaptively sized partitions, or pixel blocks with irregular geometric shapes. For example, the partitions may be defined by a segmentation algorithm.

[0208] In the preceding discussion, the operation of aspects of the present disclosure has been described in the context of video encoders and decoders such as those shown in FIGS. 1 and 2 and 17 and 18. Generally, these components are provided as electronic devices. The video decoder and / or controller may be embodied in integrated circuits such as application specific integrated circuits, field programmable gate arrays and / or digital signal processors. Alternatively, they may be embodied in computer programs executing on a camera device, a personal computer, a notebook computer, a tablet computer, a smartphone, or a computer server. Such computer programs typically include instructions stored on a non-transitory physical storage medium such as an electronic, magnetic, and / or optical based storage device, which are read and executed by a processor. Decoders are typically packaged in consumer electronics devices such as smartphones, tablet computers, gaming systems, DVD players, portable media players, and the like, and may also be packaged in consumer software applications such as video games, media players, media editors, and the like. Of course, these components may also be provided as hybrid systems distributing functionality between dedicated hardware components and programmed general-purpose processors, if desired.

[0209] The video encoder and decoder can exchange video over channels in various ways. They can communicate with each other over a telecommunications and / or computer network, as shown in Figure 1. In yet other applications, the video encoder can output video data to a storage device, such as an electrical, magnetic, and / or optical storage medium, which can later be provided to the decoder. In such applications, the decoder can retrieve the encoded video data from the storage device and decode it.

[0210] Although several embodiments of the invention are specifically illustrated and / or described herein, it will be understood that modifications and variations of the invention are covered by the above teachings and are within the purview of the appended claims without departing from the spirit and intended scope of the invention.

Claims

1. 1. A method of video encoding, comprising: predictive coding of an input frame; if the coded input frame is designated as a reference frame, decoding coded data of the reference frame; storing the decoded reference frame data for use as a prediction reference for a subsequently coded input frame; generating a virtual reference frame of data from a pair of stored reference frames; A method, wherein predictive coding of the input frame comprises a predictive search of the reference frame data and a hypothetical reference frame data.

2. 2. The method of claim 1, further comprising, if the predictive search results in a virtual reference frame being selected, outputting to a decoder data representing the virtual reference frame and data representing the input frame predictively coded with reference to the virtual reference frame.

3. The method of claim 1 , wherein if a virtual reference frame is not selected by the predictive search, then data representing said virtual reference frame is not output to the decoder.

4. 2. The method of claim 1, wherein a first reference frame of the pair has a time position on a first side of the time position of the virtual reference frame, and a second reference frame of the pair has a time position on a second side of the time position of the virtual reference frame.

5. The method of claim 1 , wherein the content of the virtual reference frame is generated by projection of a motion vector extending from the first reference frame of the pair to the second reference frame of the pair.

6. The method of claim 1 , wherein the content of the virtual reference frame is generated by projection of a motion vector that extends from the first reference frame of the pair across multiple reference frames to the second reference frame of the pair.

7. 2. The method of claim 1, wherein when a current pixel block of an input frame is predictively coded with respect to a virtual reference frame, a motion vector of the pixel block is derived by reference to motion vectors of other pixel blocks adjacent to the current pixel block that use a common set of reference frames for prediction as the current pixel block.

8. The method of claim 1 , further comprising: providing data representing the virtual reference frame to a channel, the data including a temporal interpolation mode identifier indicating a decoder use of the virtual reference frame.

9. The time interpolation mode identifier is in the following state: a first state indicating that the decoder should use the virtual reference frame as a reference frame; a second state indicating that the decoder should output the virtual reference frame for display; and a third state indicating that the decoder should output the virtual reference frame for display augmented with additional information provided by an encoder.

10. The method of claim 1 , further comprising providing data identifying a mode of the predictive coding of the input frame.

11. The predictive coding mode information is in the following state: a No_Skip state indicating that the predictive coding generates block-level motion and residual information for the coded input frame content; a Full_Skip state indicating that the predictive coding uses direct motion vector interpolation without using auxiliary coding data; and a Semi_Skip state indicating that the predictive coding uses direct motion vector interpolation and includes auxiliary coding data.

12. 2. The method of claim 1, further comprising: if a pixel block of the input frame is predictively coded with reference to the virtual reference frame, and a motion vector obtained from the predictive coding is less than a threshold, transmitting the coded data of the pixel block along with a syntax element that identifies the motion vector as having a value of 0.

13. The method of claim 1 , further comprising: the predictive search of pixel blocks of the input frame being constrained to a predetermined search window for a collocated location of the virtual reference frame.

14. The method of claim 1 , wherein the content of the virtual reference frame is generated by an optical flow motion vector refinement technique.

15. 1. A computer readable medium storing program instructions that, when executed by a processing device, cause the processing device to: The input frame is subjected to predictive coding, when a coded input frame is designated as a reference frame, decoding coded data of the reference frame; storing the decoded reference frame data for use as a prediction reference for a later coded input frame; generating a virtual reference frame of data from a pair of stored reference frames; A computer-readable medium, wherein predictive coding of the input frame comprises a predictive lookup of the reference frame data and a virtual reference frame data.

16. A coding terminal, comprising: a video encoder having an input for a source video; a video decoder having an input for coded video from the video encoder; a reference picture buffer for storing decoded reference frames output from the video decoder; a virtual reference picture generator having an input for a reference frame from the reference picture buffer; a virtual reference picture buffer having an input for the virtual reference frames output by the virtual reference picture generator.

17. The terminal of claim 16 , further comprising: a predictor having an input for a reference frame from the reference picture buffer and an input for a virtual reference frame from the virtual reference picture buffer.

18. 17. The terminal of claim 16, wherein a first reference frame of the pair has a time position on a first side of the time position of the virtual reference frame, and a second reference frame of the pair has a time position on a second side of the time position of the virtual reference frame.

19. The terminal of claim 16 , wherein a virtual reference picture generator generates the content of the virtual reference frame by projection of a motion vector extending from the first reference frame of the pair to the second reference frame of the pair.

20. 17. The terminal of claim 16, wherein a virtual reference picture generator generates the content of the virtual reference frame by projection of a motion vector that extends from the first reference frame of the pair across multiple reference frames to the second reference frame of the pair.

21. 17. The terminal of claim 16, wherein when a current pixel block of an input frame is predictively coded with respect to a virtual reference frame, the video encoder derives a motion vector for the pixel block by referring to motion vectors of other pixel blocks neighboring the current pixel block that use a common set of reference frames for prediction as the current pixel block.

22. 1. A video decoding method, comprising: predictively decoding the coded frame according to coding parameters provided with the coded frame; if the coded frame is designated as a reference frame, storing the decoded reference frame for use as a prediction reference for a subsequently decoded frame; If the coding parameters identify a coded frame as being coded with reference to a virtual reference frame, deriving the content of the virtual reference frame from a pair of stored reference frames; and decoding the coded frame using the virtual reference frame as a prediction reference.

23. 23. The method of claim 22, wherein a first reference frame of the pair has a time position on a first side of the time position of the virtual reference frame, and a second reference frame of the pair has a time position on a second side of the time position of the virtual reference frame.

24. 23. The method of claim 22, wherein the content of the virtual reference frame is generated by projection of a motion vector extending from the first reference frame of the pair to the second reference frame of the pair.

25. 23. The method of claim 22, wherein the content of the virtual reference frame is generated by projection of a motion vector that extends from the first reference frame of the pair across multiple reference frames to the second reference frame of the pair.

26. 23. The method of claim 22, wherein when a current pixel block of an input frame is predictively coded with respect to a virtual reference frame, a motion vector of the pixel block is derived by reference to motion vectors of other pixel blocks adjacent to the current pixel block that use a common set of reference frames for prediction as the current pixel block.

27. 1. A computer readable medium storing program instructions that, when executed by a processing device, cause the processing device to: predictively decoding the coded frame according to coding parameters provided with the coded frame; if the coded frame is designated as a reference frame, storing the decoded reference frame for use as a prediction reference for a subsequently decoded frame; If the coding parameters identify a coded frame as being coded with reference to a virtual reference frame, deriving the content of said virtual reference frame from a pair of stored reference frames; A computer-readable medium for causing the coded frame to be decoded using the virtual reference frame as a prediction reference.

28. A decoding terminal, a video decoder having an input for coded video data; a reference picture buffer for storing decoded reference frames output from the video decoder; a virtual reference picture generator having an input for a reference frame from the reference picture buffer; a virtual reference picture buffer having an input for the virtual reference frames output by the virtual reference picture generator.

29. 30. The terminal of claim 28, further comprising: a predictor having an input for a reference frame from the reference picture buffer and an input for a virtual reference frame from the virtual reference picture buffer.

30. 30. The terminal of claim 28, wherein a first reference frame of the pair has a time position on a first side of the time position of the virtual reference frame, and a second reference frame of the pair has a time position on a second side of the time position of the virtual reference frame.

31. 30. The terminal of claim 28, wherein a virtual reference picture generator generates the content of the virtual reference frame by projection of a motion vector extending from the first reference frame of the pair to the second reference frame of the pair.

32. 30. The terminal of claim 28, wherein a virtual reference picture generator generates the content of the virtual reference frame by projection of a motion vector that extends from the first reference frame of the pair across multiple reference frames to the second reference frame of the pair.

Images