Calculating optical flow based prediction refinement

Abstract

The video processing method includes determining a first motion displacement Vx(x,y) at a position (x,y) and a second motion displacement Vy(x,y) at a position (x,y) within a video block to be encoded using an optical flow-based method, where x and y are fractional numbers, and Vx(x,y) and Vy(x,y) are determined based on at least the position (x,y) and a center position of a basic video block of the video block, and performing a conversion between the video block and a bitstream representation of the current video block using the first motion displacement and the second motion displacement.

Description

[Technical field]

[0001] This is a national phase application of international patent application PCT / CN2020 / 079675, which was filed on March 17, 2020. , claiming priority to and benefit of International Patent Application No. PCT / CN2019 / 078411 filed on March 17, 2019, International Patent Application No. PCT / CN2019 / 078501 filed on March 18, 2019, International Patent Application No. PCT / CN2019 / 078719 filed on March 19, 2019, and International Patent Application No. PCT / CN2019 / 079961 filed on March 27, 2019. It is what we claim. The entire disclosure of the above application is incorporated by reference as part of the disclosure of this application.

[0002] This patent document relates to video encoding and decoding. [Background technology]

[0003] Despite advances in video compression, digital video still accounts for the largest bandwidth usage in the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth demands for digital video usage are expected to continue to increase. Summary of the Invention

[0004] Various techniques are provided that can be implemented by digital video encoders, transcoders and decoders to use optical flow based prediction refinements in processing video.

[0005] A first example of a video processing method includes a step of determining a refined prediction sample P'(x,y) at a position (x,y) in a video block by modifying the prediction sample P(x,y) at the position (x,y) with a first gradient component Gx(x,y) in a first direction estimated at the position (x,y), a second gradient component Gy(x,y) in a second direction estimated at the position (x,y), a first motion displacement Vx(x,y) estimated for the position (x,y), and a second motion displacement Vy(x,y) estimated for the position (x,y), where x and y are integers; and performing a conversion between the video block and a bitstream representation of the video block using reconstructed sample values ​​Rec(x,y) at the position (x,y) obtained based on the refined prediction sample P'(x,y) and residual sample values ​​Res(x,y).

[0006] A second example of a video processing method includes determining a refined prediction sample P'(x,y) at a position (x,y) in a video block by modifying the prediction sample P(x,y) at the position (x,y) with a first gradient component Gx(x,y) in a first direction estimated at the position (x,y), a second gradient component Gy(x,y) in a second direction estimated at the position (x,y), a first motion displacement Vx(x,y) estimated for the position (x,y), and a second motion displacement Vy(x,y) estimated for the position (x,y), where x and y are integers; and encoding a bitstream representation of the video block to include residual sample values ​​Res(x,y) based on reconstructed sample values ​​Rec(x,y) at the position (x,y) that are based on at least the refined prediction sample P'(x,y).

[0007] A third example of a video processing method includes a step of determining a first motion displacement Vx(x,y) at a position (x,y) and a second motion displacement Vy(x,y) at the position (x,y) in a video block to be encoded using an optical flow based method, where x and y are fractional numbers, Vx(x,y) and Vy(x,y) are determined based on at least the position (x,y) and a center position of a basic video block of the video block, and a step of performing a conversion between the video block and a bitstream representation of a current video block using the first motion displacement and the second motion displacement.

[0008] A fourth example of a video processing method includes determining a first gradient component Gx(x,y) in a first direction estimated at a position (x,y) within a video block and a second gradient component Gy(x,y) in a second direction estimated at the position (x,y) within the video block, where the first gradient component and the second gradient component are based on a final predicted sample value of a predicted sample P(x,y) at the position (x,y), and where x and y are integers; and performing a conversion between the video block and a bitstream representation of a current video block using a reconstructed sample value Rec(x,y) at the position (x,y) obtained based on a residual sample value Res(x,y) plus the final predicted sample value of a predicted sample P(x,y), which has been refined using the gradients Gx(x,y), Gy(x,y).

[0009] A fifth example of a video processing method includes determining a first gradient component Gx(x,y) in a first direction estimated at a position (x,y) within a video block and a second gradient component Gy(x,y) in a second direction estimated at the position (x,y) within the video block, where the first gradient component and the second gradient component are based on a final predicted sample value of a predicted sample P(x,y) at the position (x,y), and where x and y are integers; and encoding a bitstream representation of the video block to include a residual sample value Res(x,y) that is based on a reconstructed sample value Rec(x,y) at the position (x,y), where the reconstructed sample value Rec(x,y) is based on the residual sample value Res(x,y) plus the final predicted sample value of a predicted sample P(x,y) that has been refined using the gradients Gx(x,y), Gy(x,y).

[0010] A sixth example of a video processing method includes determining a first gradient component Gx(x,y) in a first direction estimated at a position (x,y) within a video block and a second gradient component Gy(x,y) in a second direction estimated at the position (x,y) within the video block, where the first gradient component and the second gradient component are based on an intermediate predicted sample value of a predicted sample P(x,y) at the position (x,y), where a final predicted sample value of the predicted sample P(x,y) is based on the intermediate predicted sample value, and where x and y are integers; and performing a conversion between the video block and a bitstream representation of a current video block using a reconstructed sample value Rec(x,y) at the position (x,y) obtained based on the final predicted sample value of the predicted sample P(x,y) and a residual sample value Res(x,y).

[0011] A seventh example of a video processing method includes determining a first gradient component Gx(x,y) in a first direction estimated at a position (x,y) within a video block and a second gradient component Gy(x,y) in a second direction estimated at the position (x,y) within the video block, where the first gradient component and the second gradient component are based on an intermediate predicted sample value of a predicted sample P(x,y) at the position (x,y), where a final predicted sample value of the predicted sample P(x,y) is based on the intermediate predicted sample value, and where x and y are integers; and encoding a bitstream representation of the video block to include residual sample values ​​Res(x,y) based on reconstructed sample values ​​Rec(x,y) at the position (x,y), where the reconstructed sample values ​​Rec(x,y) are based on the final predicted sample value of predicted sample P(x,y) and the residual sample values ​​Res(x,y).

[0012] An eighth example of a video processing method includes the steps of determining a refined prediction sample P'(x,y) at a position (x,y) in an affine-coded video block by modifying the prediction sample P(x,y) at the position (x,y) with a first gradient component Gx(x,y) in a first direction estimated at the position (x,y), a second gradient component Gy(x,y) in a second direction estimated at the position (x,y), a first motion displacement Vx(x,y) estimated for the position (x,y), and a second motion displacement Vy(x,y) estimated for the position (x,y), where the first direction is orthogonal to the second direction, and x and y are integers; determining a reconstructed sample value Rec(x,y) at the position (x,y) based on a refined prediction sample P'(x,y) and a residual sample value Res(x,y); determining a refined reconstructed sample value Rec'(x,y) at the position (x,y) within the affine-coded video block, where Rec'(x,y)=Rec(x,y)+Gx(x,y)×Vx(x,y)+Gy(x,y)×Vy(x,y); and performing a conversion between the affine-coded video block and a bitstream representation of the affine-coded video block using the refined reconstructed sample value Rec'(x,y).

[0013] A ninth example of an image processing method includes a step of determining a refined prediction sample P'(x,y) at a position (x,y) in an affine-coded image block by modifying the prediction sample P(x,y) at the position (x,y) with a first gradient component Gx(x,y) in a first direction estimated at the position (x,y), a second gradient component Gy(x,y) in a second direction estimated at the position (x,y), a first motion displacement Vx(x,y) estimated for the position (x,y), and a second motion displacement Vy(x,y) estimated for the position (x,y), where the first direction is orthogonal to the second direction, and x and y are integers. determining a reconstructed sample value Rec(x,y) at the location (x,y) based on the refined prediction sample P'(x,y) and residual sample values ​​Res(x,y), determining a refined reconstructed sample value Rec'(x,y) at the location (x,y) within the affine-coded video block, where Rec'(x,y)=Rec(x,y)+Gx(x,y)×Vx(x,y)+Gy(x,y)×Vy(x,y); and encoding a bitstream representation of the affine-coded video block to include the residual sample values ​​Res(x,y).

[0014] A tenth example of a video processing method includes the steps of determining a motion vector with 1 / N pixel accuracy for an affine mode video block, determining an estimated motion displacement vector (Vx(x,y), Vy(x,y)) for a position (x,y) within the video block, the motion displacement vector being derived with 1 / M pixel accuracy, N and M being positive integers, and x and y being integers, and using the motion vector and the motion displacement vector to perform a conversion between the video block and a bitstream representation of the video block.

[0015] An eleventh example of an image processing method includes a step of determining two sets of motion vectors for a video block or for a sub-block of the video block, each set of the two sets of motion vectors having a different motion vector pixel precision, and the two sets of motion vectors being determined using a temporal motion vector prediction (TMVP) technique or a sub-block-based temporal motion vector prediction (SbTMVP) technique, and a step of performing a conversion between the video block and a bitstream representation of the video block based on the two sets of motion vectors.

[0016] A twelfth example of a video processing method includes the steps of: performing an interweave prediction technique on a video block to be coded using an affine coding mode by dividing the video block into a plurality of partitions using K sub-block patterns, where K is an integer greater than one; generating a predicted sample for the video block by performing motion compensation using a first of the K sub-block patterns, where a predicted sample at a position (x, y) is denoted as P(x, y), where x and y are integers; and generating a predicted sample for the video block by performing motion compensation using a first of the K sub-block patterns, where a predicted sample at a position (x, y) is denoted as P(x, y), where x and y are integers; For at least one of the remaining subblock patterns, determining an offset value OL(x,y) at the position (x,y) based on a predicted sample derived with a first subblock pattern and a difference between a motion vector derived using the first of the K subblock patterns and a motion vector derived using the L pattern; determining a final predicted sample for the position (x,y) as a function of OL(x,y) and P(x,y); and using the final predicted sample to perform a conversion between a bitstream representation of the video block and the video block.

[0017] A thirteenth example of a video processing method includes a step of performing a conversion between a bitstream representation of a video block and the video block using a final prediction sample, which is derived from a refined intermediate prediction sample by (a) performing an interweave prediction technique and a subsequent optical flow-based prediction refinement technique based on a rule, or (b) performing a motion compensation technique.

[0018] A fourteenth example of a video processing method includes, when bi-prediction is applied, performing a conversion between a bitstream representation of a video block and the video block using a final prediction sample, the final prediction sample being derived from a refined intermediate prediction sample by (a) disabling an interweave prediction technique and performing an optical flow-based prediction refinement technique, or (b) performing a motion compensation technique.

[0019] A fifteenth example of a video processing method includes a step of performing a conversion between a bitstream representation of a video block and the video block using prediction samples, the prediction samples being derived from refined intermediate prediction samples by performing an optical flow-based prediction refinement technique, the performing of the optical flow-based prediction refinement technique depending on only one of a first set of motion displacements Vx(x,y) estimated in a first direction for the video block, or a second set of motion displacements Vy(x,y) estimated in a second direction for the video block, where x and y are integers, and the first direction is orthogonal to the second direction.

[0020] A sixteenth example of a video processing method includes a step of obtaining a refined motion vector for a video block by refining a motion vector of the video block, where the motion vector is refined before performing a motion compensation technique, where the refined motion vector has 1 / N pixel accuracy and the motion vector has 1 / M pixel accuracy; a step of obtaining a final prediction sample by performing an optical flow-based prediction refinement technique on the video block, where the optical flow-based prediction refinement technique is applied to the difference between the refined motion vector and the motion vector; and a step of performing a conversion between a bitstream representation of the video block and the video block using the final prediction sample.

[0021] A seventeenth example of a video processing method includes a step of determining a final motion vector for a video block using a multi-step decoder-side motion vector refinement process, the final motion vector having 1 / N pixel accuracy, and a step of performing a conversion between the current block and a bitstream representation using the final motion vector.

[0022] An 18th example of a video processing method includes a step of obtaining refined intermediate prediction samples of a video block by performing an interweave prediction technique and an optical flow based prediction refinement technique on the intermediate prediction samples of the video block, a step of deriving final prediction samples from the refined intermediate prediction samples, and a step of performing a conversion between a bitstream representation of the video block and the video block using the final prediction samples.

[0023] A 19th example of a video processing method includes a step of obtaining a refined intermediate prediction sample of a video block by performing an interweave prediction technique and a phase variation affine sub-block motion compensation (PAMC) technique on the intermediate prediction sample of the video block, a step of deriving a final prediction sample from the refined intermediate prediction sample, and a step of performing a conversion between a bitstream representation of the video block and the video block using the final prediction sample.

[0024] A twentieth example of a video processing method includes a step of obtaining refined intermediate prediction samples of a video block by performing an optical flow-based prediction refinement technique and a phase variation affine sub-block motion compensation (PAMC) technique on the intermediate prediction samples of the video block, a step of deriving final prediction samples from the refined intermediate prediction samples, and a step of performing a conversion between a bitstream representation of the video block and the video block using the final prediction samples.

[0025] A twenty-first example of a video processing method includes, in converting between a video block and a bitstream representation of the video block, determining a refined prediction sample P'(x,y) at a position (x,y) in the video block by modifying the prediction sample P(x,y) at the position (x,y) as a function of a gradient in a first direction and / or a second direction estimated at the position (x,y) and a first motion displacement and / or a second motion displacement estimated for the position (x,y), and performing the conversion using a reconstructed sample value Rec(x,y) from the refined prediction sample P'(x,y).

[0026] A 22nd example of a video processing method includes a step of determining a first displacement vector Vx(x,y) and a second displacement vector Vy(x,y) at a position (x,y) within the video block corresponding to an optical flow-based method of encoding the video block based on information from adjacent blocks or basic blocks, and a step of performing a conversion between the video block and a bitstream representation of the current video block using the first displacement vector and the second displacement vector.

[0027] A 23rd example of a video processing method includes, in converting between a video block and a bitstream representation of the video block, a step of determining a refined prediction sample P'(x,y) at a position (x,y) in the video block by modifying a prediction sample P(x,y) at the position (x,y), wherein a gradient in a first direction and a gradient in a second direction at the position (x,y) are determined based on the refined prediction sample P'(x,y) and a final prediction value determined from a residual sample value at the position (x,y), and a step of performing the conversion using the gradient in the first direction and the gradient in the second direction.

[0028] A 24th example of a video processing method includes determining a reconstructed sample Rec(x,y) at a position (x,y) in a video block to be affine-coded, refining Rec(x,y) using first and second displacement vectors and first and second gradients at the position (x,y) to obtain a refined reconstructed sample Rec'(x,y), and using the refined reconstructed sample to perform a conversion between the video block and a bitstream representation of a current video block.

[0029] A twenty-fifth example of a video processing method includes, in converting between a video block coded using an affine coding mode and a bitstream representation of the video block, performing interweave prediction of the video block by dividing the video block into a plurality of partitions using K sub-block patterns, where K is an integer greater than one; performing motion compensation using a first of the K sub-block patterns to generate a predicted sample of the video block, the predicted sample at a location (x, y) being denoted as P(x, y); for at least one of the remaining K sub-block patterns, denoted an L pattern, determining an offset value OL(x, y) at the location (x, y) based on P(x, y) and a difference between a motion vector derived using the first of the K sub-block patterns and a motion vector derived using the L pattern; determining a final predicted sample for the location (x, y) as a function of OL(x, y) and P(x, y); and performing the conversion using the final predicted sample.

[0030] In yet another exemplary embodiment, a video encoder apparatus configured to implement one of the methods described in this patent document is disclosed.

[0031] In yet another exemplary embodiment, a video decoder device configured to implement one of the methods described in this patent document is disclosed.

[0032] In yet another aspect, a computer readable medium is disclosed, having processor executable code stored thereon for implementing one of the methods described in this patent document, thus being a non-transitory computer readable medium having code for implementing any of the above methods and methods described in this patent document.

[0033] These and other aspects are described in detail herein. [Brief description of the drawings]

[0034] [Figure 1] 1 illustrates an example of a derivation process for building a merge candidate list. [Diagram 2] 13 shows examples of spatial merge candidate locations. [Diagram 3] 1 shows examples of candidate pairs considered for redundancy checking of spatial merge candidates. [Figure 4] 13 shows examples of the location of the second PU for N×2N and 2N×N partitions. [Diagram 5] 13 shows an illustrative diagram of motion vector scaling for temporal merge candidates. [Figure 6] 4 shows example candidate positions for temporal merge candidates C0 and C1. [Figure 7] 1 illustrates an example of combined bi-predictive merge candidates. [Figure 8] This summarizes the derivation process for motion vector prediction candidates. [Figure 9] 1 shows an illustrative diagram of motion vector scaling for spatial motion vector candidates. [Figure 10] 13 shows an example of advanced temporal motion vector predictor ATMVP motion prediction for a coding unit CU. [Figure 11] An example of one CU having four sub-blocks (AD) and their adjacent blocks (ad) is shown. [Figure 12] FIG. 1 is an example of an explanatory diagram of sub-blocks to which OBMC is applied. [Figure 13] 1 shows an example of adjacent samples used to derive IC parameters. [Figure 14] 1 shows a simplified affine motion model. [Figure 15] 1 shows an example of affine MVF for each sub-block. [Figure 16]An example of an MVP for AF_INTER is shown below. [Figure 17A] 17A-17B show the candidates for AF_MERGE. [Figure 17B] 17A-17B show the candidates for AF_MERGE. [Figure 18] 1 illustrates an example of bilateral matching. [Figure 19] 1 shows an example of template matching. [Figure 20] 1 shows an example of unilateral motion estimation ME in frame rate up-conversion FRUC. [Figure 21] 1 shows an example of an optical flow trajectory. [Figure 22A] 22A-22B show an example of an access location outside a block and how padding is used to avoid extra memory accesses and computations. [Figure 22B] 22A-22B show an example of an access location outside a block and how padding is used to avoid extra memory accesses and computations. [Figure 23] 1 shows an example of an interpolated sample used in BIO. [Figure 24] 1 shows an example of DMVR based on bilateral template matching. [Diagram 25] An example of a sub-block MV VSB and a pixel Δv(i,j) (indicated as an arrow) is shown. [Figure 26] 1 shows an example of how to derive Vx(x,y) and / or Vy(x,y). [Figure 27] 1 illustrates an example of a video processing device. [Figure 28A] 28A-28U are example flow charts relating to methods of video processing. [Figure 28B] 28A-28U are example flow charts relating to methods of video processing. [Figure 28C]28A-28U are example flow charts relating to methods of video processing. [Figure 28D] 28A-28U are example flow charts relating to methods of video processing. [Figure 28E] 28A-28U are example flow charts relating to methods of video processing. [Figure 28F] 28A-28U are example flow charts relating to methods of video processing. [Figure 28G] 28A-28U are example flow charts relating to methods of video processing. [Fig. 28H] 28A-28U are example flow charts relating to methods of video processing. [Figure 28I] 28A-28U are example flow charts relating to methods of video processing. [Figure 28J] 28A-28U are example flow charts relating to methods of video processing. [Figure 28K] 28A-28U are example flow charts relating to methods of video processing. [Figure 28L] 28A-28U are example flow charts relating to methods of video processing. [Figure 28M] 28A-28U are example flow charts relating to methods of video processing. [Figure 28N] 28A-28U are example flow charts relating to methods of video processing. [Figure 28O] 28A-28U are example flow charts relating to methods of video processing. [Figure 28P] 28A-28U are example flow charts relating to methods of video processing. [Figure 28Q] 28A-28U are example flow charts relating to methods of video processing. [Figure 28R] 28A-28U are example flow charts relating to methods of video processing. [Figure 28S] 28A-28U are example flow charts relating to methods of video processing. [Figure 28T]28A-28U are example flow charts relating to methods of video processing. [Figure 28U] 28A-28U are example flow charts relating to methods of video processing. [Figure 29] 1 shows an example of a split pattern in interweave prediction. [Diagram 30] 1 shows an example of phase variation horizontal filtering. [Diagram 31] An example of applying one 8-tap horizontal filtering is shown. [Diagram 32] 1 shows an example of non-uniform phase vertical filtering. [Diagram 33] FIG. 1 is a block diagram illustrating an example of a video processing system in which various techniques disclosed herein may be implemented. [Diagram 34] FIG. 1 is a block diagram illustrating a video encoding system in accordance with some embodiments of the present disclosure. [Diagram 35] FIG. 2 is a block diagram illustrating an encoder according to some embodiments of the present disclosure. [Diagram 36] FIG. 2 is a block diagram illustrating a decoder in accordance with some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] This document provides various techniques that can be used by a decoder of an image or video bitstream to improve the quality of a decompressed or decoded digital video or image. For simplicity, the term "video" is used herein to include both a series of pictures (traditionally called a video) and individual images. Video encoders may also implement these techniques in the encoding process to reconstruct decoded frames that are used for further encoding.

[0036] Section headings are used in this document for ease of understanding, but they do not limit the embodiments and techniques to the corresponding sections, and therefore embodiments from one section can be combined with embodiments from other sections.

[0037] 1. Overview The technology described in this patent document relates to video coding technology. In particular, the technology described relates to motion compensation in video coding. It may be applied to existing video coding standards such as HEVC, or to emerging standards (Versatile Video Coding). It may also be applied to future video coding standards or video codecs.

[0038] 2. Background Video coding standards have evolved primarily through the development of well-known ITU-T and ISO / IEC standards. ITU-T created H.261 and H.263, ISO / IEC created MPEG-1 and MPEG-4 Visual, and these two organizations jointly created the H.262 / MPEG-2 Video, H.264 / MPEG-4 AVC (Advanced Video Coding), and H.265 / HEVC standards. Since H.262, video coding standards have been based on a hybrid video coding structure where transform coding is utilized in addition to temporal prediction. In order to explore future video coding technologies beyond HEVC, VCEG and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, many new methods have been adopted by JVET and put into a reference software named the Joint Exploration Model (JEM). In April 2018, the Joint Video Expert Team (JVET) was launched between VCEG (Q6 / 16) and ISO / IEC JTC1 SC29 / WG11 (MPEG) to work on the VVC standard, which aims to reduce the bit rate by 50% compared to HEVC.

[0039] The latest version of the VVC draft, namely Versatile Video Coding (Draft 2), is: http: / / phenix.it-sudparis.eu / jvet / doc_end_user / documents / 11_Ljubljana / wg11 / JVET-K1001-v7.zip can be found at.

[0040] The latest reference software for VVC, called VTM, is: https: / / vcgit.hhi.fraunhofer.de / jvet / VVCSoftware_VTM / tags / VTM-2.1 can be found at.

[0041] 2.1 Inter Prediction in HEVC / H.265 Each PU with inter prediction has motion parameters for one or two reference picture lists. The motion parameters include a motion vector and a reference picture index. The use of one of the two reference picture lists can also be signaled using inter_pred_idc. The motion vector can be explicitly coded as a delta to the predictor.

[0042] If a CU is coded in skip mode, it is associated with one PU, there are no significant residual coefficients, and there are no coded motion vector deltas or reference picture indexes. A merge mode is defined, according to which motion parameters for a current PU are obtained from neighboring PUs, including spatial and temporal candidates. The merge mode can be applied to any inter-predicted PU, not just to skip mode. An alternative to the merge mode is explicit transmission of motion parameters, in which motion vectors (more precisely, motion vector difference (MVD) compared to a motion vector predictor), corresponding reference picture indexes for each reference picture list, and reference picture list usage are explicitly signaled for each PU. Such a mode is referred to as Advanced motion vector prediction (AMVP) in this disclosure.

[0043] When the signaling indicates that one of the two reference picture lists is used, the PU is generated from one block of samples. This is called "uni-prediction". Uni-prediction is available for both P slices and B slices.

[0044] When the signaling indicates that both of these reference picture lists are used, the PU is generated from samples of the two blocks. This is called "bi-prediction". Bi-prediction is only available for B slices.

[0045] The following text provides details about the inter prediction modes specified in HEVC, starting with merge mode.

[0046] Merge Mode 2.1.1.1. Deriving Candidates for Merge Modes When a PU is predicted using merge mode, an index pointing to an entry in the merge candidate list is parsed from the bitstream and used to extract the motion information. The construction of this list is specified in the HEVC standard and can be summarized according to the following sequence of steps: Step 1: Derive initial candidates - Step 1.1: Spatial candidate derivation - Step 1.2: Redundancy check on spatial candidates - Step 1.3: Derive time candidates Step 2: Insert additional candidates - Step 2.1: Creating bi-prediction candidates - Step 2.2: Insertion of zero motion candidates

[0047] These steps are also shown diagrammatically in Figure 1. For spatial merge candidate derivation, up to four merge candidates are selected among candidates at five different positions. For temporal merge candidate derivation, up to one merge candidate is selected among two candidates. Since a fixed number of candidates is assumed for each PU at the decoder, additional candidates are generated if the number of candidates obtained from step 1 does not reach the maximum number of merge candidates (MaxNumMergeCand) signaled in the slice header. Since the number of candidates is fixed, the index of the best merge candidate is coded using truncated unary binarization (TU). If the size of a CU is equal to 8, all PUs of the current CU share a single merge candidate list, which is the same as the merge candidate list of the 2Nx2N prediction unit.

[0048] The following provides a detailed description of the processes associated with the above steps.

[0049] 2.1.1.2. Spatial candidate derivation In the derivation of spatial merge candidates, up to four merge candidates are selected from the candidates at the positions shown in FIG. 2. The order of derivation is A1, B1, B0, A0, and B2. Position B2 is considered only if any PU among positions A1, B1, B0, A0 is not available (e.g., because it belongs to another slice or tile) or is intra-coded. After the candidate at position A1 is added, the addition of the remaining candidates is subjected to a redundancy check that ensures that candidates with the same motion information are removed from the list so that coding efficiency is improved. In order to reduce computational complexity, the aforementioned redundancy check does not consider all possible candidate pairs. Instead, only the pairs connected by the arrows in FIG. 3 are considered, and a candidate is added to the list only if the corresponding candidate used for the redundancy check does not have the same motion information. Another source of overlapping motion information is a “second PU” that is associated with a partition different from 2N×2N. As an example, FIG. 4 shows the second PU for N×2N and 2N×N cases, respectively. When the current PU is divided into N×2N, the candidate at A1 is not considered for list construction. In fact, adding this candidate would lead to two prediction units with the same motion information, which is redundant to having only one PU in the coding unit. Similarly, when the current PU is divided into 2N×N, position B1 is not considered.

[0050] 2.1.1.3. Time candidate derivation In this step, only one candidate is added to the list. In particular, in this derivation of the temporal merge candidate, a scaled motion vector is derived based on the co-located PU belonging to the picture with the smallest POC difference with the current picture in a given reference picture list. The reference picture list used for the derivation of the co-located PU is explicitly signaled in the slice header. The scaled motion vector for the temporal merge candidate is obtained as shown by the dotted line in FIG. 5, which is scaled from the motion vector of the co-located PU (col_PU) with POC distances tb and td, where tb is defined as the POC difference between the reference picture (curr_ref) of the current picture (curr_pic) and the current picture, and td is defined as the POC difference between the reference picture (col_ref) of the co-located picture (col_pic) and the co-located picture. The reference picture index of the temporal merge candidate is set equal to zero. The actual implementation of the scaling process is defined in the HEVC specification. For a B slice, two motion vectors are obtained, one with respect to reference picture list 0 and the other with respect to reference picture list 1, which are combined to form a bi-predictive merge candidate.

[0051] FIG. 5 shows an illustration of motion vector scaling for temporal merge candidates.

[0052] For a collocated PU(Y) belonging to a reference frame, a position for a temporal candidate is selected between candidates C0 and C1, as shown in Fig. 6. If the PU at position C0 is not available, or is intra-coded, or is outside the current CTU row, position C1 is used. Otherwise, position C0 is used to derive the temporal merge candidate.

[0053] FIG. 6 shows examples of candidate positions for temporal merge candidates C0 and C1.

[0054] 2.1.1.4. Insert additional candidates Besides spatial and temporal merge candidates, there are two more types of merge candidates: combined bi-predictive merge candidates and zero merge candidates. Combined bi-predictive merge candidates are generated by utilizing spatial and temporal merge candidates. Combined bi-predictive merge candidates are used only for B slices. Combined bi-predictive candidates are generated by combining the first reference picture list motion parameters of an original candidate with the second reference picture list motion parameters of another one. If these two tuples provide different motion hypotheses, they form a new bi-predictive candidate. As an example, FIG. 7 shows the case where two candidates in the original list (left side), which are those with mvL0 and refIdxL0, or mvL1 and refIdxL1, are used to create a combined bi-predictive merge candidate that is added to the final list (right side). There are many rules regarding the combinations that are considered to generate these additional merge candidates.

[0055] Zero motion candidates are inserted to fill the remaining entries in the merge candidate list, thus reaching the MaxNumMergeCand capacity. These candidates have a spatial displacement of zero and a reference picture index that starts at zero and increases each time a new zero motion candidate is added to the list. The number of reference frames used by these candidates is 1 and 2 for unidirectional and bidirectional prediction, respectively. Finally, no redundancy check is performed on these candidates.

[0056] 2.1.1.5 Motion Estimation Regions for Parallel Processing To speed up the encoding process, motion estimation can be performed in parallel, whereby motion vectors for all prediction units in a given region are derived simultaneously. The derivation of merge candidates from spatial neighbors can hinder parallel processing, since one prediction unit cannot derive motion parameters from neighboring PUs until its associated motion estimation is completed. To mitigate the tradeoff between encoding efficiency and processing latency, HEVC defines a motion estimation region (MER), whose size is signaled in the picture parameter set using the “log2_parallel_merge_level_minus2” syntax element. When MER is specified, merge candidates that fall in the same region are marked as unavailable and therefore not considered in the list creation.

[0057] 2.1.2 AMVP AMVP exploits the spatial-temporal correlation of motion vectors with neighboring PUs, which is used for explicit transmission of motion parameters. For each reference picture list, a motion vector candidate list is constructed by first checking the availability of the upper-left temporally neighboring PU position, removing redundant candidates, and adding zero vectors to make the candidate list a constant length. Then, the encoder can select the best predictor from the candidate list and transmit the corresponding index pointing to the selected candidate. Similarly, in merge index signaling, the index of the best motion vector candidate is coded using a truncated unary. The maximum value coded in this case is 2 (see Figure 8). The following sections provide details of the motion vector prediction candidate derivation process.

[0058] 2.1.2.1 Derivation of AMVP Candidates In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. In spatial motion vector candidate derivation, two motion vector candidates are finally derived based on the motion vectors of each PU at five different positions as shown in Figure 2.

[0059] In the temporal motion vector candidate derivation, one motion vector candidate is selected from two candidates that are derived based on two different co-located positions. After a first list of spatial-temporal candidates is created, duplicate motion vector candidates in the list are removed. If the number of possible candidates is greater than two, motion vector candidates whose reference picture index in the associated reference picture list is greater than one are removed from the list. If the number of spatial-temporal motion vector candidates is less than two, an additional zero motion vector candidate is added to the list.

[0060] 2.1.2.2. Spatial Motion Vector Candidates In the derivation of spatial motion vector candidates, a maximum of two candidates out of five possible candidates are considered, which are derived from PUs located as shown in Fig. 2, and their positions are the same as the positions of the motion merge. The derivation order for the left side of the current PU is defined as A0, A1, scaled A0, scaled A1. The derivation order for the top side of the current PU is defined as B0, B1, B2, scaled B0, scaled B1, scaled B2. Thus, for each side, there are four cases that can be used as motion vector candidates, two cases do not need to use spatial scaling, and two cases use spatial scaling. The four different cases are summarized as follows: No spatial scaling - (1) Same reference picture list and same reference picture index (same POC) - (2) Different reference picture lists, but the same reference picture (same POC) Spatial Scaling - (3) Same reference picture list, but different reference pictures (different POC) - (4) Different reference picture lists and different reference pictures (different POCs)

[0061] The case without spatial scaling is checked first, followed by spatial scaling. Spatial scaling is considered when the POC is different between the reference pictures of the neighboring PU and the reference picture of the current PU, regardless of the reference picture list. If all PUs of the left candidate are not available or are intra-coded, scaling is enabled for the above motion vector to aid in the parallel derivation of the left and above MV candidates. Otherwise, spatial scaling is not enabled for the above motion vector.

[0062] FIG. 9 shows an illustration of motion vector scaling for spatial motion vector candidates.

[0063] In the spatial scaling process, the motion vectors of neighboring PUs are scaled in a similar manner as for temporal scaling, as shown in Figure 9. The main difference is that the reference picture list and the index of the current PU are given as input, and the actual scaling process is the same as that of temporal scaling.

[0064] 2.1.2.3. Temporal Motion Vector Candidates Apart from the derivation of the reference picture index, all the processes for the derivation of temporal merge candidates are the same as for the derivation of spatial motion vector candidates (see FIG. 6). The reference picture index is signaled to the decoder.

[0065] 2.2. New Inter Prediction Method in JEM 2.2.1 Sub-CU-based motion vector prediction In JEM with QTBT, each CU can have at most one set of motion parameters for each prediction direction. In the encoder, two sub-CU level motion vector prediction methods are considered by splitting a large CU into multiple sub-CUs and deriving motion information for all sub-CUs of the large CU. The alternative temporal motion vector prediction (ATMVP) method allows each CU to fetch multiple sets of motion information from multiple blocks smaller than the current CU in the co-located reference picture. In the spatial-temporal motion vector prediction (STMVP) method, the motion vector of a sub-CU is recursively derived by using the temporal motion vector predictor and the spatial neighboring motion vector.

[0066] In order to preserve more accurate motion fields for sub-CU motion prediction, currently motion compression on reference frames is disabled.

[0067] 2.2.1.1. Alternative Temporal Motion Vector Prediction In the alternative temporal motion vector prediction (ATMVP) method, the temporal motion vector prediction (TMVP) of the motion vector is modified by fetching multiple sets of motion information (including motion vectors and reference indexes) from multiple blocks smaller than the current CU. As shown in Figure 10, a sub-CU is a square NxN block (N is set to 4 by default).

[0068] ATMVP predicts motion vectors of multiple sub-CUs in a CU in two steps. The first step is to identify corresponding blocks in a reference picture using a so-called time vector. The reference picture 1050 is also called a motion source picture. The second step is to split the current CU 1000 into sub-CUs 1001, as shown in FIG. 10, and obtain the motion vector and reference index of each sub-CU from the block corresponding to each sub-CU.

[0069] In the first step, the reference picture and corresponding block are determined by the motion information of the spatial neighboring blocks of the current CU. To avoid the repeated scanning process of the neighboring blocks, the first merge candidate in the merge candidate list of the current CU is used. The first available motion vector and its associated reference index are set to be the time vector and index to the motion source picture. Thus, in ATMVP, the corresponding block can be identified more accurately compared with TMVP, and the corresponding block (sometimes called the co-located block) is always in the bottom-right or center position with respect to the current CU.

[0070] In the second step, the corresponding block of the sub-CU is identified by the time vector in the motion source picture by adding the time vector to the coordinates of the current CU. For each sub-CU, the motion information of the corresponding block (the smallest motion grid covering the center sample) is used to derive the motion information for that sub-CU. After the motion information of the corresponding N×N block is identified, it is converted to the motion vector and reference index of the current sub-CU, similar to TMVP in HEVC, where motion scaling and other procedures are applied. For example, the decoder checks whether the low latency condition (i.e., the POC of all reference pictures of the current picture is smaller than the POC of the current picture) is met, and uses the motion vector MVx (the motion vector corresponding to the reference picture list X) to derive the motion vector MV for each sub-CU. y (X is equal to 0 or 1, and Y is equal to 1-X).

[0071] 2.2.1.2. Spatial-Temporal Motion Vector Prediction In this method, the motion vectors of sub-CUs are derived recursively according to the raster scan order. Figure 11 illustrates this concept. Consider an 8x8 CU that contains four 4x4 sub-CUs A, B, C, and D. Label the adjacent 4x4 blocks in the current frame as a, b, c, and d.

[0072] The motion derivation for sub-CU A starts by identifying its two spatial neighbors. The first neighbor is an N×N block (block c) above sub-CU A. If this block c is not available or is intra-coded, other N×N blocks above sub-CU A are examined (starting from block c and going from left to right). The second neighbor is a block (block b) to the left of sub-CU A. If block b is not available or is intra-coded, other blocks to the left of sub-CU A are examined (starting from block b and going from top to bottom). The motion information obtained from these neighboring blocks for each list is scaled relative to the first reference frame for a given list. Then, the temporal motion vector predictor (TMVP) of sub-block A is derived by following the same TMVP derivation procedure as specified in HEVC. The motion information of the co-located block at position D is fetched and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors (maximum of three) are averaged separately for each reference list. The averaged motion vector is assigned as the motion vector of the current sub-CU.

[0073] 2.2.1.3. Sub-CU Motion Prediction Mode Signaling Sub-CU modes are enabled as additional merge candidates, and no additional syntax elements are required to signal these modes. Two additional merge candidates are added to the merge candidate list of each CU to represent ATMVP and STMVP modes. Up to seven merge candidates are used when the sequence parameter set indicates that ATMVP and STMVP are enabled. The coding logic of these additional merge candidates is the same as for the merge candidates in HM, which means that for each CU in a P slice or B slice, two more RD checks are required for the two additional merge candidates.

[0074] In JEM, all bins of the merge index are context coded by CABAC, whereas in HEVC, only the first bin is context coded and the remaining bins are context-bypass coded.

[0075] 2.2.2. Adaptive Motion Vector Difference Resolution In HEVC, when use_integer_mv_flag is equal to 0 in the slice header, the motion vector difference (MVD) (between the motion vector and the predicted motion vector of the PU) is signaled in units of 1 / 4 luma samples. In JEM, locally adaptive motion vector resolution (LAMVR) is introduced. In JEM, the MVD can be coded in units of 1 / 4 luma samples, integer luma samples, and 4 luma samples. This MVD resolution is controlled at the coding unit (CU) level, and an MVD resolution flag is conditionally signaled for each CU that has at least one non-zero MVD component.

[0076] For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether ¼ luma sample MV precision is used for that CU. If the first flag (equal to 1) indicates that ¼ luma sample MV precision is not used, another flag is signaled to indicate whether integer luma sample MV precision or 4 luma sample MV precision is used.

[0077] If the first MVD resolution flag of a CU is zero or not coded for a CU (meaning all MVDs in the CU are zero), then 1 / 4 luma sample MV resolution is used for that CU. If a CU uses integer luma sample MV precision or 4 luma sample MV precision, the MVPs in the AMVP candidate list for that CU are rounded to the corresponding precision.

[0078] At the encoder, a CU-level RD check is used to determine which MVD resolution should be used for a CU, i.e., three CU-level RD checks are performed, one for each MVD resolution. To accelerate the encoder speed, the following coding scheme is applied in JEM: In RD test of a CU with normal 1 / 4 luma sample MVD resolution, the motion information (integer luma sample precision) of the current CU is stored. In RD test of the same CU with integer luma sample and 4 luma sample MVD resolution, the stored motion information (after rounding) is used as the starting point for motion vector refinement in a smaller range, to avoid repeating the time-consuming motion estimation process three times.

[0079] RD check for CUs at 4 luma sample MVD resolution is conditionally invoked: if the RD cost for a CU at integer luma sample MVD resolution is much larger than that at 1 / 4 luma sample MVD resolution, RD check at 4 luma sample MVD resolution for that CU is skipped.

[0080] 2.2.3 Higher motion vector storage accuracy In HEVC, the motion vector precision is 1 / 4 pel (1 / 4 luma sample and 1 / 8 chroma sample for 4:2:0 video). In JEM, the precision of the internal motion vector storage and merge candidates is increased to 1 / 16 pel. This higher motion vector precision (1 / 16 pel) is used in motion compensated inter prediction for CUs coded in skip / merge mode. For CUs coded in regular AMVP mode, either integer pel or 1 / 4 pel motion is used as described in Section 0.

[0081] The SHVC upsampling interpolation filter, which has the same filter length and normalization factor as the HEVC motion compensated interpolation filter, is used as the motion compensated interpolation filter for the additional fractional pel positions. The chroma component motion vector precision is 1 / 32 sample in JEM, and the additional interpolation filter for the 1 / 32-pel fractional position is obtained by using the average of the filters of two adjacent 1 / 16-pel fractional positions.

[0082] 2.2.4 Overlapped Block Motion Compensation Previously, overlapped block motion compensation (OBMC) has been used in H.263. Unlike in H.263, in JEM, OBMC can be switched on and off using CU-level syntax. When OBMC is used in JEM, it is performed for all motion compensation (MC) block boundaries except the right and bottom boundaries of a CU. Furthermore, it applies to both luma and chroma components. In JEM, an MC block corresponds to a coding block. If a CU is coded in a sub-CU mode (including sub-CU merge, affine, and FRUC modes), each sub-block of that CU is an MC block. In order to handle CU boundaries in a uniform manner, OBMC is performed at the sub-block level for all MC block boundaries by setting the sub-block size equal to 4×4 as shown in FIG. 12.

[0083] When OBMC is applied to a current sub-block, in addition to the current motion vector, the motion vectors of four connected neighboring sub-blocks, if available and not identical to the current motion vector, are also used to derive a prediction block for the current sub-block, and these multiple prediction blocks based on multiple motion vectors are combined to generate a final prediction signal for the current sub-block.

[0084] The prediction block based on the motion vector of the neighboring sub-block is P, where N indicates the indexes of the neighboring upper, lower, left and right sub-blocks. N The predicted block based on the motion vector of the current subblock is denoted as P C It is written as P N If P is based on the motion information of neighboring sub-blocks that contain the same motion information for the current sub-block, then OBMC is N Otherwise, P N All samples of P C are added to the same sample in N The four rows / columns of P C The weighting coefficients {1 / 4, 1 / 8, 1 / 16, 1 / 32} are added to P N The weighting factors {3 / 4, 7 / 8, 15 / 16, 31 / 32} are used for P C The exception is for small MC blocks (i.e., when the height or width of the coding block is equal to 4, or when the CU is coded in sub-CU mode), in which case P N Only two rows / columns of are added to Pc. In this case, the weighting coefficients {1 / 4, 1 / 8} are added to Pc. N The weighting factors {3 / 4,7 / 8} are used for P C is used for P, which is generated based on the motion vectors of vertically (horizontally) adjacent sub-blocks. N So, P N The samples in the same row (column) of P C are added together.

[0085] In JEM, for CUs with size of 256 luma samples or less, a CU-level flag is signaled to indicate whether OBMC is applied to the current CU. For CUs with size larger than 256 luma samples or CUs not coded in AMVP mode, OBMC is applied by default. When OBMC is applied to a CU in the encoder, its impact is taken into account during the motion estimation stage. The prediction signal formed by OBMC using the motion information of the upper and left neighboring blocks is used to compensate the upper and left boundaries of the original signal of the current CU, and then the normal motion estimation process is applied.

[0086] 2.2.5 Local illumination compensation Local Illumination Compensation (LIC) is based on a linear model of illumination changes with a scaling factor a and an offset b, and it is adaptively enabled or disabled for each inter-mode coded coding unit (CU).

[0087] FIG. 13 shows an example of adjacent samples used to derive the IC parameters.

[0088] When LIC is applied to a CU, a least square error method is adopted to derive parameters a and b by using neighboring samples of the current CU and their corresponding reference samples. More specifically, as illustrated in Fig. 13, subsampled (2:1 subsampled) neighboring samples of the CU and corresponding samples in the reference picture (specified by the motion information of the current CU or the current sub-CU) are used. IC parameters are derived and applied separately for each prediction direction.

[0089] If the CU is coded in merge mode, the LIC flag is copied from the neighboring block in a manner similar to the motion information copy in merge mode, otherwise the LIC flag is signaled to indicate whether LIC is applied or not for the CU.

[0090] When LIC is enabled for a picture, an additional CU-level RD check is required to determine whether LIC applies to the CU. When LIC is enabled for a CU, the mean-removed sum of absolute difference (MR-SAD) and the mean-removed sum of absolute Hadamard-transformed difference (MR-SATD) are used instead of SAD and SATD for integer-pixel motion search and fractional-pixel motion search, respectively.

[0091] To reduce the encoding complexity, an encoding scheme is applied in JEM: · LIC is disabled for the entire picture when there is no obvious illumination change between the current picture and its reference pictures. To identify this situation, the histogram of the current picture and the histograms of all reference pictures of the current picture are computed at the encoder. If the histogram difference between the current picture and all reference pictures of the current picture is smaller than a given threshold, LIC is disabled for the current picture, otherwise LIC is enabled for the current picture.

[0092] 2.2.6 Affine Motion Compensated Prediction In HEVC, only the translational motion model is applied for motion compensation prediction (MCP). Meanwhile, in the real world, there are many kinds of motions, e.g., zoom in / out, rotation, perspective motion, and other irregular motions. In JEM, a simplified affine transformation motion compensation prediction is applied. As shown in Figure 14, the affine motion field of a block is described by two control point motion vectors.

[0093] The motion vector field (MVF) of a block is given by:

number

[0094] Here, (v 0x ,v 0y ) is the motion vector of the upper left corner control point, and (v 1x ,v 1y ) is the motion vector of the upper right corner control point.

[0095] To further simplify the motion compensation prediction, sub-block based affine transformation prediction is applied. The sub-block size M×N is expressed as Equation 2:

number

[0096] After being derived by Equation 2, M and N should be adjusted downwards if necessary to make them divisors of w and h, respectively.

[0097] To derive a motion vector for each M×N sub-block, the motion vector of the center sample of each sub-block is calculated according to Equation 1 and rounded to 1 / 16 fractional precision, as shown in Figure 15. Then, the motion compensated interpolation filter described in Section 0 is applied to generate a prediction for each sub-block with the derived motion vector.

[0098] After the MCP, the high-precision motion vectors for each subblock are rounded and stored with the same precision as the regular motion vectors.

[0099] In JEM, there are two affine motion modes: AF_INTER and AF_MERGE. For CUs whose width and height are both greater than 8, the AF_INTER mode can be applied. An affine flag at the CU level is signaled in the bitstream to indicate whether the AF_INTER mode is used. In this mode, the motion vector pair {(v0,v1)|v0={v A ,v B ,v C},v1={v D ,v E A candidate list with}} is constructed using neighboring blocks. As shown in FIG. 16, v0 is selected from the motion vectors of blocks A, B or C. The motion vectors from the neighboring blocks are scaled according to the reference list and the relationship between the POC of the reference for the neighboring block, the POC of the reference for the current CU, and the POC of the current CU. Then, the approach of selecting v1 from neighboring blocks D and E is similar. If the number of candidate lists is less than two, the list is padded with motion vector pairs constructed by duplicating each of the AMVP candidates. If the candidate list is greater than two, the candidates are first sorted according to the consistency of the neighboring motion vectors (the similarity of the two motion vectors of the pair candidate), and only the first two candidates are kept. The RD cost check is used to determine which motion vector pair candidate is selected as the control point motion vector prediction (CPMVP) of the current CU. Then, an index pointing to the position of the CPMVP in the candidate list is signaled in the bitstream. After the CPMVP of the current affine CU is determined, affine motion estimation is applied to find the control point motion vector (CPMV), and the difference between the CPMV and the CPMVP is signaled in the bitstream.

[0100] When a CU is applied in AF_MERGE mode, it obtains the first block coded in affine mode from the valid neighboring reconstructed blocks. And the selection order of the candidate blocks is from the left, top, top right, bottom left, top left, as shown in Fig. 17A. When the neighboring bottom left block is coded in affine mode as shown in Fig. 17B, the motion vectors v2, v3, and v4 of the top left corner, top right corner, and bottom left corner of the CU containing block A are derived. And the top left motion vector v0 of the current CU is calculated according to v2, v3, and v4. Then, the top right motion vector v1 of the current CU is calculated.

[0101] After the CPMVs v0 and v1 of the current CU are calculated according to the simplified affine motion model, Equation 1, the MVF of the current CU is generated. To specify whether the current CU is coded in AF_MERGE mode, an affine flag is signaled in the bitstream when at least one neighboring block is coded in affine mode.

[0102] 2.2.7 Pattern Matching Motion Vector Derivation The pattern matched motion vector derivation (PMMVD) mode is a special merge mode based on the Frame-Rate Up Conversion (FRUC) technique, in which the motion information of blocks is derived at the decoder side without being signaled.

[0103] If the merge flag is true, a FRUC flag is signaled for the CU. If the FRUC flag is false, a merge index is signaled and normal merge mode is used. If the FRUC flag is true, an additional FRUC mode flag is signaled to indicate which method (bilateral matching or template matching) is used to derive motion information for the block.

[0104] At the encoder side, the decision to use the FRUC merging mode for a CU is based on the RD cost selection as it is done for normal merging candidates. That is, for a CU, two matching modes (bilateral matching and template matching) are both examined by using the RD cost selection. The one that leads to the smallest cost is further compared with the other CU modes. If the FRUC matching mode is the most efficient one, the FRUC flag is set to true for that CU and the associated matching mode is used.

[0105] The motion derivation process in FRUC merge mode has two steps. First, a CU-level motion search is performed, followed by sub-CU-level motion refinement. At the CU level, an initial motion vector is derived for the entire CU based on bilateral matching or template matching. A list of MV candidates is first generated, and the candidate that leads to the minimum matching cost is selected as the starting point for further CU-level refinement. Then, a local search based on bilateral matching or template matching is performed around the starting point, and the MV that leads to the minimum matching cost is taken as the MV for the entire CU. Then, starting from the derived CU motion vector, the motion information is further refined at the sub-CU level.

[0106] For example, for a W×H CU motion information derivation, the following derivation process is performed: In the first step, the MV for the whole W×H CU is derived. In the second step, the CU is further divided into M×M sub-CUs. The value of M is calculated as in equation (16), and D is a predefined division depth, which is set to 3 by default in JEM.

number

[0107] As shown in Figure 18, bilateral matching is used to derive the motion information of a current CU by finding the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. Under the assumption of continuous motion trajectories, the motion vectors MV0 and MV1 pointing to two reference blocks are proportional to the temporal distances between the current picture and the two reference pictures, i.e., TD0 and TD1. As a special case, when the current picture is between two reference pictures in time and the temporal distances from the current picture to the two reference pictures are the same, the bilateral matching becomes a bidirectional MV based on mirrors.

[0108] As shown in FIG. 19, template matching is used to derive the motion information of the current CU by finding the closest match between a template in the current picture (the above and / or left neighboring block of the current CU) and a block in the reference picture (the same size as the template). Except for the FRUC merge mode mentioned above, template matching is also applied to the AMVP mode. In JEM, as is done in HEVC, AMVP has two candidates. A new candidate is derived using the template matching method. If the newly derived candidate by template matching is different from the first existing AMVP candidate, it is inserted into the first part of the AMVP candidate list, and the list size is set to 2 (meaning deleting the second existing AMVP candidate). When applied to the AMVP mode, only the CU level search is applied.

[0109] 2.2.7.1 CU-level MV candidate set MV candidates set at the CU level are: (i) If the CU is currently in AMVP mode, the original AMVP candidate (ii) All merge candidates (iii) Some MVs in the interpolated MV field introduced in Section 0 (iv) Above and to the left adjacent motion vectors It is composed of:

[0110] When using bilateral matching, each valid MV of a merge candidate is used as an input to generate an MV pair assuming bilateral matching. For example, one valid MV of a merge candidate is (MVa, refa) in reference list A. Then, a reference picture refb of the bilateral MV paired with it is found in the other reference list B, such that refa and refb are on different sides of the current picture in time. If such a refb is not available in reference list B, refb is determined as a reference in list B that is different from refa and has the smallest temporal distance to the current picture. After refb is determined, MVb is derived by scaling MVa based on the temporal distance between the current picture and refa and refb.

[0111] The four MVs from the interpolated MV field are also added to the CU-level candidate list. More specifically, the interpolated MVs at positions (0,0), (W / 2,0), (0,H / 2) and (W / 2,H / 2) of the current CU are added.

[0112] When FRUC is applied in AMVP mode, the original AMVP candidates are also added to the CU-level MV candidate set.

[0113] At the CU level, up to 15 MVs for an AMVP CU and up to 13 MVs for a merged CU are added to the candidate list.

[0114] 2.2.7.2 Sub-CU level MV candidate set MV candidates set at the sub-CU level are: (i) MV determined from CU-level exploration (ii) Top, left, top-left, and top-right adjacent MVs (iii) Scaled collocated MV from a reference picture (iv) Up to four ATMVP candidates (v) Up to four STMVP candidates It is composed of:

[0115] The scaled MV from the reference picture is derived as follows: All reference pictures in both lists are considered. The MV at the co-located position of the sub-CU in the reference picture is scaled to the reference of the origin CU level MV.

[0116] ATMVP and STMVP candidates are limited to the first four candidates.

[0117] At the sub-CU level, a maximum of 17 MVs are added to the candidate list.

[0118] 2.2.7.3 Generating Interpolated MV Fields Before encoding a frame, an interpolated motion field is generated for the whole picture based on unilateral ME, and the motion field can be used later as a CU-level or sub-CU-level MV candidate.

[0119] First, the motion field of each reference picture in both reference lists is considered at the 4x4 block level. For each 4x4 block, if the motion associated with the 4x4 block in the current picture through that block (as shown in Figure 20) does not have any interpolated motion assigned, the motion of the reference block is scaled to the current picture according to the temporal distances TD0 and TD1 (the same way as MV scaling of TMVP in HEVC), and the scaled motion is assigned to the block in the current frame. If a 4x4 block does not have a scaled MV assigned, the motion of that block is marked as unavailable in the interpolated motion field.

[0120] 2.2.7.4 Interpolation and Matching Costs When motion vectors point to fractional sample positions, motion compensated interpolation is required. To reduce complexity, bilinear interpolation is used instead of the usual 8-tap HEVC interpolation for both bilateral and template matching.

[0121] The calculation of the matching cost is slightly different in different steps. When selecting a candidate from the candidate set at the CU level, the matching cost is the sum of absolute differences (SAD) of bilateral matching or template matching. After the starting MV is determined, the matching cost C of bilateral matching in the sub-CU level search is calculated as follows:

number

[0122] In FRUC mode, MV is derived using only luma samples. The derived motion will be used for both luma and chroma for MC inter prediction. After the MV is determined, the final MC is performed using an 8-tap interpolation filter for luma and a 4-tap interpolation filter for chroma.

[0123] 2.2.7.5 MV Refinement MV refinement is a pattern-based MV search with bilateral matching cost or template matching cost criteria. Two search patterns are supported in JEM: unrestricted center-biased diamond search (UCBDS) and adaptive cross search for MV refinement at CU and sub-CU levels. In both CU and sub-CU level MV refinement, MVs are directly searched with 1 / 4 luma sample MV precision, followed by 1 / 8 luma sample MV refinement. The search range of MV refinement for CU and sub-CU steps is set equal to 8 luma samples.

[0124] 2.2.7.6 Selection of prediction direction in template matching FRUC merge mode In bilateral matching merge mode, bi-prediction is always applied since the motion information of the CU is derived based on the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. There is no such restriction in template matching merge mode. In template matching merge mode, the encoder can choose between uni-prediction from list0, uni-prediction from list1, or bi-prediction for the CU. The choice is based on the cost of template matching as follows: If costBi<=factor*min(cost0,cost1), Bi-prediction is used; Otherwise, if cost0<=cost1, One prediction from list0 is used; In other cases, A single prediction from list1 is used; where cost0 is the SAD of list0 template matching, cost1 is the SAD of list1 template matching, and costBi is the SAD of bi-predictive template matching. The value of factor is equal to 1.25, which means that this selection process is biased towards bi-prediction.

[0125] The inter prediction direction selection only applies to the CU level template matching process.

[0126] 2.2.8 Generalized Bi-Prediction Improvement The Generalized Bi Prediction Improvement (GBi) proposed in JVET-L0646 has been adopted in VTM--3.0.

[0127] GBi was proposed in JVET-C0047. JVET-K0248 improved the gain-complexity tradeoff in GBi and it was adopted in BMS2.1. BMS2.1 GBi applies unequal weights to predictors from L0 and L1 in bi-prediction mode. In inter-prediction mode, multiple weight pairs including equal weight pair (1 / 2,1 / 2) are evaluated based on rate-distortion optimization (RDO) and the GBi index of the selected weight pair is signaled to the decoder. In merge mode, the GBi index is inherited from the neighboring CU. In BMS2.1 GBi, the predictor generation in bi-prediction mode is shown in Equation (1).

[0128] P GBi =(w0*P L0 +w1*P L1 +RoundingOffset GBi )>>shiftNum GBi where P GBi is the final predictor of GBi, and w0 and w1 are the selected GBi weight pairs applied to the predictors in list 0 (L0) and list 1 (L1), respectively. GBi and shiftNum GBi is used to normalize the final predictor in GBi. The supported w1 weight set is {-1 / 4, 3 / 8, 1 / 2, 5 / 8, 5 / 4}, where these five weights correspond to one equal and four unequal weight pairs. The mixture gain, i.e., the sum of w1 and w0, is fixed to 1.0. Thus, the corresponding w0 weight set is {5 / 4, 5 / 8, 1 / 2, 3 / 8, -1 / 4}. This weight pair selection is at the CU level.

[0129] For non-low latency pictures, the weight set size is reduced from 5 to 3, the w1 weight set is {3 / 8, 1 / 2, 5 / 8} and the w0 weight set is {5 / 8, 1 / 2, 3 / 8}. The weight set size reduction for non-low latency pictures applies to BMS2.1 GBi and all GBi tests in this contribution.

[0130] In this JVET-L0646, a combined solution based on JVET-L0197 and JVET-L0296 is proposed to further improve GBi performance. Specifically, the following changes are applied on top of the existing GBi design of BMS2.1:

[0131] 2.2.8.1 GBi encoder bug fix To reduce GBi encoding time, in the current encoder design, the encoder stores the uni-predictive motion vectors estimated from the GBi weights equal to 4 / 8 and reuses them for uni-predictive searches of other GBi weights. This fast encoding method applies to both translational and affine motion models. In VTM2.0, the 6-parameter affine model was adopted along with the 4-parameter affine model. The BMS2.1 encoder does not distinguish between the 4-parameter affine model and the 6-parameter affine model when storing the uni-predictive affine MV when the GBi weights equal to 4 / 8. Therefore, after encoding with the GBi weights 4 / 8, the 4-parameter affine MV may be overwritten by the 6-parameter affine MV. The stored 6-parameter affine MV can be used for the 4-parameter affine ME for other GBi weights, or the stored 4-parameter affine MV can be used for the 6-parameter affine ME. The proposed GBi encoder bug fix separates the 4-parameter affine MV storage from the 6-parameter affine MV storage. The encoder stores the affine MVs for GBi weights based on the affine model type if they are equal to 4 / 8, and for other GBi weights, reuses the corresponding affine MVs based on the affine model type.

[0132] 2.2.8.2 GBi encoder speedup To reduce the encoding time when GBi is enabled, five encoder speed-up methods are proposed.

[0133] (1) Conditionally skip affine motion estimation for some GBi weights.

[0134] In BMS2.1, affine ME, including 4-parameter and 6-parameter affine ME, is performed for all GBi weights. We propose to conditionally skip affine ME for unequal GBi weights (weights not equal to 4 / 8). Specifically, affine ME is performed for other GBi weights only if the affine mode is selected as the current best mode after evaluating 4 / 8 GBi weights and it is not the affine merge mode. If the current picture is a non-low latency picture, bi-predictive ME of the translation model is skipped for unequal GBi weights when affine ME is performed. If the affine mode is not selected as the current best mode or if the affine merge is selected as the current best mode, affine ME is skipped for all other GBi weights.

[0135] (2) Reduce the number of weights for RD cost check for low-delay pictures in encoding with 1-pel and 4-pel MVD precision.

[0136] For low-latency pictures, there are five weights for RD cost check at all MVD precisions including 1 / 4-pel, 1-pel and 4-pel. The encoder will first check the RD cost at 1 / 4-pel MVD precision. We propose to skip some of the GBi weights for RD cost check at 1-pel and 4-pel MVD precision. We order the unequal weights according to their RD cost at 1 / 4-pel MVD precision. Only the first two weights with the smallest RD cost will be evaluated in encoding at 1-pel and 4-pel MVD precision, along with the GBi weight 4 / 8. Thus, for low-latency pictures, at most three weights will be evaluated for 1-pel and 4-pel MVD precision.

[0137] (3) Conditionally skip bi-predictive search if the L0 and L1 reference pictures are the same.

[0138] For some pictures in RA, the same picture may occur in both reference picture lists (list-0 and list-1). For example, in a random access coding configuration in CTC, the reference picture structure for the first group of pictures (GOP) is listed as follows: POC:16, TL:0, [L0: 0 ][L1: 0 ] POC:8, TL:1, [L0: 0 16 ] [L1: 16 0 ] POC:4, TL:2, [L0:0 8 ] [L1: 8 16] POC:2,TL:3,[L0:0 4 ] [L1: 4 8] POC:1, TL:4, [L0:0 2 ] [L1: 2 4] POC:3, TL:4, [L0:2 0] [L1:4 8] POC:6,TL:3,[L0:4 0] [L1:8 16] POC:5, TL:4, [L0:4 0] [L1:68] POC:7, TL:4, [L0:6 4] [L1:8 16] POC:12, TL:2, [L0: 8 0] [L1:16 8 ] POC:10, TL:3, [L0:8 0] [L1:12 16] POC:9, TL:4, [L0:8 0] [L1:10 12] POC:11, TL:4, [L0:10 8] [L1:12 16] POC:14, TL:3, [L0: 12 8] [L1: 12 16] POC:13, TL:4, [L0:12 8] [L1:14 16] POC:15, TL:4, [L0: 14 12] [L1:1614 ]

[0139] It can be seen that pictures 16, 8, 4, 2, 1, 12, 14, and 15 have the same reference picture in both lists. In bi-prediction for these pictures, it is possible that the reference picture is the same in L0 and L1. We propose that the encoder skips bi-prediction ME for unequal GBi weights if 1) the two reference pictures in bi-prediction are the same, and 2) the temporal layer is greater than 1, and 3) the MVD precision is ¼ pel. In affine bi-prediction ME, this fast skip method is only applied to 4-parameter affine ME.

[0140] (4) Skip RD cost check for unequal GBi weights based on the temporal layer and the POC distance between the reference picture and the current picture.

[0141] We propose to skip RD cost evaluation for unequal GBi weights when the temporal stratum is equal to 4 (the highest temporal stratum in RA) or the POC distance between the reference picture (either list-0 or list-1) and the current picture is equal to 1 and the coding QP is greater than 32.

[0142] (5) Change floating-point calculations to fixed-point calculations for unequal GBi weights in ME.

[0143] In the existing bi-predictive search, the encoder fixes the MV of one list and refines the MV of the other list. To reduce the computational complexity, the target is modified before ME. For example, if the MV of list-1 is fixed and the encoder refines the MV of list-0, the target of MV refinement of list-0 is modified by Equation (5). O is the original signal, P1 is the predicted signal of list-1, and w is the GBi weight for list-1.

number

[0144] Here, the term (1 / (8-w)) is stored with floating-point precision, which increases the computational complexity. We convert (5) to (6):

number

number

[0145] 2.2.8.3 CU Size Constraints for GBi In this method, GBi is disabled for small CUs. In inter prediction mode, when bi-prediction is used and the CU region is smaller than 128 luma samples, GBi is disabled without any signaling.

[0146] 2.2.9 Bidirectional Optical Flow 2.2.9.1 Theoretical analysis In BIO, motion compensation is first performed to generate a first prediction (in each prediction direction) of the current block. The first prediction is used to derive spatial gradients, temporal gradients, and optical flow for each subblock / pixel in the block, which are then used to generate a second prediction, i.e., the final prediction of the subblock / pixel. The details are as follows:

[0147] Bidirectional Optical Flow (BIO) is a sample-by-sample motion refinement that is performed on top of block-by-block motion compensation for bidirectional prediction. This sample-level motion refinement does not use signaling.

[0148] FIG. 21 shows an example of an optical flow trajectory.

[0149] The luma value from reference k (k=0,1) after block motion compensation is I (k) Let ∂I (k) / ∂x, ∂I (k) / ∂y are I (k) are the horizontal and vertical components of the gradient. Assuming that the optical flow is valid, the motion vector field (v x ,v y ) is the formula:

number

[0150] Combining this optical flow equation with Hermite interpolation for the motion trajectory of each sample, we obtain the function value I (k) and the derivative ∂I (k) / ∂x, ∂I (k) We obtain a unique cubic polynomial that fits both t = 0 and t = 0. The value of this polynomial at t = 0 is the BIO prediction:

number

[0151] Here, τ0 and τ1 represent the distance to the reference frame as shown in FIG. 21. The τ0 and τ1 are calculated based on the POC for Ref0 and Ref1 as τ0=POC(current)-POC(Ref0), τ1=POC(Ref1)-POC(current). If both predictions are from the same temporal direction (either both from the past or both from the future), the signs are different (i.e., τ0·τ1<0). In this case, BIO is applied only if the predictions are not from the same time point (i.e., τ0≠τ1), both reference regions have non-zero motion (MVx0,MVy0,MVx1,MVy1≠0), and the block motion vectors are proportional to the temporal distance (MVx0 / MVx1=MVy0 / MVy1=-τ0 / τ1).

[0152] Motion vector field (v x ,v y ) is determined by minimizing the difference Δ in values ​​between points A (which are the intersections of the motion trajectory and the reference frame plane in FIG. 9) and B. The model is a local Taylor expansion with respect to Δ:

number

[0153] All values ​​in equation (9) depend on the sample position (i',j'), which has been omitted from the notation so far. Assuming that the motion is consistent within the local surrounding area, we minimize Δ inside a (2M+1) × (2M+1) square window Ω centered on the current prediction point (i,j), where M is equal to 2:

number

[0154] For this optimization problem, JEM uses a simplified approach that first minimizes vertically and then horizontally:

number

number

[0155] To avoid division by zero or very small values, regularization parameters r and m are introduced in equations (11) and (12).

number

[0156] where d is the bit depth of the video samples.

[0157] In order to keep the memory access for BIO the same as for regular bi-predictive motion compensation, all prediction and gradient values ​​I (k) , ∂I (k) / ∂x, ∂I (k)In equation (13), a (2M+1) × (2M+1) square window Ω centered on the current prediction point on the boundary of the prediction block needs to access positions outside the block (as shown in Figure 22A). In JEM, I (k) , ∂I (k) / ∂x, ∂I (k) The value of / ∂y is set equal to the closest available value inside the block. For example, this can be implemented as padding, as shown in Figure 22B.

[0158] With BIO, it is possible to refine the motion field for each sample. To reduce the computational complexity, a block-based design of BIO is used in JEM. The motion refinement is calculated based on 4 × 4 blocks. In block-based BIO, s in equation (13) for all samples in a 4 × 4 block is n The values ​​of are aggregated, and then s n The aggregate value of is used to derive the BIO motion vector offset for that 4x4 block. More specifically, for block-based BIO derivation, the following equation:

number

[0159] In some cases, the MV refinement of BIO may be unreliable due to noise or irregular motion. Therefore, in BIO, the magnitude of the MV refinement is clipped to a threshold thBIO. This threshold is determined based on whether the reference pictures of the current picture are all from one direction. If all the reference pictures of the current picture are from one direction, the value of the threshold is 12×214-d otherwise, it is set to 12 x 2 13-d is set to.

[0160] The gradients for BIO are calculated simultaneously with the motion compensation interpolation, using arithmetic consistent with the HEVC motion compensation process (2D separable FIR). The inputs of this 2D separable FIR are the same reference frame samples as the motion compensation process, and the fractional positions (fracX, fracY) according to the fractional part of the block motion vector. For horizontal gradient ∂I / ∂x, the signal is first vertically interpolated using BIOfilterS corresponding to fractional position fracY with a descaling shift of d-8, and then BIOfilterG corresponding to fractional position fracX with a descaling shift of 18-d is applied horizontally. For vertical gradient ∂I / ∂y, the gradient filter is first applied vertically using BIOfilterG corresponding to fractional position fracY with a descaling shift of d-8, and then the signal is displaced horizontally using BIOfilterS corresponding to fractional position fracX with a descaling shift of 18-d. The length of the interpolation filter for gradient calculation BIOfilterG and the interpolation filter for signal displacement BIOfilterF is made shorter (6 taps) to keep the complexity reasonable. Table 1 shows the filters used for gradient calculation for different fractional positions of block motion vectors in BIO.

[0161] Table 2 shows the interpolation filters used to generate the predicted signal in BIO. [Table 1] [Table 2]

[0162] In JEM, BIO is applied to all bi-predictive blocks when the two predictions are from different reference pictures. If LIC is enabled for a CU, BIO is disabled.

[0163] In JEM, OBMC is applied to blocks after the normal MC process. To reduce computational complexity, BIO is not applied during the OBMC process. This means that BIO is only applied in the MC process for a block when it uses its own MV, but not in the MC process when the MV of the neighboring block is used in the OBMC process.

[0164] 2.2.9.2 BIO in VTM-3.0 proposed in JVET-L0256 Step 1: Determine if BIO is applicable (W and H are the width and height of the current block).

[0165] BIO is not applicable if: Affine coded ATMVP encoded · (iPOC-iPOC0)*(iPOC-iPOC1)>=0 H==4 or (W==4 and H==8) Use weighted prediction · GBi weight is not (1,1).

[0166] BIO is not used if: - The total SAD between two reference blocks (denoted as R0 and R1) is less than a threshold.

number

[0167] Step 2: Data preparation For a WxH block, (W+2)x(H+2) samples are interpolated.

[0168] The inner W×H samples are interpolated with an 8-tap interpolation filter as in normal motion compensation.

[0169] The four outer lines of the samples (black circles in FIG. 23) are interpolated using a bilinear filter.

[0170] For each position, the gradient is calculated on two reference blocks (denoted R0 and R1): Gx0(x,y)=(R0(x+1,y)-R0(x-1,y))>>4 Gy0(x,y)=(R0(x,y+1)-R0(x,y-1))>>4 Gx1(x,y)=(R1(x+1,y)-R1(x-1,y))>>4 Gy1(x,y)=(R1(x,y+1)-R1(x,y-1))>>4.

[0171] For each position, the internal value T1=(R0(x,y)>>6)-(R1(x,y)>>6),T2=(Gx0(x,y)+Gx1(x,y))>>3,T3=(Gy0(x,y)+Gy1(x,y))>>3 B1(x,y)=T2*T2,B2(x,y)=T2*T3,B3(x,y)=-T1*T2,B5(x,y)=T3*T3,B6(x,y)=-T1*T3 It is calculated as:

[0172] Step 3: Calculate the prediction for each block If the SAD between two 4x4 reference blocks is less than a threshold, the BIO is skipped for the 4x4 block.

[0173] Calculate Vx and Vy.

[0174] Calculate the final prediction for each location in the 4x4 block: b(x,y)=(Vx(Gx 0 (x,y)-Gx 1 (x,y))+Vy(Gy 0 (x,y)-Gy 1 (x,y))+1)>>1 P(x,y)=(R 0 (x,y)+R 1 (x,y)+b(x,y)+offset)>>shift.

[0175] b(x,y) is known as the correction term.

[0176] 2.2.9.3 BIO in VTM-3.0 The section numbers below refer to sections in the latest edition of the VVC standard document.

[0177] 8.3.4 Inter-Block Decoding Process - if predFlagL0 and predFlagL1 are equal to 1, DiffPicOrderCnt(currentPic, refPicList0[refIdx0])*DiffPicOrderCnt(currPic, refPicList1[refIdx1])<0, MotionModelIdc[xCb][yCb] is equal to 0, and MergeModeList[merge_idx[xCb]][yCb]] is not equal to SbCol, set the value of bioAvailableg to TRUE. - Otherwise, set the value of bioAvailableFlag to FALSE. ...(original specification text continues) - If bioAvailableFlag is equal to TRUE, the following applies: - The variable Shift is set equal to Max(2,14-bitDepth). - The variables cuLevelAbsDiffThres and subCuLevelAbsDiffThres are set equal to (1<<(bitDepth-8+shift))*cbWidth*cbHeight and 1<<(bitDepth-3+shift). The variable cuLevelSumAbsoluteDiff is set to 0. - for xSbIdx=0..(cbWidth>>2)-1 and ySbIdx=0..(cbHeight>>2)-1, the variables subCuLevelSumAbsoluteDiff[xSbIdx][ySbIdx] and the bidirectional optical flow utilization flag bioUtilizationFlag[xSbIdx][ySbIdx] of the current sub-block are derived as follows: i,j=0..3, subCuLevelSumAbsoluteDiff[xSbIdx][ySbIdx]=Σ i Σ j Abs(predSamplesL0L[(xSbIdx<<2)+1+i][(ySbIdx<<2)+1+j]- predSamplesL1L[(xSbIdx<<2)+1+i][(ySbIdx<<2)+1+j]) bioUtilizationFlag[xSbIdx][ySbIdx]= subCuLevelSumAbsoluteDiff[xSbIdx][ySbIdx]>=subCuLevelAbsDiffThres cuLevelSumAbsoluteDiff+=subCuLevelSumAbsoluteDiff[xSbIdx][ySbIdx] - If cuLevelSumAbsoluteDiff is less than cuLevelAbsDiffThres, set bioAvailableFlag to FALSE. - if bioAvailableFlag is equal to TRUE, the prediction samples in the current luma coding sub-block, predSamplesL[xL+xSb][yL+ySb], for xL=0..sbWidth-1 and yL=0..sbHeight-1, are derived by invoking the bidirectional optical flow sample prediction process specified in clause 8.3.4.5 using the luma coding sub-block width sbWidth, luma coding sub-block height sbHeight, sample arrays preSamplesL0L and preSamplesL1L, and variables predFlagL0, predFlagL1, refIdxL0, refIdxL1.

[0178] 8.3.4.3 Fractional Sample Interpolation Process 8.3.4.3.1 General The inputs to this process are: - a luma position (xSb, ySb) that defines the top left sample of the currently coded sub-block relative to the top left luma sample of the current picture, a variable sbWidth that specifies the width of the currently coded sub-block in luma samples; a variable sbHeight that specifies the height of the currently coded sub-block in luma samples; - the luma motion vector mvLX given in units of 1 / 16 luma samples, - the chroma motion vector mvCLX given in units of 1 / 32 chroma samples, - the selected reference picture sample array refPicLXL and the arrays refPicLXCb and refPicLXCr, - Bidirectional optical flow enable flag bioAvailableFlag.

[0179] The output of this process is: - a (sbWidth) by (sbHeight) array predSamplesLXL of predicted luma sample values ​​if bioAvailableFlag is FALSE, or a (sbWidth+2) by (sbHeight+2) array predSamplesLXL of predicted luma sample values ​​if bioAvailableFlag is TRUE; - two (sbWidth / 2) by (sbHeight / 2) arrays of predicted chroma sample values, preSamplesLXCb and preSamplesLXCr.

[0180] Let (xIntL, yIntL) be the luma position given in full samples, and (xFracL, yFracL) be the offset given in 1 / 16 samples. These variables are only used in this section to specify the position of the fractional samples inside the reference sample arrays refPicLXL, refPicLXCb, and refPicLXCr.

[0181] If bioAvailableFlag is equal to TRUE, then for each luma sample position (xL=-1..sbWidth, yL=-1..sbHeight) inside the predicted luma sample array preSamplesLXL, the corresponding predicted luma sample value preSamplesLXL[xL][yL] is derived as follows: - The variables xIntL, yIntL, xFracL and yFracL are derived as follows: xIntL = xSb-1 + (mvLX[0]>>4) + xL yIntL = ySb - 1 + (mvLX [1] >> 4) + yL xFracL=mvLX[0]&15 yFracL=mvLX[1]&15 - The value of bilinearFiltEnabledFlag is derived as follows: - If xL is equal to -1 or sbWidth, or if yL is equal to -1 or sbHeight, set the value of bilinearFiltEnabledFlag to TRUE. - Otherwise, set the value of bilinearFiltEnabledFlag to FALSE. The predicted luma sample values ​​predSamplesLXL[xL][yL] are derived by invoking the process specified in Section 8.3.4.3.2 using (xIntL, yIntL), (xFracL, yFracL), refPicLXL, and bilinearFiltEnabledFlag as input.

[0182] If bioAvailableFlag is equal to FALSE, then for each luma sample position (xL=0..sbWidth-1, yL=0..sbHeight-1) inside the predicted luma sample array preSamplesLXL, the corresponding predicted luma sample value preSamplesLXL[xL][yL] is derived as follows: - The variables xIntL, yIntL, xFracL and yFracL are derived as follows: xIntL = xSb + (mvLX[0]>>4) + xL yIntL = ySb + (mvLX[1]>>4) + yL xFracL=mvLX[0]&15 yFracL=mvLX[1]&15 - The variable bilinearFiltEnabledFlag is set to FALSE. The predicted luma sample values ​​predSamplesLXL[xL][yL] are derived by invoking the process specified in Section 8.3.4.3.2 using (xIntL, yIntL), (xFracL, yFracL), refPicLXL, and bilinearFiltEnabledFlag as input. ...(original specification text continues) 8.3.4.5 Bidirectional Optical Flow Prediction Process The inputs to this process are: - two variables nCbW and nCbH that define the width and height of the current coding block, - two (nCbW+2) x (nCbH+2) luma prediction sample arrays predSamplesL0 and predSamplesL1, - prediction list usage flags predFlagL0 and predFlagL1, - the reference indices refIdxL0 and refIdxL1, - Bidirectional optical flow utilization flag bioUtilizationFlag[xSbIdx][ySbIdx], for xSbIdx=0..(nCbW>2)-1, ySbIdx=0...(nCbH>2)-1.

[0183] The output of this process is a (nCbW) x (nCbH) array of luma prediction sample values, pbSamples.

[0184] The variable bitDepth is set equal to BitDepthY.

[0185] The variable shift2 is set equal to Max(3,15-bitDepth) and the variable offset2 is set equal to 1<<(shift2-1).

[0186] The variable mvRefinesThres is set equal to 1<<(13-bitDepth).

[0187] For xSbIdx=0..(nCbW>>2)-1 and ySbIdx=0..(nCbH>>2)-1, - if bioUtilizationFlag[xSbIdx][ySbIdx] is FALSE, for x=xSb..xSb+3, y=ySb..ySb+3, the predicted sample values ​​of the current prediction unit are derived as follows: pbSamples[x][y]=Clip3(0,(1< <bitDepth)-1, (predSamplesL0[x][y]+predSamplesL1[x][y]+offset2)>>shift2) - Otherwise, the predicted sample values ​​for the current prediction unit are derived as follows: The location (xSb, ySb) defining the top-left sample of the current sub-block relative to the top-left samples of the prediction sample arrays predSamplesL0 and predSampleL1 is derived as follows: xSb=(xSbIdx<<2)+1 ySb=(ySbIdx<<2)+1 - For x=xSb-1..xSb+4, y=ySb-1..ySb+4, the following applies: The position (hx,vy) for each corresponding sample (x,y) inside the prediction sample array is derived as follows: hx=Clip3(1,nCbW,x) vy=Clip3(1,nCbH,y) - The variables gradientHL0[x][y], gradientVL0[x][y], gradientHL1[x][y] and gradientVL1[x][y] are derived as follows: gradientHL0[x][y]=(predSamplesL0[hx+1][vy]-predSampleL0[hx-1][vy])>>4 gradientVL0[x][y]=(predSampleL0[hx][vy+1]-predSampleL0[hx][vy-1])>>4 gradientHL1[x][y]=(predSamplesL1[hx+1][vy]-predSampleL1[hx-1][vy])>>4 gradientVL1[x][y]=(predSampleL1[hx][vy+1]-predSampleL1[hx][vy1])>>4 - The temporary temp, tempX, tempY, and other characters are as follows: temp[x][y]=(predSamplesL0[hx][vy]>>6)-(predSamplesL1[hx][vy]>>6) tempX[x][y]=(gradientHL0[x][y]+gradientHL1[x][y])>>3 tempY[x][y]=(gradientVL0[x][y]+gradientVL1[x][y])>>3 sGx2, sGy2, sGxGy, sGxdI, and sGydI. x,y=-1..4 and sGx2=Σ x Σ y (tempX[xSb+x][ySb+y]*tempX[xSb+x][ySb+y]) x,y=-1..4 and sGy2=Σ x Σ y (tempY[xSb+x][ySb+y]*tempY[xSb+x][ySb+y]) x,y=-1..4 and sGxGy=Σ x Σ(tempX[xSb+x][ySb+y]*tempY[xSb+x][ySb+y]) x,y=-1..4、sGxdIΣ x Σ(-tempX[xSb+x][ySb+y]*temp[xSb+x][ySb+y]) x,y=-1..4, sGydI=Σ x Σ y (-tempY[xSb+x][ySb+y]*temp[xSb+x][ySb+y]) - The horizontal and vertical motion refinement of the current sub-block are derived as follows: vx=sGx2>0?Clip3(-mvRefineThres,mvRefineThres,-(sGxdI<<3)>>Floor(Log2(sGx2))):0 vy=sGy2>0?Clip3(-mvRefineThres,mvRefineThres,((sGydI<<3)-((vx*sGxGym)<<12+vx*sGxGys)>>1)>>Floor(Log2(sGy2))):0 sGxGym=sGxGy>>12; sGxGys = sGxGy & ((1<<12)-1)

[0188] For x=xSb-1..xSb+2, y=ySb-1..ySb+2, the following applies: sampleEnh=Round((vx*(gradientHL1[x+1][y+1]-gradientHL0[x+1][y+1]))>>1) +Round((vy*(gradientVL1[x+1][y+1]-gradientVL0[x+1][y+1]))>>1) pbSamples[x][y]=Clip3(0,(1< <bitDepth)-1,(predSamplesL0[x+1][y+1] +predSamplesL1[x+1][y+1]+sampleEnh+offset2)>>shift2)

[0189] 2.2.10 Decoder-side Motion Vector Refinement (DMVR) DMVR is a type of Decoder-side Motion Vector Derivation (DMVD).

[0190] In bi-prediction operation, for prediction of one block region, two prediction blocks formed using the motion vector (MV) of list0 and the MV of list1 are combined to form a single prediction signal. In the decoder-side motion vector refinement (DMVR) method, the two motion vectors of bi-prediction are further refined by a bilateral template matching process. In order to obtain the refined MV without transmitting additional motion information, bilateral template matching is applied in the decoder to perform a distortion-based search between the bilateral template and the reconstructed sample in the reference picture.

[0191] In DMVR, a bilateral template is generated as a weighted combination (i.e., average) of two prediction blocks from the original, MV0 in list0 and MV1 in list1, as shown in Fig. 24. The template matching operation consists of calculating a cost metric between the generated template and a sample region (near the original prediction block) in the reference picture. For each of the two reference pictures, the MV that produces the smallest template cost is considered as the updated MV of that list to replace the original MV. In JEM, nine MV candidates are searched for each list. The nine MV candidates include the original MV and eight surrounding MVs with one luma sample offset in either horizontal or vertical direction or both with respect to the original MV. Finally, the two new MVs, i.e., MV0' and MV1' shown in Fig. 24, are used to generate the final bi-prediction result. The sum of absolute differences (SAD) is used as the cost metric. It should be noted that when calculating the cost of a predicted block generated by one surrounding MV, the rounded MV (to integer pels) is actually used instead of the actual MV to obtain the predicted block.

[0192] DMVR is applied to bi-predictive merge modes that use one MV from a past reference picture and another MV from a future reference picture without the transmission of additional syntax elements. In JEM, DMVR is not applied when LIC, affine motion, FRUC, or sub-CU merge candidates are enabled for a CU.

[0193] 2.2.11 JVET-N0236 This contribution proposes a method to refine sub-block-based affine motion compensation prediction using optical flow. After sub-block-based affine motion compensation is performed, the prediction samples are refined by adding the difference derived by the optical flow equation, which is referred to as prediction refinement with optical flow (PROF). The proposed method can achieve inter prediction at pixel-level granularity without increasing memory access bandwidth.

[0194] To achieve finer granularity of motion compensation, this contribution proposes a method to refine sub-block-based affine motion compensation prediction using optical flow. After sub-block-based affine motion compensation is performed, the luma prediction samples are refined by adding the difference derived by the optical flow equation. The proposed prediction refinement with optical flow (PROF) is described as the following four steps:

[0195] Step 1) Perform sub-block based affine motion compensation to generate the sub-block prediction I(i,j).

[0196] Step 2) At each sample position, we use a 3-tap filter [-1,0,1] to estimate the spatial gradient of the subblock prediction g x (i, j) and g y Calculate (i,j):

number

[0197] The subblock prediction is extended by one pixel on each side for gradient calculation. To reduce memory bandwidth and complexity, the pixels on the extended boundary are copied from the nearest integer pixel location in the reference picture. Thus, additional interpolation for the padding area is avoided.

[0198] Step 3) Refine luma prediction using optical flow equation:

number

[0199] Since the affine model parameters and pixel location relative to the sub-block center do not change between sub-blocks, Δv(i,j) can be calculated for the first sub-block and reused for other sub-blocks in the same CU. Let x and y be the horizontal and vertical offsets from the pixel location to the center of the sub-block, and Δv(x,y) be defined as:

number

[0200] In the four-parameter affine model,

number

[0201] In the six-parameter affine model,

number

[0202] Step 4) Finally, add the luma prediction refinement to the sub-block prediction I(i,j). The final prediction I′ is given by:

number

[0203] 2.2.12 JVET-N0510 Phase-variant affine subblock motion compensation (PAMC) To better approximate the affine motion model in affine sub-blocks, phase variational MC is applied to the sub-blocks. In the proposed method, the affine coding block is also divided into 4 × 4 sub-blocks, and a sub-block MV is derived for each sub-block as done in VTM4.0. The MC for each sub-block is divided into two stages. The first stage filters a (4+L-1) × (4+L-1) reference block window with (4+L-1) rows of horizontal filtering, where L is the filter tap length of the interpolation filter. However, unlike the translational MC, in the proposed phase variational affine sub-block MC, the filter phase for each sample row is different. For each sample row, the MVx is derived as follows: MVx=(subblockMVx<<7+dMvVerX×(rowIdx-L / 2-2))>>7 (Formula 1)

[0204] The filter phase for each sample row is derived from MVx. subblockMVx is the x-component of the MV for subblock MV, derived as done in VTM4.0. rowIdx is the sample row index. dMvVerX is (cuBottomLeftCPMVx-cuTopLeftCPMVx)<<(7-log2LumaCbHeight), where cuBottomLeftCPMVx is the x-component of the CU bottom-left control point MV, cuTopLeftCPMVx is the x-component of the CU top-left control point MV, and LumaCbHeight is the log2 of the luma coding block (CB) height.

[0205] After horizontal filtering, 4×(4+L-1) horizontally filtered samples are generated. Figure 1 shows the proposed horizontal filtering concept. The grey dots are the samples of the reference block window, and the orange dots represent the horizontally filtered samples. The blue tubular 8×1 samples represent applying one 8-tap horizontal filtering, as shown in Figure 30 and Figure 31. Each sample row needs four horizontal filterings. The filter phase of one sample row is the same. However, the filter phases of different rows are different. The skewed 4×11 samples are generated.

[0206] In the second stage, the 4×(4+L-1) horizontally filtered samples (orange samples in Figure 1) are further vertically filtered. For each sample row, MVy is derived as follows: MVy=(subblockMVy<<7+dMvHorY×(columnIdx-2))>>7 (Formula 2)

[0207] The filter phase for each sample row is derived from MVy. subblockMVy is the y component of MV for subblock MV, derived as done in VTM4.0. columnIdx is the sample column index. dMvHorY is (cuTopRightCPMVy-cuTopLeftCPMVy)<<(7-log2LumaCbWidth), where cuTopRightCPMVy is the y component of the CU top right control point MV, cuTopLeftCPMVy is the y component of the CU top left control point MV, and log2LumaCbWidth is the log2 of the luma CB width.

[0208] After vertical filtering, 4x4 affine subblock prediction samples are generated. Figure 32 shows the proposed vertical filtering concept. The light orange dots are the horizontally filtered samples from the first stage. The red dots are the vertically filtered samples as the final prediction samples.

[0209] In this proposal, the interpolation filter set used is the same as that in VTM4.0. The only difference is that the horizontal filter phase for one sample row is different and the vertical filter phase for one sample column is different. As for the number of filtering operations for each affine subblock in the proposed method, it is the same as that in VTM4.0.

[0210] 3. An example of the problem that is solved by the disclosed technical solution 1. BIO only considers bi-prediction 2. The derivation of vx and vy does not take into account the motion information of neighboring blocks. 3. JVET-N0236 uses optical flow for affine prediction, which is far from optimal.

[0211] 4. Examples of embodiments and techniques To address these issues, we propose a different configuration for deriving refinement prediction samples using optical flow. In addition, we propose to use information (e.g., reconstructed samples or motion information) of neighboring (e.g., adjacent or non-adjacent) blocks and / or gradients of one subblock / block and of its predicted blocks to obtain the final predicted block of the current subblock / block.

[0212] The techniques and embodiments listed below should be considered as examples to illustrate the general concept. These embodiments should not be construed in a narrow sense. Moreover, these techniques can be combined in any suitable manner.

[0213] The reference pictures of the current picture from list 0 and list 1 are denoted by Ref0 and Ref1, respectively, τ0=POC(current)-POC(Ref0), τ1=POC(Ref1)-POC(current), and the reference blocks of the current block from Ref0 and Ref1 are denoted by refblk0 and Refblk1, respectively. For a subblock in the current block, the MV of the corresponding reference subblock of refblk0 that points to refblk1 is (v x ,v y The MVs of the current subblock that refer to Ref0 and Ref1 are denoted by (mvL0 x ,mvL0 y ) and (mvL1 x ,mvL1 y )

[0214] In the following description, SatShift(x,n) is

number

[0215] In one example, offset0 and / or offset1 are (1<<n)> >1 or (1<<(n-1). In another example, offset0 and / or offset1 are set to 0.

[0216] In another example, offset0=offset1=((1<<n)> >1)-1 or ((1<<(n-1)))-1.

[0217] Clip3(min,max,x) is

number

[0218] In the following description, an operation between two motion vectors means that the operation is applied to both components of the motion vector. For example, MV3=MV1+MV2 means MV3 x =MV1 x +MV2 x and MV3 y =MV1 y +MV2 y Alternatively, the operation may be applied only to the horizontal or vertical components of the two motion vectors.

[0219] In the following description, the left adjacent block, the lower left adjacent block, the upper adjacent block, the upper right adjacent block, and the upper left adjacent block are represented as blocks A1, A0, B1, B0, and B2 shown in FIG. 1. It is proposed that a predicted sample P(x,y) at position (x,y) in a block can be refined as P'(x,y) = P(x,y) + Gx(x,y) × Vx(x,y) + Gy(x,y) × Vy(x,y). P'(x,y) is used together with the residual sample Res(x,y) to generate a reconstructed Rec(x,y). (Gx(x,y), Gy(x,y)) represent the gradient at position (x,y), e.g. along the horizontal and vertical directions, respectively. (Vx(x,y), Vy(x,y)) represent the motion displacement at position (x,y), which can be derived on the fly; Alternatively, a weighting function can be applied to the prediction samples, gradients, and motion displacements, for example P'(x,y) = α(x,y) x P(x,y) + β(x,y) x Gx(x,y) x Vx(x,y) + γ(x,y) x Gy(x,y) x Vy(x,y), where α(x,y), β(x,y) and γ(x,y) are weight values ​​at position (x,y) and can be integers or real numbers; i. For example, P'(x,y)=(α(x,y)×P(x,y)+β(x,y)×Gx(x,y)×Vx(x,y)+γ(x,y)×Gy(x,y)×Vy(x,y)+offsetP) / (α(x,y)+β(x,y)+γ(x,y)). In one example, offsetP is set to 0. Alternatively, the division may be replaced by a shift; ii. For example, P'(x,y)=P(x,y)-Gx(x,y)×Vx(x,y)+Gy(x,y)×Vy(x,y); iii. For example, P'(x,y)=P(x,y)-Gx(x,y)×Vx(x,y)-Gy(x,y)×Vy(x,y); iv. For example, P'(x,y)=P(x,y)+Gx(x,y)×Vx(x,y)-Gy(x,y)×Vy(x,y); v. For example, P'(x,y)=0.5×P(x,y)+0.25×Gx(x,y)×Vx(x,y)+0.25×Gy(x,y)×Vy(x,y); vi. For example, P'(x,y)=0.5×P(x,y)+0.5×Gx(x,y)×Vx(x,y)+0.5×Gy(x,y)×Vy(x,y); vii. For example, P'(x,y)=P(x,y)+0.5×Gx(x,y)×Vx(x,y)+0.5×Gy(x,y)×Vy(x,y); b. Alternatively, P'(x,y)=Shift(α(x,y)×P(x,y),n1)+Shift(β(x,y)×Gx(x,y)×Vx(x,y),n2)+Shift(γ(x,y)×Gy(x,y)×Vy(x,y),n3), where α(x,y), β(x,y) and γ(x,y) are weight values ​​at position (x,y) and are integers. n1, n2, n3 are non-negative integers, e.g., 1; c. Alternatively, P'(x,y)=SatShift(α(x,y)×P(x,y),n1)+SatShift(β(x,y)×Gx(x,y)×Vx(x,y),n2)+SatShift(γ(x,y)×Gy(x,y)×Vy(x,y),n3), where α(x,y), β(x,y) and γ(x,y) are weight values ​​at position (x,y) and are integers. n1, n2, n3 are non-negative integers, e.g., 1; d. Alternatively, P'(x,y)=Shift(α(x,y)×P(x,y)+β(x,y)×Gx(x,y)×Vx(x,y)+γ(x,y)×Gy(x,y)×Vy(x,y),n1), where α(x,y), β(x,y) and γ(x,y) are weight values ​​at position (x,y) and are integers. n1 is a non-negative integer, e.g., 1; e. Alternatively, P'(x,y)=SatShift(α(x,y)×P(x,y)+β(x,y)×Gx(x,y)×Vx(x,y)+γ(x,y)×Gy(x,y)×Vy(x,y),n1), where α(x,y), β(x,y) and γ(x,y) are weight values ​​at position (x,y) and are integers. n1 is a non-negative integer, e.g., 1; f. Alternatively, P'(x,y)=α(x,y)×P(x,y)+Shift(β(x,y)×Gx(x,y)×Vx(x,y),n2)+Shift(γ(x,y)×Gy(x,y)×Vy(x,y),n3), where α(x,y), β(x,y) and γ(x,y) are weight values ​​at position (x,y) and are integers. n2, n3 are non-negative integers, e.g., 1; g. Alternatively, P'(x,y)=α(x,y)×P(x,y)+SatShift(β(x,y)×Gx(x,y)×Vx(x,y),n2)+SatShift(γ(x,y)×Gy(x,y)×Vy(x,y),n3), where α(x,y), β(x,y) and γ(x,y) are weight values ​​at position (x,y) and are integers. n2, n3 are non-negative integers, e.g., 1; h. Alternatively, P'(x,y)=α(x,y)×P(x,y)+Shift(β(x,y)×Gx(x,y)×Vx(x,y)+γ(x,y)×Gy(x,y)×Vy(x,y),n3), where α(x,y), β(x,y) and γ(x,y) are weight values ​​at position (x,y) and are integers. n3 is a non-negative integer, e.g., 1; i. Alternatively, P'(x,y)=α(x,y)×P(x,y)+SatShift(β(x,y)×Gx(x,y)×Vx(x,y)+γ(x,y)×Gy(x,y)×Vy(x,y),n3), where α(x,y), β(x,y) and γ(x,y) are weight values ​​at position (x,y) and are integers. n3 is a non-negative integer, e.g., 1; j. Or, P'(x,y)=f0(P(x,y))+f1(Gx(x,y)×Vx(x,y))+f2(Gy(x,y)×Vy(x,y)), where f0, f1 and f2 are three functions; k. In one example, (Gx(x,y), Gy(x,y)) is computed on P(x1,y1), where x1 is in the range [x-Bx0,x+Bx1], y1 is in the range [y-By0,y+By1], and Bx0, Bx1, By0, By1 are integers; i. For example, Gx(x,y)=P(x+1,y)-P(x-1,y), Gy(x,y)=P(x,y+1)-P(x,y-1); (i) Alternatively, Gx(x,y) = Shift(P(x+1,y) - P(x-1,y), n1), Gy(x,y) = Shift(P(x,y+1) - P(x,y-1), n2), for example, n1 = n2 = 1; (ii) Alternatively, Gx(x,y) = SatShift(P(x+1,y) - P(x-1,y), n1), Gy(x,y) = SatShift(P(x,y+1) - P(x,y-1), n2), for example, n1 = n2 = 1; l. P(x,y) can be the prediction value of one-sided prediction (inter prediction with one MV); m. P(x,y) can be the final prediction value after bi-prediction (inter prediction on two MVs); i. For example, Vx(x,y) and Vy(x,y) may be derived according to the methodology defined in BIO (also known as bidirectional optical flow, BDOF); n. P(x,y) may be a multi-hypothesis inter-prediction (inter-prediction with three or more MVs); o. P(x,y) may be an affine prediction; p. P(x,y) may be an intra prediction; q. P(x,y) may be an intra-block copy (IBC) prediction; r. P(x,y) may be generated by triangular prediction mode (TPM) or geographic prediction mode (GPM) technique; s. P(x,y) may be a joint inter-intra prediction; t. P(x,y) may be a global inter prediction, where the regions share the same motion model and parameters; u. P(x,y) may be generated by a palette coding mode; v. P(x,y) may be inter-view prediction in multiview or 3D video coding; w. P(x,y) may be inter-layer prediction in scalable video coding; x. P(x,y) may be filtered before being refined; y. P(x,y) may be the final prediction to be summed with the residual sample values ​​to obtain the reconstructed sample values. In some embodiments, P(x,y) may be the final prediction when the refinement process is not applied. In some embodiments, P'(x,y) may be the final prediction when the refinement process is applied; i. In one example, for a block (or sub-block) where bi-prediction or multi-hypothesis prediction is applied, the above function is applied once to the final predicted value; ii. In one example, in a block (or sub-block) where bi-prediction or multiple hypothesis prediction is applied, for each prediction block according to one prediction direction or reference picture or motion vector, the above function is applied multiple times, so that the above process is called to update the prediction block. The updated prediction block can then be used to generate a final prediction block; iii. Alternatively, P(x,y) may be an intermediate prediction that is to be used to derive a final prediction; (i) For example, P(x,y) may be a prediction from one reference picture list when the current block is bi-predictive and inter-predicted; (ii) For example, P(x,y) may be a prediction from one reference picture list when the current block is inter-predicted with TPM or GPM techniques; (iii) For example, P(x,y) may be a prediction from one reference picture when the current block is inter-predicted with multiple hypotheses; (iv) For example, P(x,y) may be inter-predicted if the current block is jointly inter-intra predicted; (v) For example, P(x,y) may be an inter prediction before local illumination compensation (LIC) is applied if the current block uses LIC; (vi) For example, P(x,y) may be an inter prediction before DMVR (or other type of DMVD) is applied if the current block uses DMVR (or other type of DMVD); (vii) For example, P(x,y) may be an inter prediction before being multiplied by a weighting factor if the current block uses weighted prediction or generalized bi-prediction (GBi); z. A gradient, denoted G(x,y), e.g. Gx(x,y) or / and Gy(x,y), may be derived on the final prediction to be added with the residual sample values ​​to obtain the reconstructed sample values. In some embodiments, if no refinement process is applied, the final predicted sample values ​​are added to the residual sample values ​​to obtain the reconstructed sample values; i. For example, G(x,y) can be derived on P(x,y); ii. Alternatively, G(x,y) may be derived on intermediate predictions that will be used to derive the final prediction; (i) For example, G(x,y) may be derived on prediction from one reference picture list when the current block is bi-predictively inter-predicted; (ii) For example, G(x,y) may be derived on prediction from one reference picture list when the current block is inter-predicted with TPM or GPM techniques; (iii) For example, G(x,y) may be derived on a prediction from one reference picture when the current block is inter predicted with multiple hypotheses; (iv) For example, G(x,y) may be derived on inter prediction when the current block is jointly inter-intra predicted; (v) For example, G(x,y) may be derived on an inter prediction before local illumination compensation (LIC) is applied if the current block uses LIC; (vi) For example, G(x,y) may be derived on inter prediction before DMVR (or other type of DMVD) is applied if the current block uses DMVR (or other type of DMVD); (vii) For example, G(x,y) may be derived on the inter prediction before being multiplied by the weighting factor if the current block uses weighted prediction or generalized bi-prediction (GBi); aa. Alternatively, P'(x,y) may be further processed by other methods to obtain the final predicted sample; bb. Alternatively, a reconstructed sample Rec(x,y) at position (x,y) in a block may be refined as Rec'(x,y)=Rec(x,y)+Gx(x,y)×Vx(x,y)+Gy(x,y)×Vy(x,y). Rec'(x,y) will be used to replace the reconstructed Rec(x,y). (Gx(x,y),Gy(x,y)) represent the gradient at position (x,y), e.g. along the horizontal and vertical directions, respectively. (Vx(x,y),Vy(x,y)) represent the motion displacement at position (x,y), which may be derived on the fly; i. In one example, (Gx(x,y), Gy(x,y)) are derived on the reconstruction samples. 2. It is proposed that Vx(x,y) and / or Vy(x,y) used in optical flow based methods (such as in bullet 1) may depend on spatial or temporal neighboring blocks; Alternatively, Vx(x,y) and / or Vy(x,y) in the process of BIO (also known as BDOF) may depend on spatial or temporal neighboring blocks; b. In one example, the "dependence" on spatial or temporal neighboring blocks may include dependence on motion information (e.g., MV), coding mode (e.g., inter-coded or intra-coded), neighboring CU size, neighboring CU position, etc.; c. In one example, (Vx(x,y), Vy(x,y)) is the MV Mix can be equal to; i. In one example, MV Mix is Wc(x,y)×MVc+W N1 (x,y)×MV N1 +W N2 (x,y)×MV N2 +...+W Nk (x,y)×MV Nk where MVc is the MV of the current block, and MV N1 ...MV Nk are the MVs of k spatially or temporally adjacent blocks: N1...Nk. N1 ...W Nk is the weight value, which can be an integer or a real number; ii. Or, MV Mix is Shift(Wc(x,y)×MVc+W N1 (x,y)×MV N1 +W N2 (x,y)×MV N2 +...+W Nk (x,y)×MV Nk , n1), where MVc is the MV of the current block, and MV N1 ...MV Nk are the MVs of k spatially or temporally adjacent blocks: N1...Nk. N1 ...W Nk is the weight value, which is an integer; iii. Or, MV Mix is SatShift(Wc(x,y)×MVc+W N1 (x,y)×MV N1 +W N2 (x,y)×MV N2 +...+W Nk (x,y)×MV Nk , n1), where MVc is the MV of the current block, and MV N1 ...MV Nk are the MVs of k spatially or temporally adjacent blocks: N1...Nk. N1 ...W Nk is the weight value, which is an integer; iv. In one example, Wc(x,y)=0; v. In one example, k=1 and N1 is a spatially adjacent block; (i) In one example, N1 is the closest spatially neighboring block to location (x, y); (ii) In one example, W N1 (x,y) is larger the closer the position (x,y) is to N1; vi. In one example, k=1 and N1 is a time-adjacent block; (i) For example, N1 is a collocated block in a collocated picture for position (x, y); In one example, different locations may use different spatial or temporal neighboring blocks; vii. Figure 26 shows an example of how to derive Vx(x,y) and / or Vy(x,y), where each block represents a basic block (e.g. a 4x4 block). The current block is marked with a bold line; (i) Prediction samples within shaded basic blocks, e.g., those that are not at the top or left boundary, are not refined with optical flow; (ii) prediction samples in basic blocks at the upper boundary, e.g., C00, C10, C20 and C30, will be refined with optical flow; For example, the MV for the predicted samples in the basic block at the upper boundary Mix is derived depending on the adjacent upper neighboring blocks. For example, MV for the predicted samples in C10 Mix is derived depending on the upper neighboring block T1; (iii) Prediction samples in basic blocks at the left boundary, e.g., C00, C01, C02 and C03, will be refined with optical flow; For example, the MV for the predicted samples in the basic block at the left boundary Mix is derived depending on the adjacent left neighboring block. For example, MV for the predicted sample in C01 Mix is derived depending on the left neighboring block L1; viii. In one embodiment, MVc and MV N1 ...MV Nk can be scaled to the same reference picture; (i) In one example, they are scaled to the reference picture that MVc refers to; ix. In one example, MV is generated using spatial or temporal neighboring block Ns only if it is not intra-coded. Mix can be derived; x. In one example, MV is generated using spatial or temporal neighboring block Ns only if it is not IBC coded. Mix can be derived; xi. In one example, MVc is generated using spatial or temporal neighboring block Ns only if MVs references the same reference picture as MVc. Mix can be derived; d. In one example, (Vx(x,y), Vy(x,y)) is f(MV Mix ,MVc), where f is a function and MVc is the MV of the current block; i. For example, (Vx(x,y),Vy(x,y)) is MV Mix -May be equal to MVc; ii. For example, (Vx(x,y),Vy(x,y)) is MVc-MV Mix can be equal to; iii. For example, (Vx(x,y), Vy(x,y)) is p×MV Mix +q×MVc, where p and q are real numbers. Some examples of p and q are p=q=0.5, or p=1, q=−0.5, or q=1, p=−0.5, etc.; (i) Or, (Vx(x,y), Vy(x,y)) is Shift(p×MV Mix +q × MVc,n) or SatShift(p × MV Mix +p×MVc,n), where p, q, and n are integers. Some examples of n, p, and q are n=1, p=2, q=-1, or n=1, p=q=1, or n=1, p=-1, q=2, etc.; e. In one example, the current block may be uni-predictive and inter-predicted, and MVc may refer to reference picture list 0; f. In one example, the current block may be uni-predictive and inter-predicted, and MVc may refer to reference picture list 1; g. In one example, the current block is bi-predictive and inter-predicted, and MVc may refer to reference picture list 0 or reference picture list 1; i. In one example, the final prediction is refined by optical flow: (Vx(x,y), Vy(x,y)) may be derived with MVc referring to one of the reference picture lists, e.g. reference picture list 0 or reference picture list 1; ii. In one example, the prediction from reference list 0 is refined by optical flow: (Vx(x,y), Vy(x,y)) can be derived with MVc referring to reference list 0; iii. In one example, the prediction from reference list 1 is refined by optical flow: (Vx(x,y), Vy(x,y)) can be derived with MVc referring to reference list 1; iv. The predictions from reference list 0 after being refined by optical flow and the predictions from reference list 1 after being independently refined by optical flow can be combined (e.g., averaged or weighted averaged) to obtain a final prediction; h. In one example, the current block is bi-predictively inter predicted, BDOF is applied, and Vx(x,y), Vy(x,y) are modified according to spatial or temporal neighboring blocks; i. For example, if (Vx(x,y), Vy(x,y)) derived in the BDOF process is denoted as V'(x,y)=(V'x(x,y), V'y(x,y)), and (Vx(x,y)), Vy(x,y)) derived in the disclosed manner are denoted as V"(x,y)=(V"x(x,y), V"y(x,y)), then the final V(x,y)=(Vx(x,y), Vy(x,y)) can be derived as follows: (i) For example, V(x,y) = V'(x,y) x W'(x,y) + V"(x,y) x W"(x,y), where W'(x,y) and W"(x,y) are integers or real numbers. For example, W'(x,y) = 0, W"(x,y) = 1, or W'(x,y) = 1, W"(x,y) = 0, or W'(x,y) = 0.5, W"(x,y) = 0.5; (ii) For example, V(x,y) = Shift(V'(x,y) × W'(x,y) + V"(x,y) × W"(x,y), n1), where W'(x,y) and W"(x,y) are integers. n1 is a non-negative integer, e.g., 1; (iii) For example, V(x,y)=SatShift(V'(x,y)×W'(x,y)+V"(x,y)×W"(x,y),n1), where W'(x,y) and W"(x,y) are integers. n1 is a non-negative integer, e.g., 1; ii. For example, modifying (Vx(x,y), Vy(x,y)) in BDOF according to spatial or temporal neighboring blocks may depend on the position (x,y); (i) (Vx(x,y), Vy(x,y)) in the shaded basic blocks of FIG. 26, e.g., those that are not on the top or left boundary, are not modified according to spatial or temporal neighboring blocks; i. Vx(x,y) and / or Vy(x,y) may be clipped; j. Alternatively, the spatial or temporal neighboring blocks in the above method may be replaced by non-neighboring blocks of the current block; k. Alternatively, the spatially or temporally adjacent block in the above method may be replaced by a non-adjacent sub-block of the current sub-block; l. Alternatively, the spatially or temporally adjacent blocks in the above methods may be replaced by non-adjacent sub-blocks of the current block / current CTU / current VPDU / current region covering the current sub-block; m. Alternatively, the spatial or temporal neighboring blocks in the above methods may be replaced by entries in a history-based motion vector. 3. It is proposed that Vx(x,y) and / or Vy(x,y) in the refinement on affine prediction using optical flow disclosed in JVET-N0236 may be derived as follows: a. Vx(x,y)=a×(x-xc)+b×(y-yc), Vy(x,y)=c×(x-xc)+d×(y-yc), where (x,y) is the position under consideration, (xc,yc) is the center position of a basic block of size w×h (e.g. 4×4 or 8×8) that covers position (x,y), and a, b, c and d are affine parameters; i. Or, Vx(x,y)=Shift(a×(x-xc)+b×(y-yc),n1), Vy(x,y)=Shift(c×(x-xc)+d×(y-yc),n1), where n1 is an integer; ii. Or, Vx(x,y)=SatShift(a×(x-xc)+b×(y-yc),n1), Vy(x,y)=SatShift(c×(x-xc)+d×(y-yc),n1), where n1 is an integer; iii. For example, if the top left position of a basic block (e.g., 4x4 or 8x8) that covers position (x,y) is (x0,y0), then (xc,yc)=(x0+(w / 2,y0+(h / 2)); (i) Or, (xc, yc) = (x0 + (w / 2) - 1, y0 + (h / 2) - 1); (ii) Or, (xc, yc) = (x0 + (w / 2), y0 + (h / 2) - 1); (iii) Or, (xc, yc) = (x0 + (w / 2) - 1, y0 + (h / 2)); iv. In one example, if the current block applies a 4-parameter affine mode, then c=-b and d=a; v. In one example, a, b, c, and d may be derived from the CPMV along with the width (W) and height (H) of the current block. For example,

number

[0220] FIG. 27 is a block diagram of a video processing device 2700. The device 2700 may be used to implement one or more of the methods described herein. The device 2700 may be embodied in a smartphone, a tablet, a computer, an Internet of Things (IoT) receiver. The device 2700 may include one or more processors 2702, one or more memories 2704, and video processing hardware 2706. The processor(s) 2702 may be configured to execute one or more methods described herein. The memory(s) 2704 may be used to store data and code used to execute the methods and techniques described herein. The video processing hardware 2706 may be used to implement some of the techniques described herein in a hardware circuit, and may be part of the processor 2702 (e.g., a graphics processor core GPU or other signal processing circuit) in part or in whole.

[0221] 28A is a flow chart of an example video processing method. The method 2800A includes determining 2802 a refined prediction sample P'(x,y) at a position (x,y) in the video block by modifying the prediction sample P(x,y) at the position (x,y) as a function of a gradient in a first direction and / or a second direction estimated at the position (x,y) and a first motion displacement and / or a second motion displacement estimated for the position (x,y), and performing 2804 the conversion using a reconstructed sample value Rec(x,y) from the refined prediction sample P'(x,y).

[0222] FIG. 28B is a flow chart of an example of a video processing method. Method 2800B includes determining (2812) a refined prediction sample P'(x,y) at a position (x,y) within the video block by modifying the prediction sample P(x,y) at position (x,y) with a first gradient component Gx(x,y) in a first direction estimated at position (x,y), a second gradient component Gy(x,y) in a second direction estimated at position (x,y), a first motion displacement Vx(x,y) estimated for position (x,y), and a second motion displacement Vy(x,y) estimated for position (x,y), where x and y are integers, and performing (2814) a conversion between the video block and a bitstream representation of the video block using reconstructed sample values ​​Rec(x,y) at position (x,y) obtained based on the refined prediction sample P'(x,y) and residual sample values ​​Res(x,y).

[0223] 28C is a flow chart of an example video processing method. The method 2800C includes determining (2822) a refined prediction sample P'(x,y) at a position (x,y) in the video block by modifying the prediction sample P(x,y) at the position (x,y) with a first gradient component Gx(x,y) in a first direction estimated at the position (x,y), a second gradient component Gy(x,y) in a second direction estimated at the position (x,y), a first motion displacement Vx(x,y) estimated for the position (x,y), and a second motion displacement Vy(x,y) estimated for the position (x,y), where x and y are integers, and encoding (2824) a bitstream representation of the video block to include residual sample values ​​Res(x,y) based on reconstructed sample values ​​Rec(x,y) at the position (x,y) that are based on at least the refined prediction sample P'(x,y).

[0224] In some embodiments of methods 2800B and / or 2800C, the first direction and the second direction are orthogonal to each other. In some embodiments of methods 2800B and / or 2800C, the first motion displacement represents a direction parallel to the first direction and the second motion displacement represents a direction parallel to the second direction. In some embodiments of methods 2800B and / or 2800C, P'(x,y) = P(x,y) + Gx(x,y) x Vx(x,y) + Gy(x,y) x Vy(x,y). In some embodiments of methods 2800B and / or 2800C, P'(x,y) = α(x,y) x P(x,y) + β(x,y) x Gx(x,y) x Vx(x,y) + γ(x,y) x Gy(x,y) x Vy(x,y), where α(x,y), β(x,y) and γ(x,y) are weighting values ​​at position (x,y), and α(x,y), β(x,y) and γ(x,y) are integers or real numbers. In some embodiments of method 2800B and / or 2800C, P'(x,y) = (α(x,y) x P(x,y) + β(x,y) x Gx(x,y) x Vx(x,y) + γ(x,y) x Gy(x,y) x Vy(x,y) + offsetP) / (α(x,y) + β(x,y) + γ(x,y)), where α(x,y), β(x,y) and γ(x,y) are weighting values ​​at position (x,y), and α(x,y), β(x,y) and γ(x,y) are integers or real numbers. In some embodiments of method 2800B and / or 2800C, offsetP is equal to 0. In some embodiments of method 2800B and / or 2800C, P'(x,y) is obtained using a binary shift operation.

[0225] In some embodiments of methods 2800B and / or 2800C, P'(x,y) = P(x,y) - Gx(x,y) x Vx(x,y) + Gy(x,y) x Vy(x,y). In some embodiments of methods 2800B and / or 2800C, P'(x,y) = P(x,y) - Gx(x,y) x Vx(x,y) - Gy(x,y) x Vy(x,y). In some embodiments of methods 2800B and / or 2800C, P'(x,y) = P(x,y) + Gx(x,y) x Vx(x,y) - Gy(x,y) x Vy(x,y). In some embodiments of methods 2800B and / or 2800C, P'(x,y) = 0.5 x P(x,y) + 0.25 x Gx(x,y) x Vx(x,y) + 0.25 x Gy(x,y) x Vy(x,y). In some embodiments of methods 2800B and / or 2800C, P'(x,y) = 0.5 x P(x,y) + 0.5 x Gx(x,y) x Vx(x,y) + 0.5 x Gy(x,y) x Vy(x,y). In some embodiments of methods 2800B and / or 2800C, P'(x,y) = P(x,y) + 0.5 x Gx(x,y) x Vx(x,y) + 0.5 x Gy(x,y) x Vy(x,y). In some embodiments of method 2800B and / or 2800C, P'(x,y) = Shift(α(x,y) x P(x,y),n1) + Shift(β(x,y) x Gx(x,y) x Vx(x,y),n2) + Shift(γ(x,y) x Gy(x,y) x Vy(x,y),n3), where the Shift() function refers to a binary shift operation, α(x,y), β(x,y) and γ(x,y) are weighting values ​​at position (x,y), α(x,y), β(x,y) and γ(x,y) are integers, and n1, n2, n3 are non-negative integers. In some embodiments of method 2800B and / or 2800C, n1, n2, n3 are equal to 1.

[0226] In some embodiments of method 2800B and / or 2800C, P'(x,y) = SatShift(α(x,y) x P(x,y),n1) + SatShift(β(x,y) x Gx(x,y) x Vx(x,y),n2) + SatShift(γ(x,y) x Gy(x,y) x Vy(x,y),n3), where SatShift() function refers to a saturated binary shift operation, α(x,y), β(x,y) and γ(x,y) are weighting values ​​at position (x,y), α(x,y), β(x,y) and γ(x,y) are integers, and n1, n2, n3 are non-negative integers. In some embodiments of method 2800B and / or 2800C, n1, n2, n3 are equal to 1. In some embodiments of method 2800B and / or 2800C, P'(x,y) = Shift (α(x,y) x P(x,y) + β(x,y) x Gx(x,y) x Vx(x,y) + γ(x,y) x Gy(x,y) x Vy(x,y), n1), where the Shift() function refers to a binary shift operation, α(x,y), β(x,y) and γ(x,y) are weighting values ​​at location (x,y), α(x,y), β(x,y) and γ(x,y) are integers, and n1 is a non-negative integer. In some embodiments of method 2800B and / or 2800C, n1 is equal to 1.

[0227] In some embodiments of method 2800B and / or 2800C, P'(x,y) = SatShift(α(x,y) x P(x,y) + β(x,y) x Gx(x,y) x Vx(x,y) + γ(x,y) x Gy(x,y) x Vy(x,y), n1), where the SatShift() function refers to a saturated binary shift operation, α(x,y), β(x,y) and γ(x,y) are weighting values ​​at location (x,y), α(x,y), β(x,y) and γ(x,y) are integers, and n1 is a non-negative integer. In some embodiments of method 2800B and / or 2800C, n1 is equal to 1. In some embodiments of method 2800B and / or 2800C, P'(x,y) = α(x,y) x P(x,y) + Shift(β(x,y) x Gx(x,y) x Vx(x,y),n2) + Shift(γ(x,y) x Gy(x,y) x Vy(x,y),n3), where the Shift() function refers to a binary shift operation, α(x,y), β(x,y) and γ(x,y) are weighting values ​​at location (x,y), α(x,y), β(x,y) and γ(x,y) are integers, and n2 and n3 are non-negative integers. In some embodiments of method 2800B and / or 2800C, n2 and n3 are equal to 1.

[0228] In some embodiments of method 2800B and / or 2800C, P'(x,y) = α(x,y) x P(x,y) + SatShift(β(x,y) x Gx(x,y) x Vx(x,y),n2) + SatShift(γ(x,y) x Gy(x,y) x Vy(x,y),n3), where SatShift() function refers to a saturated binary shift operation, α(x,y), β(x,y) and γ(x,y) are weighting values ​​at location (x,y), α(x,y), β(x,y) and γ(x,y) are integers, and n2 and n3 are non-negative integers. In some embodiments of method 2800B and / or 2800C, n2 and n3 are equal to 1. In some embodiments of method 2800B and / or 2800C, P'(x,y) = α(x,y) x P(x,y) + Shift(β(x,y) x Gx(x,y) x Vx(x,y) + γ(x,y) x Gy(x,y) x Vy(x,y), n3), where the Shift() function refers to a binary shift operation, α(x,y), β(x,y) and γ(x,y) are weighting values ​​at location (x,y), α(x,y), β(x,y) and γ(x,y) are integers, and n3 is a non-negative integer. In some embodiments of method 2800B and / or 2800C, n3 is equal to 1.

[0229] In some embodiments of method 2800B and / or 2800C, P'(x,y) = α(x,y) x P(x,y) + SatShift(β(x,y) x Gx(x,y) x Vx(x,y) + γ(x,y) x Gy(x,y) x Vy(x,y), n3), where the SatShift() function refers to a saturated binary shift operation, α(x,y), β(x,y) and γ(x,y) are weighting values ​​at location (x,y), α(x,y), β(x,y) and γ(x,y) are integers, and n3 is a non-negative integer. In some embodiments of method 2800B and / or 2800C, n3 is equal to 1. In some embodiments of methods 2800B and / or 2800C, 1-4, P'(x,y)=f0(P(x,y))+f1(Gx(x,y)×Vx(x,y))+f2(Gy(x,y)×Vy(x,y)), where f0(), f1(), and f2() are three functions. In some embodiments of methods 2800B and / or 2800C, the first gradient component Gx(x,y) and the second gradient component Gy(x,y) are calculated using the second predicted sample P(x1,y1), where x1 is in a first range of [x-Bx0,x+Bx1], y1 is in a second range of [y-By0,y+By1], Bx0 and By0 are integers, and Bx1 and By1 are integers.

[0230] In some embodiments of methods 2800B and / or 2800C, the first gradient component Gx(x,y)=P(x+1,y)-P(x-1,y) and the second gradient component Gy(x,y)=P(x,y+1)-P(x,y-1). In some embodiments of methods 2800B and / or 2800C, the first gradient component Gx(x,y)=Shift(P(x+1,y)-P(x-1,y),n1) and the second gradient component Gy(x,y)=Shift(P(x,y+1)-P(x,y-1),n2), where the Shift() function refers to a binary shift operation. In some embodiments of method 2800B and / or 2800C, the first gradient component Gx(x,y)=SatShift(P(x+1,y)-P(x-1,y),n1) and the second gradient component Gy(x,y)=SatShift(P(x,y+1)-P(x,y-1),n2), where the SatShift() function refers to a saturated binary shift operation. In some embodiments of method 2800B and / or 2800C, n1 and n2 are equal to 1. In some embodiments of method 2800B and / or 2800C, the predicted sample P(x,y) is a single predicted sample at position (x,y). In some embodiments of method 2800B and / or 2800C, the predicted sample P(x,y) is a final result of bi-prediction.

[0231] In some embodiments of methods 2800B and / or 2800C, Vx(x,y) and Vy(x,y) are derived using a bidirectional optical flow (BIO) technique. In some embodiments of methods 2800B and / or 2800C, the prediction sample P(x,y) satisfies any one of the following: a result of a multiple hypothesis inter prediction technique; a result of an affine prediction technique; a result of an intra prediction technique; a result of an intra block copy (IBC) prediction technique; generated by a triangular prediction mode (TPM) technique; generated by a geographic prediction mode (GPM) technique; a result of a joint inter-intra prediction technique; a result of a global inter prediction technique, which includes regions sharing the same motion model and parameters; a result of a palette coding mode; a result of inter-view prediction in multi-view or 3D video coding; a result of inter-layer prediction in scalable video coding; or a result of a filtering operation prior to determining the refined prediction sample P'(x,y).

[0232] In some embodiments of methods 2800B and / or 2800C, the prediction samples P(x,y) are the final prediction sample values ​​when the refinement process is not applied, and the reconstructed sample values ​​Rec(x,y) are obtained by adding the residual sample values ​​Res(x,y) to the prediction samples P(x,y). In some embodiments of methods 2800B and / or 2800C, the refined prediction samples P′(x,y) refined from the prediction samples P(x,y) are the final prediction sample values ​​when the refinement process is applied, and the reconstructed sample values ​​Rec(x,y) are obtained by adding the residual sample values ​​Res(x,y) to the refined prediction samples P′(x,y). In some embodiments of methods 2800B and / or 2800C, a bi-prediction technique or a multiple hypothesis prediction technique is applied to the video block or to a sub-block of the video block, and the first gradient component, the second gradient component, the first motion displacement, and the second motion displacement are applied once to the final predicted sample value.

[0233] In some embodiments of methods 2800B and / or 2800C, a bi-prediction technique or a multiple hypothesis prediction technique is applied to the video block or to a sub-block of the video block, a first gradient component, a second gradient component, a first motion displacement, and a second motion displacement are applied multiple times to a prediction block of the video block to obtain multiple sets of the first gradient component, the second gradient component, the first motion displacement, and the second motion displacement, and an updated prediction block is obtained by updating each prediction block based on the refined prediction sample P'(x,y), and the updated prediction block is used to generate a final prediction block for the video block. In some embodiments of methods 2800B and / or 2800C, the first set applies to a first predicted block at a location at least one of a first gradient component, a second gradient component, a first motion displacement, or a second motion displacement that is different from a corresponding at least one of a first gradient component, a second gradient component, a first motion displacement, or a second motion displacement applied to a second predicted block at the same location in the second set.

[0234] In some embodiments of methods 2800B and / or 2800C, the prediction samples P(x,y) are intermediate prediction sample values ​​from which a final prediction sample value is derived. In some embodiments of methods 2800B and / or 2800C, in response to the video block being inter-predicted using a bi-predictive technique, the prediction samples P(x,y) are prediction samples from one reference picture list. In some embodiments of methods 2800B and / or 2800C, in response to the video block being inter-predicted using a triangular prediction mode (TPM) technique, the prediction samples P(x,y) are prediction samples from one reference picture list. In some embodiments of methods 2800B and / or 2800C, in response to the video block being inter-predicted using a geometric partition mode (GPM) technique, the prediction samples P(x,y) are prediction samples from one reference picture list. In some embodiments of methods 2800B and / or 2800C, in response to the video block being inter-predicted using a multiple hypothesis prediction technique, the prediction sample P(x,y) is a prediction sample from one reference picture list.

[0235] In some embodiments of methods 2800B and / or 2800C, the prediction sample P(x,y) is an inter-predicted sample for a video block that is inter-inter jointly predicted. In some embodiments of methods 2800B and / or 2800C, the prediction sample P(x,y) is an inter-predicted sample before a local illumination compensation (LIC) technique is applied to the video block, and the video block uses the LIC technique. In some embodiments of methods 2800B and / or 2800C, the prediction sample P(x,y) is an inter-predicted sample before a decoder-side motion vector refinement (DMVR) technique or a decoder-side motion vector derivation (DMVD) technique is applied to the video block, and the video block uses the DMVR technique or the DMVD technique. In some embodiments of methods 2800B and / or 2800C, the prediction sample P(x,y) is an inter-predicted sample before being multiplied by a weighting factor, and the video block uses a weighted prediction technique or a generalized Bi-prediction (GBi) technique.

[0236] In some embodiments of methods 2800B and / or 2800C, the first gradient component in the first direction and / or the second gradient component in the second direction are derived for a final predicted sample value, which is added to the residual sample value Res(x,y) to obtain the reconstructed sample value Rec(x,y) if no refinement process is applied. In some embodiments of methods 2800B and / or 2800C, the final predicted sample value is the predicted sample P(x,y). In some embodiments of methods 2800B and / or 2800C, the first gradient component in the first direction and / or the second gradient component in the second direction are derived for an intermediate predicted sample value from which the final predicted sample value is to be derived. In some embodiments of methods 2800B and / or 2800C, in response to the video block being inter-predicted using a bi-predictive technique, a first gradient component in a first direction and / or a second gradient component in a second direction are derived from prediction samples from one reference picture list.

[0237] In some embodiments of methods 2800B and / or 2800C, in response to the video block being inter predicted using a triangular prediction mode (TPM) technique, the first gradient component in the first direction and / or the second gradient component in the second direction are derived from one reference picture list. In some embodiments of methods 2800B and / or 2800C, in response to the video block being inter predicted using a geometric prediction mode (GPM) technique, the first gradient component in the first direction and / or the second gradient component in the second direction are derived from one reference picture list. In some embodiments of methods 2800B and / or 2800C, in response to the video block being inter predicted using a multiple hypothesis prediction technique, the first gradient component in the first direction and / or the second gradient component in the second direction are derived from one reference picture list. In some embodiments of methods 2800B and / or 2800C, the first gradient component in a first direction and / or the second gradient component in a second direction are derived from inter-predicted samples for the inter-inter joint predicted video block.

[0238] In some embodiments of methods 2800B and / or 2800C, the first gradient component in the first direction and / or the second gradient component in the second direction are derived from inter-predicted samples before a local illumination compensation (LIC) technique is applied to the video block, and the video block uses the LIC technique. In some embodiments of methods 2800B and / or 2800C, the first gradient component in the first direction and / or the second gradient component in the second direction are derived from inter-predicted samples before a decoder-side motion vector refinement (DMVR) technique or a decoder-side motion vector derivation (DMVD) technique is applied to the video block, and the video block uses the DMVR technique or the DMVD technique. In some embodiments of methods 2800B and / or 2800C, the first gradient component in the first direction and / or the second gradient component in the second direction are derived from inter-predicted samples before being multiplied with a weighting factor, and the video block uses a weighted prediction technique or a generalized Bi-prediction (GBi) technique.

[0239] In some embodiments of methods 2800B and / or 2800C, the refined prediction sample P'(x,y) is further modified to obtain a final prediction sample value. In some embodiments of methods 2800B and / or 2800C, the first gradient component in the first direction is denoted as Gx(x,y), the second gradient component in the second direction is denoted as Gy(x,y), the first motion displacement is denoted as Vx(x,y), the second motion displacement is denoted as Vy(x,y), and the reconstructed sample value Rec(x,y) at position (x,y) is replaced with the refined reconstructed sample value Rec'(x,y), where Rec'(x,y)=Rec(x,y)+Gx(x,y)×Vx(x,y)+Gy(x,y)×Vy(x,y). In some embodiments of methods 2800B and / or 2800C, the first gradient component Gx(x,y) and the second gradient component Gy(x,y) are derived based on the reconstructed sample values.

[0240] In some embodiments of method 2800B and / or 2800C, the method further includes determining a first motion displacement Vx(x,y) at a position (x,y) within the image block and a second motion displacement Vy(x,y) at a position (x,y) based on at least information from spatially neighboring video blocks of the image block or at least information from temporally neighboring video blocks located in a time-shortened manner relative to the image block. In some embodiments of method 2800B and / or 2800C, the spatially neighboring video blocks are located adjacent to the image block and the temporally neighboring video blocks are located in a time-shortened manner relative to the image block. In some embodiments of method 2800B and / or 2800C, the first motion displacement Vx(x,y) and the second motion displacement Vy(x,y) are determined using bidirectional optical flow (BIO) techniques based on at least spatially neighboring video blocks of the image block or at least temporally neighboring video blocks located in a time-shortened manner relative to the image block.

[0241] In some embodiments of methods 2800B and / or 2800C, the information from the spatially neighboring video blocks or from the temporally neighboring video blocks includes motion information, coding mode, adjacent coding unit (CU) size, or adjacent CU position. In some embodiments of methods 2800B and / or 2800C, (Vx(x,y),Vy(x,y)) is equal to MVMix, which is equal to Wc(x,y)×MVc+WN1(x,y)×MVN1+ WN2(x,y)×MVN2+...+WNk(x,y)×MVNk, where MVc is the motion vector of the video block, MVN1,...,MVNk are the motion vectors of the k spatially or temporally neighboring video blocks, N1,...,Nk are the spatially or temporally neighboring video blocks, and Wc, WN1,...,WNk are weighting values ​​that may be integers or real numbers.

[0242] In some embodiments of method 2800B and / or 2800C, (Vx(x,y), Vy(x,y)) is equal to MVMix, which is equal to Shift(Wc(x,y)×MVc+WN1(x,y)×MVN1+WN2(x,y)×MVN2+...+WNk(x,y)×MVNk,n1), where the Shift() function refers to a binary shift operation, MVc is the motion vector of the video block, MVN1, ..., MVNk are the motion vectors of k spatially adjacent video blocks or temporally adjacent video blocks, N1, ..., Nk are spatially adjacent video blocks or temporally adjacent video blocks, and Wc, WN1, ..., WNk are weighting values ​​that are integers or real numbers, and n1 is an integer. In some embodiments of method 2800B and / or 2800C, (Vx(x,y), Vy(x,y)) is equal to MVMix, which is equal to SatShift(Wc(x,y)×MVc+WN1(x,y)×MVN1+WN2(x,y)×MVN2+...+WNk(x,y)×MVNk,n1), where the SatShift() function refers to a saturated binary shift operation, MVc is the motion vector of the video block, MVN1, ..., MVNk are the motion vectors of k spatially adjacent video blocks or temporally adjacent video blocks, N1, ..., Nk are spatially adjacent video blocks or temporally adjacent video blocks, and Wc, WN1, ..., WNk are weighting values ​​that are integers or real numbers, and n1 is an integer.

[0243] In some embodiments of methods 2800B and / or 2800C, Wc(x,y) is equal to zero. In some embodiments of methods 2800B and / or 2800C, k is equal to 1 and N1 is a spatially neighboring video block. In some embodiments of methods 2800B and / or 2800C, N1 is a spatially neighboring video block that is positionally closest to the position (x,y) in the video block. In some embodiments of methods 2800B and / or 2800C, WN1(x,y) increases in value as N1 approaches the position (x,y) in the video block. In some embodiments of methods 2800B and / or 2800C, k is equal to 1 and N1 is a temporally neighboring video block. In some embodiments of methods 2800B and / or 2800C, N1 is a collocated video block in a collocated picture for the position (x,y) in the video block. In some embodiments of methods 2800B and / or 2800C, in response to the prediction sample not being located at a top or left boundary of the video block, the prediction sample in the base video block is not refined.

[0244] In some embodiments of methods 2800B and / or 2800C, the prediction sample in the basic video block is refined in response to the prediction sample being located at the top boundary of the video block. In some embodiments of methods 2800B and / or 2800C, (Vx(x,y),Vy(x,y)) is equal to MVMix, and MVMix for a basic video block located at the top boundary of the video block is derived based on a neighboring video block located above the video block. In some embodiments of methods 2800B and / or 2800C, the prediction sample in the basic video block is refined in response to the prediction sample being located at the left boundary of the video block. In some embodiments of methods 2800B and / or 2800C, (Vx(x,y),Vy(x,y)) is equal to MVMix, and MVMix for a basic video block located at the left boundary of the video block is derived based on a neighboring video block located to the left of the video block. In some embodiments of methods 2800B and / or 2800C, the video block and motion vector and the motion vectors of a number of spatially neighboring video blocks or of a number of temporally neighboring video blocks are scaled to the same reference picture.

[0245] In some embodiments of methods 2800B and / or 2800C, the same reference picture is a reference picture referenced by the motion vector of the video block. In some embodiments of methods 2800B and / or 2800C, (Vx(x,y), Vy(x,y)) is derived using spatial or temporal neighboring video blocks only if the spatial or temporal neighboring video blocks are not intra-coded. In some embodiments of methods 2800B and / or 2800C, (Vx(x,y), Vy(x,y)) is derived using spatial or temporal neighboring video blocks only if the spatial or temporal neighboring video blocks are not intra block copy (IBC) predictively coded. In some embodiments of methods 2800B and / or 2800C, (Vx(x,y), Vy(x,y)) is derived using a spatially adjacent video block or a temporally adjacent video block only if the first motion vector of the spatially adjacent video block or the temporally adjacent video block references the same reference picture as the second motion vector of the video block.

[0246] In some embodiments of method 2800B and / or 2800C, (Vx(x,y),Vy(x,y)) is equal to f(MVMix,MVc), where f is a function and MVc is a motion vector for the video block. In some embodiments of method 2800B and / or 2800C, (Vx(x,y),Vy(x,y)) is equal to MVMix-MVc. In some embodiments of method 2800B and / or 2800C, (Vx(x,y),Vy(x,y)) is equal to MVc-MVMix. In some embodiments of method 2800B and / or 2800C, (Vx(x,y),Vy(x,y)) is equal to p*MVMix+q*MVc, where p and q are real numbers. In some embodiments of methods 2800B and / or 2800C, (Vx(x,y),Vy(x,y)) is equal to Shift(p×MVMix+q×MVc,n) or SatShift(p×MVMix+p×MVc,n), where p, q and n are integers, the Shift() function refers to a binary shift operation, and the SatShift() function refers to a saturated binary shift operation. In some embodiments of methods 2800B and / or 2800C, the video block is uni-predictively inter predicted and the motion vector of the video block references reference picture list 0. In some embodiments of methods 2800B and / or 2800C, the video block is uni-predictively inter predicted and the motion vector of the video block references reference picture list 1.

[0247] In some embodiments of methods 2800B and / or 2800C, the video block is inter predicted using bi-prediction techniques and the motion vector of the video block references reference picture list 0 or reference picture list 1. In some embodiments of methods 2800B and / or 2800C, the final predicted sample value is refined based on a first motion displacement Vx(x,y) and a second motion displacement Vy(x,y) derived from the motion vector of the video block references reference picture list 0 or reference picture list 1. In some embodiments of methods 2800B and / or 2800C, the first predicted sample value from reference picture list 0 is refined based on a first motion displacement Vx(x,y) and a second motion displacement Vy(x,y) derived from the motion vector of the video block references reference picture list 0. In some embodiments of methods 2800B and / or 2800C, a second predicted sample value from reference picture list 1 is refined based on a first motion displacement Vx(x,y) and a second motion displacement Vy(x,y) derived from a motion vector of an image block that references reference picture list 1.

[0248] In some embodiments of methods 2800B and / or 2800C, a final predicted sample value is derived using the first predicted sample value refined from reference picture list 0 and the second predicted sample value refined from reference picture list 1. In some embodiments of methods 2800B and / or 2800C, the final predicted sample value is equal to an average or weighted average of the first predicted sample value refined from reference picture list 0 and the second predicted sample value refined from reference picture list 1. In some embodiments of methods 2800B and / or 2800C, the video block is inter predicted using a bi-prediction technique, a bi-directional optical flow (BIO) technique is applied to the video block, and the first motion displacement Vx(x,y) and the second motion displacement Vy(x,y) are modified based on spatially or temporally neighboring video blocks.

[0249] In some embodiments of methods 2800B and / or 2800C, a first motion displacement vector V'(x,y) = (V'x(x,y), V'y(x,y)) is derived from BIO techniques, a second motion displacement vector derived using a method other than BIO techniques is denoted as V"(x,y) = (V"x(x,y), V"y(x,y), and a third motion displacement vector V(x,y) = (Vx(x,y), Vy(x,y)) is derived from the first set of motion displacement vectors. and the second set of motion displacement vectors, and a third motion displacement vector V(x,y) is used to refine the prediction sample P(x,y). In some embodiments of methods 2800B and / or 2800C, the third set of motion displacement vectors is obtained using the following formula: V(x,y)=V'(x,y)×W'(x,y)+V"(x,y)×W"(x,y), where W'(x,y) and W"(x,y) are integers or real numbers. In some embodiments of methods 2800B and / or 2800C, the third set of motion displacement vectors is obtained using the following formula: V(x,y)=Shift(V'(x,y)×W'(x,y)+V"(x,y)×W"(x,y),n1), where the Shift() function refers to a binary shift operation, W'(x,y) and W"(x,y) are integers, and n1 is a non-negative integer.

[0250] In some embodiments of methods 2800B and / or 2800C, the third set of motion displacement vectors is obtained using the following formula: V(x,y)=SatShift(V'(x,y)×W'(x,y)+V"(x,y)×W"(x,y),n1), where the SatShift() function refers to a saturated binary shift operation, W'(x,y) and W"(x,y) are integers, and n1 is a non-negative integer. In some embodiments of methods 2800B and / or 2800C, the first motion displacement Vx(x,y) and the second motion displacement Vx(x,y) are The motion displacement Vy(x,y) is modified based on the spatially or temporally adjacent video blocks and based on the position (x,y). In some embodiments of methods 2800B and / or 2800C, the first motion displacement Vx(x,y) and the second motion displacement Vy(x,y) that are not located on the top boundary of the video block or the left boundary of the video block are not modified. In some embodiments of methods 2800B and / or 2800C, the first motion displacement Vx(x,y) and the second motion displacement Vy(x,y) are clipped.

[0251] In some embodiments of methods 2800B and / or 2800C, the spatially adjacent video block or the temporally adjacent video block is a video block that is not adjacent to the video block. In some embodiments of methods 2800B and / or 2800C, the spatially adjacent video block or the temporally adjacent video block is a sub-block that is not adjacent to the video block. In some embodiments of methods 2800B and / or 2800C, the spatially adjacent video block or the temporally adjacent video block is a sub-block that is not adjacent to any of the video block, or the coding tree unit (CTU), or the video processing and distribution unit (VPDU), or the current region covering the sub-block. In some embodiments of methods 2800B and / or 2800C, the motion vector of the spatially adjacent video block or the temporally adjacent video block comprises an entry in a history-based motion vector. In some embodiments of methods 2800B and / or 2800C, determining the first motion displacement and the second motion displacement at the decoder side includes determining the presence of information related to the first motion displacement and the second motion displacement by parsing a bitstream representation of the video block.

[0252] In some embodiments of methods 2800B and / or 2800C, the first motion displacement Vx(x,y) and the second motion displacement Vy(x,y) are derived at a sub-block level of the video block. In some embodiments of methods 2800B and / or 2800C, the first motion displacement and the second motion displacement are derived at a 2×2 block level. In some embodiments of methods 2800B and / or 2800C, the first motion displacement and the second motion displacement are derived at a 4×1 block level. In some embodiments of methods 2800B and / or 2800C, the video block is an 8×4 video block or the sub-block is an 8×4 sub-block. In some embodiments of methods 2800B and / or 2800C, the video block is a 4×8 video block or the sub-block is a 4×8 sub-block. In some embodiments of methods 2800B and / or 2800C, the video block is a 4x4 single-predicted video block, or the sub-blocks are 4x4 single-predicted sub-blocks. In some embodiments of methods 2800B and / or 2800C, the video block is an 8x4, 4x8, and 4x4 single-predicted video block, or the sub-blocks are 8x4, 4x8, and 4x4 single-predicted sub-blocks.

[0253] In some embodiments of methods 2800B and / or 2800C, the video block excludes a 4x4 bi-predictive video block or the sub-block excludes a 4x4 bi-predictive sub-block. In some embodiments of methods 2800B and / or 2800C, the video block is a luma video block. In some embodiments of methods 2800B and / or 2800C, the determining and performing are based on any one or more of a color component of the video block, a block size of the video block, a color format of the video block, a block position of the video block, a motion type, a motion vector magnitude, a coding mode, a pixel gradient magnitude, a transform type, whether a bi-directional optical flow (BIO) technique is applied, whether a bi-predictive technique is applied, and whether a decoder-side motion vector refinement (DMVR) technique is applied.

[0254] 28D is a flow chart of an example video processing method. The method 2800D includes determining (2832) a first motion displacement Vx(x,y) at a position (x,y) and a second motion displacement Vy(x,y) at a position (x,y) within a video block to be encoded using an optical flow based method, where x and y are fractional, where Vx(x,y) and Vy(x,y) are determined based on at least the position (x,y) and a center position of a fundamental video block of the video block, and performing (2834) a conversion between the video block and a bitstream representation of the current video block using the first motion displacement and the second motion displacement.

[0255] In some embodiments of method 2800D, Vx(x,y)=a×(x-xc)+b×(y-yc), Vy(x,y)=c×(x-xc)+d×(y-yc), where (xc,yc) is the center position of the basic image block of the image block, a, b, c and d are affine parameters, the basic image block has dimensions w×h, and the position of the basic image block includes position (x,y). In some embodiments of method 2800D, Vx(x,y)=Shift(a×(x-xc)+b×(y-yc),n1), Vy(x,y)=Shift(c×(x-xc)+d×(y-yc),n1), (xc,yc) is a center position of a basic image block of an image block, a, b, c and d are affine parameters, the basic image block has dimensions w×h, the position of the basic image block comprises position (x,y), the Shift() function refers to a binary shift operation, and In some embodiments of method 2800D, Vx(x,y)=SatShift(a×(x-xc)+b×(y-yc),n1), Vy(x,y)=SatShift(c×(x-xc)+d×(y-yc),n1), where (xc,yc) is a center position of an elementary image block of an image block, a, b, c and d are affine parameters, the elementary image block has dimensions w×h, the position of the elementary image block comprises position (x,y), the SatShift() function refers to a saturated binary shift operation, and n1 is an integer.

[0256] In some embodiments of method 2800D, Vx(x,y)=-a×(x-xc)-b×(y-yc), Vy(x,y)=-c×(x-xc)-d×(y-yc), where (xc,yc) is a center position of the basic image block of the image block, a, b, c and d are affine parameters, the basic image block has dimensions w×h, and the position of the basic image block comprises position (x,y). In some embodiments of method 2800D, (xc,yc)=(x0+(w / 2),y0+(h / 2)), and the top left position of the basic image block is (x0,y0). In some embodiments of method 2800D, (xc,yc)=(x0+(w / 2)-1,y0+(h / 2)-1), and the top left position of the basic image block is (x0,y0). In some embodiments of method 2800D, (xc, yc)=(x0+(w / 2), y0+(h / 2)-1), and the top-left location of the basic image block is (x0, y0).

[0257] In some embodiments of method 2800D, (xc, yc)=(x0+(w / 2)-1, y0+(h / 2)), and the top-left location of the basic video block is (x0, y0). In some embodiments of method 2800D, c=-b and d=a, responsive to the video block being encoded using a four-parameter affine mode. In some embodiments of method 2800D, a, b, c, and d may be derived from the control point motion vector (CPMV), the width (W) of the video block, and the height (H) of the video block. In some embodiments of method 2800D,

number

[0258] In some embodiments of method 2800D, a, b, c, and d may be obtained from history-based stored information. In some embodiments of method 2800D, Vx(x+1,y)=Vx(x,y)+a, and Vy(x+1,y)=Vy(x,y)+c. In some embodiments of method 2800D, Vx(x+1,y)=Shift(Vx(x,y)+a,n1), and Vy(x+1,y)=Shift(Vy(x,y)+c,n1), where the Shift() function refers to a binary shift operation, and n1 is an integer. In some embodiments of method 2800D, Vx(x+1,y)=SatShift(Vx(x,y)+a,n1), Vy(x+1,y)=SatShift(Vy(x,y)+c,n1), where the SatShift() function refers to a saturated binary shift operation, and n1 is an integer. In some embodiments of method 2800D, Vx(x+1,y)=Vx(x,y)+Shift(a,n1), Vy(x+1,y)=Vy(x,y)+Shift(c,n1), where the Shift() function refers to a binary shift operation, and n1 is an integer.

[0259] In some embodiments of method 2800D, Vx(x+1,y)=Vx(x,y)+SatShift(a,n1), Vy(x+1,y)=Vy(x,y)+SatShift(c,n1), where the SatShift() function refers to a saturated binary shift operation, and n1 is an integer. In some embodiments of method 2800D, Vx(x,y+1)=Vx(x,y)+b, and Vy(x+1,y)=Vy(x,y)+d. In some embodiments of method 2800D, Vx(x,y+1)=Shift(Vx(x,y)+b,n1), Vy(x,y+1)=Shift(Vy(x,y)+d,n1), where the Shift() function refers to a binary shift operation, and n1 is an integer. In some embodiments of method 2800D, Vx(x,y+1)=SatShift(Vx(x,y)+b,n1), Vy(x,y+1)=SatShift(Vy(x,y)+d,n1), where the SatShift() function refers to a saturated binary shift operation, and n1 is an integer. In some embodiments of method 2800D, Vx(x,y+1)=Vx(x,y)+Shift(b,n1), Vy(x,y+1)=Vy(x,y)+Shift(d,n1), where the Shift() function refers to a binary shift operation, and n1 is an integer.

[0260] In some embodiments of method 2800D, Vx(x,y+1)=Vx(x,y)+SatShift(b,n1), Vy(x,y+1)=Vy(x,y)+SatShift(d,n1), where the SatShift() function refers to a saturated binary shift operation, and n1 is an integer. In some embodiments of method 2800D, in response to the video block being affine predicted using a bi-prediction technique, a, b, c, and d refer to reference picture list 0 or reference picture list 1. In some embodiments of method 2800D, the final prediction sample is refined with a first motion displacement Vx(x,y) and a second motion displacement Vy(x,y), and the first motion displacement Vx(x,y) and the second motion displacement Vy(x,y) are derived using a, b, c, and d that refer to either reference picture list 0 or reference picture list 1.

[0261] In some embodiments of method 2800D, the prediction samples for the first motion displacement Vx(x,y) and the second motion displacement Vy(x,y) are from reference picture list 0 and are refined, and the first motion displacement Vx(x,y) and the second motion displacement Vy(x,y) are derived using a, b, c, and d that refer to reference picture list 0. In some embodiments of method 2800D, the prediction samples for the first motion displacement Vx(x,y) and the second motion displacement Vy(x,y) are from reference picture list 1 and are refined, and the first motion displacement Vx(x,y) and the second motion displacement Vy(x,y) are derived using a, b, c, and d that refer to reference picture list 1. In some embodiments of method 2800D, the first prediction sample from reference picture list 0 is a first motion displacement vector (V 0 x(x,y),V 0 y(x,y)) and a second prediction sample from reference picture list 1 is derived using a, b, c, and d that refer to reference picture list 1 as a second motion displacement vector (V 1 x(x,y),V 1y(x,y)) and the final prediction sample is obtained by combining the first prediction sample with the second prediction sample.

[0262] In some embodiments of method 2800D, the motion displacement vector (Vx(x,y), Vy(x,y)) has a first motion vector precision that is different from a second motion vector precision of the motion vector of the video block. In some embodiments of method 2800D, the first motion vector precision is 1 / 8 pixel precision. In some embodiments of method 2800D, the first motion vector precision is 1 / 16 pixel precision. In some embodiments of method 2800D, the first motion vector precision is 1 / 32 pixel precision. In some embodiments of method 2800D, the first motion vector precision is 1 / 64 pixel precision. In some embodiments of method 2800D, the first motion vector precision is 1 / 128 pixel precision. In some embodiments of method 2800D, the motion displacement vector (Vx(x,y), Vy(x,y)) is determined based on a float pixel precision technique.

[0263] 28E is a flow chart of an example video processing method. The method 2800E includes determining (2842) a first gradient component Gx(x,y) in a first direction estimated at a position (x,y) in the video block and a second gradient component Gy(x,y) in a second direction estimated at a position (x,y) in the video block, the first gradient component and the second gradient component being based on a final predicted sample value of a predicted sample P(x,y) at the position (x,y), where x and y are integers, and performing (2844) a conversion between the video block and a bitstream representation of the current video block using a reconstructed sample value Rec(x,y) at the position (x,y) obtained based on a residual sample value Res(x,y) plus a final predicted sample value of a predicted sample P(x,y) refined using gradients Gx(x,y), Gy(x,y).

[0264] FIG. 28F is a flow chart of an example video processing method. Method 2800F includes determining (2852) a first gradient component Gx(x,y) in a first direction estimated at a position (x,y) within the video block and a second gradient component Gy(x,y) in a second direction estimated at a position (x,y) within the video block, where the first gradient component and the second gradient component are based on a final predicted sample value of a prediction sample P(x,y) at position (x,y), where x and y are integers, and encoding (2854) a bitstream representation of the video block to include a residual sample value Res(x,y) that is based on a reconstructed sample value Rec(x,y) at position (x,y), where the reconstructed sample value Rec(x,y) is based on the final predicted sample value of the prediction sample P(x,y) that has been refined using the gradients Gx(x,y), Gy(x,y) plus the residual sample value Res(x,y).

[0265] 28G is a flow chart of an example video processing method. The method 2800G includes determining (2862) a first gradient component Gx(x,y) in a first direction estimated at a position (x,y) within the video block and a second gradient component Gy(x,y) in a second direction estimated at a position (x,y) within the video block, where the first gradient component and the second gradient component are based on intermediate predicted sample values ​​of predicted samples P(x,y) at the position (x,y), where the final predicted sample values ​​of the predicted samples P(x,y) are based on the intermediate predicted sample values, and x and y are integers, and performing (2864) a conversion between the video block and a bitstream representation of the current video block using reconstructed sample values ​​Rec(x,y) at the position (x,y) obtained based on the final predicted sample values ​​of the predicted samples P(x,y) and residual sample values ​​Res(x,y).

[0266] 28H is a flow chart of an example video processing method. The method 2800H includes determining (2872) a first gradient component Gx(x,y) in a first direction estimated at a position (x,y) within the video block and a second gradient component Gy(x,y) in a second direction estimated at a position (x,y) within the video block, where the first gradient component and the second gradient component are based on intermediate predicted sample values ​​of predicted samples P(x,y) at the position (x,y), where the final predicted sample values ​​of the predicted samples P(x,y) are based on the intermediate predicted sample values, where x and y are integers, and encoding (2874) a bitstream representation of the video block to include residual sample values ​​Res(x,y) based on reconstructed sample values ​​Rec(x,y) at the position (x,y), where the reconstructed sample values ​​Rec(x,y) are based on the final predicted sample values ​​of the predicted samples P(x,y) and the residual sample values ​​Res(x,y).

[0267] In some embodiments of methods 2800G and / or 2800H, in response to the video block being affine predicted using a bi-prediction technique, the first gradient component and / or the second gradient component are based on intermediate predicted sample values ​​from one reference picture list. In some embodiments of methods 2800G and / or 2800H, when the video block uses an affine mode and local illumination compensation (LIC), the first gradient component and / or the second gradient component are based on inter-predicted sample values ​​before LIC is applied. In some embodiments of methods 2800G and / or 2800H, when the video block uses an affine mode with either a weighted prediction technique or a CU-level weighted bi-prediction (BCW) technique, the first gradient component and / or the second gradient component are based on inter-predicted sample values ​​before being multiplied by a weighting factor. In some embodiments of methods 2800G and / or 2800H, when the video block uses an affine mode and local illumination compensation (LIC), the first gradient component and / or the second gradient component are based on inter-predicted sample values ​​to which LIC has been applied. In some embodiments of methods 2800G and / or 2800H, when the video block uses an affine mode with either a weighted prediction technique or a CU-level weighted bi-prediction (BCW) technique, the first gradient component and / or the second gradient component are based on inter-predicted sample values ​​multiplied by a weighting factor.

[0268] FIG. 28I is a flow chart of an example video processing method. Method 2800I includes determining (2822) a refinement prediction sample P'(x,y) at a location (x,y) in an affine-coded video block by modifying (2822) the prediction sample P(x,y) at location (x,y) with a first gradient component Gx(x,y) in a first direction estimated at location (x,y), a second gradient component Gy(x,y) in a second direction estimated at location (x,y), a first motion displacement Vx(x,y) estimated for location (x,y), and a second motion displacement Vy(x,y) estimated for location (x,y), where the first direction is orthogonal to the second direction, and x and y are integers; The method includes determining (2884) a reconstructed sample value Rec(x,y) at position (x,y) based on the sample P'(x,y) and the residual sample values ​​Res(x,y), determining (2886) a refined reconstructed sample value Rec'(x,y) at position (x,y) within the affine-coded video block, where Rec'(x,y)=Rec(x,y)+Gx(x,y)×Vx(x,y)+Gy(x,y)×Vy(x,y), and performing (2888) a conversion between the affine-coded video block and the bitstream representation of the affine-coded video block using the refined reconstructed sample value Rec'(x,y).

[0269] 28J is a flow chart of an example video processing method. Method 2800J determines (2892) a refined prediction sample P'(x,y) at a location (x,y) in an affine-coded video block by modifying (2892) the prediction sample P(x,y) at location (x,y) with a first gradient component Gx(x,y) in a first direction estimated at location (x,y), a second gradient component Gy(x,y) in a second direction estimated at location (x,y), a first motion displacement Vx(x,y) estimated for location (x,y), and a second motion displacement Vy(x,y) estimated for location (x,y), where the first direction is orthogonal to the second direction, and where x and y are and determining (2894) a reconstructed sample value Rec(x,y) at position (x,y) based on the refined prediction sample P'(x,y) and the residual sample value Res(x,y), where P'(x,y) is an integer, determining (2896) a refined reconstructed sample value Rec'(x,y) at position (x,y) within the affine-coded video block, where Rec'(x,y)=Rec(x,y)+Gx(x,y)×Vx(x,y)+Gy(x,y)×Vy(x,y), and encoding (2898) a bitstream representation of the affine-coded video block to include the residual sample value Res(x,y).

[0270] In some embodiments of methods 2800I and / or 2800J, the first gradient component Gx(x,y) and / or the second gradient component Gy(x,y) are based on the reconstructed sample values ​​Rec(x,y). In some embodiments of methods 2800I and / or 2800J, the first motion displacement Vx(x,y) and the second motion displacement Vy(x,y) are derived at the 2x2 block level of the affine-coded video block.

[0271] 28K is a flow chart of an example video processing method. Method 2800K includes determining (28102) a motion vector with 1 / N pixel accuracy for a video block in affine mode, determining (28104) an estimated motion displacement vector (Vx(x,y), Vy(x,y)) for a position (x,y) within the video block, the motion displacement vector being derived with 1 / M pixel accuracy, where N and M are positive integers, and x and y are integers, and performing (28106) a conversion between the video block and a bitstream representation of the video block using the motion vector and the motion displacement vector.

[0272] In some embodiments of method 2800K, N is equal to 8 and M is equal to 16, or N is equal to 8 and M is equal to 32, or N is equal to 8 and M is equal to 64, or N is equal to 8 and M is equal to 128, or N is equal to 4 and M is equal to 8, or N is equal to 4 and M is equal to 16, or N is equal to 4 and M is equal to 32, or N is equal to 4 and M is equal to 64, or N is equal to 4 and M is equal to 128.

[0273] 28L is a flowchart of an example video processing method. The method 2800L includes determining (28112) two sets of motion vectors for a video block or for a sub-block of the video block, where each set of the two sets of motion vectors has a different motion vector pixel precision, and the two sets of motion vectors are determined using a temporal motion vector prediction (TMVP) technique or a sub-block-based temporal motion vector prediction (SbTMVP) technique, and performing (28114) a conversion between the video block and a bitstream representation of the video block based on the two sets of motion vectors.

[0274] In some embodiments of method 2800L, the two sets of motion vectors include a first set of motion vectors and a second set of motion vectors, where the first set of motion vectors has 1 / N pixel accuracy and the second set of motion vectors has 1 / M pixel accuracy, and N and M are positive integers. In some embodiments of method 2800L, N is equal to 8 and M is equal to 16, or N is equal to 8 and M is equal to 32, or N is equal to 8 and M is equal to 64, or N is equal to 8 and M is equal to 128, or N is equal to 4 and M is equal to 8, or N is equal to 4 and M is equal to 16, or N is equal to 4 and M is equal to 32, or N is equal to 4 and M is equal to 64, or N is equal to 4 and M is equal to 128, or N is equal to 16 and M is equal to 32, or N is equal to 16 and M is equal to 64, or N is equal to 16 and M is equal to 128. In some embodiments of method 2800L, refinement is applied to two sets of motion vectors by applying an optical flow based method, the two sets of motion vectors including a first set of motion vectors and a second set of motion vectors, a prediction sample in the optical flow based method is obtained using the first set of motion vectors, and an estimated motion displacement for a position within the image block in the optical flow based method is obtained by subtracting a second motion vector of the second set of motion vectors from a first motion vector of the first set of motion vectors.

[0275] In some embodiments of method 2800L, an optical flow based method is applied by determining a refined prediction sample P'(x,y) at a position (x,y) in the video block by modifying the prediction sample P(x,y) at the position (x,y) with a first gradient component Gx(x,y) in a first direction estimated at the position (x,y), a second gradient component Gy(x,y) in a second direction estimated at the position (x,y), a first motion displacement Vx(x,y) estimated for the position (x,y), and a second motion displacement Vy(x,y) estimated for the position (x,y), where x and y are integers, and a reconstructed sample value Rec(x,y) at the position (x,y) is obtained based on the refined prediction sample P'(x,y) and the residual sample value Res(x,y). In some embodiments of method 2800L, the first direction and the second direction are mutually orthogonal. In some embodiments of method 2800L, the first motion displacement represents a direction parallel to the first direction and the second motion displacement represents a direction parallel to the second direction. In some embodiments of method 2800L, P'(x,y) = P(x,y) + Gx(x,y) x Vx(x,y) + Gy(x,y) x Vy(x,y).

[0276] In some embodiments of any one or more of methods 2800C through 2800L, the video block is an 8x4 video block or the subblock is an 8x4 subblock. In some embodiments of any one or more of methods 2800C through 2800L, the video block is a 4x8 video block or the subblock is a 4x8 subblock. In some embodiments of any one or more of methods 2800C through 2800L, the video block is a 4x4 single-predicted video block or the subblock is a 4x4 single-predicted subblock. In some embodiments of any one or more of methods 2800C through 2800L, the video block is an 8x4, 4x8, and 4x4 single-predicted video block or the subblock is an 8x4, 4x8, and 4x4 single-predicted subblock. In some embodiments of any one or more of methods 2800C through 2800L, the video block excludes a 4x4 bi-predictive video block or the subblock excludes a 4x4 bi-predictive subblock. In some embodiments of any one or more of methods 2800C through 2800L, the video block is a luma video block. In some embodiments of any one or more of methods 2800C through 2800L, the determining and performing are performed based on any one or more of: a color component of the video block, a block size of the video block, a color format of the video block, a block position of the video block, a motion type, a motion vector magnitude, a coding mode, a pixel gradient magnitude, a transform type, whether a bi-directional optical flow (BIO) technique is applied, whether a bi-prediction technique is applied, and whether a decoder-side motion vector refinement (DMVR) technique is applied.

[0277] 28M is a flow chart of an example video processing method. Method 2800M performs 28122 an interweave prediction technique on a video block to be coded using an affine coding mode by partitioning the video block into a plurality of partitions using K subblock patterns, where K is an integer greater than 1, generates 28124 a predicted sample for the video block by motion compensation using a first of the K subblock patterns, denoted as P(x,y), where x and y are integers, and generates 28125 a predicted sample for the video block by motion compensation using a first of the K subblock patterns, denoted as P(x,y), where x and y are integers, and generates 28126 a predicted sample for the video block by motion compensation using a second of the K subblock patterns, denoted as P(x,y), where x and y are integers. For at least one of the K subblock patterns, determining (28126) an offset value OL(x,y) at position (x,y) based on the predicted sample derived with the first subblock pattern and a difference between the motion vector derived using a first of the K subblock patterns and the motion vector derived using the Lth pattern, determining (28128) a final predicted sample for position (x,y) as a function of OL(x,y) and P(x,y), and performing (28130) a conversion between the bitstream representation of the video block and the video block using the final predicted sample.

[0278] In some embodiments of method 2800M, K=2, L=1, and the final predicted sample is determined using P(x,y)+((O1(x,y)+1)>>1), where >> represents a binary shift operation. In some embodiments of method 2800M, K=2, L=1, and the final predicted sample is determined using P(x,y)+(O1(x,y)>>1), where >> represents a binary shift operation. In some embodiments of method 2800M, the final predicted sample is determined using P(x,y)+(O1(x,y)+...+OK(x,y)+K / 2) / K. In some embodiments of method 2800M, the final predicted sample is determined using P(x,y)+(O1(x,y)+...+OK(x,y)) / K. In some embodiments of method 2800M, OL(x,y) is generated from prediction samples derived in the first subblock pattern after performing horizontal and vertical interpolation and before converting the prediction samples to the bit depth of the input samples. In some embodiments of method 2800M, OL(x,y) is generated for each prediction direction. In some embodiments of method 2800M, a motion displacement at position (x,y) in the Lth pattern is derived as a difference between a first motion vector at position (x,y) from the Lth pattern and a second motion vector at position (x,y) from a first of the K subblock patterns.

[0279] In some embodiments of method 2800M, the motion vector derived using a first of the K subblock patterns has 1 / N pixel accuracy and the motion vector derived using the Lth pattern has 1 / ML pixel accuracy, where N and M are integers. In some embodiments of method 2800M, N=16 and ML=32, or N=16 and ML=64, or N=16 and ML=128, or N=8 and ML=16, or N=8 and ML=32, or N=8 and ML=64, or N=8 and ML=128, or N=4 and ML=8, or N=4 and ML=16, or N=4 and ML=32, or N=4 and ML=64, or N=4 and ML=128. In some embodiments of method 2800M, ML is different for each of the remaining K subblock patterns.

[0280] 28N is a flow chart of an example of a video processing method. The method 2800N includes performing (28132) a conversion between a bitstream representation of the video block and the video block using a final prediction sample, the final prediction sample being derived from the refined intermediate prediction sample by (a) performing an interweave prediction technique and a subsequent optical flow-based prediction refinement technique based on a rule, or (b) performing a motion compensation technique.

[0281] In some embodiments of method 2800N, the video block is coded using an affine coding mode by dividing the video block into multiple partitions using K different subblock patterns, where K is an integer greater than one, and for each subblock pattern, a prediction sample is generated for each subblock by performing a motion compensation technique, the prediction sample is refined using an optical flow-based prediction refinement technique to obtain an updated prediction sample, and a final prediction sample is generated by combining the refined prediction samples of each subblock pattern. In some embodiments of method 2800N, the video block is a uni-predictive video block, and an interweave prediction technique and an optical flow-based prediction refinement technique are applied to the video block. In some embodiments of method 2800N, a first tap filter is applied to a video block that is encoded with an interweave prediction technique and / or an optical flow-based prediction refinement technique, and the first tap filter is shorter than a second tap filter used in an interpolation filter used for other video blocks that are not encoded with an interweave prediction technique and / or an optical flow-based prediction refinement technique.

[0282] 83. The method of claim 82, wherein the affine sub-block size for the video block is 8x4 or 4x8. In some embodiments of method 2800N, a first sub-block size is used for a video block that is encoded with interweave prediction and / or optical flow based prediction refinement techniques, the first sub-block size being different from a second sub-block size for other video blocks that are not encoded with interweave prediction and / or optical flow based prediction refinement techniques. In some embodiments of method 2800N, the rule is based on coding information of the video block, the coding information including a prediction direction for the video block, reference picture information for the video block, or color components of the video block.

[0283] 28O is a flowchart of an example video processing method. The method 2800O includes, when bi-prediction is applied, performing (28142) a conversion between a bitstream representation of a video block and the video block using a final prediction sample, the final prediction sample being derived from refinement intermediate prediction samples by (a) disabling an interweave prediction technique and performing an optical flow-based prediction refinement technique, or (b) performing a motion compensation technique.

[0284] 28P is a flow chart of an example of a video processing method. Method 2800P includes performing (28152) a conversion between a bitstream representation of a video block and the video block using prediction samples, the prediction samples being derived from refined intermediate prediction samples by performing an optical flow-based prediction refinement technique, the performing of the optical flow-based prediction refinement technique depending on only one of a first set of motion displacements Vx(x,y) estimated in a first direction for the video block, or a second set of motion displacements Vy(x,y) estimated in a second direction for the video block, where x and y are integers, and the first direction is orthogonal to the second direction.

[0285] In some embodiments of method 2800P, the prediction sample is based only on the first set of motion displacements Vx(x,y), and the second set of motion displacements Vy(x,y) is zero. In some embodiments of method 2800P, the prediction sample is based only on the second set of motion displacements Vy(x,y), and the first set of motion displacements Vx(x,y) is zero. In some embodiments of method 2800P, in response to the sum of the absolute values ​​of the first set of motion displacements Vx(x,y) being equal to or greater than the sum of the absolute values ​​of the second set of motion displacements Vy(x,y), the prediction sample is based only on the first set of motion displacements Vx(x,y). In some embodiments of method 2800P, in response to the sum of the absolute values ​​of the first set of motion displacements Vx(x,y) being equal to or less than the sum of the absolute values ​​of the second set of motion displacements Vy(x,y), the prediction sample is based only on the second set of motion displacements Vy(x,y).

[0286] In some embodiments of method 2800P, in response to the sum of the absolute values ​​of the first gradient components in the first direction being greater than or equal to the sum of the absolute values ​​of the second gradient components in the second direction, the prediction samples are based only on the first set of motion displacements Vx(x,y). In some embodiments of method 2800P, in response to the sum of the absolute values ​​of the first gradient components in the first direction being greater than or equal to the sum of the absolute values ​​of the second gradient components in the second direction, the prediction samples are based only on the second set of motion displacements Vy(x,y).

[0287] 28Q is a flow chart of an example video processing method. Method 2800Q includes obtaining a refined motion vector for the video block by refining a motion vector of the video block (28162), where the motion vector is refined before performing a motion compensation technique, the refined motion vector has 1 / N pixel accuracy, the motion vector has 1 / M pixel accuracy, obtaining a final prediction sample by performing an optical flow-based prediction refinement technique on the video block (28164), applying the optical flow-based prediction refinement technique to a difference between the refined motion vector and the motion vector, and performing a conversion between the bitstream representation of the video block and the video block using the final prediction sample (28166).

[0288] In some embodiments of method 2800Q, M is 16 and N is 1, or M is 8 and N is 1, or M is 4 and N is 1, or M is 16 and N is 2, or M is 8 and N is 2, or M is 4 and N is 2. In some embodiments of method 2800Q, the motion vector is in either a first direction or a second direction, the first direction being orthogonal to the second direction, and the optical flow based prediction refinement technique is performed in either the first direction or the second direction. In some embodiments of method 2800Q, the video block is a bi-predictive video block, and the motion vector is refined in a first direction or a second direction, the first direction being orthogonal to the second direction. In some embodiments of method 2800Q, the video block is a bi-predictive video block, and the motion vector is refined in a first direction and a second direction, the first direction being orthogonal to the second direction. In some embodiments of method 2800Q, the optical flow based prediction refinement technique is performed on a first number of partial motion vector components of the motion vector, the first number of partial motion vector components being less than or equal to a second number of partial motion vector components of the motion vector.

[0289] 28R is a flow chart of an example video processing method, including determining (28172) a final motion vector using a multi-step decoder-side motion vector refinement process for a video block, the final motion vector having 1 / N pixel accuracy, and performing (28174) a conversion between the current block and a bitstream representation using the final motion vector.

[0290] In some embodiments of method 2800R, N is equal to 32 or 64 or 128. In some embodiments of method 2800R, the final motion vector is a refined motion vector obtained by refining a motion vector for the video block, where the motion vector is refined prior to performing a motion compensation technique, and the final prediction sample is obtained by performing an optical flow based prediction refinement technique on the video block, where the optical flow based prediction refinement technique is applied to the difference between the final motion vector and the motion vector. In some embodiments of method 2800R, an optical flow based prediction refinement technique is applied by determining a refined prediction sample P'(x,y) at a position (x,y) in the video block by modifying the prediction sample P(x,y) at the position (x,y) with a first gradient component Gx(x,y) in a first direction estimated at the position (x,y), a second gradient component Gy(x,y) in a second direction estimated at the position (x,y), a first motion displacement Vx(x,y) estimated for the position (x,y), and a second motion displacement Vy(x,y) estimated for the position (x,y), where x and y are integers, and a final prediction sample Rec(x,y) at the position (x,y) is obtained based on the refined prediction sample P'(x,y) and the residual sample value Res(x,y). In some embodiments of method 2800R, the first direction and the second direction are mutually orthogonal. 105. The method of claim 104, wherein in some embodiments of method 2800R, the first movement displacement represents a direction parallel to a first direction and the second movement displacement represents a direction parallel to a second direction.

[0291] In some embodiments of method 2800R, P'(x,y) = P(x,y) + Gx(x,y) x Vx(x,y) + Gy(x,y) x Vy(x,y). In some embodiments of method 2800R, the refinement motion vector with 1 / 32 pixel precision is rounded to 1 / 16 pixel precision prior to performing the motion compensation technique. In some embodiments of method 2800R, the refinement motion vector with 1 / 32 pixel precision is rounded to 1 pixel precision prior to performing the motion compensation technique. In some embodiments of method 2800R, the refinement motion vector with 1 / 64 pixel precision is rounded to 1 / 16 pixel precision prior to performing the motion compensation technique. In some embodiments of method 2800R, the refinement motion vector with 1 / 64 pixel precision is rounded to 1 pixel precision prior to performing the motion compensation technique.

[0292] 28S is a flow chart of an example video processing method. The method 2800S includes obtaining (28182) refined intermediate prediction samples of a video block by performing an interweave prediction technique and an optical flow-based prediction refinement technique on the intermediate prediction samples of the video block, deriving (28184) final prediction samples from the refined intermediate prediction samples, and performing (28186) a conversion between a bitstream representation of the video block and the video block using the final prediction samples.

[0293] In some embodiments of the method 2800S, an optical flow-based prediction refinement technique is first performed, and then an interweave prediction technique is performed. In some embodiments of the method 2800S, refined intermediate prediction samples having two different subblock division patterns are first obtained using the optical flow-based prediction refinement technique, and the refined intermediate prediction samples are weighted-averaged using the interweave prediction technique to obtain a final prediction sample. In some embodiments of the method 2800S, an interweave prediction technique is first performed, and then an optical flow-based prediction refinement technique is performed. In some embodiments of the method 2800S, intermediate prediction samples having two different subblock division patterns are weighted-averaged using the interweave prediction technique to obtain a refined intermediate prediction sample, and then an optical flow-based prediction refinement technique is performed on the refined intermediate prediction sample to obtain a final prediction sample. In some embodiments of any one or more of methods 2800N through 2800S, the optical flow based prediction refinement technique is a prediction refinement with optical flow (PROF) technique.

[0294] 28T is a flow chart of an example video processing method. Method 2800T includes obtaining (28192) refined intermediate prediction samples of a video block by performing an interweave prediction technique and a phase variation affine sub-block motion compensation (PAMC) technique on the intermediate prediction samples of the video block, deriving (28194) final prediction samples from the refined intermediate prediction samples, and performing (28196) a conversion between a bitstream representation of the video block and the video block using the final prediction samples.

[0295] In some embodiments of the method 2800T, the PAMC technique is performed first, and then the interweave prediction technique is performed. In some embodiments of the method 2800T, the refined intermediate prediction samples with two different subblock division patterns are first obtained using the interpolation method of the PAMC technique, and then the refined intermediate prediction samples are weighted-averaged using the interweave prediction technique to obtain a final prediction sample. In some embodiments of the method 2800T, the interweave prediction technique is performed first, and then the PAMC technique is performed.

[0296] FIG. 28U is a flow chart of an example video processing method. The method 2800U includes obtaining (28202) refined intermediate prediction samples of a video block by performing an optical flow-based prediction refinement technique and a phase variation affine sub-block motion compensation (PAMC) technique on the intermediate prediction samples of the video block, deriving (28204) final prediction samples from the refined intermediate prediction samples, and performing (28206) a conversion between a bitstream representation of the video block and the video block using the final prediction samples. In some embodiments of the method 2800U, the PAMC technique is performed first, and then the optical flow-based prediction refinement technique is performed. In some embodiments of the method 2800U, the refinement intermediate prediction sample with two different subblock division patterns is obtained first using the interpolation method of the PAMC technique, and then the refinement intermediate prediction sample is processed using the optical flow-based prediction refinement technique to obtain a final prediction sample. In some embodiments of the method 2800U, the optical flow-based prediction refinement technique is performed first, and then the PAMC technique is performed.

[0297] In this document, the term "video processing" may refer to video encoding (including transcoding), video decoding, video compression, or video decompression. For example, a video compression algorithm may be applied during conversion from a pixel representation of a video to a corresponding bitstream representation or vice versa. The bitstream representation of a current video block may correspond to bits that are collocated or spread differently within the bitstream, e.g., as specified by a syntax. For example, a macroblock may be coded with respect to a transformed coded error residual value, and may also be coded using bits in the header and other fields in the bitstream.

[0298] It will be appreciated that the disclosed methods and techniques will be beneficial in embodiments of video encoders and / or decoders integrated into video processing devices such as smartphones, laptops, desktops, and similar devices by enabling the use of the techniques disclosed herein.

[0299] The following list of sections provides further mechanisms and embodiments that use the techniques disclosed in this document.

[0300] 1. A method of video processing, comprising the steps of: determining a refined prediction sample P'(x,y) at a position (x,y) in the video block by modifying the prediction sample P(x,y) at the position (x,y) as a function of a gradient in a first direction and / or a second direction estimated at the position (x,y) and a first motion displacement and / or a second motion displacement estimated for the position (x,y); and performing the conversion using a reconstructed sample value Rec(x,y) from the refined prediction sample P'(x,y).

[0301] 2. The method of claim 1, wherein the first direction and the second direction are perpendicular to each other.

[0302] 3. The method of any of claims 1 or 2, wherein the first movement displacement is in a direction parallel to the first direction and the second movement displacement is in a direction parallel to the second direction.

[0303] 4. The method according to any one of clauses 1 to 3, wherein the gradient in the first direction is expressed as Gx(x,y), the gradient in the second direction is expressed as Gy(x,y), the first motion displacement is expressed as Vx(x,y), the second motion displacement is expressed as Vy(x,y), and P'(x,y) = P(x,y) + Gx(x,y) × Vx(x,y) + Gy(x,y) × Vy(x,y).

[0304] 5. The method according to any of clauses 1 to 3, wherein the gradient in a first direction is denoted as Gx(x,y), the gradient in the second direction is denoted as Gy(x,y), the first motion displacement is denoted as Vx(x,y), the second motion displacement is denoted as Vy(x,y), and P'(x,y) = α(x,y) x P(x,y) + β(x,y) x Gx(x,y) x Vx(x,y) + γ(x,y) x Gy(x,y) x Vy(x,y), where α(x,y), β(x,y) and γ(x,y) are weighting values ​​at the position (x,y) and are integers or real numbers.

[0305] 6. The method according to any one of claims 1 to 5, wherein the predicted sample P(x, y) is a one-way predicted sample at the position (x, y).

[0306] 7. The method according to any one of claims 1 to 5, wherein the predicted sample P(x, y) is the final result of bi-prediction at the position (x, y).

[0307] 8. The method according to any one of claims 1 to 5, wherein the predicted sample P(x,y) satisfies one of the following: The result of multiple hypothesis inter prediction (inter prediction using three or more MVs) Affine prediction results: Intra prediction results, Intra-Block Copy (IBC) prediction results, Generated by triangular prediction mode (TPM), Inter-intra joint prediction results, Global inter prediction results, where the regions share the same motion model and parameters. Palette coding mode results in Inter-view prediction results in multi-view or 3D video coding, · Inter-layer prediction results in scalable video coding, The result of the filtering operation.

[0308] The result of a refinement process to improve the accuracy of the prediction at said location (x,y).

[0309] 9. The method of any of clauses 1 to 8, wherein the reconstructed sample values ​​Rec(x,y) are further refined prior to use in the transformation.

[0310] Item 1 listed in Section 4 provides further example embodiments for Methods 1-9.

[0311] 10. A method of video processing comprising the steps of determining a first displacement vector Vx(x,y) and a second displacement vector Vy(x,y) at a position (x,y) within a video block corresponding to an optical flow-based method of encoding the video block based on information from adjacent blocks or basic blocks, and performing a conversion between the video block and a bitstream representation of a current video block using the first displacement vector and the second displacement vector.

[0312] 11. The method of claim 10, wherein the adjacent blocks are spatially adjacent blocks.

[0313] 12. The method of claim 10, wherein the adjacent blocks are time-adjacent blocks.

[0314] 13. The method of any of clauses 10 to 12, wherein the neighboring blocks are selected based on a coding unit size, or a coding mode, or a position of a possible candidate block neighboring the video block.

[0315] 14. The method according to any of clauses 10 to 13, wherein the first displacement vector and the second displacement vector are calculated using a combination of motion vectors for a number of adjacent blocks.

[0316] 15. The method of any one of claims 11 to 13, wherein the basic blocks have a predetermined size.

[0317] 16. The method according to claim 10, wherein Vx(x,y) and Vy(x,y) are determined as Vx(x,y)=a×(x-xc)+b(y-yc), Vy(x,y)=c×(x-xc)+d(y-yc), where (xc,yc) is the center position of the basic block of dimensions w×h that covers the position (x,y).

[0318] 17. The method of claim 10, wherein Vx(x,y) and Vy(x,y) are determined as Vx(x,y)=Shift(a×(x-xc)+b(y-yc),n1), Vy(x,y)=Shift(c×(x-xc)+d(y-yc),n1, where n1 is an integer and (xc,yc) is the center position of the basic block of dimensions w×h that covers the position (x,y).

[0319] 18. The method of claim 10, wherein Vx(x,y) and Vy(x,y) are determined as Vx(x+1,y)=Vx(x,y)+a and Vy(x+1,y)=Vy(x,y)+c, and parameters a and c are obtained based on information from the adjacent blocks or from a history-based storage device.

[0320] 19. The method according to any of clauses 10 to 18, wherein the precision of Vx(x,y) and Vy(x,y) is different from the precision of the motion vectors of the basic blocks.

[0321] Section 4, item 2 provides further examples of the embodiments of items 10-19.

[0322] 20. A method of video processing, comprising: determining a refined prediction sample P'(x,y) at a position (x,y) in the video block by modifying a prediction sample P(x,y) at the position (x,y), wherein a gradient in a first direction and a gradient in a second direction at the position (x,y) are determined based on the refined prediction sample P'(x,y) and a final prediction value determined from a residual sample value at the position (x,y); and performing the conversion using the gradient in the first direction and the gradient in the second direction.

[0323] 21. The method of claim 20, wherein the gradient in the first direction and the gradient in the second direction correspond to a horizontal gradient and a vertical gradient.

[0324] 22. The method according to any of clauses 20 to 21, wherein the gradient in the first direction and the gradient in the second direction are derived from an intermediate prediction result.

[0325] 23. The method of any of clauses 20-21, wherein the gradient in the first direction and the gradient in the second direction are derived from one reference picture list for the video block that is an affine predicted bi-predictive video block.

[0326] Items 3 and 4 of Section 4 provide further example embodiments of the techniques described in Items 20-23.

[0327] 24. A method of video processing comprising the steps of determining a reconstructed sample Rec(x,y) at a position (x,y) in a video block to be affine coded, refining Rec(x,y) using first and second displacement vectors and first and second gradients at the position (x,y) to obtain a refined reconstructed sample Rec'(x,y), and using the refined reconstructed sample to perform a conversion between the video block and a bitstream representation of a current video block.

[0328] 25. The method according to clause 24, wherein Rec'(x,y)=Rec(x,y)+Gx(x,y)×Vx(x,y)+Gy(x,y)×Vy(x,y), where Gx(x,y) and Gy(x,y) represent the first and second gradients, and Vx(x,y) and Vy(x,y) represent the first and second displacement vectors at the position (x,y).

[0329] 26. The method of claim 25, wherein Vx(x,y) and Vy(x,y) are derived at the sub-block level.

[0330] 27. The method of claim 25, wherein Vx(x,y) and Vy(x,y) are derived with 1 / M pel precision, where M is an integer.

[0331] 28. The method of any of clauses 1 to 27, wherein the method is applied as a consequence of the video block having a particular size and / or a particular coding mode.

[0332] 29. The method of claim 28, wherein the specific dimensions are 8x4.

[0333] 30. The method of claim 28, wherein the image block is a 4x4 single-prediction image block.

[0334] Items 5-12 of Section 4 provide examples of the embodiments described in Items 24-30.

[0335] 31. A method according to any of clauses 1 to 27, wherein the method is applied as a result of the image block having particular color components or a particular color format, or having a particular position within an image picture, or using a particular transformation type.

[0336] 32. The method of any of clauses 1 to 31, wherein the transforming includes generating the current block from the bitstream representation or generating the bitstream representation from the current block.

[0337] 33. A method of image processing comprising: 25. A video processing method comprising: performing interweave prediction of a video block by dividing the video block into a plurality of partitions using K subblock patterns, where K is an integer greater than one, performing motion compensation using a first of the K subblock patterns to generate a predicted sample for the video block, the predicted sample at a location (x, y) being denoted as P(x, y); determining, for at least one of the remaining K subblock patterns, denoted as an L pattern, an offset value OL(x, y) at the location (x, y) based on P(x, y) and a difference between a motion vector derived using the first of the K subblock patterns and a motion vector derived using the L pattern; determining a final predicted sample for the location (x, y) as a function of OL(x, y) and P(x, y); and performing the conversion using the final predicted sample.

[0338] 34. The method of claim 33, wherein K=2, L=1, and the final predicted sample is determined using P(x,y)+((O1(x,y)+1)>>1), where >> denotes a binary shift operation.

[0339] 35. The method of claim 33, wherein K=2, L=1, and the final predicted sample is determined using P(x,y)+(O1(x,y)>>1), where >> denotes a binary shift operation.

[0340] 36. The method of any of clauses 33 to 35, wherein OL(x,y) is generated from P(x,y) after performing horizontal and vertical interpolation.

[0341] 37. The method of any of clauses 33 to 35, wherein motion compensation for the first pattern can use 1 / N pixel accuracy, and motion compensation for the L pattern can use 1 / ML pixel accuracy, where N and M are integers.

[0342] 38. The method of claim 37, wherein N=16 and ML=32, 64 or 128.

[0343] 39. The method of claim 37, wherein N=8 and ML=16, 32, 64 or 128.

[0344] 40. The method of claim 37, wherein N=4 and ML=8, 16, 32, 64 or 128.

[0345] 41. A video encoder or re-encoder having a processor configured to implement a method according to any one of clauses 1 to 40.

[0346] 42. A video decoder having a processor configured to implement a method according to any one of clauses 1 to 40.

[0347] 43. A computer readable medium having code for implementing the method of any one of clauses 1 to 40.

[0348] 33 is a block diagram illustrating an example of a video processing system 2100 in which various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of the system 2100. The system 2100 may include an input 2102 for receiving video content. The video content may be received in a raw or uncompressed format, such as 8-bit or 10-bit multi-component pixel values, or in a compressed or encoded format. The input 2102 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, passive optical network (PON), and wireless interfaces such as Wi-Fi or cellular interfaces.

[0349] The system 2100 may include an encoding component 2104 that may implement various coding or encoding methods described herein. The encoding component 2104 may reduce the average bit rate of the video from the input 2102 to the output of the encoding component 2104 to generate an encoded representation of the video. Encoding techniques are therefore sometimes referred to as video compression techniques or video transcoding techniques. The output of the encoding component 2104 may be stored or transmitted via a communication connection as represented by component 2106. The stored or communicated bitstream (or encoded) representation of the video received at the input 2102 may be used by component 2108 to generate pixel values ​​or a displayable image that is sent to the display interface 2110. The process of generating a user-viewable image from the bitstream representation is sometimes referred to as video decompression. Also, although certain video processing operations may be referred to as “encoding” operations or tools, it is understood that the encoding tools or operations are used at the encoder and the corresponding decoding tools or operations that reverse the results of the encoding are performed at the decoder.

[0350] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB) or High Definition Multimedia Interface (HDMI) or Displayport, etc. Examples of storage interfaces include serial advanced technology attachment (SATA), PCI, IDE interfaces, etc. The techniques described in this document may be embodied in a variety of electronic devices, such as, for example, mobile phones, laptops, smartphones, or other devices capable of performing digital data processing and / or video display.

[0351] Some embodiments of the disclosed technology include making a decision to enable a video processing tool or mode. In one example, when a video processing tool or mode is enabled, the encoder will use or implement that tool or mode in processing blocks of video, but will not necessarily modify the resulting bitstream based on the use of that tool or mode. That is, the conversion of blocks of video to a bitstream representation of video will use the video processing tool or mode when it is enabled based on the decision. In another example, when a video processing tool or mode is enabled, the decoder will process the bitstream knowing that the bitstream has been modified based on the video processing tool or mode. That is, the conversion of a bitstream representation of video to blocks of video will be performed using the video processing tool or mode that was enabled based on the decision.

[0352] Some embodiments of the disclosed technology include making a decision to disable a video processing tool or mode. In one example, when a video processing tool or mode is disabled, an encoder will not use that tool or mode in converting blocks of video into a video bitstream. In another example, when a video processing tool or mode is disabled, a decoder will process the bitstream knowing that the bitstream has not been modified using the video processing tool or mode that was disabled based on the decision.

[0353] FIG. 34 is a block diagram illustrating an example of a video encoding system 100 that may utilize the techniques of this disclosure. As shown in FIG. 34, the video encoding system 100 may include a source device 110 and a destination device 120. The source device 110 generates encoded video data and may be referred to as a video encoding device. The destination device 120 may decode the encoded video data generated by the source device 110 and may be referred to as a video decoding device. The source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0354] The video source 112 may include a source, such as, for example, a video capture device, an interface that receives video data from a video content provider, and / or a computer graphics system that generates video data, or a combination of such sources. The video data may have one or more pictures. The video encoder 114 encodes the video data from the video source 112 to generate a bitstream. The bitstream may include a series of bits that form an encoded representation of the video data. The bitstream may include encoded pictures and associated data. An encoded picture is an encoded representation of a picture. The associated data may include sequence parameter sets, picture parameter sets, and other syntax structures. The I / O interface 116 may include a modulator / demodulator (modem) and / or a transmitter. The encoded video data may be transmitted via the I / O interface 116 directly over the network 130a to the destination device 120. The encoded video data may also be stored on a storage medium / server 130b for access by the destination device 120.

[0355] The destination device 120 may include an I / O interface 126 , a video decoder 124 , and a display device 122 .

[0356] The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain encoded video data from the source device 110 or the storage medium / server 130b. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to a user. The display device 122 may be integrated with the destination device 120 or may be external to the destination device 120 configured to interface with an external display device.

[0357] The video encoder 114 and the video decoder 124 may operate according to a video compression standard, such as, for example, the High Efficiency Video Coding (HEVC) standard, the Versatile Video Coding (VVC) standard, and other current and / or future standards.

[0358] FIG. 35 is a block diagram illustrating an example of a video encoder 200, which may be video encoder 114 in system 100 shown in FIG.

[0359] Video encoder 200 may be configured to perform any or all of the techniques of this disclosure. In the example of FIG. 35, video encoder 200 includes multiple functional components. The techniques described in this disclosure may be shared among various components of video encoder 200. In some examples, a processor may be configured to perform any or all of the techniques described in this disclosure.

[0360] Functional components of the video encoder 200 may include a division unit 201, a prediction unit 202, which may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205, and an intra prediction unit 206, a residual generation unit 207, a transform unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transform unit 211, a reconstruction unit 212, a buffer 213, and an entropy coding unit 214.

[0361] In other examples, video encoder 200 may include more, fewer, or different functional components. In one example, prediction unit 202 may include an intra block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode, where at least one reference picture is the picture in which the current video block is located.

[0362] Also, some components, such as the motion estimation unit 204 and the motion compensation unit 205, although depicted separately in the example of FIG. 35 for illustrative purposes, may be highly integrated.

[0363] Division unit 201 may divide a picture into one or more video blocks. Video encoder 200 and video decoder 300 may support a variety of video block sizes.

[0364] The mode selection unit 203 may select one of a number of coding modes, intra or inter, for example based on the error result, and provide the resulting intra or inter coded block to a residual generation unit 207 that generates residual block data and to a reconstruction unit 212 that reconstructs the coded block for use as a reference picture. In some examples, the mode selection unit 203 may select a combination of intra and inter predication (CIIP) mode, in which prediction is based on an inter prediction signal and an intra prediction signal. The mode selection unit 203 may also select the resolution of the motion vector for the block (e.g., sub-pixel or integer pixel precision) in the case of inter prediction.

[0365] To perform inter prediction on the current video block, motion estimation unit 204 may generate motion information for the current video block by comparing one or more reference frames from buffer 213 to the current video block. Motion compensation unit 205 may determine a prediction video block for the current video block based on the motion information and decoded samples of pictures from buffer 213 other than the picture associated with the current video block.

[0366] Motion estimation unit 204 and motion compensation unit 205 may perform different operations on the current video block depending on whether the current video block is in an I slice, a P slice, or a B slice, for example.

[0367] In some examples, motion estimation unit 204 may perform unidirectional prediction on the current video block, and motion estimation unit 204 may search reference pictures of list 0 or list 1 for a reference video block for the current video block. Motion estimation unit 204 may then generate a reference index that points to a reference picture in list 0 or list 1 that contains the reference video block, and a motion vector that indicates a spatial displacement between the current video block and the reference video block. Motion estimation unit 204 may output the reference index, a prediction direction indicator, and the motion vector as motion information of the current video block. Based on the reference video block indicated by the motion information of the current video block, motion compensation unit 205 may generate a prediction video block for the current block.

[0368] In another example, motion estimation unit 204 may perform bidirectional prediction on the current video block, and motion estimation unit 204 may search reference pictures in list 0 for a reference video block for the current video block, and may also search reference pictures in list 1 for another reference video block for the current video block. Motion estimation unit 204 may then generate reference indexes that point to the reference pictures in list 0 and list 1 that contain the reference video blocks, and motion vectors that indicate spatial displacements between the reference video blocks and the current video block. Motion estimation unit 204 may output the reference indexes and the motion vector of the current video block as motion information of the current video block. Based on the reference video blocks indicated by the motion information of the current video block, motion compensation unit 205 may generate a prediction video block for the current block.

[0369] In some examples, the motion estimation unit 204 may output a complete set of motion information for the decoding process of the decoder.

[0370] In some examples, motion estimation unit 204 may not output a complete set of motion information for the current picture. Rather, motion estimation unit 204 may signal motion information of the current video block by reference to motion information of another video block. For example, motion estimation unit 204 may determine that the motion information of the current video block is sufficiently similar to the motion information of a neighboring video block.

[0371] In one example, motion estimation unit 204 may point to a value within a syntax structure associated with the current video block that indicates to video decoder 300 that the current video block has the same motion information as another video block.

[0372] In another example, motion estimation unit 204 may identify another video block and a motion vector difference (MVD) within a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the pointed-to video block. Video decoder 300 may use the motion vector of the pointed-to video block and the motion vector difference to determine the motion vector of the current video block.

[0373] As mentioned above, video encoder 200 may predictively signal motion vectors. Two examples of predictive signaling techniques that may be implemented by video encoder 200 include advanced motion vector prediction (AMVP) and merge mode signaling.

[0374] The intra prediction unit 206 may perform intra prediction on the current video block. When the intra prediction unit 206 performs intra prediction on the current video block, the intra prediction unit 206 may generate predictive data for the current video block based on decoded samples of other video blocks in the same picture. The predictive data for the current video block may include a predictive video block and various syntax elements.

[0375] Residual generation unit 207 may generate residual data for the current video block by subtracting (e.g., as indicated by a minus sign) the prediction video block(s) of the current video block from the current video block. The residual data for the current video block may include residual video blocks that correspond to different sample components of the samples in the current video block.

[0376] In other examples, such as in skip mode, residual data for the current video block may not exist and residual generation unit 207 may not perform a subtraction operation.

[0377] Transform processing unit 208 may generate one or more transform coefficient image blocks for the current video block by applying one or more transforms to a residual video block related to the current video block.

[0378] After transform processing unit 208 generates the transform coefficient image block for the current video block, quantization unit 209 may quantize the transform coefficient image block for the current video block based on one or more quantization parameter (QP) values ​​for the current video block.

[0379] Inverse quantization unit 210 and inverse transform unit 211 may apply inverse quantization and inverse transform, respectively, to the transform coefficient video block to reconstruct a residual video block from the transform coefficient video block. Reconstruction unit 212 may add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by prediction unit 202 to generate a reconstructed video block for the current block, which is stored in buffer 213.

[0380] After reconstruction unit 212 reconstructs the video block, a loop filtering operation may be performed to reduce image blocking artifacts in the video block.

[0381] An entropy encoding unit 214 may receive data from other functional components of the video encoder 200. Once the entropy encoding unit 214 receives the data, it may perform one or more entropy encoding operations to generate entropy encoded data and output a bitstream including the entropy encoded data.

[0382] FIG. 36 is a block diagram illustrating an example of a video decoder 300, which may be the video decoder 124 in the system 100 shown in FIG.

[0383] The video decoder 300 may be configured to perform any or all of the techniques described in this disclosure. In the example of FIG. 36, the video decoder 300 includes multiple functional components. The techniques described in this disclosure may be shared among various components of the video decoder 300. In some examples, a processor may be configured to perform any or all of the techniques described in this disclosure.

[0384] In the example of Figure 36, the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, a reconstruction unit 306, and a buffer 307. The video decoder 300 may, in some examples, perform a decoding pass that is generally inverse to the encoding pass described with respect to the video encoder 200 (Figure 35).

[0385] An entropy decoding unit 301 may retrieve an encoded bitstream. The encoded bitstream may include entropy encoded video data (e.g., encoded blocks of video data). The entropy decoding unit 301 may decode the entropy encoded video data, and from the entropy decoded video data, a motion compensation unit 302 may determine motion information including motion vectors, motion vector precision, reference picture list index, and other motion information. The motion compensation unit 302 may determine such information, for example, by implementing AMVP and merge mode.

[0386] The motion compensation unit 302 may perform interpolation, possibly based on an interpolation filter, to generate the motion compensated blocks. An identifier for the interpolation filter used with sub-pixel precision may be included in the syntax element.

[0387] Motion compensation unit 302 may calculate interpolated values ​​for sub-integer pixels of the reference block using an interpolation filter used by video encoder 200 during encoding of the video block. Motion compensation unit 302 may determine the interpolation filter used by video encoder 200 according to the received syntax information and generate a predictive block using the interpolation filter.

[0388] The motion compensation unit 302 may use a portion of the syntax information to determine the size of the blocks used to code frames and / or slices of the coded video sequence, partition information describing how each macroblock of a picture of the coded video sequence is divided, a mode indicating how each partition is coded, one or more reference frames (and reference frame lists) for each inter-coded block, and other information for decoding the coded video sequence.

[0389] The intra prediction unit 303 may form a prediction block from spatially neighboring blocks using an intra prediction mode, e.g., received in the bitstream. The inverse quantization unit 303 inverse quantizes, i.e., dequantizes, the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 303 applies an inverse transform.

[0390] A reconstruction unit 306 may sum the residual block with a corresponding prediction block generated by the motion compensation unit 302 or the intra prediction unit 303 to form a decoded block. If desired, a deblocking filter may also be applied to filter the decoded block to remove blocking artifacts. The decoded video block is then stored in a buffer 307, which provides a reference block for later motion compensation / intra prediction and also generates a decoded video for presentation on a display device.

[0391] In this patent document, the term "sample" or "samples" may refer to one or more samples of a video block. From the foregoing, it will be understood that, although specific embodiments of the presently disclosed technology have been described herein for purposes of illustration, various modifications may be made without departing from the scope of the invention. Accordingly, the presently disclosed technology is not to be limited except as by the appended claims.

[0392] The disclosed and other solutions, examples, embodiments, modules and functional operations described in this document, including the structures disclosed in this document and their structural equivalents, can be implemented in digital electronic circuitry, or computer software, firmware, or hardware, or in a combination of one or more of these. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., as one or more modules of computer program instructions encoded on a computer-readable medium for execution by or for controlling the operation of a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter producing a machine-readable propagated signal, or a combination of one or more of these. The term "data processing apparatus" encompasses any apparatus, device, and machine that processes data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, such as code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these. A propagated signal is an artificially generated signal, for example a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to an appropriate receiver device.

[0393] A computer program (also known as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple cooperating files (e.g., a file that stores one or more modules, subprograms, or portions of code). A computer program may be deployed to be executed on one computer or to be executed on multiple computers, located at one site, or distributed across multiple sites and interconnected by a communication network.

[0394] The processes and logic flows described in this document may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. These processes and logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0395] Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, a processor receives instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices for storing data, e.g., magnetic disks, magneto-optical disks, or optical disks, or is operatively coupled to receive data from or transfer data to the mass storage devices. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, dedicated logic circuitry.

[0396] Although this patent document contains numerous details, they should not be construed as limitations on any subject matter or the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular technology. Certain features described in this patent document in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in a particular combination, and even initially claimed as such, in some cases one or more features from a claimed combination may be removed from the combination, or the claimed combination may be brought into a subcombination or a variation of the subcombination.

[0397] Similarly, although the figures show operations in a particular order, this should not be understood as requiring that those operations be performed in the particular order or sequence shown, or that all of the operations shown be performed, to achieve desired results, nor should the separation of various system components in the embodiments described in this patent document be understood as requiring such separation in all embodiments.

[0398] Only a few implementations and examples have been described; other implementations, extensions and variations may be made based on what is described and illustrated in this patent document.

Claims

1. 1. A method for processing video data, comprising the steps of: determining at least one control point motion vector for an affine coded video block of a video; determining a motion vector for a sub-block having a position (x, y) of the affine coded video block based on the at least one control point motion vector; determining a first motion displacement Vx(x,y) in a first direction and a second motion displacement Vy(x,y) in a second direction for the position (x,y) based on the at least one control point motion vector, where Vx(x,y)=a×(x−xc)+b×(y−yc), Vy(x,y)=c×(x−xc)+d×(y−yc), where (xc,yc) is based on a center position or size of the sub-block, and a, b, c, and d are affine parameters; determining a first gradient component Gx(x,y) in the first direction and a second gradient component Gy(x,y) in the second direction for the location (x,y); determining a refined prediction sample P′(x,y) for the location (x,y) by modifying a derived prediction sample P(x,y) for the location (x,y) with the first gradient component Gx(x,y), the second gradient component Gy(x,y), the first motion displacement Vx(x,y), and the second motion displacement Vy(x,y), wherein the prediction sample P(x,y) is derived based on the motion vector for the sub-block; using the refined prediction samples P′(x,y) to convert between the affine-coded video block and the video bitstream; having the precision of the first motion displacement Vx(x,y) and the second motion displacement Vy(x,y) is different from the precision of the motion vector for the sub-block, the precision of the first motion displacement Vx(x,y) and the second motion displacement Vy(x,y) being 1 / 32 pixel precision, and the precision of the motion vector for the sub-block being 1 / 16 pixel precision; and the decoder-side motion vector refinement method and the bidirectional optical flow method are not applied to the affine-coded video block. method.

2. The method of claim 1 , wherein the color components of the affine-coded video block are luma components.

3. The method according to claim 1 or 2, wherein Vx(x,y) and Vy(x,y) are determined based on at least the position (x,y) and a center position of the sub-block.

4. The method of claim 1 , wherein Vx(x,y) and Vy(x,y) are determined based on at least the position (x,y) and the size of the sub-block.

5. 2. The method of claim 1, responsive to the affine coded video block being coded using a four parameter affine mode, c=-b and d=a.

6. 2. The method of claim 1, wherein a, b, c, and d can be derived from the control point motion vector, a width (W) of the affine coded video block, and a height (H) of the affine coded video block. 【Request 7】 【Number 1】 and mv ０ , mv １ , and mv ２ is the control point motion vector, a motion vector component having a superscript h indicates that the motion vector component is in the first direction; a further motion vector component having a superscript v indicates that the further motion vector component is in the second direction; The first direction is perpendicular to the second direction. The method according to claim 6.

8. The method of claim 7, wherein a, b, c and d are shifted.

9. The method according to claim 1 , wherein the method is used for the luma component and not for the chroma components.

10. 10. A method according to any one of claims 1 to 9, wherein the method is used for affine modes and not for non-affine modes.

11. 11. The method of claim 1, wherein the transforming comprises encoding the affine-coded video block into the bitstream.

12. 11. The method of claim 1, wherein the converting comprises decoding the affine-coded video blocks from the bitstream.

13. 1. An apparatus for processing video data comprising a processor and a non-transitory memory having instructions that, when executed by the processor, cause the processor to: determining at least one control point motion vector for an affine coded video block of the video; determining a motion vector for a sub-block having a position (x, y) of the affine-coded video block based on the at least one control point motion vector; determining a first motion displacement Vx(x,y) in a first direction and a second motion displacement Vy(x,y) in a second direction for said position (x,y) based on said at least one control point motion vector, where Vx(x,y)=a×(x−xc)+b×(y−yc), Vy(x,y)=c×(x−xc)+d×(y−yc), (xc,yc) is based on a center position or size of said sub-block, and a, b, c and d are affine parameters; determining a first gradient component Gx(x,y) in the first direction and a second gradient component Gy(x,y) in the second direction for the location (x,y); modifying the derived prediction sample P(x,y) for the position (x,y) with the first gradient component Gx(x,y), the second gradient component Gy(x,y), the first motion displacement Vx(x,y), and the second motion displacement Vy(x,y) to obtain a refined prediction sample P'(x,y) for the position (x,y), wherein the prediction sample P(x,y) is derived based on the motion vector for the sub-block; converting between the affine-coded video block and the video bitstream using the refined prediction samples P′(x,y); the precision of the first motion displacement Vx(x,y) and the second motion displacement Vy(x,y) is different from the precision of the motion vector for the sub-block, the precision of the first motion displacement Vx(x,y) and the second motion displacement Vy(x,y) being 1 / 32 pixel precision, and the precision of the motion vector for the sub-block being 1 / 16 pixel precision; and the decoder-side motion vector refinement method and the bidirectional optical flow method are not applied to the affine-coded video block. Device.

14. A non-transitory computer-readable storage medium having instructions stored thereon, the instructions causing a processor to: determining at least one control point motion vector for an affine coded video block of the video; determining a motion vector for a sub-block having a position (x, y) of the affine-coded video block based on the at least one control point motion vector; determining a first motion displacement Vx(x,y) in a first direction and a second motion displacement Vy(x,y) in a second direction for said position (x,y) based on said at least one control point motion vector, where Vx(x,y)=a×(x−xc)+b×(y−yc), Vy(x,y)=c×(x−xc)+d×(y−yc), (xc,yc) is based on a center position or size of said sub-block, and a, b, c and d are affine parameters; determining a first gradient component Gx(x,y) in the first direction and a second gradient component Gy(x,y) in the second direction for the location (x,y); modifying the derived prediction sample P(x,y) for the position (x,y) with the first gradient component Gx(x,y), the second gradient component Gy(x,y), the first motion displacement Vx(x,y), and the second motion displacement Vy(x,y) to obtain a refined prediction sample P'(x,y) for the position (x,y), wherein the prediction sample P(x,y) is derived based on the motion vector for the sub-block; converting between the affine-coded video block and the video bitstream using the refined prediction samples P′(x,y); the precision of the first motion displacement Vx(x,y) and the second motion displacement Vy(x,y) is different from the precision of the motion vector for the sub-block, the precision of the first motion displacement Vx(x,y) and the second motion displacement Vy(x,y) being 1 / 32 pixel precision, and the precision of the motion vector for the sub-block being 1 / 16 pixel precision; and the decoder-side motion vector refinement method and the bidirectional optical flow method are not applied to the affine-coded video block. A computer-readable storage medium.

15. 1. A method for storing a video bitstream, comprising: determining at least one control point motion vector for an affine coded video block of a video; determining a motion vector for a sub-block having a position (x, y) of the affine coded video block based on the at least one control point motion vector; determining a first motion displacement Vx(x,y) in a first direction and a second motion displacement Vy(x,y) in a second direction for the position (x,y) based on the at least one control point motion vector, where Vx(x,y)=a×(x−xc)+b×(y−yc), Vy(x,y)=c×(x−xc)+d×(y−yc), where (xc,yc) is based on a center position or size of the sub-block, and a, b, c, and d are affine parameters; determining a first gradient component Gx(x,y) in the first direction and a second gradient component Gy(x,y) in the second direction for the location (x,y); determining a refined prediction sample P′(x,y) for the location (x,y) by modifying a derived prediction sample P(x,y) for the location (x,y) with the first gradient component Gx(x,y), the second gradient component Gy(x,y), the first motion displacement Vx(x,y), and the second motion displacement Vy(x,y), wherein the prediction sample P(x,y) is derived based on the motion vector for the sub-block; generating the bitstream using the refined prediction samples P′(x,y); storing the bitstream on a non-transitory computer readable recording medium; having the precision of the first motion displacement Vx(x,y) and the second motion displacement Vy(x,y) is different from the precision of the motion vector for the sub-block, the precision of the first motion displacement Vx(x,y) and the second motion displacement Vy(x,y) being 1 / 32 pixel precision, and the precision of the motion vector for the sub-block being 1 / 16 pixel precision; and the decoder-side motion vector refinement method and the bidirectional optical flow method are not applied to the affine-coded video block. method.

Images