Decoder side motion vector derivation

Abstract

A method for processing video includes performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion of the current block includes determining whether use of one or both of a bi-directional optical flow (BIO) technique or a decoder-side motion vector refinement (DMVR) technique is enabled or disabled for the current block, and wherein the determination of use of the BIO technique or the DMVR technique is based on a cost criterion associated with the current block.

Description

[0001] This application is a divisional application of the patent application with application number 202080025886.9, titled “Decoder-side motion vector derivation” and filed on April 2, 2020. TECHNICAL FIELD

[0002] This document relates to video and image coding and decoding techniques. BACKGROUND

[0003] Digital video accounts for the largest bandwidth use on the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the demand for bandwidth to use for digital video is expected to continue growing. SUMMARY

[0004] In one example aspect, a method of processing video is disclosed. The method includes performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion of the current block includes determining whether use of one or both of a bi-directional optical flow (BIO) technique or a decoder-side motion vector refinement (DMVR) technique by the current block is enabled or disabled, and wherein the determination of use of the BIO technique or the DMVR technique is based on a cost criterion associated with the current block.

[0005] In another example aspect, a method of processing video is disclosed. The method includes performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion of the current block includes determining whether use of a decoder-side motion vector refinement (DMVR) technique by the current block is enabled or disabled, and wherein the DMVR technique includes refining motion information of the current block based on a cost criterion other than a mean removed sum of absolute difference (MRSAD) cost criterion.

[0006] In another example aspect, a method of processing video is disclosed. The method includes performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion of the current block includes determining whether use of one or both of a bi-directional optical flow (BIO) technique or a decoder-side motion vector refinement (DMVR) technique by the current block is enabled or disabled, and wherein the determination of use of the BIO technique or the DMVR technique is based on a calculation that a mean difference of a pair of reference blocks associated with the current block exceeds a threshold.

[0007] In another example aspect, a method of processing video is disclosed. The method includes modifying a first reference block to generate a first modified reference block and modifying a second reference block to generate a second modified reference block, wherein the first reference block and the second reference block are both associated with a current block of the video data; determining a difference between the first modified reference block and the second modified reference block, the difference comprising one or more of: a sum of absolute transformed difference (SATD), a mean removed sum of absolute transformed difference (MRSATD), a sum of squares error (SSE), a mean removed sum of squares error (MRSSE), a mean difference, or a gradient value; and performing a conversion between the current block of the video data and a corresponding coded representation of the video data, wherein the conversion includes using the difference between the first modified reference block and the second modified reference block generated by modifying the first reference block and the second reference block, respectively.

[0008] In another example aspect, a method of processing video is disclosed. The method includes determining a temporal gradient or a modified temporal gradient using reference pictures associated with a current block of the video data, the temporal gradient or the modified temporal gradient being indicative of a difference between the reference pictures; and performing a conversion between the current block of the video data and a corresponding coded representation of the video data, wherein the conversion includes using a bi-directional optical flow (BIO) technique based in part on the temporal gradient or the modified temporal gradient.

[0009] In another example aspect, a method of processing video is disclosed. The method includes determining a first temporal gradient using reference pictures associated with a first video block or a sub-block thereof; determining a second temporal gradient using reference pictures associated with a second video block or a sub-block thereof; performing a modification to the first temporal gradient and a modification to the second temporal gradient to generate a modified first temporal gradient and a modified second temporal gradient, wherein the modification to the first temporal gradient associated with the first video block is different from the modification to the second temporal gradient associated with the second video block; and performing a conversion of the first video block and the second video block to their corresponding coded representations.

[0010] In another example aspect, a method of processing video is disclosed. The method includes modifying one or both of a first inter- frame reference block and a second inter- frame reference block associated with a current block; determining a spatial gradient associated with the current block based on using one or both of the modified first inter- frame reference block and / or the modified second inter- frame reference block in accordance with applying a bi-directional optical flow (BIO) technique; and performing a conversion between the current block and a corresponding coded representation, wherein the conversion includes using the spatial gradient associated with the current block.

[0011] In another example aspect, a method of processing video is disclosed. The method includes performing, by a processor, a determination that a flag signaled at a block level indicates that one or both of a decoder-side motion vector refinement (DMVR) technique or a bi-directional optical flow (BIO) technique is to be enabled for a current block; and performing a conversion between the current block and a corresponding coded representation, wherein the coded representation includes a flag indicating whether one or both of the DMVR technique and / or the BIO technique is enabled.

[0012] In another example aspect, a method of processing video is disclosed. The method includes performing, by a processor, a determination that a decoder-side motion vector refinement (DMVR) technique is to be enabled for a current block, wherein the determination is based exclusively on a height of the current block; and performing a conversion between the current block and a corresponding coded representation.

[0013] In another example aspect, a method of processing video is disclosed. The method includes performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion includes using, on the current block, a rule associated with one or both of a decoder-side motion vector refinement (DMVR) technique or a bi-directional optical flow (BIO) technique, wherein the rule associated with the DMVR technique is consistent with an application of the BIO technique; and wherein a determination of whether one or both of the BIO technique or the DMVR technique is enabled or disabled on the current block is based on the application of the rule.

[0014] In another example aspect, the above method can be implemented by a video decoder apparatus comprising a processor.

[0015] In another example aspect, the above method can be implemented by a video codec apparatus comprising a processor.

[0016] In yet another example aspect, the methods can be embodied in the form of processor- executable instructions stored on a computer-readable program medium.

[0017] These and other aspects are further described in the present document. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1An example of bilateral matching is shown.

[0019] Figure 2 An example of template matching is shown.

[0020] Figure 3 An example of single-sided motion estimation (ME) in frame-rate up conversion (FRUC) is shown.

[0021] Figure 4 An example of optical flow trajectories is shown.

[0022] Figure 5A and Figure 5B An example of bi-directional optical flow (BIO) without block expansion is shown.

[0023] Figure 6 An example of bilateral matching with 6-point search is shown.

[0024] Figure 7 An example of adaptive integer search mode and half-sample search mode is shown.

[0025] Figure 8 is a block diagram of an example of a video processing apparatus.

[0026] Figure 9 is a block diagram of an example of a video encoder.

[0027] Figure 10 is a flowchart of an example of a video processing method.

[0028] Figure 11 is a flowchart of an example of a video processing method.

[0029] Figure 12 is a flowchart of an example of a video processing method.

[0030] Figure 13 is a flowchart of an example of a video processing method.

[0031] Figure 14 is a flowchart of an example of a video processing method.

[0032] Figure 15 is a flowchart of an example of a video processing method.

[0033] Figure 16 is a flowchart of an example of a video processing method.

[0034] Figure 17 is a block diagram of an example video processing system in which the disclosed technology can be implemented.

[0035] Figure 18 is an example of a video processing method.

[0036] Figure 19 is an example of a video processing method.

[0037] Figure 20 is an example of a video processing method.

[0038] Figure 21 is an example of a video processing method.

[0039] Figure 22 is an example of a video processing method.

[0040] Figure 23 is an example of a video processing method.

[0041] Figure 24 is an example of a video processing method.

[0042] Figure 25 is an example of a video processing method.

[0043] Figure 26 is an example of a video processing method.

[0044] Figure 27 is an example of a video processing method. DETAILED DESCRIPTION

[0045] To improve the compression rate of videos, researchers are constantly looking for new techniques by which to encode videos. This document provides various techniques that can be used by a video bitstream decoder to improve the quality of the decompressed or decoded digital video. In addition, a video encoder can also implement these techniques during the encoding process in order to reconstruct decoded frames for further encoding.

[0046] Section headings are used in this document for readability and do not limit the scope of the techniques and embodiments described in each section to that section only. In addition, while certain terminology from various existing video codec standards is used, the disclosed techniques are not limited to just those video standards or their successor standards, and are applicable to other video codec standards as well. Furthermore, in some cases, techniques are disclosed using corresponding encoding steps, and it will be understood that corresponding decoding steps will be performed at a decoder in reverse order. In addition, encoding and decoding can also be used to perform transcoding, where a video is represented from one codec representation (e.g., one bitrate) to another codec representation (e.g., a different bitrate).

[0047] 1. OVERVIEW

[0048] This patent document relates to video coding techniques. In particular, it relates to motion compensation in video coding. It can be applied to existing video coding standards, such as HEVC, or standards that are being finalized (e.g., Versatile Video Coding (VVC)). It can also be applied to future video coding standards or video coders.

[0049] 2. BACKGROUND

[0050] Video coding standards have evolved mainly through the development of the well-known ITU-T and ISO / IEC standards. ITU-T produced H.261 and H.263, ISO / IEC produced MPEG-1 and MPEG-4 Visual, and the two organizations jointly produced the H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Coding (AVC) and H.265 / HEVC standards. Since H.262, the video coding standards are based on the hybrid video coding structure, where temporal prediction plus transform coding is utilized. To explore future video coding technologies beyond HEVC, the Joint Video Exploration Team (JVET) was founded by VCEG and MPEG jointly in 2015. Since then, many new methods have been adopted by JVET and put into the reference software named Joint Exploration Model (JEM). In April 2018, the Joint Video Team (JVT) between VCEG (Q6 / 16) and ISO / IEC JTC1 SC29 / WG11 (MPEG) was established to work on the Versatile Video Coding (VVC) standard, targeting 50% bitrate reduction compared to HEVC.

[0051] The latest version of the VVC draft, namely Versatile Video Coding (Draft 2), can be found at:

[0052] http: / / phenix.it-sudparis.eu / jvet / doc_end_user / documents / 11_Ljubljana / wg11 / JVET-K1001-v7.zip

[0053] The latest reference software of VVC is named VTM, which can be found at:

[0054] https: / / vcgit.hhi.fraunhofer.de / jvet / VVCSoftware_VTM / tags / VTM-2.1

[0055] Figure 9 is a block diagram of an example implementation of a video encoder. Figure 9 The encoder implementation is shown to have a built-in feedback path in which the video encoder also performs a video decoding function (reconstructing a compressed representation of video data for encoding of next video data).

[0056] 2.1 Pattern Matched Motion Vector Derivation

[0057] Pattern Matched Motion Vector Derivation (PMMVD) mode is a special Merge mode based on the Frame-Rate Up Conversion (FRUC) technique. With this mode, the motion information of a block is derived at the decoder side instead of being signaled.

[0058] When the Merge flag of a CU is true, a FRUC flag is signaled to the CU. When the FRUC flag is false, the Merge index is signaled and the regular Merge mode is used. When the FRUC flag is true, an additional FRUC mode flag is signaled to indicate which method (bilateral matching or template matching) will be used to derive the motion information of the block.

[0059] At the encoder side, the decision on whether to use the FRUC Merge mode for a CU is based on the RD cost selection made for the normal Merge candidate. That is, both the two matching modes (bilateral matching and template matching) for the CU are examined by using the RD cost selection. The matching mode that results in the minimum cost is further compared with other CU modes. If the FRUC matching mode is the most efficient mode, the FRUC flag is set to true for the CU and the related matching mode is used.

[0060] The motion derivation process in the FRUC Merge mode has two steps: first a CU level motion search is performed, then a sub-CU level motion refinement is conducted. At the CU level, the initial motion vector of the whole CU is derived based on bilateral matching or template matching. First, a list of MV candidates is generated, and the candidate that results in the minimum matching cost is selected as the starting point for further CU level refinement. Then, a local search based on bilateral matching or template matching is performed around the starting point, and the MV that produces the minimum matching cost is taken as the MV of the whole CU. Subsequently, with the derived CU motion vector as the starting point, the motion information is further refined at the sub-CU level.

[0061] For example, the following derivation process is performed for WxH CU motion information derivation. In the first stage, the MV of the whole WxH CU is derived. In the second stage, the CU is further partitioned into MxM sub-CUs. The value of M is calculated as shown in (1), and D is a predefined partition depth, which is set to 3 by default in JEM. Then the MV of each sub-CU is derived as:

[0062]

[0063] As shown in Figure 1 , bilateral matching is used to derive the motion information of a current CU by finding the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. Under the assumption of continuous motion trajectory, the motion vectors MV0 and MV1 pointing to the two reference blocks should be proportional to the temporal distances (i.e., TD0 and TD1) between the current picture and the two reference pictures. As a special case, when the current picture is located in between the two reference pictures in the temporal domain and the temporal distances from the current picture to the two reference pictures are the same, bilateral matching becomes mirror-based bi-directional MV.

[0064] As shown in Figure 2 , template matching is used to derive the motion information of a current CU by finding the closest match between a template (top and / or left neighboring blocks of the current CU) in the current picture and a block (of the same size as the template) in a reference picture. In addition to the FRUC Merge mode described above, template matching is also applied to the AMVP mode. In JEM, as done in HEVC, AMVP has two candidates. A new candidate can be derived by the template matching method. If the new candidate derived by template matching is different from the first existing AMVP candidate, it is inserted at the beginning of the AMVP candidate list, and then the list size is set to 2 (meaning the second existing AMVP candidate is removed). When applied to the AMVP mode, only CU-level search is applied.

[0065] CU-level MV candidate set

[0066] The CU-level MV candidate set can include:

[0067] • the original AMVP candidates, if the current CU is in AMVP mode,

[0068] • all Merge candidates,

[0069] • several MVs in the interpolated MV field introduced in section 2.1.1.3, and

[0070] • the top and left neighboring motion vectors,

[0071] When bilateral matching is used, each valid MV of a Merge candidate is used as input to generate a pair of MVs under the hypothesis of bilateral matching. For example, in reference list A, one valid MV of a Merge candidate is (MVa, refa). Then, in the other reference list B, the reference picture refb of its counterpart bilateral MV is found such that refa and refb are located at different sides of the current picture in the temporal domain. If such refb is not available in reference list B, refb is determined to be a different reference from refa and its temporal distance to the current picture is the smallest in list B. After refb is determined, MVb is derived by scaling MVa based on the temporal distance between the current picture and refa, refb.

[0072] Four MVs from the interpolated MV field are also added to the CU-level candidate list. More specifically, the interpolated MVs at the locations (0, 0), (W / 2, 0), (0, H / 2), and (W / 2, H / 2) of the current CU are added.

[0073] When FRUC is applied to AMVP mode, the original AMVP candidates are also added to the CU-level MV candidate set.

[0074] At the CU level, up to 15 MVs are added to the candidate list for AMVP CUs and up to 13 MVs are added to the candidate list for Merge CUs.

[0075] Sub-CU level MV candidate set

[0076] The sub-CU level MV candidate set can include:

[0077] • MVs determined from the CU level search,

[0078] • top, left, top-left, and top-right neighboring MVs,

[0079] • scaled versions of collocated MVs from reference pictures,

[0080] • up to 4 ATMVP candidates, and

[0081] • up to 4 STMVP candidates.

[0082] Scaled MVs from reference pictures are derived as follows. All reference pictures in both lists are traversed. The MV at the collocated location of the sub-CU in the reference picture is scaled to the reference of the starting CU level MV.

[0083] ATMVP and STMVP candidates are limited to the first four.

[0084] At the sub-CU level, up to 17 MVs are added to the candidate list.

[0085] Generation of Interpolated MV Field

[0086] Before coding a frame, an interpolated motion field is generated for the whole picture based on uni-directional ME. The motion field can then be used later as CU-level or sub-CU-level MV candidates.

[0087] First, the motion field of each reference picture in both reference lists is traversed at 4x4 block level. For each 4x4 block, if the motion associated with the block passes through the 4x4 block in the current picture (as shown in Figure 3 ) and the block is not assigned any interpolated motion, the motion of the reference block is scaled to the current picture according to the temporal distances TD0 and TD1 (in the same way as the MV scaling of TMVP in HEVC) and the scaled motion is assigned to the block in the current frame. If no scaled MV is assigned to the 4x4 block, the motion of the block is marked as unavailable in the interpolated motion field.

[0088] Interpolation and Matching Cost

[0089] When the motion vector points to a fractional sample position, motion compensated interpolation can be performed. To reduce complexity, bilinear interpolation is used for bilateral matching and template matching instead of the regular 8-tap HEVC interpolation.

[0090] The calculation of the matching cost is different at different steps. When a candidate is selected from the candidate set at CU level, the matching cost is the absolute sum difference (SAD) of bilateral matching or template matching. After the initial MV is determined, the matching cost C of bilateral matching for sub-CU level search is calculated as follows:

[0091]

[0092] where w is a weighting factor set to 4 empirically, MV and MV s indicate the current MV and the initial MV, respectively. SAD is still used as the matching cost of template matching for sub-CU level search.

[0093] In FRUC mode, the MV is derived by using only the luma samples. The derived motion will be used for both luma and chroma for MC inter prediction. After the MV is determined, the final MC is performed using an 8-tap interpolation filter for luma and a 4-tap interpolation filter for chroma.

[0094] MV Refinement

[0095] MV refinement is a pattern-based MV search with bilateral matching cost or template matching cost as the criterion. In JEM, two search patterns are supported—Unrestricted Center-Biased Diamond Search (UCBDS) and Adaptive Cross Search, at CU level and sub-CU level, respectively, for MV refinement. For both CU and sub-CU level MV refinement, MV is searched directly at quarter luma sample MV precision, and then followed by eighth luma sample MV refinement. The search range for MV refinement for both CU and sub-CU level is set to equal to 8 luma samples.

[0096] Selection of prediction direction in template matching FRUC Merge mode

[0097] In bilateral matching Merge mode, bi-prediction is always applied because the motion information of a CU is derived based on the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. There is no such restriction for template matching Merge mode. In template matching Merge mode, the encoder can choose for a CU from uni-prediction in List 0, uni-prediction in List 1, or bi-prediction. The choice is based on the template matching cost as follows:

[0098] If costBi <= factor * min(cost0, cost1)

[0099] bi-prediction is used;

[0100] Else if cost0 <= cost1

[0101] uni-prediction in List 0 is used;

[0102] Else,

[0103] uni-prediction in List 1 is used;

[0104] where cost0 is the SAD of List 0 template matching, cost1 is the SAD of List 1 template matching, and costBi is the SAD of bi-prediction template matching. The value of factor is equal to 1.25, which means the selection process is biased towards bi-prediction.

[0105] Inter prediction direction selection is only applied to CU level template matching process.

[0106] Hybrid intra and inter prediction

[0107] In JVET-L0100, multi-hypothesis prediction is proposed, where hybrid intra and inter prediction is one way of generating multi-hypotheses.

[0108] When multi-hypothesis prediction is applied to improve intra mode, multi-hypothesis prediction combines one intra prediction and one Merge index prediction. In Merge CU, a flag is signaled for Merge mode to select an intra mode from the intra candidate list when the flag is true. For luma component, the intra candidate list is derived from 4 intra prediction modes including DC mode, Planar mode, Horizontal mode and Vertical mode, and the size of the intra candidate list can be 3 or 4 depending on the block shape. When the CU width is more than twice the CU height, the Horizontal mode is not included in the intra mode list, and when the CU height is more than twice the CU width, the Vertical mode is removed from the intra mode list. A weighted average is used to combine one intra prediction mode selected by the intra mode index and one Merge index prediction selected by the Merge index. For chroma component, DM is always applied without extra signaling. The weights used for combining the predictions are described as follows. When DC mode or Planar mode is selected, or CB width or height is less than 4, equal weights are applied. For those CBs with CB width and height greater than or equal to 4, when Horizontal / Vertical mode is selected, one CB is first divided into four equal-area regions by vertical / horizontal. Each weight set, denoted as (w_intra i ,w_inter i ), where i is 1 to 4, and (w_intra1, w_inter1) = (6, 2), (w_intra2, w_inter2) = (5, 3), (w_intra3, w_inter3) = (3, 5) and (w_intra4, w_inter4) = (2, 6) will be applied to the corresponding region. (w_intra1, w_inter1) is used for the region closest to the reference sample, and (w_intra4, w_inter4) is used for the region farthest from the reference sample. The combined prediction can then be computed by adding the two weighted predictions and right shifting by 3 bits. In addition, the intra prediction mode of the intra hypothesis of the prediction value can be saved for the later neighboring CUs to reference.

[0109] Bi-directional optical flow

[0110] In BIO, motion compensation is first performed to generate the first prediction (in each prediction direction) of the current block. The first prediction is used to derive the spatial gradient, temporal gradient and optical flow of each sub-block / pixel within the block, which are used to generate the second prediction, i.e., the final prediction of the sub-block / pixel. The details are described as follows.

[0111] Bi-directional optical flow (BIO) is a sample-wise motion refinement on top of the block-wise motion compensation used for bi-directional prediction. The sample-level motion refinement does not use signaling.

[0112] Let I (k)It is the brightness value from the reference k (k=0,1) after block motion compensation, and They are I (k) The horizontal and vertical components of the gradient. Assuming optical flow is effective, the motion vector field (v...) x ,v y The equation gives the following:

[0113]

[0114] By combining this optical flow equation with Hermite interpolation, the motion trajectory of each sample point is obtained, ultimately yielding the result with respect to the function value I. (k) and derivative The only matching third-order polynomial. The value of this polynomial at t=0 is the BIO prediction:

[0115]

[0116] Here, τ0 and τ1 represent the distances to the reference frame, such as... Figure 4 As shown. The distances τ0 and τ1 are calculated based on the POC of Ref0 and Ref1: τ0 = POC(current) - POC(Ref0), τ1 = POC(Ref1) - POC(current). If the two predictions come from the same time direction (either both from the past or both from the future), the signals are different (i.e., τ0·τ1 < 0). In this case, BIO is applied only when the predictions do not come from the same time (i.e., τ0 ≠ τ1), both reference regions have non-zero motion (MVx0, MVy0, MVx1, MVy1 ≠ 0γ), and the block motion vector is proportional to the time distance (MVx0 / MVx1 = MVy0 / MVy1 = -τ0 / τ1).

[0117] By minimizing points A and B ( Figure 9 The difference Δ between the values ​​of the points where the upper motion trajectory intersects the reference frame plane and the upper motion trajectory plane is used to determine the motion vector field (v). x ,v y The model uses only the first linear term of the local Taylor expansion for Δ:

[0118]

[0119] All values ​​in Equation 5 depend on the sample point location (i′,j′), which has been ignored from the notation so far. Assuming the motion is consistent in the local surrounding region, the value of Δ can be minimized within a (2M+1)×(2M+1) square window Ω centered at the current predicted point (i,j), where M equals 2:

[0120]

[0121] For this optimization problem, JEM uses a simplified approach, first minimizing in the vertical direction and then in the horizontal direction. This results in:

[0122]

[0123]

[0124] where

[0125]

[0126] To avoid division by zero or very small values, regularization parameters r and m are introduced in equations 7 and 8.

[0127] r = 500 · 4 d-8 (10)

[0128] m = 700 · 4 d-8 (11) where d is the bit-depth of the video samples.

[0129] To keep the memory access of the BIO the same as regular bi-predictive motion compensation, all prediction and gradient values I (k) , are calculated only for positions inside the current block. In equation 9, a (2M+1) x (2M+1) square window Ω centered on the current prediction point on the boundary of the prediction block can access positions outside the block (as shown in Fig. 5(a)). In JEM, the values of I (k) , outside the block are set to equal the closest available value inside the block. This can be implemented as padding, for example, as shown in Fig. 5(b).

[0130] With the BIO, it is possible to refine the motion field for each sample. To reduce the computational complexity, a block-based BIO design is used in JEM. The motion refinement is calculated on a 4x4 block basis. In the block-based BIO, the s n values in equation 9 for all samples in a 4x4 block are aggregated, and then the aggregated value of s n is used to derive the BIO motion vector offset for the 4x4 block. More specifically, the following formula is used for the block-based BIO derivation:

[0131]

[0132] where b k denotes the set of samples belonging to the k-th 4x4 block of the prediction block. The s n values in equations 7 and 8 are replaced by ((s n,bk )>>4) to derive the associated motion vector offset.

[0133] In some cases, the MV refinement of a BIO can be unreliable due to noise or irregular motion. Therefore, in a BIO, the magnitude of the MV refinement is clipped to a threshold thBIO. The threshold is determined based on whether all the reference pictures of the current picture come from one direction. If all the reference pictures of the current picture come from one direction, the threshold is set to 12x2 14-d ; otherwise, it is set to 12x2 13-d .

[0134] The gradient of a BIO is computed simultaneously with the motion compensated interpolation using the same operation as the HEVC motion compensation process (2D separable FIR (Finite Impulse Response)). The input of this 2D separable FIR is the same reference frame sample as the motion compensation process and the fractional position (fracX, fracY) according to the fractional part of the block motion vector. In the case of a horizontal gradient , the signal is first vertically interpolated using BIOfilterS corresponding to the fractional position fracY with a de-scaling shift d-8 and then the gradient filter BIOfilterG corresponding to the fractional position fracX with a de-scaling shift 18-d is applied in the horizontal direction. In the case of a vertical gradient , the gradient filter is first applied vertically using BIOfilterG corresponding to the fractional position fracY with a de-scaling shift d-8 and then the signal is shifted in the horizontal direction using BIOfilterS corresponding to the fractional position fracX with a de-scaling shift of 18-d. The interpolation filter BIOfilterG and the signal shift BIOfilterF used for the gradient computation are short (6 taps) in order to maintain a reasonable complexity. Table 1 shows the filters used for the gradient computation for different fractional positions of the block motion vector in a BIO. Table 2 shows the interpolation filters used for the prediction signal generation in a BIO.

[0135] Table 1: Filters used for gradient computation in a BIO

[0136]

[0137]

[0138] Table 2: Interpolation filters used for prediction signal generation in a BIO

[0139] Fractional pixel position Bi- filter S of a prediction signal 0 {0,0,64,0,0,0} 1 / 16 {1,-3,64,4,-2,0} 1 / 8 {1,-6,62,9,-3,1} 3 / 16 {2,-8,60,14,-5,1} 1 / 4 {2,-9,57,19,-7,2} 5 / 16 {3,-10,53,24,-8,2} 3 / 8 {3,-11,50,29,-9,2} 7 / 16 {3,-11,44,35,-10,3} 1 / 2 {3,-10,35,44,-11,3}

[0140] In JEM, BIO is applied to all bi-predicted blocks when two predictions come from different reference pictures. BIO is disabled when LIC (Local Illumination Compensation) is enabled for the CU.

[0141] In JEM, OBMC is applied to the blocks after the normal MC process. To reduce the computational complexity, BIO is not applied during the OBMC process. This means that BIO is only applied to the MC process of the block when its own MV is used, while BIO is not applied to the MC process when the MV of the neighboring block is used during the OBMC process.

[0142] The two-stage early termination method is used to conditionally disable the BIO operation according to the similarity between two prediction signals. Early termination is first applied at the CU level, and then applied at the sub-CU level. Specifically, the proposed method first calculates the SAD between the L0 prediction signal and the L1 prediction signal at the CU level. Assuming that BIO is only applied to luma, the SAD calculation can only consider luma samples. If the CU-level SAD is not greater than a pre-defined threshold, the BIO process is completely disabled for the entire CU. The CU-level threshold is set to 2 (BDepth-9) . If the BIO process is not disabled at the CU level, and if the current CU includes multiple sub-CUs, the SAD of each sub-CU inside the CU will be calculated. Then, based on a pre-defined sub-CU-level SAD threshold, which is set to 3 * 2 (BDepth-10) .

[0143] 2.4 Specification of BDOF in VVC

[0144] The specification of BDOF (Bidirectional optical flow) in (JVET-N1001-v2) is as follows:

[0145] 8.5.7.4 Bidirectional optical flow prediction process

[0146] The inputs of this process are:

[0147] - two variables nCbW and nCbH specifying the width and height of the current coding block,

[0148] - two (nCbW+2) x (nCbH+2) luma prediction sample arrays predSamplesLo and predSamplesLi,

[0149] - prediction list utilization flags predFlagLo and predFlagLi,

[0150] - reference indices refldxL0 and refldxL1,

[0151] - the bi-directional optical flow utilization flag bdofUtilizationFlag[ xldx ][ yldx ], with xldx = 0..( nCbW » 2 ) - 1, yldx = 0..( nCbH » 2 ) - 1.

[0152] The output of the process is an ( nCbW ) x ( nCbH ) array of luma prediction sample values, pbSamples.

[0153] The variables bitDepth, shiftl, shift2, shift3, shift4, offset4 and mvRefineThres are derived as follows:

[0154] - The variable bitDepth is set equal to BitDepth Y .

[0155] - The variable shiftl is set equal to Max( 2, 14 - bitDepth ).

[0156] - The variable shift2 is set equal to Max( 8, bitDepth - 4 ).

[0157] - The variable shift3 is set equal to Max( 5, bitDepth - 7 ).

[0158] - The variable shift4 is set equal to Max( 3, 15 - bitDepth ), and the variable offset4 is set equal to 1 « ( shift4 - 1 ).

[0159] - The variable mvRefineThres is set equal to Max( 2, 1 « ( 13 - bitDepth ) ).

[0160] For xldx = 0..( nCbW » 2 ) - 1 and yldx = 0..( nCbH » 2 ) - 1, the following applies:

[0161] - The variable xSb is set equal to ( xldx « 2 ) + 1, and ySb is set equal to ( yldx « 2 ) + 1.

[0162] - If bdofUtilizationFlag[ xSbldx ][ yldx ] is equal to FALSE, then for x = xSb - 1..xSb + 2, y = ySb - 1..ySb + 2, the prediction sample values of the current sub-block are derived as follows:

[0163] pbSamples[x][y] = Clip3(0,(2) bitDepth )-1,(predSamplesL0[x+1][y+1]+offset2+predSamplesL1[x+1][y+1])>>shift2) (8-852)

[0164] Otherwise (bdofUtilizationFlag[xSbIdx][yIdx] equals TRUE), the predicted sample values ​​for the current sub-block are derived as follows:

[0165] -For x = xSb-1..xSb+4, y = ySb-1..ySb+4, the following ordered steps apply:

[0166] 1. Predict the position (h) of each corresponding sample point (x, y) within the sample point array. x ,v y The derivation is as follows:

[0167] h x =Clip3(1,nCbW,x) (8-853)

[0168] v y =Clip3(1,nCbH,y) (8-854)

[0169] 2. The variables gradientHL0[x][y], gradientVL0[x][y], gradientHL1[x][y], and gradientVL1[x][y] are derived as follows:

[0170] gradientHL0[x][y]=(predSamplesL0[h x +1][v y ]-predSampleL0[h x -1][v y ])>>shift1 (8-855)

[0171] gradientVL0[x][y]=(predSampleL0[h x ][v y +1]-predSampleL0[h x ][v y -1])>>shift1 (8-856)

[0172] gradientHL1[x][y]=(predSamplesL1[h x +1][v y] - predSampleL1 [h x -1][v y ]) >> shift1 (8-857)

[0173] gradientVL1 [x][y] = (predSampleL1 [h x ][v y +1] - predSampleL1 [h x ][v y -1]) >> shift1 (8-858)

[0174] 3. The variables temp[x][y], tempH[x][y] and tempV[x][y] are derived as follows: diff[x][y] = (predSamplesL0[h x ][v y ] >> shift2) - (predSamplesL1[h x ][v y ] >> shift2) (8-859)

[0175] tempH[x][y] = (gradientHL0[x][y] + gradientHL1[x][y]) >> shift3 (8-860)

[0176] tempV[x][y] = (gradientVL0[x][y] + gradientVL1[x][y]) >> shift3 (8-861)

[0177] The variables sGx2, sGy2, sGxGy, sGxdI and sGydI are derived as follows:

[0178] sGx2 = Σ i Σ j (tempH[xSb+i][ySb+j] * tempH[xSb+i][ySb+j]), for i,j = -1..4 (8-862)

[0179] sGy2 = Σ i Σ j (tempV[xSb+i][ySb+j] * tempV[xSb+i][ySb+j]), for i,j = -1..4 (8-863)

[0180] sGxGy = Σ i Σ j( -tempH[xSb+i][ySb+j] * diff[xSb+i][ySb+j] ), where i,j = -1..4 (8-865)

[0181] sGxdI = å i å j ( tempH[xSb+i][ySb+j] * diff[xSb+i][ySb+j] ), where i,j = -1..4 (8-865)

[0182] sGydI = å i å j ( -tempV[xSb+i][ySb+j] * diff[xSb+i][ySb+j] ), where i,j = -1..4 (8-866)

[0183] - The horizontal and vertical motion offsets of the current sub-block are derived as follows:

[0184] v x = sGx2 > 0? Clip3( -mvRefineThres, mvRefineThres, - ( sGxdI « 3 ) > > Floor( Log2( sGx2 ) ) ) : 0 (8-867)

[0185] v y = sGy2 > 0? Clip3( -mvRefineThres, mvRefineThres, ( ( sGydI « 3 ) - ( v x * sGxGy m ) « 12 + v x * sGxGy s ) > > 1 ) > > Floor( Log2( sGy2 ) ) ) : 0 (8-868)

[0186] - The predicted sample values of the current sub-block for x = xSb-1..xSb+2, y = ySb-1..ySb+2 are derived as follows:

[0187] bdofOffset = Round( ( v x * ( gradientHL1[ x + 1 ][ y + 1 ] - gradientHL0[ x + 1 ][ y + 1 ] ) ) > > 1 ) + Round( ( v y * ( gradientVL1[ x + 1 ][ y + 1 ] - gradientVL0[ x + 1 ][ y + 1 ] ) ) > > 1 ) (8-869)

[0188] [Ed.(JC): The Round() operation is defined for floating-point inputs. The Round() operation seems redundant here as the inputs are integer values. To be confirmed by the proposer]

[0189] pbSamples[x][y] = Clip3(0, (2 bitDepth - 1, (predSamplesL0[x + 1][y + 1] + offset4 + predSamplesL1[x + 1][y + 1] + bdofOffset) » shift4) (8-870)

[0190] The spatial gradient is computed as follows:

[0191] gradientHL0[x][y] = (predSamplesL0[h x + 1][v y ] - predSampleL0[h x - 1][v y ]) » shift1 (8-855)

[0192] On the other hand, the temporal gradient is computed as follows:

[0193] diff[x][y] = (predSamplesL0[h x ][v y ] » shift2) - (predSamplesL1[h x ][v y ] » shift2) (8-859)

[0194] Therefore, the computation of the aligned spatial and temporal gradients is not necessary.

[0195] 2.5 Decoder-side motion vector refinement

[0196] In bi-prediction operation, for the prediction of a block region, two prediction blocks formed using the Motion Vector (MV) of List 0 and the MV of List 1, respectively, are combined to form a single prediction signal. In JVET-K0217, the decoder-side motion vector refinement (DMVR) method, the two motion vectors of bi-prediction are further refined through a bilateral matching process.

[0197] In the proposed method, DMVR is applied only in Merge and Skip modes if the following condition is true:

[0198] (POC - POC0) * (POC - POC1) < 0,

[0199] POC = picture order count of the picture to be coded, POC0 and POC1 are the picture order counts of the references of the current picture.

[0200] The signaled Merge candidate pair is used as input to the DMVR process and is denoted as initial motion vector pair (MV0, MV1). The search points searched by the DMVR follow the motion vector difference mirroring condition. In other words, any point checked by the DMVR, denoted by the candidate motion vector pair (MV0', MV1'), follows the following two equations:

[0201] MV0' = MV0 + MV diff

[0202] MV1' = MV1 - MV diff

[0203] where MV diff denotes a point in the search space in one of the reference pictures.

[0204] After constructing the search space, the single-sided prediction is constructed using the regular 8-tap DCT IF interpolation filter. The bilateral matching cost function is computed by using the MRSAD (Mean Removal and Sum of Absolute Differences) Figure 6 ) between the two predictions, and the search point that yields the minimum cost is selected as the refined MV pair. For the MRSAD computation, 16-bit precision of the samples is used (which is the output of the interpolation filtering), and no clipping and rounding operations are applied before the MRSAD computation. The reason for not applying rounding and clipping is to reduce the internal buffer requirements.

[0205] Figure 6 An example of bilateral matching with 6-point search is shown.

[0206] In the proposed method, the integer precision search points are selected by an adaptive pattern method. First, the cost corresponding to the center point (pointed by the initial motion vector) is computed. The other 4 costs (symbol shape) are computed by two predictions, where the two predictions are located on opposite sides of each other by the center point. The last 6thpoint of this angle is selected by the gradient of the previously computed costs Figure 7 ).

[0207] Figure 7 An example of the adaptive integer search pattern and the half-sample search pattern is shown.

[0208] The output of the DMVR process is the refined motion vector pair corresponding to the minimum cost.

[0209] If after one iteration the minimum cost is achieved at the center point of the search space, i.e. the motion vector does not change, and the refinement process is terminated. Otherwise, the best cost is further considered as the center and the process continues, while the minimum cost does not correspond to the center point and the search range is not exceeded.

[0210] The half-sample precision search is applied only if the application of the half-pel search does not exceed the search range. In this case, only 4 MRSAD calculations are performed, corresponding to the plus-shaped points around the center point, which is selected as the best during the integer precision search. Finally, the refined motion vector pair corresponding to the minimum cost point is output.

[0211] Some simplifications and improvements are further proposed in JVET-L0163.

[0212] Reference sample padding

[0213] Reference sample padding is applied in order to extend the reference sample block pointed by the initial motion vector. Assuming that the size of the coded block is given by "w" and "h", a block of size w+7 and h+7 is retrieved from the reference picture buffer. The retrieved buffer is then extended by 2 samples in each direction by using the repeated samples of the nearest samples padding. After that, once the refined motion vector is obtained (which can deviate from the initial motion vector by 2 samples in each direction), the extended reference sample block is used to generate the final prediction.

[0214] Note that this modification completely eliminates the external memory access requirement of DMVR without any coding loss.

[0215] Bilinear interpolation instead of 8-tap DCTIF

[0216] According to this proposal, bilinear interpolation is applied during the DMVR search process, which means that the prediction used in the MRSAD calculation is generated using bilinear interpolation. Once the final refined motion vector is obtained, the regular 8-tap DCTIF interpolation filter is applied to generate the final prediction.

[0217] Disable DMVR for small blocks

[0218] DMVR is disabled for blocks 4x4, 4x8 and 8x4.

[0219] Early termination based on MV difference between Merge candidates

[0220] An additional condition is imposed on DMVR to limit the MV refinement process. With this condition, DMVR is conditionally disabled when the following condition is satisfied.

[0221] The MV difference between the selected Merge candidate and any previous Merge candidate in the same Merge list is smaller than a pre-defined threshold (i.e. 1 / 4-pel wide interval, 1 / 2-pel wide interval and 1-pel wide interval for CUs with less than 64 pixels, less than 256 pixels and at least 256 pixels, respectively).

[0222] Early termination based on SAD cost at center search coordinates

[0223] The sum of absolute differences (SAD) between two prediction signals (L0 and L1 prediction) using the initial motion vector of the current CU is calculated. If the SAD is not larger than a pre-defined threshold, i.e. 2 (BDepth-9) , then DMVR is skipped; otherwise, DMVR is still applied to refine the two motion vectors of the current block.

[0224] DMVR application condition

[0225] The DMVR application condition is (POC - POC1) x (POC - POC2) < 0, which is replaced by the new condition (POC - POC1) == (POC2 - POC) when it is implemented in BMS2.1. This means that DMVR is only applied when the reference pictures are in opposite temporal directions and equidistant to the current picture.

[0226] MRSAD computation using each second row

[0227] The MRSAD cost is only calculated for odd rows of the block, without considering even sample rows. Therefore, the number of operations for MRSAD computation is halved.

[0228] 2.6 Related methods

[0229] In the patent application identified by Application No. PCT / CN2018 / 098691, filed on August 4, 2018, entitled “Motion Refinement for Visual Media Coding,” which is incorporated by reference herein, a MV update method and a two-step inter prediction method are proposed. The derived MV between reference block 0 and reference block 1 in BIO is scaled and added to the original motion vectors of list 0 and list 1. Meanwhile, the updated MV is used to perform motion compensation, and a second inter prediction is generated as the final prediction. The temporal gradient is modified by removing the average difference between reference block 0 and reference block 1.

[0230] 2.7 DMVR Draft 4

[0231] The use of DMVR in JVET-M1001_v7 (VVC Working Draft 4, version 7) is defined as follows:

[0232] - dmvrFlag is set equal to 1 when all the following conditions are true:

[0233] - sps_dmvr_enabled_flag is equal to 1

[0234] - the current block is not coded with triangle prediction mode, AMVR affine mode, subblock mode (including Merge affine mode and ATMVP mode)

[0235] - merge_flag[ xCb ][ yCb ] is equal to 1

[0236] - both predFlagL0[ 0 ][ 0 ] and predFlagL1[ 0 ][ 0 ] are equal to 1

[0237] - mmvd_flag[ xCb ][ yCb ] is equal to 0

[0238] - DiffPicOrderCnt( currPic, RefPicList[ 0 ][ refldxL0 ] ) is equal to DiffPicOrderCnt( RefPicList[ 1 ][ refldxL1 ], currPic )

[0239] - cbHeight is greater than or equal to 8

[0240] - cbHeight * cbWidth is greater than or equal to 64

[0241] 3. Example of the problem solved by the embodiments

[0242] In BIO, the difference between two reference blocks or subblocks is calculated in the early termination stage, while the temporal gradient is also calculated. Since the temporal gradient is actually the difference (or right-shifted difference) between two reference pixels, it is not meaningful to calculate both the difference and the temporal gradient.

[0243] In DMVR, the MRSAD calculation is used to decide the refined motion vector of a block.

[0244] In BIO, the SAD calculation is used to decide whether BIO should be enabled / disabled for a block or a subblock using all the samples of a block / subblock, which increases the computational complexity.

[0245] The calculation method is different for the spatial gradient and the temporal gradient.

[0246] 4. Example of the embodiments

[0247] SATD is denoted as sum of absolute transform difference, MRSATD is denoted as mean removed sum of absolute transform difference, SSE is denoted as sum of squared error, and MRSSE is denoted as mean removed sum of squared error.

[0248] The following detailed description should be considered in connection with the accompanying drawings, of which:

[0249] In the following discussion, SatShift(x, n) is defined as

[0250]

[0251] Shift(x, n) is defined as Shift(x, n) = (x + offset0) » n.

[0252] In one example, offset0 and / or offset1 is set to (1 « n) » 1 or (1 « (n-1)).

[0253] In another example, offset0 = offset1 = ((1 « n) » 1) - 1 or ((1 « (n-1)) - 1.

[0254] In gradient calculation of BDOF, the difference between two neighboring (spatial neighboring or temporal neighboring) samples and / or non-neighboring samples can be calculated, and right shift can be performed during gradient calculation. Assume two neighboring samples are neig0 and neig1, and right shift value is shift1, and the gradient to be calculated is grad. Note that shift1 can be different for spatial gradient and temporal gradient.

[0255] 1. Propose to align the methods for calculating spatial gradient and temporal gradient.

[0256] a. In one example, the gradient is calculated from shifted sample difference.

[0257] i. Optionally, the gradient is calculated from modified sample (e.g., via shifting) difference.

[0258] b. In one example, in gradient calculation, subtraction can be performed before right shift. For example, grad = (neig0 - neig1) » shift1.

[0259] c. In one example, in gradient calculation, subtraction can be performed after right shift. For example, grad = (neig0 » shift1) - (neig1 » shift1).

[0260] d. In one example, during gradient calculation, subtraction can be performed before the right shift, and an offset can be added before the right shift. For example, grad = (neig0 – neig1 + offset) >> shift1. The offset can be equal to 1 << (shift1 – 1) or 1 < <shift1>>1.

[0261] e.In one example, in the gradient calculation, the subtraction can be performed after the right shift, and the offset can be added before the right shift. For example, grad = ((neig0 + offset) » shift1) - ((neig1 + offset) » shift1). The offset can be equal to 1 « (shift1 - 1) or 1 « (shift1 - 2). <shift1>>1.

[0262] f. In one example, the gradient can be calculated as SatShift(neig0 - neig1, shift1).

[0263] i. Alternatively, the gradient can be calculated as SatShift(neig0, shift1) - SatShift(neig1, shift1).

[0264] 2. It is proposed to use other criteria to decide to enable / disable BIO or / and DMVR in early termination stage, such as SATD or MRSATD or SSE or MRSSE or average difference or gradient value.

[0265] a. In one example, the block level and sub-block level enable / disable decision can choose different criteria, e.g. one uses SAD, while the other uses SATD.

[0266] b. In one example, for a block / sub-block, if the gradient value (horizontal and / or vertical) or average gradient value or range of gradient value meets the condition (e.g. larger than a threshold or out of a given range), BIO and / or DMVR can be disabled.

[0267] c. It is proposed that the criteria used to decide to enable / disable BIO / DMVR can be signaled from encoder to decoder in VPS / SPS / PPS / slice header / tile group header.

[0268] 3. It is proposed to use other criteria to decide the refined motion vector of a block in DMVR process, such as SATD or MRSATD or SSE or MRSSE instead of MRSAD.

[0269] a. In one example, the refined motion vector of a sub-block in DMVR process, such as SATD or MRSATD or SSE or MRSSE instead of MRSAD.

[0270] b. In one example, if SATD (or MRSATD) is applied, the whole block is divided into MxN sub-blocks, and SATD (or MRSATD) is calculated for each sub-block. The SATD (or MRSATD) of all or some sub-blocks are summed up to get the SATD (or MRSATD) value of the whole block.

[0271] 4. BIO or / and DMVR can be disabled when the average difference of two reference blocks of a block is larger than a threshold (T1).

[0272] a. BIO can be disabled when the average difference of two reference sub-blocks of a sub-block is larger than a threshold (T2).

[0273] b. The threshold T1 and / or T2 can be predefined.

[0274] c. The threshold T1 and / or T2 can depend on the block dimension.

[0275] 5. It is proposed that in the early termination stage of BIO, before the difference (e.g., SAD / SATD / SSE, etc.) between two reference blocks / sub-blocks is calculated, the reference block or / and sub-block can be first modified.

[0276] a. In one example, the mean of the reference block or / and sub-block can be calculated and then subtracted from the reference block or / and sub-block.

[0277] b. In one example, the method disclosed in the application No. PCT / CN2018 / 096384 entitled “Motion Prediction Based on Updated Motion Vectors” filed on July 20, 2018 (which is incorporated by reference herein) can be used to calculate the mean of the reference block or / and sub-block, i.e., the mean is calculated for some representative positions.

[0278] 6. It is proposed that in the early termination stage of BIO technique or / and DMVR, the difference (e.g., SAD / SATD / SSE / MRSAD / MRSATD / MRSSE, etc.) between two reference blocks or / and sub-blocks can be calculated only for some representative positions.

[0279] a. In one example, the difference is calculated only for even rows of the block or / and sub-block.

[0280] b. In one example, the difference is calculated only for four corner samples of one block / sub-block of the block or / and sub-block.

[0281] c. In one example, the method disclosed in the U.S. Provisional Application No. 62 / 693,412 entitled “Decoder Side Motion Vector Derivation in Video Coding” filed on July 2, 2018 (which is incorporated by reference herein) can be used to select the representative positions.

[0282] d. In one example, the difference (e.g., SAD / SATD / SSE / MRSAD / MRSATD / MRSSE, etc.) between two reference blocks can be calculated only for some representative sub-blocks.

[0283] e. In one example, the sum of the differences (e.g., SAD / SATD / SSE / MRSAD / MRSATD / MRSSE, etc.) computed for representative positions or sub-blocks is taken to get the difference for the whole block / sub-block.

[0284] 7. The temporal gradient (temporal gradient at position (x, y) is defined as G(x, y) = P0(x, y) - P1(x, y), where P0(x, y) and P1(x, y) represent the prediction at (x, y) from two different reference pictures) or modified temporal gradient is proposed to be used as the difference (instead of SAD) for the early termination stage of BIO, and the threshold for early termination can be adjusted accordingly.

[0285] a. In one example, the absolute sum of the temporal gradient is computed and used as the difference for two reference blocks or / and sub-blocks.

[0286] b. In one example, the absolute sum of the temporal gradient is computed only at some representative positions of the block or / and sub-block.

[0287] c. In one example, the method disclosed in U.S. Provisional Application No. 62 / 693,412, titled “Decoder Side Motion Vector Derivation in Video Coding,” filed on July 2, 2018 (which is incorporated by reference herein) can be used to select the representative positions.

[0288] 8. It is proposed that the temporal gradient modification process can be performed adaptively for different blocks / sub-blocks.

[0289] a. In one example, the temporal gradient is modified only when the absolute average difference (or SAD / SATD / SSE, etc.) between two parameter blocks is greater than a threshold T, e.g., T = 4.

[0290] b. In one example, the temporal gradient is modified only when the absolute average difference (or SAD / SATD / SSE, etc.) between two parameter blocks is less than a threshold T, e.g., T = 20.

[0291] c. In one example, the temporal gradient is modified only when the absolute average difference (or SAD / SATD / SSE, etc.) between two parameter blocks is in the range [T1, T2], e.g., T1 = 4, T2 = 20.

[0292] d. In one example, if the absolute average difference (or SAD / SATD / SSE, etc.) between two parameter blocks is greater than a threshold T (e.g., T = 40), then BIO is disabled.

[0293] e. In one example, these thresholds can be implicitly predefined.

[0294] f. In one example, these thresholds can be signaled at SPS / PPS / picture / slice / tile level.

[0295] g. In one example, these thresholds can be different for different CUs, LCUs, slices, tiles or pictures.

[0296] i. In one example, these thresholds can be designed based on decoded / encoded pixel values.

[0297] ii. In one example, these thresholds can be designed differently for different reference pictures.

[0298] h. In one example, the temporal gradient is modified only when the (absolute) mean of the two reference blocks (or either of the two) is larger than a threshold T, e.g., T = 40.

[0299] i. In one example, the temporal gradient is modified only when the (absolute) mean of the two reference blocks (or either of the two) is smaller than a threshold T, e.g., T = 100.

[0300] j. In one example, the temporal gradient is modified only when the (absolute) mean of the two reference blocks (or either of the two) is in the range [T1, T2], e.g., T1 = 40, T2 = 100.

[0301] k. In one example, the temporal gradient is modified only when the (absolute) mean of the two reference blocks (or either of the two) is larger / smaller than the absolute mean difference (or SAD / SATD, etc.) multiplied by T, in one example, T = 4.5.

[0302] l. In one example, the temporal gradient is modified only when the (absolute) mean of the two reference blocks (or either of the two) is in the range of the absolute mean difference (or SAD / SATD, etc.) multiplied by [T1, T2], in one example, T1 = 4.5, T2 = 7.

[0303] 9. It is proposed that in the hybrid intra and inter prediction mode, when the spatial gradient is calculated in BIO, either the two inter reference blocks can be modified, or they can be modified before the whole BIO process is performed.

[0304] a. In one example, the intra and inter prediction blocks in each prediction direction are weighted averaged (using the same weighting method as in the hybrid intra and inter prediction) to generate two new prediction blocks, denoted as wAvgBlkL0 and wAvgBlkL1, which are used to derive the spatial gradient in BIO.

[0305] b. In one example, wAvgBlkL0 and wAvgBlkL1 are used to generate a prediction block for the current block, denoted as predBlk. Then, wAvgBlkL0, wAvgBlkL1 and predBlk are further used in the BIO process, and the refined prediction block generated in BIO is used as the final prediction block.

[0306] 10. It is proposed that a DMVR or / and BIO flag can be signaled at block level to indicate whether DMVR or / and BIO is enabled for the block.

[0307] a. In one example, such a flag can be signaled only for AMVP mode, and in Merge mode, such a flag can be inherited from spatial or / and temporal neighboring blocks.

[0308] b. In one example, whether BIO or / and DMVR is enabled can be jointly decided by a signaled flag and a dynamic (on-the-fly) decision (e.g., a SAD-based decision based on early termination). The signaled flag can indicate whether the dynamic decision is correct.

[0309] c. Such a flag can not be signaled for uni-prediction blocks.

[0310] d. Such a flag can not be signaled for bi-prediction blocks whose two reference pictures are both in front or in back in display order.

[0311] e. Such a flag can not be signaled for bi-prediction blocks if POC_diff(curPic, ref0) is not equal to POC_diff(ref1, curPic), where POC_diff() calculates the POC difference between two pictures, and ref0 and ref1 are the reference pictures of the current picture.

[0312] f. Such a flag can not be signaled for intra-coded blocks. In addition, optionally, such a flag can not be signaled for blocks coded in mixed intra and inter prediction mode.

[0313] Optionally, such a flag can not be signaled for blocks that refer to the current picture.

[0314] g. Whether to signal the flag can depend on the block dimension. For example, such a flag can not be signaled if the block size is smaller than a threshold. Optionally, such a flag can not be signaled if the block width and / or height is equal to or larger than a threshold.

[0315] h. Whether to signal the flag can depend on the motion vector precision. For example, such a flag can not be signaled if the motion vector is in integer precision.

[0316] i. If such a flag is not signaled, it can be implicitly derived as true or false.

[0317] j. A flag can be signaled at slice header / tile header / PPS / SPS / VPS to indicate whether the method is enabled or not.

[0318] k. Such a signaling method can depend on the temporal layer of the picture, e.g., the method can be disabled for pictures with high temporal layer.

[0319] l. Such a signaling method can depend on the QP of the picture, e.g., the method can be disabled for pictures with high QP.

[0320] 11. Instead of checking both block height and block size, it is proposed to decide whether to enable or disable DMVR only based on block height.

[0321] a. In one example, DMVR can be enabled when the block height is greater than T1 (e.g., T1 = 4).

[0322] b. In one example, DMVR can be enabled when the block height is equal to or greater than T1 (e.g., T1 = 8).

[0323] 12. The above methods applied to DMVR / BIO can only apply to other Decoder-Side Motion Vector Derivation (DMVD) methods, such as affine mode’s optical flow based prediction refinement.

[0324] a. In one example, the condition checks for usage determination of DMVR and BIO can be aligned, such as whether the block height satisfies the same threshold.

[0325] i. In one example, DMVR and BIO can be enabled when the block height is equal to or greater than T1 (e.g., T1 = 8).

[0326] ii. In one example, DMVR and BIO can be enabled when the block height is greater than T1 (e.g., T1 = 4).

[0327] 5. Embodiments

[0328] 5.1 Embodiment #1

[0329] The usage of DMVR in JVET-M1001_v7 (VVC Working Draft 4, version 7) is modified as follows:

[0330] - dmvrFlag is set equal to 1 when all of the following conditions are true:

[0331] - sps_dmvr_enabled_flag is equal to 1

[0332] - the current block is not coded with triangle prediction mode, AMVR affine mode, subblock mode (including Merge affine mode and ATMVP mode)

[0333] - merge_flag[ xCb ][ yCb ] is equal to 1

[0334] - both predFlagL0[ 0 ][ 0 ] and predFlagL1[ 0 ][ 0 ] are equal to 1

[0335] - mmvd_flag[ xCb ][ yCb ] is equal to 0

[0336] - DiffPicOrderCnt( currPic, RefPicList[ 0 ][ refIdxL0 ] ) is equal to DiffPicOrderCnt( RefPicList[ 1 ][ refIdxL1 ], currPic )

[0337] - cbHeight is greater than or equal to 8

[0338]

[0339] This means that "cbHeight * cbWidth is greater than or equal to 64" is removed.

[0340] 5.2 Embodiment #2

[0341] The newly added parts are highlighted in bold italic and the removed parts are highlighted in strikeout.

[0342] i. One example

[0343] 8.5.7.4 Bi-directional optical flow prediction process

[0344] The inputs of the process are:

[0345] - two variables nCbW and nCbH specifying the width and height of the current coding block,

[0346] - two (nCbW+2) x (nCbH+2) luma prediction samples arrays predSamplesL0 and predSamplesL1,

[0347] - the prediction list utilization flags predFlagL0 and predFlagL1,

[0348] - the reference indices refIdxL0 and refIdxL1,

[0349] - bidirectional optical flow utilization flag bdofUtilizationFlag[ xIdx ][ yIdx ], with xIdx = 0..(nCbW » 2) - 1, yIdx = 0..(nCbH » 2) - 1.

[0350] The output of this process is an (nCbW) x (nCbH) array of luma prediction sample values, pbSamples.

[0351] The variables bitDepth, shiftl, shift2, shift3, shift4, offset4 and mvRefineThres are derived as follows:

[0352] - The variable bitDepth is set equal to BitDepth Y .

[0353] - The variable shiftl is set equal to Max(2, 14 - bitDepth).

[0354] - The variable shift2 is set equal to Max(8, bitDepth - 4).

[0355] - The variable shift3 is set equal to Max(5, bitDepth - 7).

[0356] - The variable shift4 is set equal to Max(3, 15 - bitDepth), and the variable offset4 is set equal to 1 « (shift4 - 1).

[0357] - The variable mvRefineThres is set equal to Max(2, 1 « (13 - bitDepth)).

[0358] For xldx = 0..(nCbW » 2) - 1 and yldx = 0..(nCbH » 2) - 1, the following applies:

[0359] - The variable xSb is set equal to (xldx « 2) + 1, and ySb is set equal to (yldx « 2) + 1.

[0360] - If bdofUtilizationFlag[ xSbldx ][ yldx ] is equal to FALSE, then for x = xSb - 1..xSb + 2, y = ySb - 1..ySb + 2, the prediction sample values of the current sub-block are derived as follows:

[0361] pbSamples[ x ][ y ] = Clip3( 0, (2 bitDepth )-1,(predSamplesL0[x+1][y+1]+offset2+predSamplesL1[x+1][y+1])>>shift2) (8-852)

[0362] Otherwise (bdofUtilizationFlag[xSbIdx][yIdx] equals TRUE), the predicted sample values ​​for the current sub-block are derived as follows:

[0363] -For x = xSb-1..xSb+4, y = ySb-1..ySb+4, the following ordered steps apply:

[0364] 4. Predict the position (h) of each corresponding sample point (x, y) within the sample point array. x ,v y The derivation is as follows:

[0365] h x =Clip3(1,nCbW,x) (8-853)

[0366] v y =Clip3(1,nCbH,y) (8-854)

[0367] 5. The variables gradientHL0[x][y], gradientVL0[x][y], gradientHL1[x][y], and gradientVL1[x][y] are derived as follows:

[0368] gradientHL0[x][y]=(predSamplesL0[h x +1][v y ]-predSampleL0[h x -1][v y ])>>shift1 (8-855)

[0369] gradientVL0[x][y]=(predSampleL0[h x ][v y +1]-predSampleL0[h x ][v y -1])>>shift1 (8-856)

[0370] gradientHL1[x][y]=(predSamplesL1[h x +1][v y ]-predSampleL1[h x -1][v y ]) >> shift1 (8-857)

[0371] gradientVL1[x][y] = ( predSampleL1[ h x ][v y +1] - predSampleL1[ h x ][v y -1] ) >> shift1 (8-858)

[0372] 6. The variables temp[x][y], tempH[x][y] and tempV[x][y] are derived as follows:

[0373]

[0374] diff[x][y] = ( predSamplesL0[ h x ][v y ] - predSamplesL1[ h x ][v y ] ) >> shift2 (8-859)

[0375] tempH[x][y] = ( gradientHL0[x][y] + gradientHL1[x][y] ) >> shift3 (8-860)

[0376] tempV[x][y] = ( gradientVL0[x][y] + gradientVL1[x][y] ) >> shift3 (8-861)

[0377] The variables sGx2, sGy2, sGxGy, sGxdI and sGydI are derived as follows:

[0378] sGx2 = Σ i Σ j ( tempH[ xSb+i ][ ySb+j ] * tempH[ xSb+i ][ ySb+j ] ), where i,j = -1..4 (8-862)

[0379] sGy2 = Σ i Σ j ( tempV[ xSb+i ][ ySb+j ] * tempV[ xSb+i ][ ySb+j ] ), where i,j = -1..4 (8-863)

[0380] sGxGy = Σ i Σ j (tempH[xSb+i][ySb+j]*tempV[xSb+i][ySb+j]), where i,j-1..4(8-864)

[0381] sGxdI=Σ i Σ j (-tempH[xSb+i][ySb+j]*diff[xSb+i][ySb+j]), where i,j=-1..4(8-865)

[0382] sGydI=Σ i Σ j (-tempV[xSb+i][ySb+j]*diff[xSb+i][ySb+j]), where i,j=-1..4(8-866)

[0383] -The horizontal and vertical motion offsets of the current sub-block are derived as follows:

[0384] v x =sGx2>0? Clip3(-mvRefineThres,mvRefineThres,-(sGxdI<<3)>>Floor(Log2(sGx2))):0 (8-867)

[0385] v y =sGy2>0? Clip3(-mvRefineThres,mvRefineThres,((sGydI<<3)–((v x *sGxGy m )<<12+v x *sGxGy s )>>1)>>Floor(Log2(sGx2))):0 (8-868)

[0386] - For x = xSb-1..xSb+2 and y = ySb-1..ySb+2, the predicted sample values ​​for the current sub-block are derived as follows:

[0387] bdofOffset = Round((v x *(gradientHL1[x+1][y+1]-gradientHL0[x+1][y+1]))>>1)+Round((v y *(gradientVL1[x+1][y+1]-gradientVL0[x+1][y+1]))>>1) (8-869)

[0388] [Ed.(JC): The Round() operation is defined for floating-point inputs. The Round() operation seems redundant here as the inputs are integer values. To be confirmed by the proposer]

[0389] pbSamples[x][y] = Clip3(0, (2 bitDepth - 1, (predSamplesL0[x + 1][y + 1] + offset4 + predSamplesL1[x + 1][y + 1] + bdofOffset) » shift4) (8-870)

[0390] ii. One example

[0391] 8.5.7.4 Bi-directional optical flow prediction process

[0392] The input of this process are:

[0393] - two variables nCbW and nCbH specifying the width and height of the current coding block,

[0394] - two (nCbW+2) x (nCbH+2) luma prediction samples arrays predSamplesL0 and predSamplesL1,

[0395] - the prediction list utilization flags predFlagL0 and predFlagL1,

[0396] - the reference indices refIdxL0 and refIdxL1,

[0397] - the bi-directional optical flow utilization flags bdofUtilizationFlag[xIdx][yIdx] with xIdx = 0..(nCbW>>2)-1 and yIdx = 0..(nCbH>>2)-1.

[0398] The output of this process is an (nCbW) x (nCbH) array of luma prediction sample values pbSamples.

[0399] The variables bitDepth, shift1, shift2, shift3, shift4, offset4 and mvRefineThres are derived as follows:

[0400] - the variable bitDepth is set equal to BitDepth Y .

[0401] - the variable shift1 is set equal to Max(2, 14 - bitDepth).

[0402] - The variable shift2 is set equal to Max(8, bitDepth - 4).

[0403] - The variable shift3 is set equal to Max(5, bitDepth - 7).

[0404] - The variable shift4 is set equal to Max(3, 15 - bitDepth), and the variable offset4 is set equal to 1 « (shift4 - 1).

[0405] - The variable mvRefineThres is set equal to Max(2, 1 « (13 - bitDepth)). For xldx = 0..(nCbW » 2) - 1 and yldx = 0..(nCbH » 2) - 1, the following applies:

[0406] - The variable xSb is set equal to (xldx « 2) + 1, and ySb is set equal to (yldx « 2) + 1.

[0407] - If bdofUtilizationFlag[ xSbldx ][ yldx ] is equal to FALSE, then for x = xSb - 1..xSb + 2, y = ySb - 1..ySb + 2, the prediction sample values of the current subblock are derived as follows:

[0408] pbSamples[ x ][ y ] = Clip3( 0, (2 bitDepth - 1, ( predSamplesLo[ x + 1 ][ y + 1 ] + offset2 + predSamplesLl[ x + 1 ][ y + 1 ] ) » shift2 ) (8-852)

[0409] - Else ( bdofUtilizationFlag[ xSbldx ][ yldx ] is equal to TRUE ), the prediction sample values of the current subblock are derived as follows:

[0410] - For x = xSb - 1..xSb + 4, y = ySb - 1..ySb + 4, the following ordered steps apply:

[0411] 7. The position (h x , v y ) of each corresponding sample position (x, y) inside the prediction sample array is derived as follows:

[0412] h x = Clip3( 1, nCbW, x ) (8-853)

[0413] v y = Clip3( 1, nCbH, y ) (8-854)

[0414] 8. The variables gradientHL0[ x ][ y ], gradientVL0[ x ][ y ], gradientHL1[ x ][ y ], and gradientVL1[ x ][ y ] are derived as follows:

[0415]

[0416]

[0417] gradientHL0[ x ][ y ] = ( predSamplesL0[ h x + 1 ][ v y ] » shiftl ) - ( predSampleL0[ h x - 1 ][ v y ] » shiftl ) (8-855)

[0418] gradientVL0[ x ][ y ] = ( predSampleL0[ h x ][ v y + 1 ] » shiftl ) - ( predSampleL0[ h x ][ v y - 1 ] » shiftl ) (8-856)

[0419] gradientHL1[ x ][ y ] = ( predSamplesL1[ h x + 1 ][ v y ] » shiftl ) - ( predSampleL1[ h x - 1 ][ v y ] » shiftl ) (8-857)

[0420] gradientVL1[ x ][ y ] = ( predSampleL1[ h x ][ v y + 1 ] » shiftl ) - ( predSampleL1[ h x ][ v y - 1 ] » shiftl ) (8-858)

[0421] 9. The variables temp[ x ][ y ], tempH[ x ][ y ], and tempV[ x ][ y ] are derived as follows: diff[ x ][ y ] = ( predSamplesL0[ h x ][ v y ] >> shift2) - ( predSamplesLl[ h x ][v y ] >> shift2) (8-859)

[0422] tempH[x][y] = (gradientHL0[x][y] + gradientHLl[x][y]) » shift3 (8-860)

[0423] tempV[x][y] = (gradientVL0[x][y] + gradientVLl[x][y]) » shift3 (8-861)

[0424] The variables sGx2, sGy2, sGxGy, sGxdI and sGydI are derived as follows:

[0425] sGx2 = Σ i Σ j (tempH[xSb+i][ySb+j] * tempH[xSb+i][ySb+j]), for i,j = -1..4 (8-862)

[0426] sGy2 = Σ i Σ j (tempV[xSb+i][ySb+j] * tempV[xSb+i][ySb+j]), for i,j = -1..4 (8-863)

[0427] sGxGy = Σ i Σ j (tempH[xSb+i][ySb+j] * tempV[xSb+i][ySb+j]), for i,j = -1..4 (8-864)

[0428] sGxdI = Σ i Σ j (-tempH[xSb+i][ySb+j] * diff[xSb+i][ySb+j]), for i,j = -1..4 (8-865)

[0429] sGydI = Σ i Σ j (-tempV[xSb+i][ySb+j] * diff[xSb+i][ySb+j]), for i,j = -1..4 (8-866)

[0430] The horizontal and vertical motion offsets for the current sub-block are derived as follows:

[0431] v x = sGx2 > 0? Clip3( -mvRefineThres, mvRefineThres, - ( sGxdl « 3 ) >> Floor( Log2( sGx2 ) ) ) : 0 (8-867)

[0432] v y = sGy2 > 0? Clip3( -mvRefineThres, mvRefineThres, ( ( sGydl « 3 ) - ( v x * sGxGy m << 12 + v x * sGxGy s ) >> 1 ) >> Floor( Log2( sGy2 ) ) ) : 0 (8-868)

[0433] - For x = xSb-1..xSb+2, y = ySb-1..ySb+2, the predicted sample values of the current sub-block are derived as follows:

[0434] = predSamplesL0[ x + 1 ][ y + 1 ] + offset4 + predSamplesL1[ x + 1 ][ y + 1 ] + bdofOffset (8-870)

[0435] = predSamplesL0[ x + 1 ][ y + 1 ] + offset4 + predSamplesL1[ x + 1 ][ y + 1 ] + bdofOffset (8-870) x * ( gradientHL1[ x + 1 ][ y + 1 ] - gradientHL0[ x + 1 ][ y + 1 ] ) ) >> 1 ) + Round( ( v y * ( gradientVL1[ x + 1 ][ y + 1 ] - gradientVL0[ x + 1 ][ y + 1 ] ) ) >> 1 ) (8-869)

[0436] [Ed.(JC): The Round() operation is defined for floating-point inputs. The Round() operation seems redundant here as the inputs are integer values. To be confirmed by the proposer]

[0437] = predSamplesL0[ x + 1 ][ y + 1 ] + offset4 + predSamplesL1[ x + 1 ][ y + 1 ] + bdofOffset (8-870) bitDepth -1, ( predSamplesL0[ x + 1 ][ y + 1 ] + offset4 + predSamplesL1[ x + 1 ][ y + 1 ] + bdofOffset ) >> shift4 ) (8-870)

[0438] iii. One example

[0439] 8.5.7.4 Bi-directional optical flow prediction process

[0440] The input to this process is:

[0441] - Two variables nCbW and nCbH specifying the width and height, respectively, of the current coding block,

[0442] - two (nCbW + 2) x (nCbH + 2) luma prediction sample arrays predSamplesLo and predSamplesLi,

[0443] - a prediction list utilization flag predFlagLo and predFlagLi,

[0444] - a reference index refldxLo and refldxLi,

[0445] - a bi-directional optical flow utilization flag bdofUtilizationFlag[xldx][yldx] with xldx = 0..(nCbW » 2) - 1, yldx = 0..(nCbH » 2) - 1.

[0446] The output of the process is an (nCbW) x (nCbH) array of luma prediction sample values pbSamples.

[0447] The variables bitDepth, shiftl, shift2, shift3, shift4, offset4, offset5, offset6 and mvRefineThres are derived as follows:

[0448] - the variable bitDepth is set equal to BitDepth Y .

[0449] - the variable shiftl is set equal to Max(2, 14 - bitDepth).

[0450] - the variable shift2 is set equal to Max(8, bitDepth - 4).

[0451] - the variable shift3 is set equal to Max(5, bitDepth - 7).

[0452] - the variable shift4 is set equal to Max(3, 15 - bitDepth) and the variable offset4 is set equal to 1 « (shift4 - 1).

[0453] - the variable mvRefineThres is set equal to Max(2, 1 « (13 - bitDepth)).

[0454] - the variable offset5 is set equal to (1 « (shiftl - 1)).

[0455] - the variable offset6 is set equal to (1 « (shift2 - 1)).

[0456] For xldx = 0..(nCbW » 2) - 1 and yldx = 0..(nCbH » 2) - 1, the following applies:

[0457] - The variable xSb is set equal to (xldx « 2) + 1 and ySb is set equal to (yldx « 2) + 1.

[0458] - If bdofUtilizationFlag[ xSbldx ][ yldx ] is equal to FALSE, the prediction sample values of the current sub-block are derived as follows for x = xSb - 1..xSb + 2, y = ySb - 1..ySb + 2:

[0459] pbSamples[ x ][ y ] = Clip3( 0, (2 bitDepth - 1, ( predSamplesL0[ x + 1 ][ y + 1 ] + offset2 + predSamplesL1[ x + 1 ][ y + 1 ] ) » shift2 ) (8-852)

[0460] - Otherwise (bdofUtilizationFlag[ xSbldx ][ yldx ] is equal to TRUE), the prediction sample values of the current sub-block are derived as follows:

[0461] - For x = xSb - 1..xSb + 4, y = ySb - 1..ySb + 4, the following ordered steps apply:

[0462] 10. The position (h x , v y ) of each corresponding sample position (x, y) inside the prediction sample array is derived as follows:

[0463] h x = Clip3( 1, nCbW, x ) (8-853)

[0464] v y = Clip3( 1, nCbH, y ) (8-854)

[0465] 11. The variables gradientHL0[ x ][ y ], gradientVL0[ x ][ y ], gradientHL1[ x ][ y ] and gradientVL1[ x ][ y ] are derived as follows:

[0466] gradientHL0[ x ][ y ] = ( predSamplesL0[ h x + 1 ][ v y ] - predSampleL0[ h x ] ) * ( 1 « ( shift2 - 1 ) ) (8-855)-1][v y + 1 + offset5) » shift1 (8-855)

[0467] gradientVL0[x][y] = ( predSampleL0[ h x + 1 ][ v y - predSampleL0[ h x - 1 ][ v y + offset5) » shift1 (8-856)

[0468] gradientHL1[x][y] = ( predSampleL1[ h x + 1 ][ v y - predSampleL1[ h x - 1 ][ v y + offset5) » shift1 (8-857)

[0469] gradientVL1[x][y] = ( predSampleL1[ h x + 1 ][ v y - predSampleL1[ h x - 1 ][ v y + offset5) » shift1 (8-858)

[0470] 12. The variables temp[x][y], tempH[x][y] and tempV[x][y] are derived as follows:

[0471]

[0472] diff[x][y] = ( predSamplesL0[ h x ][ v y - predSamplesL1[ h x ][ v y + offset6) » shift2 (8-859)

[0473] tempH[x][y] = ( gradientHL0[x][y] + gradientHL1[x][y] ) » shift3 (8-860)

[0474] tempV[x][y] = ( gradientVL0[x][y] + gradientVL1[x][y] ) » shift3 (8-861)

[0475] - The variables sGx2, sGy2, sGxGy, sGxdI and sGydI are derived as follows:

[0476] sGx2 = ∑ i ∑ j (tempH[xSb+i][ySb+j] * tempH[xSb+i][ySb+j]), where i,j = -1..4 (8-862)

[0477] sGy2 = ∑ i ∑ j (tempV[xSb+i][ySb+j] * tempV[xSb+i][ySb+j]), where i,j = -1..4 (8-863)

[0478] sGxGy = ∑ i ∑ j (tempH[xSb+i][ySb+j] * tempV[xSb+i][ySb+j]), where i,j = -1..4 (8-864)

[0479] sGxdI = ∑ i ∑ j (-tempH[xSb+i][ySb+j] * diff[xSb+i][ySb+j]), where i,j = -1..4 (8-865)

[0480] sGydI = ∑ i ∑ j (-tempV[xSb+i][ySb+j] * diff[xSb+i][ySb+j]), where i,j = -1..4 (8-866)

[0481] - The horizontal and vertical motion offsets of the current sub-block are derived as follows:

[0482] v x = sGx2 > 0? Clip3(-mvRefineThres, mvRefineThres, -(sGxdI « 3) >> Floor(Log2(sGx2))): 0 (8-867)

[0483] v y = sGy2 > 0? Clip3(-mvRefineThres, mvRefineThres, ((sGydI « 3) - ((v x *sGxGy m ) « 12 + v x *sGxGy s >>1)>>Floor(Log2(sGx2))):0 (8-868)

[0484] - For x = xSb-1..xSb+2, y = ySb-1..ySb+2, the predicted sample values of the current sub-block are derived as follows:

[0485] bdofOffset = Round((v x *(gradientHL1[x+1][y+1]-gradientHL0[x+1][y+1]))>>1) + Round((v y *(gradientVL1[x+1][y+1]-gradientVL0[x+1][y+1]))>>1) (8-869)

[0486] [Ed.(JC): The Round() operation is defined for floating-point inputs. The Round() operation seems redundant here as the inputs are integer values. To be confirmed by the proposer]

[0487] pbSamples[x][y] = Clip3(0, (2 bitDepth )-1, (predSamplesL0[x+1][y+1] + offset4 + predSamplesL1[x+1][y+1] + bdofOffset) >> shift4) (8-870)

[0488] Figure 8 is a block diagram of a video processing device 800. The device 800 can be used to implement one or more methods described herein. The device 800 can be included in a smartphone, a tablet computer, a computer, an Internet of Things (IoT) receiver, and the like. The device 800 can comprise one or more processors 802, one or more memories 804, and video processing hardware 806. The processor(s) 802 can be configured to implement one or more methods described in the present document. The memory (memories) 804 can be used for storing data and code used for implementing the methods and techniques described herein. The video processing hardware 806 can be used to implement, in hardware circuitry, some of the techniques described in the present document. The video processing hardware 806 can be partially or entirely included within the processor(s) 802 in the form of dedicated hardware, a Graphical Processor Unit (GPU), or a special-purpose signal processing block.

[0489] Figure 10 is a flowchart of a method 1000 of processing video. The method 1000 includes performing a determination of a characteristic of a first video block (1005), the characteristic including a difference between reference blocks associated with the first video block, the difference including one or more of a sum of absolute transformed differences (SATD), a mean removed sum of absolute transformed differences (MRSATD), a sum of squared errors (SSE), a mean removed sum of squared errors (MRSSE), a mean difference, or a gradient value; determining (1010) an operational status of one or both of a bi-directional optical flow (BIO) technique or a decoder-side motion vector refinement (DMVR) technique based on the characteristic of the first video block, the operational status being one of enabled or disabled; and performing (1015) further processing of the first video block in accordance with the operational status of one or both of the BIO technique or the DMVR technique.

[0490] Figure 11 is a flowchart of a method 1100 of processing video. The method 1100 includes modifying (1105) a first reference block to generate a first modified reference block and modifying a second reference block to generate a second modified reference block, the first and second reference blocks being associated with a first video block; determining (1110) a difference between the first and second modified reference blocks, the difference including one or more of a sum of absolute transformed differences (SATD), a mean removed sum of absolute transformed differences (MRSATD), a sum of squared errors (SSE), a mean removed sum of squared errors (MRSSE), a mean difference, or a gradient value; and performing (1115) further processing of the first video block based on the difference between the first and second modified reference blocks.

[0491] Figure 12 is a flowchart of a method 1200 of processing video. The method 1200 includes determining (1205) a difference between a portion of a first reference block and a portion of a second reference block associated with a first video block, the difference including one or more of a sum of absolute transformed differences (SATD), a mean removed sum of absolute transformed differences (MRSATD), a sum of squared errors (SSE), a mean removed sum of squared errors (MRSSE), a mean difference, or a gradient value; and performing (1210) further processing of the first video block based on the difference.

[0492] Figure 13 is a flowchart of a method 1300 of processing video. The method 1300 includes determining (1305) a temporal gradient or a modified temporal gradient using reference pictures associated with a first video block, the temporal gradient or the modified temporal gradient indicating a difference between the reference pictures; and performing (1310) further processing of the first video block using a bi-directional optical flow (BIO) coding tool in accordance with the difference.

[0493] Figure 14 is a flowchart of a method 1400 of processing a video. The method 1400 includes determining (1405) a temporal gradient using a reference picture associated with a first video block, modifying (1410) the temporal gradient to generate a modified temporal gradient, and performing (1415) further processing of the first video block using the modified temporal gradient.

[0494] Figure 15 is a flowchart of a method 1500 of processing a video. The method 1500 includes modifying (1505) one or both of a first inter- frame reference block and a second inter- frame reference block associated with a first video block, determining (1510) a spatial gradient according to a bi-directional optical flow coding tool (BIO) using one or both of the modified first inter- frame reference block or the modified second inter- frame reference block, and performing (1515) further processing of the first video block based on the spatial gradient.

[0495] Figure 16 is a flowchart of a method 1600 of processing a video. The method 1600 includes performing (1605) a determination of a flag signaled at a block level indicating whether one or both of a decoder-side motion vector refinement (DMVR) or a bi-directional optical flow (BIO) is to be enabled for a first video block, and performing (1610) further processing of the first video block including applying one or both of the DMVR or the BIO in accordance with the flag.

[0496] With reference to the methods 1000, 1100, 1200, 1300, 1400, 1500, and 1600, some examples of determining to use bi-directional optical flow (BIO) or decoder-side motion vector refinement (DMVR) are described in Section 4 of this document. For example, as described in Section 4, a difference between reference blocks can be determined and the difference can be used to enable or disable BIO or DMVR.

[0497] With reference to the methods 1000, 1100, 1200, 1300, 1400, 1500, and 1600, a video block can be encoded in a video bitstream, where bit efficiency can be achieved by using bitstream generation rules related to motion information prediction.

[0498] The method can include where an operational status of the BIO technique or the DMVR technique is different between a block level and a sub-block level.

[0499] The method can include determining that one or more of a gradient value, an average of gradient values, or a range of gradient values is within a threshold range, where determining the operational status is based on a determination that the gradient value, the average of gradient values, or the range of gradient values is within the threshold range.

[0500] The method can include where determining the operational status is further based on information signaled from the encoder to the decoder in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, or a slice header.

[0501] The method can include determining a refined motion vector for the first video block based on SATD, MRSATD, SSE, or MRSS E, and where performing the further processing is based on the refined motion vector.

[0502] The method can include where determining the refined motion vector is based on SATD or MRSATD, the method further comprising: determining a SATD or MRSATD for each sub-block of the first video block; and generating a SATD or MRSATD for the first video block based on a sum of the SATD or MRSATD for each sub-block, where the further processing of the first video block is based on the generated SATD or MRSATD.

[0503] The method can include determining that an average difference of two reference blocks of the first video block is greater than a threshold, and where based on the average difference of the two reference blocks, one or both of BIO or DMVR is in a disabled operational status.

[0504] The method can include determining that an average difference of two reference sub-blocks of a sub-block of the first video block is greater than a threshold, and where based on the average difference of the two reference sub-blocks, one or both of BIO or DMVR is in a disabled operational status.

[0505] The method can include where the threshold is predefined.

[0506] The method can include determining a dimension of the first video block, and where the threshold is based on the dimension of the first video block.

[0507] The method can include where modifying the first reference block and the second reference block includes subtracting a mean value of the first reference block from the first reference block.

[0508] The method can include where the portion of the first reference block and the second reference block includes even rows.

[0509] The method can include where the portion of the first reference block and the second reference block includes corner samples.

[0510] The method can include where the portion of the first reference block and the second reference block includes representative sub-blocks.

[0511] The method can include where a difference between the representative sub-blocks is summed to generate a difference of the first reference block or the second reference block.

[0512] The method can include where the difference is correlated with an absolute sum of temporal gradients.

[0513] The method can include wherein modifying the temporal gradient is based on the absolute average difference between the reference blocks being greater than a threshold.

[0514] The method can include wherein the threshold is 4.

[0515] The method can include wherein modifying the temporal gradient is based on the absolute average difference between the reference blocks being less than a threshold.

[0516] The method can include wherein the threshold is 20.

[0517] The method can include wherein modifying the temporal gradient is based on the absolute average difference between the reference blocks being within a threshold range.

[0518] The method can include wherein based on the absolute average difference being greater than the threshold, the BIO is in a disabled operating state.

[0519] The method can include wherein the threshold or threshold range is indicated at a VPS, SPS, PPS, picture, slice, or tile level.

[0520] The method can include wherein the threshold or threshold range is different for different coding units (CUs), largest coding units (LCUs), slices, tiles, or pictures.

[0521] The method can include wherein the threshold or threshold range is based on decoded or encoded pixel values.

[0522] The method can include wherein the threshold or threshold range is based on a reference picture.

[0523] The method can include wherein determining the spatial gradient includes determining a weighted average of intra-predicted blocks and inter-predicted blocks in each prediction direction.

[0524] The method can include wherein the flag is provided in an advanced motion vector prediction (AMVP) mode, and in a Merge mode, the flag is inherited from one or both of a spatial neighboring block or a temporal neighboring block.

[0525] The method can include wherein the flag is not signaled for a uni-predicted block.

[0526] The method can include wherein the flag is not signaled for a bi-predicted block having a reference picture that is a preceding picture or a following picture in display order.

[0527] The method can include wherein the flag is not signaled for a bi-predicted block.

[0528] The method can include wherein the flag is not signaled for an intra-coded block.

[0529] The method can include where the flag is signaled for blocks coded in the mixed intra and inter prediction mode.

[0530] The method can include where the flag is signaled based on dimensions of the first video block.

[0531] The method can include where the flag is signaled in a VPS, SPS, or PPS.

[0532] The method can include where the flag is based on a temporal layer of a picture associated with the first video block.

[0533] The method can include where the flag is based on a quantization parameter (QP) of a picture associated with the first video block.

[0534] Figure 17 is a block diagram illustrating an example video processing system 1700 in which various techniques disclosed herein can be implemented. Various implementations can include some or all of the components of the system 1700. The system 1700 can include an input 1702 for receiving video content. The video content can be received in a raw or uncompressed format, e.g., 8 or 10 bit multi-component pixel values, or can be in a compressed or coded format. The input 1702 can represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, passive optical networks (PONs), etc., and wireless interfaces such as Wi-Fi or cellular interfaces.

[0535] The system 1700 can include a coding component 1704 that can implement various coding or transcoding methods described in this document. The coding component 1704 can reduce the average bitrate of video from the input 1702 to an output of the coding component 1704 to produce a coded representation of the video. Coding techniques are thus sometimes referred to as video compression or video transcoding techniques. The output of the coding component 1704 can be stored, or transmitted via a communication as represented by the connection to the component 1706. The stored or communicated bitstream (or coded) representation of the video received at the input 1702 can be used by the component 1708 to generate pixel values or a displayable video that is sent to a display interface 1710. The process of generating user-viewable video from a bitstream representation is sometimes referred to as video decompression. Moreover, while certain video processing operations are referred to as "coding" operations or tools, it should be understood that the coding tools or operations are used at an encoder and corresponding decoding tools or operations that reverse the results of the coding will be performed by a decoder.

[0536] Examples of peripheral bus interfaces or display interfaces can include universal serial bus (USB) or high definition multimedia interface (HDMI) or display port, etc. Examples of storage interfaces include SATA (serial advanced technology attachment), PCI, IDE interfaces, etc. The technology described in this document can be embodied in various electronic devices such as mobile phones, laptops, smart phones, or other devices capable of performing digital data processing and / or video display.

[0537] It should be appreciated that the disclosed technology can be embodied in a video encoder or decoder to improve compression efficiency when compressed coding units have a shape that is significantly different from traditional square blocks or semi-square rectangular blocks. For example, new coding tools that use long or high coding units such as 4x32 or 32x4 size units can benefit from the disclosed technology.

[0538] In some implementations, a method of video processing can be performed as follows:

[0539] During a conversion between a video block and a bitstream representation of the video block, a filtering method for computing a spatial gradient and a temporal gradient is used, and

[0540] The conversion is performed using the filtering.

[0541] Here, the conversion includes generating the bitstream representation from pixel values of the video block or generating the pixel values from the bitstream representation.

[0542] In some embodiments, the spatial gradient and the temporal gradient are computed using shifted sample differences.

[0543] In some embodiments, the spatial gradient and the temporal gradient are computed using modified samples.

[0544] Additional details of the method are provided in item 1 discussed in section 4.

[0545] Figure 18 A flowchart of an example of a method of video processing. Steps of the method are discussed in example 2 of section 4 of this document. The method includes (at step 1805) performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion of the current block includes determining whether one or both of a bi-directional optical flow (BIO) technique or a decoder-side motion vector refinement (DMVR) technique is enabled or disabled for use on the current block, and wherein the determination of whether to use the BIO technique or the DMVR technique is based on a cost criterion associated with the current block.

[0546] Figure 19 is a flowchart of an example of a video processing method. Steps of the method are discussed in Example 3 of Section 4 of this document. The method includes performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data, where the conversion of the current block includes determining whether use of a decoder-side motion vector refinement (DMVR) technique is enabled or disabled for the current block, and where the DMVR technique includes refining motion information of the current block based on a cost criterion other than a mean removal of sum of absolute difference (MRSAD) cost criterion.

[0547] Figure 20 is a flowchart of an example of a video processing method. Steps of the method are discussed in Example 4 of Section 4 of this document. The method includes performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data, where the conversion of the current block includes determining whether use of one or both of a bi-directional optical flow (BIO) technique or a decoder-side motion vector refinement (DMVR) technique is enabled or disabled for the current block, and where determining whether to use the BIO technique or the DMVR technique is based on a calculation of a mean difference of a pair of reference blocks associated with the current block exceeding a threshold.

[0548] Figure 21 is a flowchart of an example of a video processing method. Steps of the method are discussed in Example 6 of Section 4 of this document. The method includes modifying a first reference block to generate a first modified reference block and modifying a second reference block to generate a second modified reference block, where the first reference block and the second reference block are both associated with a current block of visual media data. The method also includes determining a difference between the first modified reference block and the second modified reference block, the difference including one or more of: a sum of absolute transformed differences (SATD), a mean removal of sum of absolute transformed differences (MRSATD), a sum of squared errors (SSE), a mean removal of sum of squared errors (MRSSE), a mean difference, or a gradient value. The method includes performing a conversion between the current block of visual media data and a corresponding coded representation of the visual media data, where the conversion includes using the difference between the first modified reference block and the second modified reference block generated by modifying the first reference block and the second reference block, respectively.

[0549] Figure 22 is a flowchart of an example of a video processing method. Steps of the method are discussed in example 7 of section 4 of this document. The method includes determining (at step 2205), using reference pictures associated with a current block of visual media data, a temporal gradient or a modified temporal gradient that is indicative of a difference between the reference pictures. The method includes performing (at step 2210) a conversion between the current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion includes using a bi-directional optical flow (BIO) technique based partly on the temporal gradient or the modified temporal gradient.

[0550] Figure 23 is a flowchart of an example of a video processing method. Steps of the method are discussed in example 8 of section 4 of this document. The method includes determining (at step 2305), using reference pictures associated with a first video block or a sub-block thereof, a first temporal gradient. The method includes determining (at step 2310), using reference pictures associated with a second video block or a sub-block thereof, a second temporal gradient. The method includes performing a modification to the first temporal gradient and a modification to the second temporal gradient to generate a modified first temporal gradient and a modified second temporal gradient, wherein the modification to the first temporal gradient associated with the first video block is different from the modification to the second temporal gradient associated with the second video block (at step 2315). The method includes performing a conversion of the first video block and the second video block to their corresponding coded representations (at step 2320).

[0551] Figure 24 is a flowchart of an example of a video processing method. Steps of the method are discussed in example 9 of section 4 of this document. The method includes modifying (at step 2405) one or both of a first inter- frame reference block and a second inter- frame reference block associated with a current block. The method includes determining (at step 2410), based on using the modified one or both of the first inter- frame reference block and the second inter- frame reference block, a spatial gradient associated with the current block in accordance with applying a bi-directional optical flow (BIO) technique. The method includes performing (at step 2415) a conversion between the current block and a corresponding coded representation, wherein the conversion includes using the spatial gradient associated with the current block.

[0552] Figure 25 is a flowchart of an example of a video processing method. Steps of the method are discussed in example 10 of section 4 of this document. The method includes (at step 2505) performing, by a processor, a determination that a flag signaled at a block level indicates at least partially that one or both of a decoder-side motion vector refinement (DMVR) technique or a bi-directional optical flow (BIO) technique is to be enabled for a current block. The method includes (at step 2510) performing a conversion between the current block and a corresponding coded representation, wherein the coded representation includes a flag that indicates whether one or both of the DMVR technique and / or the BIO technique is enabled.

[0553] Figure 26 is a flowchart of an example of a video processing method. Steps of the method are discussed in example 11 of section 4 of this document. The method includes (at step 2605) performing, by a processor, a determination that a decoder-side motion vector refinement (DMVR) technique is to be enabled for a current block, wherein the determination is based exclusively on a height of the current block. The method includes (at step 2610) performing a conversion between the current block and a corresponding coded representation.

[0554] Figure 27 is a flowchart of an example of a video processing method. Steps of the method are discussed in example 12 of section 4 of this document. The method includes (at step 2705) performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion includes using a rule associated with one or both of a decoder-side motion vector refinement (DMVR) technique or a bi-directional optical flow (BIO) technique on the current block, wherein the rule associated with the DMVR technique is consistent with an application of the BIO technique, and wherein a determination of whether one or both of the BIO technique or the DMVR technique is enabled or disabled for use on the current block is based on the application of the rule.

[0555] Some embodiments of the present technology are discussed in a clause-based format.

[0556] 1. A method of visual media processing, comprising:

[0557] performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data,

[0558] wherein the conversion of the current block includes a determination of whether one or both of a bi-directional optical flow (BIO) technique or a decoder-side motion vector refinement (DMVR) technique is enabled or disabled for use on the current block, and

[0559] wherein the determination of whether to use the BIO technique or the DMVR technique is based on a cost criterion associated with the current block.

[0560] 2. The method of clause 1, wherein the cost criterion is based on one or more of: sum of absolute transformed differences (SATD), mean removed sum of absolute transformed differences (MRSATD), sum of squared errors (SSE), mean removed sum of squared errors (MRSSE), mean difference, or gradient value.

[0561] 3. The method of any one or more of clauses 1-2, wherein the cost criterion is associated with a sub-block of the current block.

[0562] 4. The method of clause 3, wherein the sub-block level cost criterion is different than the block level cost criterion.

[0563] 5. The method of any one or more of clauses 1-4, further comprising:

[0564] disabling the application of the BIO technique and / or the DMVR technique upon determining that one or more of the gradient value, the mean of the gradient values, or the range of the gradient values is outside of a threshold range.

[0565] 6. The method of clause 1, wherein the cost criterion associated with the current block is signaled in a coded representation.

[0566] 7. The method of clause 6, wherein the cost criterion is signaled in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, or a slice header.

[0567] 8. A method of visual media processing, comprising:

[0568] performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data,

[0569] wherein the conversion of the current block includes determining whether use of a decoder-side motion vector refinement (DMVR) technique on the current block is enabled or disabled, and

[0570] wherein the DMVR technique includes refining motion information of the current block based on a cost criterion other than a mean removed sum of absolute differences (MRSAD) cost criterion.

[0571] 9. The method of clause 8, wherein the cost criterion associated with the current block is based on one or more of: sum of absolute transformed differences (SATD), mean removed sum of absolute transformed differences (MRSATD), sum of squared errors (SSE), or mean removed sum of squared errors (MRSSE).

[0572] 10. The method of any one or more of clauses 8-9, wherein the cost criterion is associated with a sub-block of the current block.

[0573] 11. The method of clause 10, further comprising:

[0574] dividing the current block into a plurality of sub-blocks of size M x N, wherein the cost criterion is based on motion information associated with each of the plurality of sub-blocks; and

[0575] generating a cost corresponding to each of the plurality of sub-blocks.

[0576] 12. The method of clause 11, further comprising:

[0577] summing at least a subset of the costs corresponding to each of the plurality of sub-blocks to generate a resulting cost associated with the current block.

[0578] 13. A method of visual media processing, comprising:

[0579] performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data,

[0580] wherein the conversion of the current block includes determining whether use of one or both of a bi-directional optical flow (BIO) technique or a decoder-side motion vector refinement (DMVR) technique is enabled or disabled for the current block, and

[0581] wherein determining use of the BIO technique or the DMVR technique is based on computing that a mean difference of a pair of reference blocks associated with the current block exceeds a threshold.

[0582] 14. The method of clause 13, wherein the threshold is a first threshold, further comprising:

[0583] upon determining that a mean difference of a pair of reference sub-blocks associated with a sub-block of the current block exceeds a second threshold, disabling application of the BIO technique and / or the DMVR technique.

[0584] 15. The method of clause 14, wherein the first threshold and / or the second threshold is a predefined number.

[0585] 16. The method of clause 14, wherein the first threshold and / or the second threshold is based on a dimension of the current block.

[0586] 17. A method of visual media processing, comprising:

[0587] modifying a first reference block to generate a first modified reference block and modifying a second reference block to generate a second modified reference block, wherein the first reference block and the second reference block are both associated with a current block of visual media data;

[0588] determining a difference between the first modified reference block and the second modified reference block, the difference comprising one or more of: a sum of absolute transformed differences (SATD), a mean removed sum of absolute transformed differences (MRSATD), a sum of squared errors (SSE), a mean removed sum of squared errors (MRSSE), a mean difference, or a gradient value; and

[0589] performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion comprises using a difference between a first modified reference block and a second modified reference block generated by respectively modifying a first reference block and a second reference block.

[0590] 18. The method of clause 17, wherein modifying the first reference block and the second reference block comprises:

[0591] calculating a first arithmetic mean based on sample values included in the first reference block and a second arithmetic mean based on sample values included in the second reference block;

[0592] subtracting the first arithmetic mean from samples included in the first reference block and the second arithmetic mean from samples included in the second reference block.

[0593] 19. The method of clause 18, wherein the first arithmetic mean and the second arithmetic mean are based on a subset of samples respectively included in the first reference block and the second reference block.

[0594] 20. The method of any one or more of clauses 17-19, wherein the first reference block and / or the second reference block is a sub-block associated with the current block.

[0595] 21. A method of visual media processing, comprising:

[0596] determining a temporal gradient or a modified temporal gradient using reference pictures associated with a current block of visual media data, the temporal gradient or the modified temporal gradient being indicative of a difference between the reference pictures; and

[0597] performing a conversion between the current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion comprises using a bi-directional optical flow (BIO) technique based in part on the temporal gradient or the modified temporal gradient.

[0598] 22. The method of clause 21, further comprising:

[0599] in response to determining that the temporal gradient or the modified temporal gradient is less than or equal to a threshold value, early-terminating the BIO technique.

[0600] 23. The method of clause 22, further comprising:

[0601] adjusting a threshold based on a number of samples used to compute the time-domain gradient or the modified gradient.

[0602] 24. The method of any one or more of clauses 21-23, wherein the difference relates to an absolute sum of time-domain gradients.

[0603] 25. The method of any one or more of clauses 21-24, wherein the difference between the reference pictures corresponds to a difference between a first portion of a first reference picture and a second portion of a second reference picture.

[0604] 26. The method of any one or more of clauses 21-25, wherein the reference pictures are associated with sub-blocks of the current block.

[0605] 27. A method of visual media processing, comprising:

[0606] determining a first time-domain gradient using reference pictures associated with a first video block or sub-blocks thereof;

[0607] determining a second time-domain gradient using reference pictures associated with a second video block or sub-blocks thereof;

[0608] performing a modification to the first time-domain gradient and a modification to the second time-domain gradient to generate a modified first time-domain gradient and a modified second time-domain gradient, wherein the modification to the first time-domain gradient associated with the first video block is different than the modification to the second time-domain gradient associated with the second video block; and

[0609] performing a conversion of the first video block and the second video block to their corresponding coded representations.

[0610] 28. The method of clause 27, wherein the modification to the first time-domain gradient and / or the modification to the second time-domain gradient is conditionally based on an absolute average difference between the reference pictures associated with the first video block and / or the second video block being greater than a threshold.

[0611] 29. The method of clause 28, wherein the threshold is 4.

[0612] 30. The method of clause 27, wherein the modification to the first time-domain gradient and / or the modification to the second time-domain gradient is conditionally based on an absolute average difference between the reference pictures associated with the first video block and / or the second video block being less than a threshold.

[0613] 31. The method of clause 30, wherein the threshold is 20.

[0614] 32. The method of clause 27, wherein the modification of the first temporal gradient and / or the modification of the second temporal gradient is conditionally based on an absolute mean difference between reference pictures associated with the first video block and / or the second video block being within a threshold range.

[0615] 33. The method of any of clauses 27-32, further comprising:

[0616] based on the absolute mean difference between reference pictures associated with the first video block and / or the second video block being greater than a threshold, disabling use of a bi-directional optical flow (BIO) technique on the first video block and / or the second video block.

[0617] 34. The method of any one or more of clauses 27-33, wherein the threshold or threshold range is indicated at a VPS, SPS, PPS, picture, slice, or tile level associated with the first video block and / or the second video block.

[0618] 35. The method of any one or more of clauses 27-33, wherein the threshold or threshold range is an implicitly predefined parameter.

[0619] 36. The method of any one or more of clauses 27-33, wherein the threshold or threshold range is different for different coding units (CUs), largest coding units (LCUs), slices, tiles, or pictures associated with the first video block and / or the second video block.

[0620] 37. The method of any one or more of clauses 27-33, wherein the threshold or threshold range is based on decoded or encoded pixel values associated with the first video block and / or the second video block.

[0621] 38. The method of any one or more of clauses 27-33, wherein the threshold or threshold range for the first set of reference pictures is different than the threshold or threshold range for the second set of reference pictures.

[0622] 39. The method of clause 27, wherein the modification of the first temporal gradient and / or the modification of the second temporal gradient is conditionally based on an absolute mean of reference pictures associated with the first video block and / or the second video block being greater than a threshold.

[0623] 40. The method of clause 39, wherein the threshold is 40.

[0624] 41. The method of clause 27, wherein the modification of the first temporal gradient and / or the modification of the second temporal gradient is conditionally based on an absolute mean of reference pictures associated with the first video block and / or the second video block being less than a threshold.

[0625] 42. The method of clause 41, wherein the threshold is 100.

[0626] 43. The method of clause 27, wherein the modification of the first temporal gradient and / or the modification of the second temporal gradient is conditionally based on an absolute mean of a reference picture associated with the first video block and / or the second video block being within a threshold range.

[0627] 44. The method of clause 27, wherein the modification of the first temporal gradient and / or the modification of the second temporal gradient is conditionally based on an absolute mean of a reference picture associated with the first video block and / or the second video block being greater than an absolute mean difference of the reference picture associated with the first video block and / or the second video block multiplied by a multiplication factor.

[0628] 45. The method of clause 27, wherein the modification of the first temporal gradient and / or the modification of the second temporal gradient is conditionally based on an absolute mean of a reference picture associated with the first video block and / or the second video block being less than an absolute mean difference of the reference picture associated with the first video block and / or the second video block multiplied by a multiplication factor.

[0629] 46. The method of any one or more of clauses 44-45, wherein the multiplication factor is 4.5.

[0630] 47. A method of video processing, comprising:

[0631] modifying one or both of a first inter-frame reference block and a second inter-frame reference block associated with a current block;

[0632] determining a spatial domain gradient associated with the current block based on using one or both of the modified first inter-frame reference block and / or the modified second inter-frame reference block in accordance with applying a bi-directional optical flow (BIO) technique; and

[0633] performing a conversion between the current block and a corresponding coded representation, wherein the conversion includes using the spatial domain gradient associated with the current block.

[0634] 48. The method of clause 47, wherein determining the spatial domain gradient comprises:

[0635] generating two prediction blocks based on a weighted average of an intra-prediction block and an inter-prediction block associated with the current block; and

[0636] using the two prediction blocks for determining the spatial domain gradient associated with the current block.

[0637] 49. The method of clause 48, further comprising:

[0638] generating a refined prediction block from the two prediction blocks using the BIO technique; and

[0639] using a refined prediction block for predicting a sub-block and / or a sample of the current block.

[0640] 50. A method of visual media processing, comprising:

[0641] determining, by a processor, that a flag signaled at a block level indicates at least in part that one or both of a decoder-side motion vector refinement (DMVR) technique or a bi-directional optical flow (BIO) technique is to be enabled for a current block; and

[0642] performing a conversion between the current block and a coded representation, wherein the coded representation includes the flag indicating whether one or both of the DMVR technique and / or the BIO technique is enabled.

[0643] 51. The method of clause 50, wherein the flag is signaled in the coded representation in response to detecting that an advanced motion vector prediction (AMVP) technique is enabled for the current block.

[0644] 52. The method of clause 50, wherein the flag is derived from one or both of spatially neighboring blocks or temporally neighboring blocks associated with the current block in response to detecting that a Merge mode is enabled for the current block.

[0645] 53. The method of clause 52, wherein the flag is inherited from a selected Merge candidate if the selected Merge candidate is a spatial Merge candidate.

[0646] 54. The method of clause 52, wherein the flag is inherited from a selected Merge candidate if the selected Merge candidate is a temporal Merge candidate.

[0647] 55. The method of clause 50, wherein a cost criterion associated with the current block is used to determine whether one or both of the DMVR technique and / or the BIO technique is enabled, and the flag signaled in the coded representation is used to indicate whether such determination is correct.

[0648] 56. The method of clause 55, wherein the cost criterion associated with the current block is a sum of absolute differences (SAD) between two reference blocks of the current block, and wherein the determination that one or both of the DMVR technique and / or the BIO technique is enabled applies when the cost criterion is greater than a threshold value.

[0649] 57. The method of clause 50, further comprising:

[0650] skipping signaling the flag in the coded representation upon determining that the current block is a uni-prediction block.

[0651] 58. The method of clause 50, further comprising:

[0652] skipping signaling the flag in the coded representation when the current block is determined to be a bi-predicted block associated with a pair of reference pictures that are both preceding or succeeding in display order.

[0653] 59. The method of clause 50, further comprising:

[0654] skipping signaling the flag in the coded representation when the current block is determined to be a bi-predicted block associated with a pair of reference pictures that have a picture order count (POC) distance different from a current picture associated with the current block.

[0655] 60. The method of clause 50, further comprising:

[0656] skipping signaling the flag in the coded representation when the current block is determined to be an intra coded block.

[0657] 61. The method of clause 50, further comprising:

[0658] skipping signaling the flag in the coded representation when the current block is determined to be a mixed intra and inter predicted block.

[0659] 62. The method of clause 50, further comprising:

[0660] skipping signaling the flag in the coded representation when the current block is determined to be associated with at least one block in the picture that is the same as a reference block.

[0661] 63. The method of clause 50, further comprising:

[0662] skipping signaling the flag in the coded representation when the current block is determined to have a dimension less than a threshold.

[0663] 64. The method of clause 50, further comprising:

[0664] skipping signaling the flag in the coded representation when the current block is determined to have a dimension greater than or equal to a threshold.

[0665] 65. The method of clause 50, further comprising:

[0666] skipping signaling the flag in the coded representation when the current block is determined to have motion information associated therewith that is integer precision.

[0667] 66. The method of clause 50, further comprising:

[0668] The flag is skipped from being signaled in the coded representation upon determining that a temporal layer associated with a picture containing the current block exceeds a threshold.

[0669] 67. The method of clause 50, further comprising:

[0670] The flag is skipped from being signaled in the coded representation upon determining that a quantization parameter associated with the current block exceeds a threshold.

[0671] 68. The method of any one or more of clauses 50-67, further comprising:

[0672] In response to determining that the flag is skipped from being signaled in the coded representation, a value of the flag is derived to be Boolean true or false.

[0673] 69. The method of any one or more of clauses 50-67, further comprising:

[0674] Upon determining that the flag is Boolean true, one or both of the DMVR technique or the BIO technique is enabled.

[0675] 70. The method of any one or more of clauses 50-67, further comprising:

[0676] Upon determining that the flag is Boolean false, one or both of the DMVR technique or the BIO technique is disabled.

[0677] 71. The method of any one or more of clauses 50-67, further comprising:

[0678] Upon determining that the flag is Boolean true, a determination of one or both of enabling or disabling the DMVR technique or the BIO technique is determined to be correct based on at least one cost criterion.

[0679] 72. The method of any one or more of clauses 50-67, further comprising:

[0680] Upon determining that the flag is Boolean false, a determination of one or both of enabling or disabling the DMVR technique or the BIO technique is determined to be incorrect based on at least one cost criterion.

[0681] 73. The method of any one or more of clauses 50-67, wherein the flag is signaled in a slice header, a tile header, a video parameter set (VPS), a sequence parameter set (SPS), or a picture parameter set (PPS).

[0682] 74. The method of clause 50, wherein a first flag is signaled to indicate whether the DMVR technique is disabled and a second flag is signaled to indicate whether the BIO technique is disabled.

[0683] 75. The method of any one or more of clauses 64-74, further comprising:

[0684] disabling the DMVR technique for the slice, tile, video, sequence, or picture upon determining that the flag for the DMVR technique is Boolean true.

[0685] 76. The method of any one or more of clauses 64-74, further comprising:

[0686] enabling the DMVR technique for the slice, tile, video, sequence, or picture upon determining that the flag for the DMVR technique is Boolean false.

[0687] 77. The method of any one or more of clauses 64-74, further comprising:

[0688] disabling the BIO technique for the slice, tile, video, sequence, or picture upon determining that the flag for the BIO technique is Boolean true.

[0689] 78. The method of any one or more of clauses 64-74, further comprising:

[0690] enabling the BIO technique for the slice, tile, video, sequence, or picture upon determining that the flag for the BIO technique is Boolean false.

[0691] 79. A method of visual media processing, comprising:

[0692] performing, by a processor, a determination that a decoder-side motion vector refinement (DMVR) technique is to be enabled for a current block, wherein the determination is based exclusively on a height of the current block; and

[0693] performing a conversion between the current block and a corresponding coded representation.

[0694] 80. The method of clause 79, further comprising:

[0695] in response to determining that the DMVR technique is enabled, verifying that the height of the current block is greater than or exceeds a threshold parameter.

[0696] 81. The method of clause 80, wherein the threshold parameter is equal to 4.

[0697] 82. The method of clause 80, wherein the threshold parameter is equal to 8.

[0698] 83. A method of visual media processing, comprising:

[0699] performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion includes using a rule associated with one or both of a decoder-side motion vector refinement (DMVR) technique or a bi-directional optical flow (BIO) technique on the current block, wherein the rule associated with the DMVR technique is consistent with an application of the BIO technique; and

[0700] wherein determining whether one or both of the BIO technique or the DMVR technique is enabled for use on the current block is based on applying the rule.

[0701] 84. The method of clause 83, wherein the rule to determine whether the DMVR technique is enabled is the same as the rule to determine whether the BIO technique is enabled.

[0702] 85. The method of clause 84, wherein the rule to determine whether the BIO technique and / or the DMVR technique is enabled specifies verifying that a height of the current block is greater than or equal to a threshold.

[0703] 86. The method of clause 84, wherein the rule to determine whether the BIO technique and / or the DMVR technique is enabled specifies verifying that a width and a height of the current block are both greater than or equal to a threshold.

[0704] 87. The method of any one or more of clauses 85 or 86, wherein the threshold is 4 or 8.

[0705] 88. The method of clause 84, wherein the rule to determine whether the BIO technique and / or the DMVR technique is enabled specifies verifying that a size of the current block is greater than or equal to a threshold.

[0706] 89. The method of clause 86, wherein the threshold is 64 or 128.

[0707] 90. The method of clause 84, wherein the rule to determine whether the BIO technique and / or the DMVR technique is enabled specifies verifying that the current block is not coded under a CU-level weighted bi-prediction (BCW) mode in which unequal weights are used for two reference blocks from two reference lists.

[0708] 91. The method of clause 84, wherein the rule to determine whether the BIO technique and / or the DMVR technique is enabled specifies verifying that the current block is a bi-predicted block associated with a pair of reference pictures having a same picture order count (POC) distance as a current picture associated with the current block.

[0709] 92. The method of clause 91, wherein the pair of reference pictures includes a preceding picture and a following picture in display order of the current picture associated with the current block.

[0710] 93. A video decoding apparatus comprising a processor, wherein the processor is configured to implement a method recited in one or more of clauses 1 to 92.

[0711] 94. A video coding apparatus comprising a processor, wherein the processor is configured to implement a method recited in one or more of clauses 1 to 92.

[0712] 95. A computer program product having computer code stored thereon, the code, when executed by a processor, causing the processor to implement a method recited in any one of clauses 1 to 92.

[0713] 96. A method, apparatus or system described in this document.

[0714] The disclosed and other aspects, examples, implementations, modules and functional operations described in this document can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this document and their structural equivalents, or in combinations of one or more of them. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. The propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus.

[0715] A computer program, which can also be referred to or referred to as a program, software, a software application, an app, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and are interconnected by a communication network.

[0716] The processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0717] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0718] While this patent document contains many details, these should not be construed as limiting the subject matter or the scope of any claims to the specific embodiments described, but rather as describing features that are particular to certain embodiments. Certain features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a sub-combination or variation of a sub-combination.

[0719] Similarly, while operations are described in a particular order, this should not be understood as requiring such order nor requiring all illustrated operations be performed, to achieve desirable results. Moreover, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0720] Only a few implementations and examples are described and other implementations, enhancements and variations can be made based on what is described and illustrated in this patent document.

Claims

1. A method of processing video data, comprising: determining, for a current block of a video, initial prediction samples; refining the initial prediction samples with a prediction sample offset based on an optical flow refinement technique to obtain final prediction samples; and performing a conversion between the current block and a bitstream of the video based on the final prediction samples, wherein the prediction sample offset is determined based on at least one spatial gradient of the initial prediction samples, wherein the spatial gradient is calculated based on at least a difference between two first prediction samples from a same reference picture list, and wherein values of the two first prediction samples are right shifted by a first value before the difference between the two first prediction samples is calculated; wherein the prediction sample offset is further determined based on at least one temporal gradient, wherein the temporal gradient is calculated based on at least a difference between two second prediction samples from different reference picture lists, and wherein a shift rule for the difference between the two second prediction samples is the same as a shift rule for the difference between the two first prediction samples, and the shift rule comprises a sequence of a right shift operation and a subtraction operation. for a sample position (x, y) in the current block, the two first prediction samples have positions (hx+1, vy) and (hx-1, vy) corresponding to the same reference picture list X or positions (hx, vy+1) and (hx, vy-1) corresponding to the same reference picture list X, and 2. The method of claim 1, wherein, wherein X = 0 or 1, hx = Clip3(l, nCbW, x) and vy = Clip3(l, nCbH, y), nCbW and nCbH are a width and a height of the current block, and wherein Clip3 is a clipping function defined as follows:

3. The method of claim 1, values of the two second prediction samples are right shifted by a second value before the difference between the two second prediction samples is calculated. wherein for a sample position (x, y) in the current block, the two second prediction samples have a position (hx, vy) corresponding to reference picture list 0 and reference picture list 1, and 4. The method of claim 3, wherein, wherein hx = Clip3(l, nCbW, x) and vy = Clip3(l, nCbH, y), nCbW and nCbH are a width and a height of the current block, and wherein Clip3 is a clipping function defined as follows: the first value is different from the second value.

5. The method of claim 3, wherein, an optical flow refinement process is enabled based on a condition related to a size of the current block.

6. The method of claim 1, wherein, enabling a decoder-side motion vector refinement technique for the current block is based on the same condition, wherein the decoder-side motion vector refinement technique is used to derive refined motion information of the current block based on a cost between at least one prediction sample obtained based on at least one reference sample according to reference picture list 0 and at least one prediction sample obtained based on at least one reference sample according to reference picture list 1.

7. The method of claim 6, wherein, the optical flow refinement technique and the decoder-side motion vector refinement technique are enabled based on at least a condition that a height of the current block is equal to or greater than Tl.

8. The method of claim 7, wherein, performing the conversion comprises decoding the current block from the bitstream.

9. The method of claim 8, wherein, T1＝8。 10. The method of claim 1, wherein, ​ 11. The method of claim 1, wherein, Performing the conversion includes encoding the current block into the bitstream.

12. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: for a current block of a video, determine initial prediction samples; based on an optical flow refinement technique, refine the initial prediction samples with a prediction sample offset to obtain final prediction samples; and perform a conversion between the current block and a bitstream of the video based on the final prediction samples, wherein the prediction sample offset is determined based on at least one spatial gradient of the initial prediction samples, wherein the spatial gradient is calculated based on at least a difference between two first prediction samples from a same reference picture list, and wherein, before calculating the difference between the two first prediction samples, values of the two first prediction samples are right shifted by a first value; wherein the prediction sample offset is further determined based on at least one temporal gradient, wherein the temporal gradient is calculated based on at least a difference between two second prediction samples from different reference picture lists, and wherein a shift rule for the difference between the two second prediction samples is the same as a shift rule for the difference between the two first prediction samples, and the shift rule comprises a sequence of a right shift operation and a subtraction operation.

13. The apparatus of claim 12, wherein, for a sample position (x, y) in the current block, the two first prediction samples have positions (hx+1, vy) and (hx-1, vy) corresponding to the same reference picture list X or positions (hx, vy+1) and (hx, vy-1) corresponding to the same reference picture list X, and wherein X = 0 or 1, hx = Clip3(1, nCbW, x) and vy = Clip3(1, nCbH, y), nCbW and nCbH are a width and a height of the current block, and wherein Clip3 is a clipping function defined as follows:

14. The apparatus of claim 12, wherein before calculating the difference between the two second prediction samples, values of the two second prediction samples are right shifted by a second value.

15. The apparatus of claim 14, wherein, for a sample position (x, y) in the current block, the two second prediction samples have a position (hx, vy) corresponding to reference picture list 0 and reference picture list 1, and wherein hx = Clip3(1, nCbW, x) and vy = Clip3(1, nCbH, y), nCbW and nCbH are a width and a height of the current block, and wherein Clip3 is a clipping function defined as follows:

16. The apparatus of claim 14, wherein, the first value is different from the second value.

17. A non-transitory computer-readable storage medium storing instructions, the instructions causing a processor to: for a current block of a video, determine initial prediction samples; based on an optical flow refinement technique, refine the initial prediction samples with a prediction sample offset to obtain final prediction samples; and perform a conversion between the current block and a bitstream of the video based on the final prediction samples, wherein The prediction sample offset is determined based on at least one spatial gradient of the initial prediction sample, wherein the spatial gradient is computed based on at least a difference between two first prediction samples from a same reference picture list, and wherein, before the difference between the two first prediction samples is computed, values of the two first prediction samples are right shifted by a first value; wherein the prediction sample offset is further determined based on at least one temporal gradient, wherein the temporal gradient is computed based on at least a difference between two second prediction samples from a different reference picture list, and wherein a shift rule for the difference between the two second prediction samples is the same as a shift rule for the difference between the two first prediction samples, and the shift rule comprises a sequence of a right shift operation and a subtraction operation.

18. The non-transitory computer-readable storage medium of claim 17, wherein, For a sample position (x, y) in the current block, the two first prediction samples have positions (hx+1, vy) and (hx-1, vy) corresponding to the same reference picture list X or positions (hx, vy+1) and (hx, vy-1) corresponding to the same reference picture list X, and wherein X = 0 or 1, hx = Clip3(1, nCbW, x) and vy = Clip3(1, nCbH, y), nCbW and nCbH are a width and a height of the current block, and wherein Clip3 is a clipping function defined as follows:

19. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing apparatus, wherein, The method comprises: determining, for a current block of a video, an initial prediction sample; refining the initial prediction sample with a prediction sample offset based on an optical flow refinement technique to obtain a final prediction sample; and generating the bitstream based on the final prediction sample, wherein the prediction sample offset is determined based on at least one spatial gradient of the initial prediction sample, wherein the spatial gradient is computed based on at least a difference between two first prediction samples from a same reference picture list, and wherein, before the difference between the two first prediction samples is computed, values of the two first prediction samples are right shifted by a first value; wherein the prediction sample offset is further determined based on at least one temporal gradient, wherein the temporal gradient is computed based on at least a difference between two second prediction samples from a different reference picture list, and wherein a shift rule for the difference between the two second prediction samples is the same as a shift rule for the difference between the two first prediction samples, and the shift rule comprises a sequence of a right shift operation and a subtraction operation.

20. A method of storing a bitstream of a video, comprising: determining, for a current block of a video, an initial prediction sample; refining the initial prediction sample with a prediction sample offset based on an optical flow refinement technique to obtain a final prediction sample; generating the bitstream based on the final prediction sample, and storing the bitstream in a non-transitory computer-readable recording medium, wherein the prediction sample offset is determined based on at least one spatial gradient of the initial prediction sample, wherein the spatial gradient is computed based on at least a difference between two first prediction samples from a same reference picture list, and wherein, before the difference between the two first prediction samples is computed, values of the two first prediction samples are right shifted by a first value; wherein a value of the two first prediction samples is right shifted by a first value before a difference between the two first prediction samples is calculated; wherein the prediction sample offset is further determined based on at least one temporal gradient, wherein the temporal gradient is calculated based on at least a difference between two second prediction samples from different reference picture lists, and wherein a shift rule for the difference between the two second prediction samples is the same as a shift rule for the difference between the two first prediction samples, and the shift rule comprises a sequence of a right shift operation and a subtraction operation.

21. A video processing device comprising a processor, wherein, The processor is configured to implement the method according to any one of claims 1 to 11.

22. A non-transitory computer-readable medium having instructions stored thereon, wherein, The instructions, when executed by the processor, cause the processor to implement the method according to any one of claims 1 to 11.

Images