Gradient calculation for different motion vector refinements

Abstract

To provide a video image processing method for performing an inclination calculation by performing a fine adjustment of a different motion vector, and provide a device and a program.SOLUTION: A video image processing method contains steps of: determining a final prediction block against the current video image block by performing a fine adjustment of one or more initial predictions to the current video image block by using an optical flow calculation on the basis of an inclination of an initial prediction sample in accordance with an accuracy regulation to the current video image block of the video image; and performing a conversion between the current video image block and a coded expression by using the final prediction block. The optical flow calculation contains a prediction fine adjustment by a prediction fine adjustment with an optical flow (PROF) order or a binal-direction optical flow (BDOF) order, where the accuracy regulation is provided so as to regulate that the same accuracy is used for expressing the inclination to both of the PROF order and the BDOF order.SELECTED DRAWING: Figure 29D

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application is a divisional application of Japanese Patent Application No. 2021-560232, which is the national phase application of International Patent Application No. PCT / CN2020 / 085662 filed on April 20, 2020. It is intended to timely claim priority to and the benefit of International Patent Application No. PCT / CN2019 / 083434, filed April 19, 2019, and International Patent Application No. PCT / CN2019 / 092762, filed June 25, 2019, the entire disclosures of which are incorporated by reference as part of the disclosure herein.

[0002] This patent specification relates to image processing techniques, devices and systems. [Background technology]

[0003] Despite advances in video compression, digital video remains the largest bandwidth user on the Internet and other digital communications networks. The bandwidth demands for digital video use are expected to continue to grow as the number of connected user devices capable of receiving and displaying video increases. Summary of the Invention

[0004] Apparatus, system and method for digital video processing The described methods may be applied to both existing video coding standards (eg, High Efficiency Video Coding (HEVC)) and future video coding standards or video codecs.

[0005] In one exemplary aspect, the disclosed techniques may be used to provide a method of video processing that includes generating a prediction block for a current video block of a video using affine mode motion compensation performed at a sub-block level of the current video block, performing gradient calculations on a region of the current video block to refine the prediction block using a Prediction Refinement with Optical Flow (PROF) procedure, where a size (M×N) of the region differs from a size of a sub-block of the current video block, where M, N are positive integers, and converting between the current video block and a coded representation of the video based on the gradient calculations.

[0006] In another representative aspect, the disclosed techniques may be used to provide a method of video processing, the method including deriving a final predicted block calculated by fine-tuning one or more intermediate predicted blocks using gradient calculations performed on an image region having a second size according to a rule for a current video block of an image having a first size, the fine-tuning using an optical flow procedure, and using the final predicted block to convert between the current video block and a coded representation of the video.

[0007] In another representative aspect, the disclosed techniques may be used to provide a method of video processing that includes deriving motion information for a current video block of a video using bidirectional optical flow (BDOF) or prediction refinement with optical flow (PROF), performing gradient calculations on samples in a region of the current video block such that at least one sample in the region is omitted from the gradient calculation, and converting between the current video block and a coded representation of a video including the current video block based on the gradient calculations, where one or more initial predictions for the current video block are calculated at a sub-block level and refined using optical flow calculations in the PROF, or one or more initial predictions are refined using spatial and temporal gradients in the BDOF.

[0008] In another representative aspect, the disclosed techniques may be used to provide a method of video processing, the method including: for a current video block of a video, determining a final prediction block for the current video block by using optical flow computation to fine-tune one or more initial predictions for the current video block based on gradients of initial prediction samples according to precision rules, and converting between the current video block and a coded representation using the final prediction block, the optical flow computation including a prediction fine-tuning with optical flow (PROF) procedure or a bidirectional optical flow (BDOF) procedure, the precision rules specifying using the same precision to represent the gradients for both the PROF procedure and the BDOF procedure.

[0009] In another representative aspect, the disclosed techniques may be used to provide a method of video processing, the method including determining a final predicted block for a current video block of a video by fine-tuning one or more initial predictions for the current video block using optical flow computations for conversion between the current video block and a coded representation of the video, the optical flow computations including prediction fine-tuning with optical flow (PROF) procedure and / or bidirectional optical flow (BDOF) procedure, the optical flow computations further including a padding operation applicable to at least one of the PROF procedure or the BDOF procedure according to a rule to derive padding samples.

[0010] In yet another exemplary embodiment, the method is embodied in a processor executable code and stored on a computer readable program medium.

[0011] In yet another exemplary aspect, a device configured or operable to perform the above-described method is disclosed. The device may include a processing device programmed to implement the method.

[0012] In yet another exemplary aspect, a video decoder device may implement the methods as described herein.

[0013] These and other aspects and features of the disclosed technology are explained in more detail in the drawings, description and claims. [Brief description of the drawings]

[0014] [Figure 1] Here is an example of building a merge candidate list: [Diagram 2] 1 shows an example of spatial candidate locations. [Diagram 3] 1 shows an example of candidate pairs subject to redundancy checking of spatial merge candidates. [Figure 4A] 13 shows an example of a position of a second prediction unit (PU) based on a size and shape of a current block. [Figure 4B] 13 shows an example of a position of a second prediction unit (PU) based on a size and shape of a current block. [Diagram 5] 13 illustrates an example of scaling of motion vectors for temporal merge candidates. [Figure 6] 13 shows an example of candidate positions for temporal merge candidates. [Figure 7] 13 illustrates an example of generating joint bi-predictive merge candidates. [Figure 8] 1 shows an example of constructing motion vector prediction candidates. [Figure 9] 13 illustrates an example of motion vector scaling for spatial motion vector candidates. [Figure 10] 1 illustrates an example of motion prediction using an alternative temporal motion vector prediction (ATMVP) algorithm for a coding unit (CU). [Figure 11] 1 illustrates an example of a coding unit (CU) with sub-blocks and neighboring blocks used in a spatial-temporal motion vector prediction (STMVP) algorithm. [Figure 12A] 1 illustrates an exemplary snapshot of sub-blocks when using an overlapped block motion compensation (OBMC) algorithm. [Figure 12B] 1 illustrates an exemplary snapshot of sub-blocks when using an overlapped block motion compensation (OBMC) algorithm. [Figure 13] 1 shows an example of a neighborhood of samples used to derive parameters for a local illumination compensation (LIC) algorithm. [Figure 14] An example of a simplified affine motion model is shown. [Figure 15] 1 shows an example of an affine motion vector field (MVF) for each subblock. [Figure 16] 13 shows an example of motion vector prediction (MVP) in AF_INTER affine motion mode. [Figure 17A]Examples of AF_MERGE affine motion mode candidates are shown. [Figure 17B] Examples of AF_MERGE affine motion mode candidates are shown. [Figure 18] We present an example of bilateral matching in the Pattern Matching Motion Vector Derivation (PMMVD) mode, which is a special merging mode based on the Frame Rate Up Conversion (FRUC) algorithm. [Figure 19] An example of template matching in the FRUC algorithm is shown below. [Figure 20] 1 shows an example of one-sided motion estimation in the FRUC algorithm. [Figure 21] 1 shows an example of an optical flow trajectory used in the Bidirectional Optical Flow (BIO) algorithm. [Figure 22A] 13 shows an example snapshot using the Bidirectional Optical Flow (BIO) algorithm without block extension. [Figure 22B] 13 shows an example snapshot using the Bidirectional Optical Flow (BIO) algorithm without block extension. [Diagram 23] Here is an example of an interpolated sample used in BIO: [Figure 24] 1 illustrates an example of a decoder-side motion vector refinement (DMVR) algorithm based on bilateral template matching. [Diagram 25] An example of one sub-block MV VSB and pixel Δv(i,j) is shown. [Figure 26] 13 shows an example of phase-variable horizontal plane filtering. [Figure 27] 1 shows an example of applying 8-tap horizontal filtering. [Figure 28] 13 shows an example of non-uniform phase vertical filtering. [Figure 29A] 1 shows a flowchart of an exemplary method for video processing. [Figure 29B] 1 shows a flowchart of an exemplary method for video processing. [Figure 29C]1 shows a flowchart of an exemplary method for video processing. [Figure 29D] 1 shows a flowchart of an exemplary method for video processing. [Figure 29E] 1 shows a flowchart of an exemplary method for video processing. [Figure 30A] FIG. 2 is a block diagram illustrating an example of a hardware platform for implementing the visual media decoding or visual media encoding techniques described herein. [Figure 30B] FIG. 2 is a block diagram illustrating an example of a hardware platform for implementing the visual media decoding or visual media encoding techniques described herein. [Diagram 31] An example of 16 4x4 sub-blocks in a 16x16 region is shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Due to the increasing demand for higher resolution video, video processing methods and techniques are ubiquitous in modern technology. Video codecs generally include electronic circuits or software that compress or decompress digital video and are constantly being improved to provide higher coding efficiency. Video codecs convert uncompressed video to compressed formats or vice versa. There is a complex relationship between the quality of the video, the amount of data used to represent the video (determined by the bit rate), the complexity of the encoding and decoding algorithms, the sensitivity to data loss and errors, the ease of editing, random access, and the end-to-end delay (latency). The compression format usually complies with a standard video compression specification, such as the High Efficiency Video Coding (HEVC) standard (also known as H.265 or MPEG-H Part 2), a general-purpose video coding standard to be completed, or other current and / or future video coding standards.

[0016] Embodiments of the disclosed technology may be applied to existing video coding standards (e.g., HEVC, H.265) and future standards to improve compression performance. Section headings are used herein to improve readability of the description and are not intended to limit the description or embodiments (and / or implementations) to only the respective sections.

[0017] 1. Example of Inter Prediction in HEVC / H.265 Video coding standards have improved significantly over the years and now, in part, achieve high coding efficiency and support higher resolutions. Recent standards such as HEVC and H.265 are based on a hybrid video coding structure in which temporal prediction plus transform coding is utilized.

[0018] 1.1 Prediction mode examples Each inter-predicted PU (Prediction Unit) has motion parameters for one or two reference picture lists. In some embodiments, the motion parameters include a motion vector and a reference picture index. In other examples, the use of one of the two reference picture lists may be signaled using inter_pred_idc. In yet other embodiments, the motion vector may be explicitly coded as a delta to the predictor.

[0019] If a CU is coded in skip mode, one PU is associated with this CU, there are no significant residual coefficients, no coded motion vector differentials, and no reference picture indexes. A merge mode is specified, which obtains motion parameters for the current PU from neighboring PUs, including spatial and temporal candidates. The merge mode can be applied to any inter-predicted PU, not just for skip mode. An alternative to the merge mode is explicit transmission of motion parameters, explicitly signaling, for each PU, a motion vector, which is each reference picture list and a reference picture index corresponding to the use of the reference picture list.

[0020] If the signaling indicates to use one of the two reference picture lists, the PU is generated from a block of one sample. This is called "uni-prediction." Uni-prediction is available for both P and B slices.

[0021] If the signaling indicates to use both reference picture lists, generate the PU from a block of two samples. This is called "bi-prediction." Bi-prediction is only available for B slices.

[0022] 1.1.1 Implementation of Constructing Merge Mode Candidates When predicting a PU using merge mode, an index pointing to an entry in the merge candidate list is parsed from the bitstream and used to look up the motion information. The construction of this list can be summarized based on the following sequence of steps:

[0023] Step 1: Initial candidate selection

[0024] Step 1.1: Spatial candidate derivation

[0025] Step 1.2: Redundancy check of spatial candidates

[0026] Step 1.3: Temporal candidate derivation

[0027] Step 2: Additional candidate insertions

[0028] Step 2.1: Creating bidirectional prediction candidates

[0029] Step 2.2: Inserting zero motion candidates

[0030] Figure 1 shows an example of constructing a merge candidate list based on the sequence of steps above. For spatial merge candidate derivation, we select up to four merge candidates among candidates at five different positions. For temporal merge candidate derivation, we select up to one merge candidate among two candidates. Since the decoder side expects a constant number of candidates per PU, we generate additional candidates if the number of candidates does not reach the maximum number of merge candidates (MaxNumMergeCand) signaled in the slice header. Since the number of candidates is constant, we use a shortened unary binarization (TU) to encode the index of the best merge candidate. If the size of a CU is equal to 8, all PUs of the current CU share one merge candidate list, which is the same as the merge candidate list of the 2N × 2N prediction units.

[0031] 1.1.2 Constructing spatial merging candidates In the derivation of spatial merge candidates, up to four merge candidates are selected from the candidates at the positions shown in FIG. 2. The order of derivation is A1, B1, B0, A0, B2. Position B2 is considered only if any PU at positions A1, B1, B0, A0 is not available (e.g., because it belongs to another slice or tile) or is intra-coded. After adding the candidate at position A1, adding the remaining candidates is subject to a redundancy check, which can ensure that candidates with the same motion information are removed from the list, improving coding efficiency.

[0032] In order to reduce computational complexity, the above redundancy check does not consider all possible candidate pairs. Instead, only the pairs linked by arrows in FIG. 3 are considered, and a candidate is added to the list only if the corresponding candidate used in the redundancy check does not have the same motion information. Another source of duplicated motion information is a "second PU" associated with a division different from 2N×2N. FIG. 4A and FIG. 4B depict the second PU for N×2N and 2N×N, respectively. When dividing the current PU into N×2N, the candidate at position A1 is not considered for list construction. In some embodiments, adding this candidate makes two prediction units have the same motion information, and it is redundant to have only one PU in one coding unit. Similarly, when dividing the current PU into 2N×N, position B1 is not considered.

[0033] 1.1.3 Building Temporal Merge Candidates In this step, only one candidate is added to the list. Specifically, in the derivation of this temporal merge candidate, a scaled motion vector is derived based on the co-located PU belonging to the picture with the smallest POC difference with the current picture in a given reference picture list. In the slice header, the reference picture list used for the derivation of the co-located PU is explicitly signaled.

[0034] FIG. 5 shows an example of derivation of a scaled motion vector (dotted line) for a temporal merge candidate, where t is the POC difference between the reference picture of the current picture and the current picture, and t is the POC difference between the reference picture of the co-located picture and the co-located picture, scaled from the motion vector of the co-located PU using POC distances t, t. Set the reference picture index of the temporal merge candidate equal to zero. For B slice, we take two motion vectors, one for reference picture list 0 and one for reference picture list 1, and combine them to form a bi-predictive merge candidate.

[0035] For the co-located PU(Y) belonging to the reference frame, select the location of the temporal candidate between candidates C0 and C1 as shown in Figure 6. If the PU at location C0 is not available, is intra-coded, or is outside the current CTU, location C1 is used. Otherwise, location C0 is used to derive the temporal merge candidate.

[0036] 1.1.4 Building Additional Types of Merge Candidates Besides the spatial-temporal merge candidate, there are two additional types of merge candidates: joint bi-predictive merge candidate and zero merge candidate. The spatial-temporal merge candidate is utilized to generate a joint bi-predictive merge candidate. The joint bi-predictive merge candidate is used only for B slices. A joint bi-predictive candidate is generated by combining a first reference picture list motion parameter of an initial candidate with a second reference picture list motion parameter of another candidate. If these two tuples provide different motion hypotheses, these tuples form a new bi-predictive candidate.

[0037] FIG. 7 shows an example of this process, where two candidates in the original list (710, left) with mvL0, refIdxL0 or mvL1, refIdxL1 are used to generate a combined bidirectional prediction merge candidate that is added to the final list (720, right).

[0038] The MaxNumMergeCand capacity is hit by inserting zero motion candidates to fill the remaining entries in the merge candidate list. These candidates have a spatial displacement of zero and a reference picture index that starts from zero and increases each time a new zero motion candidate is added to the list. The number of reference frames used by these candidates is one for unidirectional prediction and two for bidirectional prediction, respectively. In some embodiments, no redundancy check is performed on these candidates.

[0039] 1.1.5 Example of motion estimation region for parallel processing To speed up the encoding process, motion estimation can be done in parallel, thereby deriving motion vectors for all prediction units in a given region simultaneously. Derivation of merge candidates from spatial neighborhoods can interfere with parallel processing, since one prediction unit cannot derive motion parameters from neighboring PUs until its associated motion estimation is completed. To mitigate the tradeoff between encoding efficiency and processing latency, a motion estimation region (MER) can be specified. The "log2_parallel_merge_level_minus2" syntax element may be used to signal the size of the MER in the picture parameter set (PPS). When specifying one MER, merge candidates in the same region are marked as unavailable and therefore not considered in the list construction.

[0040] 1.2 Advanced Motion Vector Prediction (AMVP) Implementation AMVP exploits the spatial-temporal correlation between motion vectors and neighboring PUs and uses it for unambiguous transmission of motion parameters. First, it builds a motion vector candidate list by checking the availability of the left, upper, temporally neighboring PU positions, removing redundant candidates, and adding zero vectors to keep the length of the candidate list constant. Then, the encoder can select the best predictor from the candidate list and transmit the corresponding index indicating the selected candidate. Similar to the merge index signaling, the index of the best motion vector candidate is encoded using a shortened unary term. The maximum value encoded in this case is 2 (see Figure 8). The following sections will explain the details of the motion vector prediction candidate derivation process.

[0041] 1.2.1 Example of constructing motion vector prediction candidates FIG. 8 summarizes the process of deriving motion vector prediction candidates, which may be implemented for each reference picture list with refidx as input.

[0042] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. To derive spatial motion vector candidates, as shown in Figure 2 above, we finally derive two motion vector candidates based on the motion vectors of each PU in five different positions.

[0043] To derive a temporal motion vector candidate, select one motion vector candidate from two candidates derived based on two different co-located positions. After creating an initial list of spatial-temporal candidates, remove duplicate motion vector candidates in the list. If the number of candidates is greater than two, remove from the list the motion vector candidates whose reference picture index in the associated reference picture list is greater than 1. If the number of spatial-temporal motion vector candidates is less than two, add an additional zero motion vector candidate to the list.

[0044] 1.2.2 Construction of spatial motion vector candidates In deriving spatial motion vector candidates, we consider up to two candidates among five potential candidates derived from PUs located as shown in Fig. 2 above, which are located at the same position as the motion merge. The order of derivation for the left side of the current PU is specified as A0, A1, scaled A0, scaled A1. The order of derivation for the top side of the current PU is specified as B0, B1, B2, scaled B0, scaled B1, scaled B2. So, for each edge, there are four possible cases for motion vector candidates: two cases that do not require spatial scaling and two cases that use spatial scaling. The four different cases can be summarized as follows:

[0045] -No spatial scaling

[0046] (1) The same reference picture list and the same reference picture index (same POC)

[0047] (2) Different reference picture lists, but the same reference picture (same POC)

[0048] -Spatial Scaling

[0049] (3) Same reference picture list, but different reference pictures (different POC)

[0050] (4) Different reference picture lists and different reference pictures (different POCs)

[0051] First, check the case of non-spatial scaling, then check the case of enabling spatial scaling. Regardless of the reference picture list, consider spatial scaling if the POC is different between the reference picture of the neighboring PU and the reference picture of the current PU. If all PUs of the left candidate are not available or are intra-coded, scaling of the top motion vector is useful for parallel derivation of the left and top MV candidates. Otherwise, spatial scaling is not allowed for the top motion vector.

[0052] As shown in the example of Figure 9, in the case of spatial scaling, the motion vectors of neighboring PUs are scaled in the same manner as in temporal scaling. One difference is that the reference picture list and index of the current PU are given as input, and the actual scaling process is the same as in temporal scaling.

[0053] 1.2.3 Construction of Temporal Motion Vector Candidates Other than deriving the reference picture index, all the processes for deriving the temporal merge candidates are the same as the processes for deriving the spatial motion vector candidates (shown in the example of FIG. 6). In some embodiments, the reference picture index is signaled to the decoder.

[0054] 2. Example of Inter-Prediction Method in Joint Exploration Model (JEM) In some embodiments, future video coding techniques are explored using a reference software known as the Joint Search Model (JEM), in which sub-block-based prediction is applied in several coding tools, such as affine prediction, alternative temporal motion vector prediction (ATMVP), spatio-temporal motion vector prediction (STMVP), bidirectional optical flow (BIO), frame rate up-conversion (FRUC), locally adaptive motion vector resolution (LAMVR), overlapping block motion compensation (OBMC), local illumination compensation (LIC), and decoder-side motion vector refinement (DMVR).

[0055] 2.1 Example of motion vector prediction based on sub-CU In the quadtree plus binary tree JEM (QTBT), each CU can have at most one set of motion parameters for each prediction direction. In some embodiments, the encoder considers two sub-CU level motion vector prediction methods by splitting the large CU into sub-CUs and deriving motion information for all sub-CUs of the large CU. The alternative temporal motion vector prediction (ATMVP) method allows each CU to retrieve multiple sets of motion information from multiple blocks smaller than the current CU in the aligned reference picture. In the spatial-temporal motion vector prediction (STMVP) method, the temporal motion vector predictor and spatial neighborhood motion vectors are used to recursively derive the motion vectors of the sub-CUs. In some embodiments, the motion compression of the reference frame may be disabled to maintain a more accurate motion field for sub-CU motion prediction.

[0056] 2.1.1 Alternative Temporal Motion Vector Prediction (ATMVP) Example In the ATMVP method, the temporal motion vector prediction (TMVP) method is modified by retrieving multiple sets of motion information (including motion vectors and reference indexes) from blocks smaller than the current CU.

[0057] 10 shows an example of ATMVP motion prediction processing in a CU 1000. The ATMVP method predicts a motion vector of a sub-CU 1001 in a CU 1000 in two steps. In the first step, a corresponding block 1051 in a reference picture 1050 is identified by a temporal vector. The reference picture 1050 is also called a motion source picture. In the second step, the current CU 1000 is divided into sub-CUs 1001, and the motion vector and reference index of each sub-CU are obtained from the block corresponding to each sub-CU.

[0058] In the first step, the reference picture 1050 and corresponding blocks are determined according to the motion information of the spatially neighboring blocks of the current CU 1000. To avoid the repeated scanning process of the neighboring blocks, the first merging candidate in the merging candidate list of the current CU 1000 is used. The first available motion vector and its associated reference index are set to the temporal vector and the index of the motion source picture. In this way, the corresponding blocks can be identified more accurately compared to TMVP, and the corresponding blocks (sometimes called aligned blocks) are always in the bottom-right or center position with respect to the current CU.

[0059] In a second step, the corresponding block of the sub-CU 1051 is identified by the temporal vector in the motion source picture 1050 by adding the temporal vector to the coordinate of the current CU. For each sub-CU, the motion information of its corresponding block (e.g., the smallest motion grid covering the center sample) is used to derive the motion information of the sub-CU. After identifying the motion information of the corresponding N×N block, it is converted into the motion vector and reference index of the current sub-CU, and motion scaling and other procedures are applied, similar to TMVP of HEVC. For example, the decoder checks whether a low-delay condition (e.g., the POC of all reference pictures of the current picture is smaller than the POC of the current picture) is met, and possibly predicts the motion vector MVy (e.g., X is equal to 0 or 1, and Y is equal to 1-X) of each sub-CU using the motion vector MVx (e.g., the motion vector corresponding to the reference picture list X).

[0060] 2.1.2 Spatial-Temporal Motion Vector Prediction (STMVP) Example In the STMVP method, the motion vectors of sub-CUs are derived recursively along the raster scan order. Figure 11 shows an example of one CU with four sub-blocks and neighboring blocks. Consider an 8x8 CU 1100, which contains four 4x4 sub-CUs, A (1101), B (1102), C (1103), and D (1104). The neighboring 4x4 blocks in the current frame are labeled a (1111), b (1112), c (1113), and d (1114).

[0061] The motion derivation of sub-CU A starts by identifying its two spatial neighbors. The first neighbor is the N×N block above sub-CU A 1101 (block c 1113). If this block c (1113) is not available or is intra-coded, check the other N×N blocks above sub-CU A (1101) (starting from block c 1113 and going from left to right). The second neighbor is the block to the left of sub-CU A 1101 (block b 1112). If block b (1112) is not available or is intra-coded, check the other blocks to the left of sub-CU A 1101 (starting from block b 1112 and going from top to bottom). The motion information obtained from the neighboring blocks in each list is scaled to the first reference frame of a given list. Then, derive the temporal motion vector predictor (TMVP) of sub-block A 1101 following the same procedure as the TMVP derivation specified in HEVC. The motion information of the ordered blocks in block D1104 is fetched and scaled accordingly. Finally, after retrieving and scaling the motion information, average all available motion vectors for each reference list separately. This averaged motion vector is the motion vector of the current sub-CU.

[0062] 2.1.3 Example of Sub-CU Motion Prediction Mode Signaling In some embodiments, sub-CU modes are enabled as additional merge candidates, and no additional syntax elements are required to signal the mode. Add two additional merge candidates to the merge candidate list of each CU to represent ATMVP and STMVP modes. In other embodiments, up to seven merge candidates may be used if the sequence parameter set indicates that ATMVP and STMVP are enabled. The encoding logic of the additional merge candidates is the same as that of the merge candidates in HM, that is, for each CU in a P or B slice, two or more RD checks may be required for the two additional merge candidates. In some embodiments, such as JEM, all bins of the merge index are context coded by Context-based Adaptive Binary Arithmetic Coding (CABAC). In other embodiments, such as HEVC, only the first bin is context coded, and the remaining bins are context bypass coded.

[0063] 2.2 Example of Adaptive Motion Vector Difference Resolution In an embodiment of the present invention, when use_integer_mv_flag is 0 in the slice header, the motion vector difference (MVD) (the difference between the motion vector and the predicted motion vector of the PU) is signaled in units of quarter luma samples. In JEM, locally adaptive motion vector resolution (LAMVR) is introduced. In JEM, MVD can be coded in units of quarter luma samples, integer luma samples or four luma samples. MVD resolution is controlled at the coding unit (CU) level, and the MVD resolution flag is conditionally signaled for each CU that has at least one non-zero MVD module.

[0064] For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether ¼ luma sample MV precision is used in the CU. If the first flag (equal to 1) indicates that ¼ luma sample MV precision is not used, another flag is signaled to indicate whether integer luma sample MV precision or 4 luma sample MV precision is used.

[0065] If the first MVD resolution flag of a CU is zero or is not coded for the CU (i.e., all MVDs in the CU are zero), then ¼ luma sample MV resolution is used for the CU. If the CU uses integer luma sample MV precision or 4 luma sample MV precision, round the MVPs in the CU's AMVP candidate list to the corresponding precision.

[0066] In the encoder, the CU-level RD check is used to determine which MVD resolution to use for the CU. That is, the CU-level RD check is performed three times for each MVD resolution. To increase the speed of the encoder, the following coding method is applied in JEM.

[0067] During RD check of a CU with normal 1 / 4 luma sample MVD resolution, the motion information (integer luma sample precision) of the current CU is stored. During RD check of the same CU with integer luma sample and 4 luma sample MVD resolution, the stored motion information (after rounding) is used as starting point for further small range motion vector refinement, so that the time consuming motion estimation process is not duplicated three times.

[0068] -Conditionally invoke RD check for CUs with 4 luma sample MVD resolution. For a CU, if the RD cost of integer luma sample MVD resolution is much larger than that of 1 / 4 luma sample MVD resolution, then the RD check of 4 luma sample MVD resolution for the CU is omitted.

[0069] 2.3 Example of improving the accuracy of motion vector storage In HEVC, the precision of motion vectors is 1 / 4 pixel (1 / 4 luma sample and 1 / 8 chroma sample for 4:2:0 video). In JEM, the precision of internal motion vector storage and merge candidates is increased to 1 / 16 pixel. The higher motion vector precision (1 / 16 pixel) is used for motion compensated inter prediction for CUs coded in skip / merge mode. For CUs coded in regular AMVP mode, either integer pixel or quarter pixel motion is used.

[0070] The SHVC upsampling interpolation filter, which has the same filter length and normalization factor as the HEVC motion compensation interpolation filter, is used as the motion compensation interpolation filter for the additional fractional pixel position. In JEM, the accuracy of the motion vector of the chroma component is 1 / 32 sample, and the average of the two neighboring 1 / 16 pixel fractional position filters is used to derive the additional 1 / 32 pixel fractional position interpolation filter.

[0071] 2.4 Example of Overlapped Block Motion Compensation (OBMC) In JEM, OBMC can be switched on / off using CU-level syntax. When using OBMC in JEM, OBMC is performed on all motion compensation (MC) block boundaries except the bottom-right boundary of the CU. It is also applied to both luma and chroma components. In JEM, an MC block corresponds to a coding block. If the CU is coded in a sub-CU mode (including sub-CU merge, affine, and FRUC modes), each sub-block of the CU is one MC block. In order to handle CU boundaries uniformly, OBMC is performed at the sub-block level for all MC block boundaries, where the sub-block size is set equal to 4×4, as shown in Figures 12A and 12B.

[0072] Figure 12A shows the sub-blocks at the CU / PU boundary, where the shaded sub-blocks are where OBMC is applied. Similarly, Figure 12B shows the sub-PUs in ATMVP mode.

[0073] When OBMC is applied to the current subblock, in addition to the current motion vector, the motion vectors of the four connected neighboring subblocks are also available and used to derive a prediction block for the current subblock if they are not the same as the current motion vector. These multiple prediction blocks based on multiple motion vectors are combined to generate a final prediction signal for the current subblock.

[0074] Let PN (N is the index of the neighboring subblocks above, below, left, and right) be the prediction block based on the motion vector of the neighboring subblock, and PC be the prediction block based on the motion vector of the current subblock. If PN is based on the motion information of the neighboring subblocks that contain the same motion information as the current subblock, OBMC is not performed from PN. Otherwise, add all samples of PN to the same samples in PC, i.e., add four rows / columns of PN to PC. Use weighting factors {1 / 4, 1 / 8, 1 / 16, 1 / 32} for PN and weighting factors {3 / 4, 7 / 8, 15 / 16, 31 / 32} for PC. The exception is small MC blocks (i.e., when the height or width of the coding block is equal to 4 or one CU is coded in sub-CU mode), in which case only two rows / columns of PN are added to PC. In this case, weighting factors {1 / 4, 1 / 8} are used for PN and weighting factors {3 / 4, 7 / 8} are used for PC. For a PN generated based on the motion vector of a neighboring subblock in the vertical (horizontal) direction, a sample in the same row (column) of the PN is added to the PC with the same weighting coefficient.

[0075] In JEM, for CUs whose size is 256 luma samples or less, a CU level flag is signaled to indicate whether OBMC is applied to the current CU. For CUs whose size is larger than 256 luma samples or not coded in AMVP mode, OBMC is applied by default. In the encoder, if OBMC is applied to a CU, its influence is considered during the motion estimation stage. The prediction signal formed by OBMC using the motion information of the upper neighboring block and the left neighboring block is used to compensate the upper and left boundaries of the original signal of the current CU, and then the normal motion estimation process is applied.

[0076] 2.5 Local Illumination Compensation (LIC) Example LIC is based on a linear model of illumination changes, with a scale factor a and an offset b, and is adaptively enabled or disabled for each inter mode coding unit (CU).

[0077] When LIC is applied to a CU, a least square error method is used to derive parameters a and b by using the neighboring samples of the current CU and their corresponding reference samples. Figure 13 shows an example of the neighboring samples used to derive the parameters of the IC algorithm. Specifically, as shown in Figure 13, the subsampled (2:1 subsampled) neighboring samples of the CU and the corresponding samples in the reference picture (identified by the motion information of the current CU or sub-CU) are used. The IC parameters are derived and applied separately for each prediction direction.

[0078] If a CU is coded in merge mode, it copies the LIC flag from the neighboring block, similar to copying motion information in merge mode, otherwise it signals the LIC flag to the CU to indicate whether LIC is applied or not.

[0079] When LIC is enabled for a picture, an additional CU-level RD check is required to determine whether LIC applies for a CU. If LIC is enabled for a CU, the Average Removed Sum of Absolute Diffusion (MR-SAD) and the Average Removed Sum of Absolute Hadamard Transform Differences (MR-SATD) are used instead of SAD and SATD for integer pixel and fractional pixel motion search, respectively.

[0080] In order to reduce the coding complexity, the following coding scheme is applied in JEM:

[0081] - If there is no clear illumination change between the current picture and its reference pictures, LIC is disabled for the entire picture. To identify this situation, in the encoder, we calculate the histograms of the current picture and all its reference pictures. If the histogram difference between the current picture and all its reference pictures is smaller than a given threshold, we disable LIC for the current picture, otherwise we enable LIC for the current picture.

[0082] 2.6 Example of Affine Motion Compensation Prediction In HEVC, only a translational motion model is applied for motion compensated prediction (MCP). However, the camera and the object may have various kinds of motion, such as zoom in / out, rotation, perspective motion, and / or other irregular motion. On the other hand, JEM applies a simplified affine transformation motion compensated prediction. FIG. 14 is a diagram showing an example of an affine motion field of a block 1400 described by motion vectors V0, V1 of two control points. The motion vector field (MVF) of the block 1400 can be expressed as the following equation:

[0083]

number

[0084] As shown in Figure 14, (v 0x ,v0y ) is the motion vector of the control point in the upper left corner, and (v 1x ,v 1y ) is the motion vector of the control point in the upper right corner. To simplify the motion compensation prediction, sub-block based affine transformation prediction can be applied. The size of the sub-block M×N is derived as follows:

[0085]

number

[0086] where MvPre is the precision of the fractional part of the motion vector (e.g., 1 / 16 in JEM). 2x ,v 2y ) is the motion vector of the bottom-left control point calculated according to equation (1). If necessary, M and N can be adjusted downward to be divisors of w and h, respectively.

[0087] FIG. 15 shows an example of per-subblock affine MVF for block 1500. To derive motion vectors for each M×N subblock, the motion vector of the center sample of each subblock can be calculated according to equation (1) and rounded to fractional precision of the motion vector (e.g., 1 / 16 in JEM). A motion compensation interpolation filter can then be applied to generate a prediction for each subblock using the derived motion vector. After MCP, the high-precision motion vector for each subblock is rounded and stored with the same precision as the normal motion vector.

[0088] 2.6.1 AF_INTER mode implementation In JEM, there are two affine motion modes: AF_INTER mode and AF_MERGE mode. For CUs with both width and height greater than 8, AF_INTER mode can be applied. A CU-level affine flag is signaled in the bitstream to indicate whether AF_INTER mode is used. In AF_INTER mode, motion vector pairs are calculated using neighboring blocks.

number

[0089] FIG. 16 shows an example of motion vector prediction (MVP) for block 1600 in AF_INTER mode. As shown in FIG. 16, select v0 from the motion vectors of subblocks A, B, or C. The motion vectors from the neighboring blocks can be scaled according to the reference list. Also, the motion vectors may be scaled based on the relationship between the picture order count (POC) of the reference of the neighboring block and the POC of the reference of the current CU and the POC of the current CU. The method of selecting v1 from the neighboring subblocks D and E is similar. If the number of candidate lists is less than two, fill the list with motion vector pairs that duplicate each of the AMVP candidates. If the candidate list is greater than two, first sort the candidates based on the motion vectors of the neighbors (e.g., based on the similarity of the two motion vectors in the pair candidate). In some implementations, keep the first two candidates. In some embodiments, use a distortion ratio (RD) cost check to determine which motion vector pair candidate to select as the control point motion vector prediction (CPMVP) of the current CU. An index indicating the position of the CPMVP in the candidate list can be signaled in the bitstream. After determining the CPMVP of the current affine CU, affine motion estimation is applied to obtain a control point motion vector (CPMV). Then, the difference between the CPMV and the CPMVP is signaled in the bitstream.

[0090] 2.6.3 Implementation of AF_MERGE Mode When applying a CU in AF_MERGE mode, the CU obtains the first block coded in affine mode from the valid neighboring reconstructed blocks. FIG. 17A shows an example of a selection order of candidate blocks for a current CU 1700. As shown in FIG. 17A, the selection order can be from the left (1701), top (1702), top right (1703), bottom left (1704) to the top left (1705) of the current CU 1700. FIG. 17B shows another example of candidate blocks for a current CU 1700 in AF_MERGE mode. When coding the neighboring bottom left block 1801 in affine mode, derive the motion vectors v2, v3, and v4 of the top left corner, top right corner, and bottom left corner of the CU containing the subblock 1701, as shown in FIG. 17B. Based on v2, v3, and v4, calculate the motion vector v0 of the top left corner in the current CU 1700. Thus, the motion vector v1 of the top right of the current CU can be calculated.

[0091] After calculating the CPMV of current CUs v0, v1 according to the affine motion model in equation (1), the MVF of the current CU can be generated. To identify whether the current CU is coded in AF_MERGE mode, an affine flag can be signaled in the bitstream if there is at least one neighboring block coded in affine mode.

[0092] 2.7 Pattern Matching Motion Vector Derivation (PMMVD) Example The PMMVD mode is a special merging mode based on the Frame Rate Up-Conversion (FRUC) method, in which the motion information of the blocks is not signaled but is derived at the decoder side.

[0093] The FRUC flag may be signaled to the CU if its merge flag is true. If the FRUC flag is false, a merge index may be signaled and normal merge mode is used. If the FRUC flag is true, an additional FRUC mode flag may be signaled to indicate which method (e.g., bilateral matching or template matching) is used to derive motion information for the block.

[0094] On the encoder side, the decision of whether to use the FRUC merging mode for a CU is based on the RD cost selection, just as it is done for regular merging candidates. For example, we use the RD cost selection to check multiple matching modes (e.g., bilateral matching and template matching) for one CU. The one that leads to the smallest cost is further compared with other CU modes. If the FRUC matching mode is the most efficient one, we set the FRUC flag to true for the CU and use the associated matching mode.

[0095] Generally, the motion derivation process in FRUC merge mode has two steps: first, CU-level motion search, and then sub-CU-level motion refinement. At the CU level, an initial motion vector for the entire CU is derived based on bilateral matching or template matching. First, a list of MV candidates is generated, and the candidate that leads to the minimum matching cost is selected as the starting point for further CU-level refinement. Then, a local search based on bilateral matching or template matching is performed near the starting point. The MV result at the minimum matching cost is taken as the MV for the entire CU. Then, the derived CU motion vector is used as the starting point to further refine the motion information at the sub-CU level.

[0096] For example, to derive W×H CU motion information, the following derivation process is performed: In the first stage, an MV for the entire W×H CU is derived. In the second stage, the CU is further divided into M×M sub-CUs. The value of M is calculated as in Equation (3), where D is a predefined division depth, which is set to 3 by default in JEM. Then, an MV for each sub-CU is derived.

[0097]

number

[0098] FIG. 18 shows an example of bilateral matching used in the frame rate up-conversion (FRUC) method. This bilateral matching is used to derive the motion information of the current CU (1800) by finding the closest match between two blocks along the motion trajectory of the current CU (1800) in two different reference pictures (1810, 1811). Assuming a continuous motion trajectory, the motion vectors MV0 (1801), MV1 (1802) pointing to the two reference blocks are proportional to the temporal distance between the current picture and the two reference pictures, e.g., TD0 (1803), TD1 (1804). In some embodiments, if the current picture 1800 is temporally between the two reference pictures (1810, 1811) and the temporal distance between the current picture and the two reference pictures is the same, the bilateral matching becomes a mirror-based bidirectional MV.

[0099] FIG. 19 shows an example of template matching used in the Frame Rate Up Conversion (FRUC) method. Using template matching, motion information of the current CU 1900 can be derived by finding the closest match between a template in the current picture (e.g., a block in the upper and / or left neighbor of the current CU) and a block in the reference picture 1910 (e.g., the same size as the template). Besides the FRUC merge mode mentioned above, template matching can also be applied to the AMVP mode. In both JEM and HEVC, AMVP has two candidates. A new candidate can be derived by using the template matching method. If the newly derived candidate by template matching is different from the first existing AMVP candidate, it is inserted at the beginning of the AMVP candidate list, and then the list size is set to 2 (e.g., by removing the second existing AMVP candidate). When applied to the AMVP mode, only the CU level search is applied.

[0100] The CU-level MV candidate set may include: (1) the original AMVP candidates if the current CU is in AMVP mode, (2) all merge candidates, (3) multiple MVs in the interpolated MV field (described below), and the motion vectors of the top-left neighbor.

[0101] When using bilateral matching, each valid MV of the merge candidate can be used as input to generate MV pairs assuming bilateral matching. For example, one valid MV of the merge candidate is (MVa,ref a ) and its corresponding bilateral MV reference picture ref b is found in another reference list B, ref a and ref b are on different sides of the current picture in time. b If not available, ref b ref aThe ref is determined as a different reference from the current picture, and its temporal distance from the current picture is the smallest value in list B. b After determining the current picture and ref a ,ref b MVb is derived by scaling MVa based on the temporal distance between

[0102] In some implementations, four MVs from the interpolated MV field may be added to the CU level candidate list. Specifically, the interpolated MVs at positions (0,0), (W / 2,0), (0,H / 2), and (W / 2,H / 2) of the current CU are added. When applying FRUC in AMVP mode, the original AMVP candidates are also added to the CU level MV candidate set. In some implementations, at the CU level, 15 MVs for the AMVP CU and 13 MVs for the merged CU can be added to the candidate list.

[0103] The sub-CU level MV candidate set includes the MV determined by the CU level search, (2) the MVs of the top, left, top-left, and top-right neighbors, (3) the scaled version of the aligned MV from the reference picture, (4) one or more (e.g., up to four) ATMVP candidates, and (5) one or more (e.g., up to four) STMVP candidates. The scaled MV from the reference picture is derived as follows: Traverse the reference pictures in both lists. The MV at the aligned position of the sub-CU in the reference picture is scaled with respect to the reference of the starting CU level MV. The candidates for ATMVP and STMVP may be the first four candidates. At the sub-CU level, one or more (e.g., up to 17) MVs are added to the candidate list.

[0104] Generate Interpolated MV Fields Before encoding a frame, an interpolated motion field is generated for the whole picture based on one ME, and this motion field may be used later as a CU-level or sub-CU-level MV candidate.

[0105] In some embodiments, the motion field of each reference picture in both reference lists is traversed at the 4x4 block level. Figure 20 shows an example of unilateral motion estimation (ME) 2000 in the FRUC method. For each 4x4 block, if the motion related to the block passing through the 4x4 block of the current picture does not have an interpolated motion assigned to it yet, scale the motion of the reference block to the current picture based on the temporal distances TD0 and TD1 (similar to the MV scaling of TMVP in HEVC) and assign the scaled motion to the block of the current frame. If the 4x4 block does not have a scaled MV assigned to it, the motion of the block is marked as unavailable in the interpolated motion field.

[0106] Interpolation and Matching Costs When one motion vector points to one fractional sample position, motion compensated interpolation is necessary. To reduce complexity, bilinear interpolation can be used for both bilateral and template matching instead of the usual 8-tap HEVC interpolation.

[0107] The calculation of the matching cost is slightly different in different steps. When selecting a candidate from the CU-level candidate set, the matching cost can be the sum of absolute differences (SAD) of bilateral matching or template matching. After determining the starting MV, the matching cost C of bilateral matching in the sub-CU level search is calculated as follows:

[0108]

number

[0109] where w is a weighting factor. In some embodiments, w may be empirically set to 4. MV and MV s denote the current MV and the start MV, respectively. The SAD may still be used as the matching cost for template matching in the sub-CU level search.

[0110] In FRUC mode, the MV is derived by using only luma samples. The derived motion is used for both luma and chroma for MC inter prediction. After determining the MV, an 8-tap interpolation filter for luma and a 4-tap interpolation filter for chroma are used to perform the final MC.

[0111] MV refinement is a pattern-based MV search with the criteria of bilateral matching cost or template matching cost. JEM supports two search patterns, namely Unlimited Center Biased Diamond Search (UCBDS) and Adaptive Cross Search for MV refinement at CU level and sub-CU level, respectively. For both CU and sub-CU level MV refinement, MV is directly searched with the accuracy of 1 / 4 luma sample MV, followed by 1 / 8 luma sample MV refinement. The search range of MV refinement for CU and sub-CU steps is set equal to 8 luma samples.

[0112] In the bilateral matching merge mode, bidirectional prediction is applied because we derive the motion information of the CU based on the closest matching between two blocks along the motion trajectory of the current CU in two different reference pictures. In the template matching merge mode, the encoder can choose between single prediction from list0, single prediction from list1, or bidirectional prediction for the CU. The selection can be made based on the template matching cost as follows:

[0113] If costBi<=factor*min(cost0,cost1)

[0114] Bidirectional prediction is used.

[0115] Otherwise, if cost0<=cost1

[0116] Use a single prediction from list0.

[0117] If not,

[0118] Use a single prediction from list1.

[0119] Here, cost0 is the SAD of list0 template matching, cost1 is the SAD of list1 template matching, and costBi is the SAD of bidirectional prediction template matching. For example, a factor value of 1.25 means that the selection process is biased toward bidirectional prediction. This inter-prediction direction selection can be applied to the template matching process at the CU level.

[0120] 2.8 Generalized Bidirectional Prediction Fine-tuning (GBi) Example Generalized Bi-directional Prediction Fine-tuning (GBi) proposed in JVET-L0646 is adopted in VTM-3.0. GBi applies unequal weights to predictors from L0 and L1 in bi-directional prediction mode. In inter-prediction mode, it evaluates multiple weight pairs, including the equal weight pair (1 / 2,1 / 2), based on RDO and signals the GBi index of the selected weight pair to the decoder. In merge mode, the GBi index is inherited from the neighboring CU. The predictor generation formula is shown in Equation (5).

[0121]

number

[0122] Here, P GBi is the final predictor of GBi, and w0 and w1 are the predictors of list 0 (L0) and list 1 (L1), respectively (P L0 and P L1 ) is the selected GBi weight applied to the GBi and shiftNum GBiis used to normalize the final predictor in GBi. The supported w1 weight set is {-1 / 4, 3 / 8, 1 / 2, 5 / 8, 5 / 4}, where the five weights correspond to one equal weight pair and four unequal weight pairs. The blending gain, i.e., the sum of w1 and w0, is fixed to 1.0. Thus, the corresponding w0 weight set is {5 / 4, 5 / 8, 1 / 2, 3 / 8, -1 / 4}. The weight pair selection is at the CU level.

[0123] For non-low latency pictures, the weight set size is reduced from 5 to 3, where the w1 weight set is {3 / 8, 1 / 2, 5 / 8} and the w0 weight set is {5 / 8, 1 / 2, 3 / 8}. The weight set size reduction for non-low latency pictures applies to BMS2.1 GBi and all GBi tests in this contribution.

[0124] 2.8.1 GBi encoder bug fix To reduce GBi encoding time, in the current encoder design, the encoder stores one predicted motion vector estimated from the GBi weight being 4 / 8 and reuses them for single prediction search of other GBi weights. This fast encoding method applies to both translational and affine motion models. In VTM2.0, the 6-parameter affine model was adopted along with the 4-parameter affine model. The BMS2.1 encoder does not differentiate between the 4-parameter affine model and the 6-parameter affine model when storing single prediction affine MV when the GBi weight is 4 / 8. As a result, after encoding with GBi weight 4 / 8, the 6-parameter affine MV may be overwritten to the 4-parameter affine MV. The stored 6-parameter affine MV may be used for 4-parameter affine ME for other GBi weights, or the stored 4-parameter affine MV may be used for 6-parameter affine ME. The proposed GBi encoder bug fix is ​​to separate the 4-parameter affine MV storage area and the 6-parameter affine MV storage area. The encoder stores these affine MVs based on the affine model type if the GBi weight is 4 / 8, and reuses the corresponding affine MVs based on this affine model type if the GBi weight is other.

[0125] 2.8.2 GBi encoder speedup In the present existing implementation, when GBi is enabled, five encoder speed-up methods are proposed to reduce the encoding time.

[0126] (1) Conditionally skip affine motion estimation for certain GBi weights

[0127] In BMS2.1, we perform affine ME, including 4-parameter and 6-parameter affine ME, for all GBi weights. For these unequal GBi weights (weights not equal to 4 / 8), we propose to conditionally skip affine ME. Specifically, after evaluating 4 / 8 GBi weights, we perform affine ME for other GBi weights only if affine mode is selected as the current best mode and not affine merge mode. If the current picture is a non-low latency picture, bidirectional prediction ME for transformation models is skipped for unequal GBi weights when affine ME is performed. If affine mode is not selected as the current best mode or affine merge is selected as the current best mode, we skip affine ME for all other GBi weights.

[0128] (2) Reduce the number of weights for RD cost check of low-delay pictures in 1-pixel and 4-pixel MVD precision encoding.

[0129] For low-latency pictures, there are five weights in the RD cost check for all MVD precisions including 1 / 4-pixel, 1-pixel, and 4-pixel. The encoder first checks the RD cost at 1 / 4-pixel MVD precision. For the RD cost check for 1-pixel and 4-pixel MVD precision, we propose to skip some of the GBi weights. We order these unequal weights according to their RD cost at 1 / 4-pixel MVD precision. During encoding at 1-pixel and 4-pixel MVD precision, we evaluate only the first two weights with the smallest RD cost along with the GBi weight 4 / 8. Thus, for low-latency pictures, we evaluate a maximum of three weights for 1-pixel and 4-pixel MVD precision.

[0130] (3) Conditionally skip bidirectional prediction search if L0 and L1 reference pictures are the same

[0131] For some pictures in RA, the same picture may occur in both reference picture lists (list-0 and list-1). For example, for a random access coding configuration in CTC, the reference picture structure of the first group of pictures (GOP) is listed as follows:

[0132] POC:16,TL:0,[L0: 0 ] [L1: 0 ]

[0133] POC:8,TL:1,[L0: 0 16 ] [L1: 16 0 ]

[0134] POC:4,TL:2,[L0:0 8 ] [L1: 8 16]

[0135] POC:2,TL:3,[L0:0 4 ] [L1: 4 8]

[0136] POC:1,TL:4,[L0:0 2 ] [L1: 2 4]

[0137] POC:3,TL:4,[L0:2 0] [L1:4 8]

[0138] POC:6,TL:3,[L0:4 0] [L1:8 16]

[0139] POC:5,TL:4,[L0:4 0] [L1:6 8]

[0140] POC:7,TL:4,[L0:6 4] [L1:8 16]

[0141] POC:12,TL:2,[L0: 8 0] [L1:16 8 ]

[0142] POC:10,TL:3,[L0:8 0] [L1:12 16]

[0143] POC:9,TL:4,[L0:8 0] [L1:10 12]

[0144] POC:11,TL:4,[L0:10 8] [L1:12 16]

[0145] POC:14,TL:3,[L0: 12 8] [L1: 12 16]

[0146] POC:13,TL:4,[L0:12 8] [L1:14 16]

[0147] POC:15,TL:4,[L0: 14 12] [L1:16 14 ]

[0148] Note that pictures 16, 8, 4, 2, 1, 12, 14 and 15 have the same reference picture in both lists. For bi-prediction of these pictures, the L0 and L1 reference pictures may be the same. If 1) the two reference pictures in bi-prediction are the same, 2) the temporal layer is greater than 1, and 3) the MVD precision is 1 / 4 pixel, the encoder proposes to skip bi-prediction ME for unequal GBi weights. For affine bi-prediction ME, this fast skip method applies only to 4-parameter affine ME.

[0149] (4) Skipping RD cost check to determine whether GBi weights are unequal based on the temporal layer and the POC distance between the reference picture and the current picture.

[0150] If the temporal stratum is 4 (the highest temporal stratum in RA) or the POC distance between the reference picture (list-0 or list-1) and the current picture is equal to 1 and the coding QP is greater than 32, we propose to skip the RD cost evaluation for these unequal GBi weights.

[0151] (5) If the GBi weights during ME are not equal, change the floating point calculation to a fixed point calculation.

[0152] For the existing bidirectional prediction search, the encoder fixes the MV of one list and fine-tunes the MV in another list. To reduce the computational complexity, the object is modified before ME. For example, if the MV of list-1 is fixed and the encoder should fine-tune the MV of list-0, the MV fine-tuning target of list-0 is modified in equation (6). O is the original signal, P1 is the predicted signal of list-1, and w is the GBi weight of list-1.

[0153]

number

[0154] Where:

number

[0155]

number

[0156] where a1 and a2 are scaling factors, which are calculated as follows:

[0157]

number

[0158] 2.8.3 GBi CU size constraint In this method, for small CUs, GBi is disabled. In inter prediction mode, if bi-prediction is used and the CU region is smaller than 128 luma samples, GBi is disabled without any signaling.

[0159] 2.9 Bidirectional Optical Flow (BDOF or BIO) Example

[0160] 2.9.1 Overview of BDOF In BIO, we first perform motion compensation to generate a first prediction (in each prediction direction) of the current block. The first prediction is used to derive spatial gradients, temporal gradients, and optical flow for each sub-block or pixel in the block, which are then used to generate a second prediction, e.g., a final prediction for the sub-block or pixel. The details are described below.

[0161] The Bi-directional Optical Flow (BIO) method is a sample-by-sample motion refinement performed on top of block-by-block motion compensation for bi-prediction. In some embodiments, the sample-level motion refinement does not use signaling.

[0162] The luminance from reference k (k=0,1) after block motion compensation is I (k) Let ∂I (k) / ∂x, ∂I (k) / ∂y respectively I (k) Let,be the horizontal and vertical components of the gradient.,Assuming that optical flow is valid, the motion vector field,v, x ,v y )i is given by the following formula:

[0163]

number

[0164] By combining this optical flow equation with the motion trajectory of each sample by Hermite interpolation, both functional values ​​I (k) and the derivative ∂I(k) / ∂x, ∂I (k) We obtain a unique 3rd order polynomial that fits / ∂y. The value of this polynomial at t=0 is the BIO prediction, as follows:

[0165]

number

[0166] FIG. 24 shows an example of optical flow trajectory in the Bidirectional Optical Flow (BIO) method. Here, τ0 and τ1 indicate the distance to the reference frame. The distances τ0, τ1 are calculated based on the POC of Ref0 and Ref1: τ0=POC(current)-POC(Ref0), τ1=POC(Ref1)-POC(current). If both predictions are from the same time direction (both from the past or both from the future), the signs are different (e.g., τ0·τ1<0). In this case, BIO is applied if the predictions are not from the same time instant (e.g., τ0≠τ1). Both reference regions have non-zero motion (e.g.,

number

number

[0167] Motion vector field (v x ,v y ) is determined by minimizing the difference Δ between the values ​​at points A and B. Figures 9A-9B show examples of intersections of a motion trajectory with a reference frame surface. The model uses only the first linear term of a local Taylor expansion for Δ as follows:

[0168]

number

[0169] All values ​​in the above equation depend on the sample position, denoted as (i',J'). Assuming that the motion is consistent in the local surrounding area, Δ can be minimized inside a (2M+1)×(2M+1) square window Ω centered on the current prediction point (i,J), where M is equal to 2.

[0170]

number

[0171] For this optimization problem, JEM uses a simple approach that first minimizes vertically and then horizontally, resulting in:

[0172]

number

[0173]

number

[0174] Where:

[0175]

number

[0176] To avoid division by zero or very small numbers, we introduce regularization parameters r and m in equations (12) and (13).

[0177]

number

[0178]

number

[0179] where d is the bit depth of the video samples.

[0180] To make the memory access for BIO the same as for normal bidirectional motion compensation, all the prediction and gradient values ​​I (k) , ∂I (k) / ∂x,∂I (k) 22A shows an example of access positions outside the block 2200. As shown in FIG. 22A, in equation (12), (2M+1)×(2M+1) square windows centered on the current prediction point on the boundary of the prediction block need to access positions outside the block. In JEM, I (k) , ∂I (k) / ∂x,∂I (k) The value of / ∂y is set equal to the closest valid value inside the block. For example, this can be implemented as a padding region 2201, as shown in Figure 22B.

[0181] Using BIO, we can refine the motion field on a sample-by-sample basis. To reduce the computational complexity, a block-based BIO design is used in JEM. The motion refinement can be calculated based on a 4×4 block. In block-based BIO, the s n The value of s is then integrated. n The value of is used to derive the BIO motion vector offset for the 4x4 block. Specifically, the following equation can be used for block-based BIO derivation:

[0182]

number

[0183] where b k represents a set of samples belonging to the k-th 4×4 block of the prediction block, and sn ((s n,bk )>>4) and derive the associated motion vector offset.

[0184] In some scenarios, the MV regimen of BIO may be unreliable due to noise and irregular motion. Therefore, in BIO, the magnitude of the MV regimen is clipped to a threshold value. The threshold value is determined based on whether all reference pictures of the current picture are from one direction or not. If all reference pictures of the current picture are from one direction, the threshold value is set to 12 × 2 14-d otherwise, set the threshold to 12×2 13-d Set to.

[0185] The gradients of BIO may be calculated simultaneously with the motion compensation interpolation using arithmetic compatible with the HEVC motion compensation process (e.g., 2D separable finite impulse response (FIR)). In some embodiments, the input for the 2D separable FIR is the same reference frame sample as for the motion compensation process and fractional positions (fracX, fracY) by the fractional part of the block motion vector. For the horizontal gradient ∂I / ∂x, first interpolate the signal vertically using BIOfilterS corresponding to the fractional position at fracYwith descaling shift d-8. Then, perform gradient filter BIOfilterG horizontally corresponding to the fractional position at fracXwith descaling shift by 18-d. For the vertical gradient ∂I / ∂y, apply gradient filter vertically using BIOfilterG corresponding to fractional position fracY with descaling shift d-8. Then, perform signal displacement using horizontal BIOfilterS corresponding to fractional position fracX with descaling shift by 18-d. To keep the complexity reasonable, the length of the interpolation filters for the gradient calculation BIOfilterG and the signal displacement BIOfilterF may be shorter (e.g., 6 taps). Table 1 shows exemplary filters that can be used for gradient calculation of different fractional positions of the block motion vector in BIO. Table 2 shows exemplary interpolation filters that can be used for generating the prediction signal in BIO.

[0186] [Table 1]

[0187] [Table 2]

[0188] In this JEM, BIO can be applied to all bidirectionally predicted blocks when the two predictions are from different reference pictures. BIO can be disabled when CU's Local Lighting Compensation (LIC) is enabled.

[0189] In some embodiments, OBMC is applied to one block after normal MC processing. In order to reduce computational complexity, BIO may not be applied during OBMC processing. That is, BIO is applied in the MC processing of one block when it uses its own MV, and is not applied in the MC processing when it uses the MV of a neighboring block in the OBMC processing.

[0190] 2.9.2 Example of BIO in VTM-3.0 proposed in JVET-L0256 Step 1: Determines if the BIO is applicable (W / H is the width / height of the current block).

[0191] BIO is not applicable in the following cases:

[0192] o The current video block is either affine coded or ATMVP coded.

[0193] ○ (iPOC-iPOC0)×(iPOC-iPOC1)≧0

[0194] ○ H==4 or (W==4 and H==8)

[0195] ○ Weight prediction

[0196] ○ GBi weight is not (1,1)

[0197] If the total SAD between two reference blocks (denoted as R0 and R1) is less than a threshold, the BIO is not used.

[0198]

number

[0199] Step 2: Data Creation

[0200] For a WxH block, (W+2)x(H+2) samples are interpolated.

[0201] The inner W×H samples are interpolated with an 8-tap interpolation filter, as in the case of normal motion compensation.

[0202] The four lateral outer lines of the sample (black circles in Figure 23) are interpolated using a bilinear filter.

[0203] For each position, the gradient is calculated with respect to two reference blocks (R0 and R1).

[0204] Gx0(x,y)=(R0(x+1,y)-R0(x-1,y))>>4

[0205] Gy0(x,y)=(R0(x,y+1)-R0(x,y-1))>>4

[0206] Gx1(x,y)=(R1(x+1,y)-R1(x-1,y))>>4

[0207] Gy1(x,y)=(R1(x,y+1)-R1(x,y-1))>>4

[0208] For each position, the internal value is calculated as follows:

[0209] and

[0210] B1(x,y)=T2*T2,B2(x,y)=T2*T3,B3(x,y)=-T1*T2,B5(x,y)=T3*T3,B6(x,y)=-T1*T3

[0211] Step 3: Calculate the prediction for each block

[0212] If the SAD between two 4x4 reference blocks is less than a threshold, skip the BIO for the 4x4 block.

[0213] Calculate Vx and Vy.

[0214] The final prediction for each position in the 4x4 block is calculated as follows:

[0215] b(x,y)=(Vx(Gx 0 (x,y)-Gx 1 (x,y))+Vy(Gy 0 (x,y)-Gy 1 (x,y))+1)>>1

[0216] P(x,y)=(R 0 (x,y)+R 1 (x,y)+b(x,y)+offset)>>shift

[0217] Here, b(x, y) is the correction item.

[0218] 2.9.3 BIO in VTM-4.0 JVET-M0063, which proposes rounding the results of BDOF calculations according to bit depth, has been adopted for VTM-4.0.

[0219] We adopt JVET-M0487 in VTM-4.0, remove the bilinear filtering, take the nearest integer pixel of the reference block, and fill in the outer lines of the four side samples (black circles in Figure 23).

[0220] The working draft on BIO in VTM-4.0 is shown below (from JVET-M1001).

[0221] 2.9.4 Fractional Sample Interpolation general The inputs to this process are: - a luminance position (xSb, ySb) that defines the top-left sample of the current coding sub-block relative to the top-left luminance sample of the current picture, a variable sbWidth that defines the width of the current coding sub-block, - the variable sbHeight, which defines the height of the current coding sub-block, - the motion vector offset mvOffset, - fine-tune motion vector refMvLX, - the selected reference picture sample array refPicLX, - the bidirectional optical flow flag bdofFlag, - A variable cIdx that specifies the color component index of the current block. The output of this process is: - predSamplesLX, a (sbWidth+bdofOffset)×(sbHeight+bdofOffset) array of predicted sample values. The bidirectional optical flow boundary offset bdofOffset is derived as follows. bdofOffset=bdofFlag ? 2:0 (8-811) - If cIdx is equal to 0, the following applies: - Let (xIntL, yIntL) be the luma position given in full sample units, and (xFracL, yFracL) be the offset in 1 / 16 sample units. These variables are used in this section only to specify a fractional sample position within the reference sample array refPicLX. - For each luminance sample position (x L =0..sbWidth-1+bdofOffset,y L For each of the predicted luminance samples (x = 0..sbHeight-1 + bdofOffset), L ][y L ] is derived as follows: - Variable xInt L , yInt L , xFrac L , yFrac L is derived as follows: xInt L =xSb+(refMvLX[0]>>4)+x L (8-812) yInt L =ySb+(refMvLX[1]>>4)+y L (8-813) xFrac L =refMvLX[0]&15 (8-814) yFrac L =refMvLX[1]&15 (8-815) - if bdofFlag is TRUE and one or more of the following conditions are true, the predicted luminance sample values ​​predSamplesLX[x L ][y L ] is (xInt L ,yInt L ), (xFrac L ,yFrac L ) and refPicLX as input and invoke the luminance integer sample fetch process as specified in Section 8.5.7.3.3. - bdofFlag is TRUE. -xL is equal to 0 -x L is equal to sbWidth+1 -y L is equal to 0 -y L is equal to sbHeight+1 Otherwise the following applies: - The motion vector mvLX is set equal to (refMvLX-mvOffset). The predicted luma sample values ​​predSamplesLX[xL][yL] are derived by invoking the luma sample 8-tap interpolation filtering process as specified in Section 8.5.7.3.2 using (xIntL, yIntL), (xFracL, yFracL), refPicLX, and padVal as input. ... Luminance integer sample fetch process The inputs to this process are: - Full sample unit (xInt L ,yInt L ) Luminance position. - Luminance reference sample array refPicLX L . The output of this process is the predicted luminance sample value predSampleLX. L It is. This variable shift is Max(2,14-BitDepth Y ). The variable picW is set equal to pic_width_in_luma_samples, and the variable picH is set equal to pic_height_in_luma_samples. The luminance position in full sample units (xInt, yInt) is derived as follows: xInt=Clip3(0,picW-1,sps_ref_wraparound_enabled_flag ? (8-838) ClipH((sps_ref_wraparound_offset_minus1+1)*MinCbSizeY,picW,xInt L ):xInt L ) yInt = Clip3(0,picH-1,yInt L ) (8-839) Predicted luminance sample value predSampleLX L is derived as follows: predSampleLX L =refPicLX L [xInt][yInt]< <shift3 (8-840) Bidirectional optical flow prediction processing The inputs to this process are: - two variables nCbW and nCbH that define the width and height of the current coding block; - two (nCbW+2)×(nCbH+2) luma prediction sample arrays predSamplesL0 and predSamplesL1. - Prediction list usage flags predFlagL0 and predFlagL1. - Reference indices refIdxL0 and refIdxL1. - Bidirectional optical flow utilization flag bdofUtilizationFlag[xIdx][yIdx] for xIdx=0..(nCbW>>2)-1, yIdx=0..(nCbH>>2)-1. The output of this process is a (nCbW) by (nCbH) array of luma predicted sample values, pbSamples. The variables bitDepth, shift1, shift2, shift3, shift4, offset4, and mvRefineThres are derived as follows: - The variable bitDepth is BitDepth Y is set equal to - The variable shift1 is set equal to Max(2,14-bitDepth). - The variable shift2 is set equal to Max(8,bitDepth-4). - The variable shift3 is set equal to Max(5,bitDepth-7). - The variable shift4 is set equal to Max(3,15-bitDepth) and the variable offset4 is set equal to 1<<(shift4-1). - The variable mvRefineThres is set equal to Max(2,1<<(13-bitDepth)). For xIdx=0..(nCbW>>2)-1, yIdx=0..(nCbH>>2)-1 the following applies: - Set variable xSb equal to (xIdx<<2)+1, and set variable ySb equal to (yIdx<<2)+1. - If bdofUtilizationFlag[xSbIdx][yIdx] is FALSE, for x=xSb-1..xSb+2,y=ySb-1..ySb+2, the predicted sample value of the current sub-block is derived as follows: pbSamples[x][y]=Clip3(0,(2 bitDepth )-1(8-852),(predSamplesL0[x+1][y+1]+offset2+ predSamplesL1[x+1][y+1])>>shift2) Otherwise (bdofUtilizationFlag[xSbIdx][yIdx] is TRUE), the predicted sample value of the current sub-block is derived as follows: - for x=xSb-1..xSb+4, y=ySb-1..ySb+4, the following ordered steps apply: 1. For each location (h x ,v y ) is derived as follows: h x =Clip3(1,nCbW,x) (8-853) v y =Clip3(1,nCbH,y) (8-854) 2. The variables gradientHL0[x][y], gradientVL0[x][y], gradientHL1[x][y], gradientVL1[x][y] are derived as follows: gradientHL0[x][y]=(predSamplesL0[h x +1][v y ]-predSampleL0[h x -1][v y ])>>shift1 (8-855) gradientVL0[x][y]=(predSampleL0[h x ][v y +1]-predSampleL0[h x ][v y -1])>>shift1 (8-856) gradientHL1[x][y]=(predSamplesL1[h x +1][v y ]-predSampleL1[h x -1][v y ])>>shift1 (8-857) gradientVL1[x][y]=(predSampleL1[h x ][v y +1]-predSampleL1[h x ][v y -1])>>shift1 (8-858) 3. The variables temp[x][y], temppH[x][y], and tempV[x][y] are derived as follows: diff[x][y]=(predSamplesL0[h x ][v y ]>>shift2)-(predSamplesL1[h x ][v y ]>>shift2) (8-859) tempH[x][y]=(gradientHL0[x][y]+gradientHL1[x][y])>>shift3 (8-860) tempV[x][y]=(gradientVL0[x][y]+gradientVL1[x][y])>>shift3 (8-861) The variables sGx2, sGy2, sGxGy, sGxdI, sGydI are derived as follows: sGx2=Σ i Σ j (tempH[xSb+i][ySb+j]*tempH[xSb+i][ySb+j])with i,j=-1..4 (8-862) sGy2=Σ i Σ j (tempV[xSb+i][ySb+j]*tempV[xSb+i][ySb+j])with i,j=-1..4 (8-863) sGxGy=Σ i Σ j (tempH[xSb+i][ySb+j]*tempV[xSb+i][ySb+j])with i,j-1..4 (8-864) sGxdI=Σ i Σ j (-tempH[xSb+i][ySb+j]*diff[xSb+i][ySb+j])with i,j=-1..4 (8-865) sGydI=Σ i Σ j (-tempV[xSb+i][ySb+j]*diff[xSb+i][ySb+j])with i,j=-1..4 (8-866) - The horizontal and vertical motion offsets for the current sub-block are derived as follows: v x =sGx2>0?Clip3(-mvRefineThres,mvRefineThres, (8-867) -(sGxdI<<3)>>Floor(Log2(sGx2))):0 v y =sGy2>0?Clip3(-mvRefineThres,mvRefineThres,((sGydI<<3)- (8-868) ((v x *sGxGy m )<<12+vx *sGxGy s )>>1)>>Floor(Log2(sGx2))):0 - For x=xSb-1..xSb+2, y=ySb-1..ySb+2, the predicted sample values ​​of the current sub-block are derived as follows: bdofOffset = Round((v x *(gradientHL1[x+1][y+1]-gradientHL0[x+1][y+1]))>>1) (8-869) +Round((v y *(gradientVL1[x+1][y+1]-gradientVL0[x+1][y+1]))>>1) [Define Ed.(JC):Round() operation for float inputs. The Round() operation seems redundant here since the inputs are integer values. Recommenders should confirm.] pbSamples[x][y]=Clip3(0,(2 bitDepth )-1(8-870),(predSamplesL0[x+1][y+1]+offset4+ predSamplesL1[x+1][y+1]+bdofOffset)>>shift4)

[0222] 2.10 Decoder-side Motion Vector Refinement (DMVR) Example In the bi-prediction operation, two prediction blocks are combined to form one prediction signal, which are constructed by using the motion vector (MV) in list0 and the MV in list1, respectively, to predict one block region. In the decoder-side motion vector refinement (DMVR) method, the two motion vectors of bi-prediction are further refined by a bilateral template matching process. To obtain the refined MV without transmitting additional motion information, bilateral template matching is applied in the decoder to perform a distortion-based search between the bilateral template and the reconstructed sample in the reference picture.

[0223] In DMVR, we generate a bilateral template from the first MV0 in list0 and MV1 in list1 as a weighted combination (i.e., average) of two prediction blocks, respectively, as shown in Fig. 24. The template matching operation consists of calculating a cost measure between the generated template and a sample region in the reference picture (near the first prediction block). For each of the two reference pictures, the MV with the smallest template cost is considered as the updated MV of that list and replaces the original MV. In JEM, we search for nine MV candidates for each list. The nine MV candidates include the original MV and eight surrounding MVs with one luma sample offset with respect to the original MV in either horizontal or vertical direction or both directions. Finally, we use two new MVs, namely MV0' and MV1' as shown in Fig. 24, to generate the final bidirectional prediction result. We use the sum of absolute differences (SAD) as the cost measure. Note that when calculating the cost of a prediction block generated by one surrounding MV, we use the rounded MV (integer pixels) instead of the actual MV to obtain the prediction block.

[0224] DMVR is applied to the bi-prediction merge mode between one MV from a past reference picture and one MV from a future reference picture without transmitting additional syntax elements. In JEM, DMVR is not applied when LIC, affine motion, FRUC, or sub-CU merge candidates are enabled for a CU.

[0225] 2.11 JVET-N0236 In this contribution, we propose a method to fine-tune sub-block-based affine motion compensation prediction using optical flow. After performing sub-block-based affine motion compensation, we fine-tune the predicted samples by adding the difference derived from the optical flow equation, which we call prediction fine-tuning with optical flow (PROF). The proposed method can realize inter prediction at pixel-level granularity without increasing memory access bandwidth.

[0226] To achieve finer granularity in motion compensation, this contribution proposes a method to fine-tune sub-block-based affine motion compensation prediction using optical flow. After performing sub-block-based affine motion compensation, we fine-tune the luma prediction samples by adding the difference derived from the optical flow equation. The proposed prediction fine-tuning with optical flow (PROF) is explained in four steps: Step 1) Perform sub-block based affine motion compensation to generate the sub-block prediction I(i,j). Step 2) At each sample location, we use a 3-tap filter [-1,0,1] to find the spatial gradient of the subblock prediction g x (i, j) and g y Calculate (i,j). g x (i,j) = I(i+1,j) - I(i-1,j) g y (i,j)=I(i,j+1)-I(i,j-1) The subblock prediction is expanded by one pixel on each side for gradient calculation. To reduce memory bandwidth and complexity, the pixels on the expanded boundary are copied from the nearest integer pixel location in the reference picture. Thus, additional interpolation for the padding area is avoided. Step 3) Calculate the fine adjustment to the brightness prediction (call it ΔI) via the optical flow equation. ΔI(i,j)=g x (i,j)*Δv x (i,j)+g x (i,j)*Δv x (i,j) Here, delta MV (denoted as Δv(i,j)) is the difference between the pixel MV calculated for sample position (i,j), represented by v(i,j), as shown in FIG. 25, and the sub-block MV of the sub-block MV to which pixel (i,j) belongs. Since the affine model parameters and pixel location relative to the subblock center do not change from subblock to subblock, Δv(i,j) can be calculated for the first subblock and reused for other subblocks in the same CU. If the horizontal and vertical offsets from the pixel location to the center of the subblock are x, y, Δv(x,y) can be derived as follows:

[0227]

number

[0228] For a four-parameter affine model,

[0229]

number

[0230] For a six-parameter affine model,

[0231]

number

[0232] Here, (v 0x ,v 0y ), (v 1x ,v 1y ), (v 2x ,v 2y ) are the top-left, top-right, and bottom-left control point motion vectors, and w, h are the width and height of the CU. Step 4) Finally, the refinement of the luma prediction is added to the sub-block prediction I(i,j). The final prediction I' is generated according to the following equation: I'(i,j) = I(i,j) - ΔI(i,j) Details of JVET-N0236 a) How to derive the slope of PROF In JVET-N0263, we compute the gradient of each subblock (4x4 subblocks in VTM-4.0) in each reference list. For each subblock, we take the closest integer sample of the reference block and fill the four lateral borders of the sample (black circles in Figure 23). Let the MV of the current subblock be (MVx,MVy). Then, calculate its fractional part as (FracX,FracY)=(MVx&15,MVy&15). The integer part is calculated as (IntX,IntY)=(MVx>>4,MVy>>>4). The offset (OffsetX,OffsetY) is derived as follows: OffsetX=FracX>7 ?1:0; OffsetY=FracY>7 ?1:0; Let the top left coordinate of the current sub-block be (xCur, yCur) and the dimensions of the current sub-block be W×H. Next, calculate (xCor0,yCor0), (xCor1,yCor1), (xCor2,yCor2), and (xCor3,yCor3) as follows: (xCor0,yCor0)=(xCur+IntX+OffsetX-1,yCur+IntY+OffsetY-1); (xCor1,yCor1)=(xCur+IntX+OffsetX-1,yCur+IntY+OffsetY+H); (xCor2,yCor2)=(xCur+IntX+OffsetX-1,yCur+IntY+OffsetY); (xCor3,yCor3)=(xCur+IntX+OffsetX+W,yCur+IntY+OffsetY); Let PredSample[x][y], with x=0..W-1, y=0..H-1, store the predicted samples for a sub-block. Then, derive the padding samples as follows: PredSample[x][-1]=(Ref(xCor0+x,yCor0)< <Shift0)-Rounding,for x=-1..W; PredSample[x][H]=(Ref(xCor1+x,yCor1)< <Shift0)-Rounding,for x=-1..W; PredSample[-1][y]=(Ref(xCor2,yCor2+y)< <Shift0)-Rounding,for y=0..H-1; PredSample[W][y]=(Ref(xCor3,yCor3+y)< <Shift0)-Rounding,for y=0..H-1; Here, Rec represents the reference picture. The rounding is an integer, and in the example PROF implementation, it is 2 13 Shift0=Max(2,(14-BitDepth)); PROF attempts to improve the accuracy of the gradients, unlike BIO in VTM-4.0, where the gradients are output with the same accuracy as the input luminance samples. The gradient in PROF is calculated as follows: Shift1=Shift0-4. gradientH[x][y]=(predSamples[x+1][y]-predSample[x-1][y])>>Shift1 gradientV[x][y]=(predSample[x][y+1]-predSample[x][y-1])>>Shift1 Note that predSamples[x][y] maintains the accuracy after interpolation. b) How to derive Δv in PROF The derivation of Δv (expressed as dMvH[posX][posY] and dMvV[posX][posY], where posX=0..W-1, posY=0..H-1) can be explained as follows. Let the dimensions of the current block be cbWidth×cbHeight, the number of control point motion vectors be numCpMv, and the control point motion vector be cpMvLX[cpIdx], where cpIdx=0..numCpMv−1, and X represents the two reference lists and is 0 or 1. The variables log2CbW and log2CbH are derived as follows: log2CbW = Log2(cbWidth) log2CbH = Log2(cbHeight) The variables mvScaleHor, mvScaleVer, dHorX, and dVerX are derived as follows: mvScaleHor=cpMvLX[0][0]<<7 mvScaleVer=cpMvLX[0][1]<<7 dHorX=(cpMvLX[1][0]-cpMvLX[0][0])<<(7-log2CbW) dVerX=(cpMvLX[1][1]-cpMvLX[0][1])<<(7-log2CbW) The variables dHorY and dVerY are derived as follows. - if numCpMv is equal to 3, the following applies: dHorY=(cpMvLX[2][0]-cpMvLX[0][0])<<(7-log2CbH) dVerY=(cpMvLX[2][1]-cpMvLX[0][1])<<(7-log2CbH) - Otherwise (numCpMv equals 2), the following applies: dHorY=-dVerX dVerY=dHorX The variables qHorX, qVerX, qHorY, and qVerY are derived as follows. qHorX = dHorX << 2; qVerX = dVerX << 2; qHorY = dHorY << 2; qVerY=dVerY<<2; dMvH[0][0] and dMvV[0][0] are calculated as follows: dMvH[0][0]=((dHorX+dHorY)<<1)-((qHorX+qHorY)<<1); dMvV[0][0]=((dVerX+dVerY)<<1)-((qVerX+qVerY)<<1); For xPos ranging from 1 to W-1, dMvH[xPos][0] and dMvV[xPos][0] are derived as follows. dMvH[xPos][0]=dMvH[xPos-1][0]+qHorX; dMvV[xPos][0]=dMvV[xPos-1][0]+qVerX; For yPos from 1 to H-1, the following applies: dMvH[xPos][yPos]=dMvH[xPos][yPos-1]+qHorY with xPos=0..W-1 dMvV[xPos][yPos]=dMvV[xPos][yPos-1]+qVerY with xPos=0..W-1 Finally, dMvH[xPos][yPos] and dMvV[xPos][yPos] for posX=0..W-1 and posY=0..H-1 are right-shifted as follows: dMvH[xPos][yPos]=SatShift(dMvH[xPos][yPos],7+2-1); dMvV[xPos][yPos]=SatShift(dMvV[xPos][yPos],7+2-1); Here, SatShift(x,n) and Shift(x,n) are defined as follows:

[0233]

number

[0234] Shift(x,n)=(x+offset0)>>n In one example, offset0 and / or offset1 are (1<<<n)> >1. c) How to derive ΔI for PROF For a position (posX,posY) in a subblock, its corresponding Δv(i,j) is denoted as (dMvH[posX][posY],dMvV[posX][posY]). Its corresponding gradient is denoted as (gradientH[posX][posY],gradientV[posX][posY]). Then, ΔI(posX, posY) is derived as follows: (dMvH[posX][posY],dMvV[posX][posY]) are clipped as follows: dMvH[posX][posY]=Clip3(-32768,32767,dMvH[posX][posY]); dMvV[posX][posY]=Clip3(-32768,32767,dMvV[posX][posY]); ΔI(posX,posY)=dMvH[posX][posY]×gradientH[posX][posY]+dMvV[posX][posY]×gradientV[posX][posY]; ΔI(posX,posY)=Shift(ΔI(posX,posY),1+1+4); ΔI(posX,posY)=Clip3(-(2 13 -1),2 13 -1,ΔI(posX,posY)); d) How to derive I' for PROF If the current block is not coded as bi-predictive or weighted prediction, I'(posX,posY)=Shift((I(posX,posY)+ΔI(posX,posY)),Shift0), I'(posX,posY)=ClipSample(I'(posX,posY)), Here, ClipSample clips the sample values ​​to valid output sample values. Then, I'(posX,posY) is output as an inter predicted value. Otherwise (the current block is coded as bi-predictive or weighted prediction). I'(posX,posY) is stored and used to generate an inter prediction value according to other prediction values ​​and / or weighting values.

[0235] 2.12 JVET-N0510 In JVET-N0510, a phase-variant affine sub-block motion compensation (MC) is proposed, which applies a traditional two-stage horizontal-vertical interpolation. However, unlike the MC based on phase-invariant blocks, which uses the same horizontal filter for all sample rows and the same vertical filter for all sample columns, different phases of the filter can be applied to different sample rows and different sample columns in the affine sub-block.

[0236] To better approximate the affine motion model in an affine sub-block, we apply phase-variable MC to this sub-block. In the proposed method, the affine coding block is also divided into 4×4 sub-blocks, and one sub-block MV is derived for each sub-block in the same way as in VTM4.0. The MC for each sub-block is divided into two stages. The first stage is to filter the (4+L-1)×(4+L-1) reference block windows with (4+L-1) rows of horizontal filtering, where L is the filter tap length of the interpolation filtering module. However, unlike the translational MC, in the proposed phase-variable affine sub-block MC, the filter phase of each sample row is different. For each sample row, the MVx is derived as follows:

[0237] MVx=(subblockMVx<<7+dMvVerX×(rowIdx-L / 2-2))>>7

[0238] The filter phase for each sample row is derived from MVx. subblockMVx is the x-component of the MV of the derived subblock MV as described in VTM4.0. rowIdx is the sample row index. dMvVerX is (cuBottomLeftCPMVx-cuTopLeftCPMVx)<<(7-log2LumaCbHeight), where cuBottomLeftCPMVx is the x-component of the CU bottom-left control point MV, cuTopLeftCPMVx is the x-component of the CU top-left control point MV, and LumaCbHeight is the log2 of the height of the luma coding block (CB).

[0239] After horizontal filtering, it produces 4×(4+L-1) horizontally filtered samples. Figure 26 shows the proposed horizontal filtering concept. The grey dots are the samples of the reference block window, and the orange dots show the horizontally filtered samples. The blue tube of 8×1 samples means applying 8-tap horizontal filtering once, as shown in Figure 26 and Figure 27 respectively. Each sample row needs 4 horizontal filterings. The filter phase in one sample row is the same. However, the filter phase in different rows is different. The skewed 4×11 samples are produced.

[0240] In the second stage, the 4×(4+L−1) horizontally filtered samples (orange samples in FIG. 26) are further filtered vertically. For each sample column, MVy is derived as follows:

[0241] MVy=(subblockMVy<<7+dMvHorY×(columnIdx-2))>>7 (Formula 2)

[0242] The filter phase for each sample column is derived from MVy. subblockMVy is the y component of the MV of the derived subblock MV, as was done in VTM4.0. columnIdx is the sample column index. dMvHorY is (cuTopRightCPMVy-cuTopLeftCPMVy)<<(7-log2LumaCbWidth), where cuTopRightCPMVy is the y component of the CU top right control point MV, cuTopLeftCPMVy is the y component of the CU top left control point MV, and log2LumaCbWidth is the log2 of the width of the luma CB.

[0243] After vertical filtering, we generate 4x4 affine subblock prediction samples. Figure 28 illustrates the proposed vertical filtering concept. The light orange dots are the horizontally filtered samples from the first stage. The red dots are the vertically filtered samples as the final prediction samples.

[0244] In this proposal, the interpolation filter set used is the same as that in VTM4.0. The only difference is that the horizontal filter phase in one sample row is different and the vertical filter phase in one sample column is different. The number of filtering operations for each affine subblock in the proposed method is the same as that in VTM4.0.

[0245] 3. Shortcomings of existing implementations Some existing implementations have the following drawbacks:

[0246] (1) The slope calculation method is not the same for BDOF and PROF.

[0247] (a) In BDOF, we compute the gradient for the whole block and pad once. In PROF, we compute the gradient for each subblock and pad N times (assuming there are N subblocks).

[0248] (b) PROF requires higher gradient accuracy than BDOF.

[0249] (2) The interaction between PROF and other tools is unclear.

[0250] (3) It is unclear how PROF is applied to the chroma component.

[0251] (4) The process of deriving Δv may be incorrect.

[0252] (5) PROF may be performed conditionally for higher coding performance.

[0253] (6) It is unclear how the methods of JVET-N0236 and JVET-N0510 will be combined.

[0254] (7) The bit width of dMvH and dMvV may be too large.

[0255] 4. Exemplary method for predictive fine-tuning with optical flow (PROF) The embodiments of the disclosed technology overcome the shortcomings of existing implementations, thereby providing video coding with higher coding efficiency. The method of optical flow-based prediction fine-tuning based on the disclosed technology can improve both existing and future video coding standards, and is elucidated in the following examples for various implementations. The examples of the disclosed technology provided below are intended to illustrate the general concept and should not be construed as limiting. In an example, various features described in these examples can be combined, unless expressly indicated to the contrary.

[0256] Denote the reference pictures of the current image from list0 and list1 by Ref0 and Ref1, respectively, denote τ0=POC(current)-POC(Ref0), τ1=POC(Ref1)-POC(current), and denote the reference blocks of the current block from Ref0 and Ref1 by refblk0 and refblk1, respectively. For one sub-block in the current block, the MV of its corresponding sub-block in refblk0 that points to refblk1 is (v x ,v y The MVs of the sub-blocks in Ref0 and Ref1 are expressed as (mvL0 x ,mvL0 y ), (mvL1 x ,mvL1 y ) is expressed as

[0257] Shift(x,s) is defined as Shift(x,s)=(x+off)>>s.

[0258] SignShift(x,s) is defined as:

[0259]

number

[0260] In one example, offset0 and / or offset1 are (1<<n)> >1 or (1<<(n-1)). In another example, offset0 and / or offset1 are set to 0. In yet another example, offset0=offset1=((1<<n)> >1)-1 or ((1<<(n-1)))-1.

[0261] Clip3(x,min,max) is defined as follows:

[0262]

number

[0263] where Max(a,b)=a>=b ? a:b and Min(a,b)=a<=b ? a:b.

[0264] In the following description, an operation between two motion vectors means that the operation is applied to both components of the motion vector. For example, MV3=MV1+MV2 means MV3 x =MV1 x +MV2 x and MV3 y =MV1 y +MV2 y Alternatively, this operation may be applied only to the horizontal or vertical components of the two motion vectors. The term "absolute value" of MV(MVx,MVy) may also be expressed as abs(MVx), or abs(MVy), or max(abs(MVx), abs(MVy)) or abs(MVx)+abs(MVy), where the function abs(x) returns the absolute value of x and the function max(x,y) returns the greater of x and y.

[0265] In the following description, the left neighboring block, the lower left neighboring block, the upper neighboring block, the upper right neighboring block, and the upper left neighboring block are denoted as blocks A1, A0, B1, B0, and B2, as shown in FIG. 1. It is proposed that the gradient computation in PROF may be performed at an MxN region level, which is different from the sub-block size used for motion compensation in affine mode. In one example, the gradient calculation in PROF may be performed over an M×N region that is larger than one sub-block. b. In one example, M and N may be some predefined numbers, such as M=N=8 or M=N=16. c. In one example, M and N are defined based on the width / height of the sub-block size, and may be any number, for example, M=N=2*Wmc, where Wmc is the width / height of the sub-block size used for motion compensation. d. The padding process used to derive the gradients in PROF is done at the MxN region level. e. In all the above examples, M and N are defined as follows: i. In one example, M=min(K0, block width), where K0 is an integer value. ii. In one example, N=min(K1, the height of the block), where K0 is an integer value. iii. In the above example, K0=K1=16. iv. In one example, K0 and K1 are aligned with those used for BDOF. f. The gradient of a first sample in the first sub-block may be derived by a second sample in the second sub-block. i. In one example, the second sub-block is adjacent to the first sub-block. ii. In one example, the second sample is used to derive the gradient of the first sample, whether it is in the first sub-block or the second sub-block. iii. The above method may be applied when M×N is larger than a sub-block. g. For the padding process of each M×N region, one or more MVs may be derived. In one example, one specific MV is derived for the padding process for an M×N region. Integer reference samples may be located at the specific MV and then used to pad samples outside the M×N region. (i) In one example, the specific MV may be one MV of one subblock in an M×N region, such as the upper left corner subblock or the central subblock in the M×N region. An example is shown in Figure 31. The MVs of subblocks A, B, C, D, and E may be selected as the specific MV. (ii) In one example, a particular MV may be derived from an affine model towards a particular location (eg, center) of the M×N domain. (iii) In one example, a particular MV may be derived from the MVs of sub-blocks in an M×N region. For example, a particular MV may be derived as the average of the MVs of all sub-blocks in an M×N region. b. For example, a particular MV may be derived as an average of multiple MVs of the central sub-blocks. i. For example, a particular MV may be derived as the average value of multiple MVs B, C, D, and E in FIG. ii. For example, a particular MV may be derived as the average value of multiple MVs at B and E in FIG. iii. For example, a particular MV may be derived as the average value of multiple MVs at C and D in FIG. c. For example, a particular MV may be derived as a function of multiple MVs (eg, CPMVs or MVs of sub-blocks). In one example, multiple MVs are derived for padding operations for an M×N region, and an integer reference sample may be placed in one of the multiple MVs and then used to pad samples outside the M×N region. (i) In one example, when padding a first sample into a first sub-block of an M×N region, the first MV of the first sub-block may be used to locate an integer reference sample for padding the first sample. iii. The above method applies when M×N is larger than one sub-block, and the padding process used to derive the gradients in PROF is done for each M×N region. 2. Gradient calculations in PROF / BIO may be performed at M×N region levels, where M / N may be adaptively changed. In one example, M and N may depend on the dimensions W×H of the current block. i. For example, this region may be the entire current block, i.e., M=W, N=H. ii. For example, M=W / T1, N=H / T2, where T1, T2 are integers, e.g., T1=T2=2. iii. In one example, M and / or N may be signaled from the encoder to the decoder in the VPS / DPS / SPS / PPS / APS / slice header / tile group header / tile / CTU / CU, etc. (i) Alternatively, M and / or N may be specified in a profile / level / layer of a video coding standard. iv. In one example, M=Min(W,T1) and N=Min(H,T2). For example, T1=T2=16. (i) In one example, T1 and / or T2 may be signaled from the encoder to the decoder, such as in the VPS / DPS / SPS / PPS / APS / slice header / tile group header / tile / CTU / CU. (ii) Alternatively, T1 and / or T2 may be specified in a profile / level / layer of a video coding standard. 3. The following may further apply to the above method: In one example, M is at least equal to Mmin and N is at least equal to Nmin, for example, Mmin=Nmin=8. b. In one example, the padding process is performed once for each M×N region to obtain a padded (M+dM)×(N+dN) region, e.g., dM=dN=2. i. In one example, samples within the regions (eg, open circles in FIG. 23) may be derived from motion compensation by interpolation filtering. (i) In one example, the samples within the region may be derived from motion compensation of multiple sub-blocks in the region. ii. In one example, the four lateral outer lines of the sample (eg, the black circles in FIG. 23) may be padded. (i) In one example, the padded sample may copy the intensity of the nearest integer sample in the reference block. (ii) In one example, a padded sample may copy the intensity of the nearest sample in the unpadded region. 4. For each region where the gradient calculation in PROF / BIO is applied, instead of calculating the gradient value for each sample, it is proposed to calculate the gradient based on a subset of samples. a. In one example, in PROF / BIO, the gradient associated with the sample at a given coordinate may be used, e.g., (2x,y) or (x,2y) or (2x+1,2y+1) or (2x,2y), where (m,n) are the coordinates for the top-left sample in the current block. b. In one example, the samples may first be modified (eg, downsampled) and the modified samples may be used to derive the gradient. 5. It is proposed that the accuracy of the gradient values ​​calculated in BDOF and PROF may be the same. In one example, the sample differences may be shifted by the same value. i. In one example, the horizontal and / or vertical gradients (denoted as gradientH, gradientV, respectively) may be calculated as follows: gradientH[x][y]=(predSamples[x+1][y]-predSample[x-1][y])>>Shift0 gradientV[x][y]=(predSample[x][y+1]-predSample[x][y-1])>>Shift1 or, gradientH[x][y]=Shift((predSamples[x+1][y]-predSample[x-1][y]),Shift0) gradientV[x][y]=Shift((predSample[x][y+1]-predSample[x][y-1]),Shift1) or, gradientH[x][y]=SatShift((predSamples[x+1][y]-predSample[x-1][y]),Shift0) gradientV[x][y]=SatShift((predSample[x][y+1]-predSample[x][y-1]),Shift1) ii. In one example, the horizontal and / or vertical gradients (denoted as gradientH, gradientV, respectively) may be calculated as follows: gradientH[x][y]=(predSamples[x][y]*2-predSamples[x+1][y]-predSample[x-1][y])>>Shift0 gradientV[x][y]=(predSamples[x][y]*2-predSample[x][y+1]-predSample[x][y-1])>>Shift1 or, gradientH[x][y]=Shift((predSamples[x][y]*2-predSamples[x+1][y]-predSample[x-1][y]),Shift0) gradientV[x][y]=Shift((predSamples[x][y]*2-predSample[x][y+1]-predSample[x][y-1]),Shift1) or, gradientH[x][y]=SatShift((predSamples[x][y]*2-predSamples[x+1][y]-predSample[x-1][y]),Shift0) gradientV[x][y]=SatShift((predSamples[x][y]*2-predSample[x][y+1]-predSample[x][y-1]),Shift1) iii. In one example, Shift0 and / or Shift1 may be set to Max(2,(14-BitDepth)), where BitDepth is the bit depth of the reconstructed samples / input samples. 6. The following method for padding the outer lines of samples (called padding samples, such as black circles in FIG. 23) may be applied to PROF, BIO, or both PROF and BIO. The padding samples may be padded in the same manner as PROF and / or BIO. "Same manner" may be any padding method disclosed below. b. In one example, the padding samples may be derived (eg, copied) from integer samples in a reference picture for PROF and / or BIO. i. In one example, the integer samples used to derive the padding samples may be positioned by adding an MV according to the position of the padding sample and rounded to an integer MV in the addition operation. (i) In one example, MV(MvX, MvY) may be rounded to a floor integer MV(IntX, IntY). For example, IntX=MvX>>P and IntY=MvY>>P, where P is the MV precision. (ii) In one example, MV(MvX, MvY) may be rounded to the nearest integer MV(IntX, IntY). For example, if P is the precision of MV, then FracX=MvX&((1< <P)-1),FracY=MvY & ((1<<P)-1),OffX=(FracX> Setting HalfFrac=(1<<(P-1))) ? 1:0,OffY=(FracY>=(1<<(P-1))) ? 1:0 results in IntX=(MvX>>P)+OffX,IntY=(MvY>>P)+OffY. HalfFrac may be equal to 1<<(P-1), or in other examples, (1<<(P-1))-1 or (1<<(P-1))+1. (iii) In one example, where P is the MV precision, MV(MvX, MvY) may be rounded to an integer MV(IntX, IntY), where IntX = SatShift(MvX, P) and IntY = SatShift(MvY, P). (iv) In the above black circle, the MV precision P may depend on the color format and / or color components. For example, the MV precision of the Cb / Cr components may be equal to the MV precision of the luma component plus K in the 4:2:0 color format. For example, K may be equal to 1. (v) The method of padding may be signaled from the encoder to the decoder in the VPS / DPS / SPS / PPS / APS / slice header / tile group header / tile / CTU / CU, etc. Alternatively, the method of padding may be specified in the profile / level / layer of the video coding standard. (vi) The manner in which padding is done may depend on the block dimensions. 7. If PROF is applied, it is suggested that coding tool X cannot be applied. Alternatively, if encoding tool X is applied, then PROF cannot be applied. b. In one example, if encoding tool X cannot be applied, then a syntax element to indicate encoding tool X may not be signaled. c. In one example, the encoding tool X may be a Generalized Bi-Prediction (GBI). i. For example, if GbiIdx is not equal to 0, then PROF does not apply. ii. Alternatively, if PROF applies, GbiIdx shall be 0. iii. Alternatively, if PROF applies, GbiIdx is not signaled and is inferred to be 0. iv. Alternatively, if PROF applies, then GBI does not apply, whether GbiIdx is 0 or not. d. In one example, coding tool X may be local illumination compensation. e. In one example, the encoding tool X may be a Multiple Transform Set (MTS). For example, when applying PROF, only the default transformations may be applied. (i) For example, if PROF applies, syntax elements related to MTS do not apply. f. In one example, coding tool X may be a weighted prediction. i. For example, when applying unequal weights and / or unequal offsets with weighted prediction to a block, PROF is not applied. 8. It is proposed that the method of application of PROF may depend on the color format and / or the use of separate plane coding. In one example, if the color format is 4:0:0, PROF cannot be applied to the chroma component. b. In one example, if the color format is 4:4:4, PROF may be applied to the chroma component. c. In one example, PROF may be applied to the chroma component if the color format is not equal to 4:0:0. d. In one example, the method of deriving delta MV (eg, Δv in section 2.11) may depend on the color format. 9. It is proposed that the application method of PROF may depend on the color components. In one example, the gradient may be calculated for each color component independently. i. Alternatively, the gradient calculated for the first color component may be used for the second color component. ii. Alternatively, the gradient may be calculated twice, once for the luma / dominant color component and once for the two chroma / dependent color components. b. In one example, delta MV (eg, Δv in section 2.11) may be calculated independently for each color component. i. Alternatively, the delta MV calculated for the first color component may be used for the second color component. c. In one example, a predicted refinement (eg, ΔI in section 2.11) may be calculated for each color component independently. i. Alternatively, the predicted refinement (eg, ΔI in section 2.11) calculated for the first color component may be used by the second color component. d. In one example, the accuracy of the gradient in PROF may depend on the color component. e. In one example, the accuracy of delta MV in PROF (eg, Δv in section 2.11) may be color component dependent. f. In one example, whether and how clipping operations are performed in PROF may depend on the color components. g. In one example, whether and how a shift operation occurs in PROF may depend on the color component. h. In one example, PROF may be applied to only the luminance component. i. In one example, PROF may be applied to different color components with different sub-block sizes. i. Alternatively, PROF may be applied to different color components of the same sub-block size. j. In one example, PROF may be applied to chroma components of M*N sub-block sizes. i. For example, set M and N equal to 4. k. The above method (hj) may further depend on the color format (e.g., 4:2:0 or 4:4:4). 10. It is proposed that the derivation of delta MV (eg, Δv in section 2.11) may depend on the width and / or height of the sub-block. In one example, dMvH[0][0] and dMvV[0][0] are calculated as follows: qHorX = dHorX * P0; qVerX = dVerX * P0; qHorY = dHorY * P0; qVerY = dVerY * P0; dMvH[0][0]=((iDMvHorX+iDMvVerX)*P1)-(quadHorX*(blockWidth>>1)+quadVerX*(blockHeight*P1)); dMvV[0][0]=((iDMvHorY+iDMvVerY)*P1)-(quadHorY*(blockWidth>>1)+quadVerY*(blockHeight*P1)); Here, blockWidth and blockHeight represent the width and height of the sub-block respectively. P0 and P1 are two numbers that control the precision. i. For example, when P0 = 4 and P1 = 2, dMvH[0][0] and dMvV[0][0] are calculated as follows. qHorX=dHorX<<2; qVerX=dVerX<<2; qHorY=dHorY<<2; qVerY=dVerY<<2; dMvH[0][0]=((iDMvHorX+iDMvVerX)<<1)-(quadHorX*(blockWidth>>1)+quadVerX*(blockHeight>>1)); dMvV[0][0]=((iDMvHorY+iDMvVerY)<<1)-(quadHorY*(blockWidth>>1)+quadVerY*(blockHeight>>1)); 11. It is proposed that for the affine-encoded block, instead of always applying PROF, it may be conditionally performed. a. In one example, the presence or absence of the execution of PROF and its method may depend on the dimensions W×H of the current block. i. For example, PROF may not be applied when W <= T1 and / or H <= T2, for example, T1 = T2 = 16. ii. For example, PROF may not be applied when W < T1 and / or H < T2, for example, T1 = T2 = 16. iii. For example, when W >= T1 and / or H >= T2, for example, when T1 = T2 = 64, PROF may not need to be applied. iv. For example, when W > T1 and / or H > T2, for example, when T1 = T2 = 64, it may not need to be applied. v. For example, when W * H > T1, for example, when T1 = 64 * 64, PROF may not need to be applied. vi. For example, when W * H >= T1, for example, when T1 = 64 * 64, PROF may not need to be applied. vii. For example, when W * H < T1, for example, when T1 = 16 * 16, PROF may not need to be applied. viii. For example, when W * H <= T1, for example, when T1 = 16 * 16, PROF may not need to be applied. ix. For example, when min(W, H) >= T1, for example, when T1 = 64, PROF may not need to be applied. x. For example, when min(W, H) > T1, for example, when T1 = 64, PROF may not need to be applied. xi. For example, when max(W, H) <= T1, for example, when T1 = 16, PROF may not need to be applied. xii. For example, when max(W, H) < T1, for example, when T1 = 16, PROF may not need to be applied. b. In one example, whether to execute PROF and / or how to do so may depend on the control point motion vector. c. In one example, whether to perform PROF and / or how to perform it may depend on the affine parameter and / or the number of affine parameters. i. In the case of a 4-parameter affine model,

[0266]

Number

[0267] in this case, whether to execute PROF and how to do so may depend on parameters a and b. ii. For a four-parameter affine model,

[0268]

number

[0269] Whether and how PROF is run may depend on parameters such as a, b, c, and d. iii. In one example, if the maximum affine parameter is less than (or not greater than) a threshold, then PROF may not be applied. (i) Alternatively, if all (eg, four or six) affine parameters are less than (or not greater than) a threshold, then PROF may not be applied. (ii) Alternatively, if at least one affine parameter is less than (or not greater than) a threshold, then PROF may not be applied. iv. In one example, if the maximum absolute value of the affine parameters is less than (or not greater than) a threshold, PROF may not be applied. (i) Alternatively, if the absolute values ​​of all affine parameters are less than (or not greater than) a threshold, then PROF may not be applied. (ii) Alternatively, PROF may be applied only if at least one of the absolute values ​​of all affine parameters is greater than (or not smaller than) a threshold value. v. In one example, if the minimum affine parameter is greater than (or not less than) a threshold, PROF may not be applied. (i) Alternatively, if all (eg, four or six) affine parameters are greater (or less) than a threshold, then PROF may not be applied. (ii) Alternatively, if at least one affine parameter is greater than (or not less than) a threshold, then PROF may not be applied. vi. In one example, if the minimum absolute value of the affine parameters is greater (or not less) than a threshold, PROF may not be applied. (i) Alternatively, if the absolute values ​​of all affine parameters are greater (or less) than a threshold, then PROF may not be applied. (ii) Alternatively, PROF may be applied only if at least one of the absolute values ​​of the affine parameters is less than (or not greater than) a threshold value. vii. In one example, if the maximum value of the "absolute value" of delta MV as disclosed in JVET-N0236 is less than (or not greater than) a threshold value, PROF may not be applied. (i) Alternatively, if the "absolute value" of all delta MVs is less than (or not greater than) a threshold, then PROF may not be applied. (ii) Alternatively, PROF may be applied only if at least one of the "absolute values" of the delta MVs is greater than (or not less than) a threshold value. viii. In one example, if the minimum "absolute value" of delta MV is greater (or not less) than a threshold, then PROF may not be applied. (i) Alternatively, if the "absolute value" of all delta MVs is greater than (or not less than) a threshold, then PROF may not be applied. (ii) Alternatively, PROF may be applied only if at least one of the "absolute values" of the delta MVs is greater than (or not less than) a threshold value. ix. In one example, the PROF may be applied to a specific location. (i) For example, a PROF may be applied to a position if the "absolute value" of its corresponding delta MV is less than (or not greater than) a threshold value. (ii) For example, a PROF may be applied to a position if the "absolute value" of its corresponding delta MV is greater (or not less) than a threshold. x. In one example, the affine parameters may be expressed as integers dHorX, dVerX, dHorY, dVerY to a particular precision as described in JVET-M1001. xi. In one example, the threshold may be bit-depth dependent. (i) In one example, the threshold may be derived as 1 << BitDepth. (ii) Further alternatively, this threshold may depend on whether bidirectional prediction is applied or whether single prediction is applied. a. For example, the threshold may be derived as (1 << BitDepth) + (Bi - prediction? 1 : 0). xii. In one example, the presence or absence of application of the method disclosed in black circle 11, and / or the method may depend on the structure of the reference picture. (i) For example, if all reference pictures of the current picture are pre - placed in the display order of the current picture, i.e., if the POC of all reference pictures is smaller than the POC of the current picture, one or more of the methods disclosed herein may not be applied. (ii) Alternatively, the presence or absence of provision of the method disclosed in black circle 11, and / or the method may depend on the slice / picture type (e.g., I - slice or B - slice). (iii) Alternatively, the presence or absence of provision of the method disclosed in black circle 11, and / or the method may depend on the temporal layer. xiii. In black circle 11, instead of the coding method "PROF", other coding methods for improving affine prediction, such as the interleaved prediction or phase - variable affine sub - block motion compensation as disclosed in JVET - N0216, may be used. 12. It is proposed that phase - variable affine sub - block motion compensation as proposed in JVET - N0510 may be first applied to obtain a predicted value, and then PROF may be used. 13. It is proposed that the bit - width of any variable used to derive dMvH[x][y] and / or dMvV[x][y] for any valid x and y cannot exceed a specific number, e.g., 32. In one example, dMvH[x][y] and / or dMvV[x][y] are clipped before being used to derive other dMvH[t][z] and / or dMvV[t][z], where (t,z) is not equal to (x,y). In one example, dMvH[x][y] and / or dMvV[x][y] are shifted to the right before being used to derive another dMvH[t][z] and / or dMvV[t][z], where (t,z) is not equal to (x,y). 14. It is proposed that dMvH and / or dMvV may have the same precision as the stored motion vectors. For example, dMvH[xPos][yPos]=SatShift(dMvH[xPos][yPos],7+M); dMvV[xPos][yPos]=SatShift(dMvV[xPos][yPos],7+M); where M is the additional precision for deriving dMvH and / or hMvV, e.g., M=2. 15. It is proposed that clipping on dMvH and / or dMvV before being used to derive the predicted fine adjustment ΔI may depend on the accuracy of dMvH and / or dMvV. For example, dMvH[posX][posY]=Clip3(-2 K-1 ,2 K-1 -1,dMvH[posX][posY]); dMvV[posX][posY]=Clip3(-2 K-1 ,2 K-1 -1,dMvV[posX][posY]); Here, K depends on the accuracy of dMvH and / or dMvV. b. Alternatively, dMvH[x][y] and / or dMvV[x][y] are not clipped before being used to derive the prediction refinement. 16. It is proposed that the right shift of the prediction fine adjustment ΔI(posX,posY) may depend on the sign of ΔI(posX,posY). For example, ΔI(posX,posY)=SatShift(ΔI(posX,posY),N), where N is an integer. 17. It is proposed that the clipping in the prediction refinement ΔI(posX,posY) may depend on the sample bit depth. a. For example, ΔI(posX,posY)=Clip3(-(2 3+BitDepth -1),2 3+BitDpeth -1,ΔI(posX,posY)); 18. Whether and / or how deblocking is performed for sub-block boundaries within an affine mode block (e.g., inner sub-block boundaries) may depend on whether interwoven prediction, as disclosed in JVET-N0216, and / or PROF, and / or phase-variable affine sub-block motion compensation is applied to the block. In one example, if interleaved prediction or / and PROF or / and phase-variant affine sub-block motion compensation is applied to a block, deblocking may be disabled. Alternatively, the deblocking filter may be weaker at sub-block boundaries where interleaved prediction and / or PROF and / or phase-variable affine sub-block motion compensation is applied to the block, e.g., the boundary strength may be set small at such boundaries. b. In one example, deblocking may be enabled if interleaved prediction or / and PROF or / and phase-variable affine sub-block motion compensation is not applied to the block.

[0270] The above examples may be included in the context of the methods described below, such as methods 2910-2950, ​​which may be implemented in a video decoder or a video encoder.

[0271] 29A shows a flowchart of an example video processing method. Method 2910 includes, at step 2912, generating a prediction block for a current video block of a video using affine mode motion compensation performed at a sub-block level of the current video block. Method 2910 further includes, at step 2914, performing gradient calculations on a region of the current video block to refine the prediction block using a prediction refinement with optical flow (PROF) procedure, the size of the region (M×N) being different from the size of a sub-block of the current video block, where M and N are positive integers. Method 2910 further includes, at step 2916, converting between the current video block and a coded representation of the video based on the gradient calculations.

[0272] 29B shows a flowchart of an exemplary video processing method. The method 2920 includes, in step 2922, deriving a final predicted block calculated by fine-tuning one or more intermediate predicted blocks using gradient calculations performed on an image region having a second size according to a rule for a current video block of an image having a first size, the fine-tuning using an optical flow procedure. The method 2920 further includes, in step 2924, using the final predicted block to convert between the current video block and a coded representation of the video.

[0273] FIG. 29C shows a flowchart of an example video processing method. Method 2930 includes deriving motion information for a current video block of a video using bidirectional optical flow (BDOF) or prediction refinement with optical flow (PROF) at step 2932. Method 2930 further includes performing gradient calculations on samples in a region of the current video block such that at least one sample in the region is omitted from the gradient calculation at step 2934. Method 2930 further includes converting between the current video block and a coded representation of the video that constitutes the current video block based on the gradient calculation at step 2936. In some implementations, one or more initial predictions for the current video block are calculated at a sub-block level and refined using optical flow calculations in the PROF, or one or more initial predictions are refined using spatial and temporal gradients in BDOF.

[0274] 29D shows a flowchart of an exemplary video processing method. The method 2940 includes, for a current video block of a video, determining a final prediction block for the current video block by fine-tuning one or more initial predictions for the current video block using an optical flow calculation based on gradients of initial prediction samples according to a precision rule in step 2942. The method 2940 further includes, in step 2944, converting between the current video block and a coded representation using the final prediction block, the optical flow calculation including a prediction refinement with optical flow (PROF) procedure or a prediction refinement by a bidirectional optical flow (BDOF) procedure, and the precision rule specifies using the same precision for expressing the gradients for both the PROF procedure and the BDOF procedure.

[0275] 29E shows a flowchart of an example video processing method. Method 2950 includes, at step 2952, determining a final predicted block of a current video block for conversion between a current video block of a video and a coded representation of the video by fine-tuning one or more initial predictions of the current video block using an optical flow calculation. Method 2950 further includes, at step 2954, performing the conversion using the final predicted block. In some implementations, the optical flow calculation includes a prediction fine-tuning with optical flow (PROF) procedure and / or a prediction fine-tuning with a bidirectional optical flow (BDOF) procedure, and the optical flow calculation further includes a padding operation that can be applied to at least one of the PROF procedure or the BDOF procedure according to a rule to derive padding samples.

[0276] 5. Exemplary implementations of the disclosed technology

[0277] FIG. 30A is a block diagram of a video processing device 3000. The device 3000 may be used to implement one or more of the methods described herein. The device 3000 may be implemented by a smartphone, a tablet, a computer, an IoT (Internet of Things) receiver, etc. The device 3000 may include one or more processing devices 3002, one or more memories 3004, and video processing hardware 3006. The one or more processing devices 3002 may be configured to implement one or more methods described herein, including but not limited to the method 2900. The memory(s) 3004 may be used to store data and codes used to implement the methods and techniques described herein. The video processing hardware 3006 may be used to implement the techniques described herein in a hardware circuit.

[0278] FIG. 30B is another example of a block diagram illustrating an example video processing system in which the disclosed technology can be implemented. FIG. 30B is a block diagram illustrating an example video processing system 4100 in which various technologies disclosed herein can be implemented. Various implementations may include some or all of the modules of the system 4100. The system 4100 may include an input unit 4102 for receiving video content. The video content may be received in a raw or uncompressed format, for example, 8 or 10 bit multi-module pixel values, or may be received in a compressed or encoded format. The input unit 4102 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, passive optical network (PON), and wireless interfaces such as Wi-Fi or cellular interfaces.

[0279] The system 4100 may include an encoding module 4104 that may implement various encoding or coding methods described herein. The encoding module 4104 may reduce the average bit rate of the video from the input unit 4102 to the output of the encoding module 4104, generating an encoded representation of the video. This encoding technique may therefore be referred to as a video compression or video transcoding technique. The output of the encoding module 4104 may be stored or transmitted via a connected communication, as represented by module 4106. The bitstream (or encoded) representation of the video received at the input unit 4102, stored or communicated, may be used by module 4108 to generate pixel values ​​or displayable video that are transmitted to the display interface unit 4110. The process of generating a user-viewable video from the bitstream representation may be referred to as video decompression. Furthermore, although certain video processing operations are referred to as "encoding" operations or tools, it will be understood that the encoding tools or operations are performed by an encoder and corresponding decoding tools or operations that reverse the results of the decoding, performed by a decoder.

[0280] Examples of peripheral bus interface units or display interface units may include Universal Serial Bus (USB) or High-Definition Multimedia Interface (HDMI) or DisplayPort, etc. Examples of storage interfaces include Serial Advanced Technology Attachment (SATA), PCI, IDE interfaces, etc. The techniques described herein may be implemented in various electronic devices such as mobile phones, laptops, smartphones, or other devices capable of digital data processing and / or video display.

[0281] Some embodiments of the disclosed technology include determining or determining to enable a video processing tool or mode. In one example, if a video processing tool or mode is enabled, the encoder uses or implements the tool or mode when processing one video block, but does not necessarily modify the resulting bitstream based on the use of the tool or mode. That is, the conversion from a block of video to a bitstream representation of video uses the video processing tool or mode if the video processing tool or mode is enabled based on the decision or determination. In another example, if the video processing tool or mode is enabled, the decoder processes the bitstream knowing that the bitstream has been modified based on the video processing tool or mode. That is, the conversion from the bitstream representation of video to a block of video is performed using the video processing tool or mode enabled based on the decision or determination.

[0282] Some embodiments of the disclosed techniques include deciding or determining to disable a video processing tool or mode. In one example, when a video processing tool or mode is disabled, an encoder does not use the tool or mode when converting blocks of a video into a bitstream representation of the video. In another example, when a video processing tool or mode is disabled, a decoder processes the bitstream with the knowledge that the bitstream has not been modified using the video processing tool or mode that was disabled based on the decision or determination.

[0283] As used herein, the term "video processing" can refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm may be applied during conversion of a pixel representation of a video to a corresponding bitstream representation, or vice versa. The bitstream representation of a current video block may correspond to bits that are spread to the same or different locations in the bitstream, e.g., as specified by a syntax. For example, one macroblock may be coded in terms of transformed and coded error residual values, and using bits in the header and other fields in the bitstream.

[0284] It will be appreciated that the disclosed methods and techniques will be beneficial in embodiments of video encoders and / or decoders that are integrated into video processing devices such as smart phones, laptops, desktop computers, and similar appliances, by enabling the use of the techniques disclosed herein.

[0285] The various techniques and embodiments may be described using the following section-based format.

[0286] The first set of sections describes certain features and aspects of the technology disclosed in the previous section.

[0287] 1. A video processing method, comprising: performing gradient calculations in a first region of a current video block, the first region (M×N) being different from a size of a sub-block of the current video block used for motion compensation in affine mode, where M and N are positive integers (>0); and converting between the current video block and a bitstream representation of a video including the current video block based on the gradient calculations.

[0288] 2. The method according to item 1, wherein the size of the first region is greater than the size of the sub-block.

[0289] 3. The method according to item 1 or 2, wherein M and N are predefined positive integers.

[0290] 4. The method according to claim 1 or 2, wherein the size of the first region is based on the size of the sub-block.

[0291] 5. The method according to item 1, wherein M / N is adaptively changed.

[0292] 6. The method of claim 1, wherein M and N are based on dimensions of the current image block.

[0293] 7. The method according to any one of items 1 to 6, wherein M has a minimum value Mmin and N has a minimum value Nmin.

[0294] 8. The method according to item 7, wherein Mmin=Nmin=8.

[0295] 9. The method according to any of items 1 to 6, further comprising padding the first region to generate a first padding region of size (M+dM)×(N+dN).

[0296] 10. The method according to item 9, wherein samples in the first region or the first padding region are derived based on motion compensation by interpolation filtering.

[0297] 11. The method of claim 1, further comprising omitting at least one sample in the first region when performing the gradient calculation.

[0298] 12. The method of claim 1, wherein the gradient calculation is performed with a first accuracy in bidirectional optical flow (BDOF) and with a second accuracy in prediction refinement with optical flow (PROF), the first accuracy and the second accuracy being equal.

[0299] 13. A method of video processing comprising: making a decision regarding selective application of an encoding tool to a current video block in accordance with selective application of prediction refinement using optical flow-assisted prediction refinement (PROF) for a current video block, the encoding tool being different from the PROF; and converting between the current video block and a bitstream representation of a video including the current video block based on the decision.

[0300] 14. The method according to claim 13, wherein the PROF is not applied and the encoding tool is applied.

[0301] 15. The method of claim 13, wherein the encoding tool includes generalized bidirectional prediction.

[0302] 16. The method of claim 15, wherein the PROF is not applied and an index associated with the generalized bidirectional prediction is not zero.

[0303] 17. The method according to claim 13, wherein the coding tool is local illumination compensation.

[0304] 18. The method according to claim 13, wherein the encoding tool is a multiple transform set (MTS).

[0305] 19. The method of claim 18, wherein the PROF is applied and only a default transformation from the MTS is applied to the current video block.

[0306] 20. The method according to claim 13, wherein the encoding tool is a weighted prediction.

[0307] 21. A method of video processing comprising: determining, during conversion between a current video block and a bitstream representation of a video constituting the current video block, a selective application of a prediction refinement with optical flow (PROF) operation, the determination being based on color information of the current video block.

[0308] 22. The method of claim 21, wherein the PROF operation is not applied to one or more chroma components of the current image block, and the color information includes a 4:0:0 color format.

[0309] 23. The method of claim 21, wherein the PROF operation is applied to one or more chroma components of the current image block, and the color information includes a 4:4:4 color format.

[0310] 24. The method of claim 21, wherein the PROF operation is applied to one or more chroma components of the current image block, and the color information includes a 4:0:0 color format.

[0311] 25. The method of claim 21, wherein the PROF operation is applied and the color information includes multiple color components.

[0312] 26. The method of claim 25, wherein one or more gradients of the PROF calculation are calculated independently for each of a plurality of color components.

[0313] 27. The method of claim 25, further comprising calculating one or more gradients of the PROF operation for a first color component of the plurality of color components and reusing the gradients for a second color component of the plurality of color components.

[0314] 28. The method of claim 26 or 27, wherein the accuracy of the gradient is based on at least one of the plurality of color components.

[0315] 29. A method of video processing comprising: determining based on a height (H) or width (W) of a current video block, regarding selective application of a prediction refinement with optical flow (PROF) operation, and converting between the current video block and a bitstream representation of a video including the current video block based on the determination.

[0316] 30. The method of claim 29, wherein the PROF operation is applied to a luminance component of the current image block.

[0317] 31. The method of claim 29, wherein the current video block is encoded using an affine mode.

[0318] 32. The method according to item 31, wherein the PROF calculation is not applied, W≦T1 and / or H≦T2, and T1=T2=16.

[0319] 33. The method according to item 31, wherein the PROF calculation is not applied, W≧T1 and / or H≧T2, and T1=T2=64.

[0320] 34. The method according to item 31, wherein the PROF calculation is not applied, W×H≦T or max(W,H)≦T, and T=16.

[0321] 35. The method according to item 31, wherein the PROF calculation is not applied, W×H≧T or minimum(W,H)≧T, and T=64.

[0322] 36. The method of claim 1 or 2, wherein the size of the current image block is W×H, and M=min(K,W), where K is an integer.

[0323] 37. The method of claim 1 or 2, wherein the size of the current image block is W×H, and N=min(K,H), where K is an integer.

[0324] 38. The method according to item 36 or 37, wherein K=16.

[0325] 39. The method of claim 1 or 2, further comprising performing a padding operation on the first region of the current image block before performing the gradient calculation.

[0326] 40. The method of claim 39, wherein performing the padding process includes deriving one or more motion vectors.

[0327] 41. The method of claim 40, wherein the one or more motion vectors include a motion vector derived from an affine model toward a particular position of the first region.

[0328] 42. The method of claim 40, wherein the one or more motion vectors include a motion vector derived from at least one motion vector of at least one subblock of the first region.

[0329] 43. The method of claim 39, wherein performing the padding process is based on a height or width of the current image block.

[0330] 44. The method according to item 39, wherein the padding process is performed based on signaling in a video parameter set (VPS), a decoder parameter set (DPS), a sequence parameter set (SPS), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, a tile, a tile group header, a coding tree unit (CTU) or a coding unit (CU).

[0331] 45. The method according to item 5 or 6, wherein M and N are signaled in a video parameter set (VPS), a decoder parameter set (DPS), a sequence parameter set (SPS), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, a tile, a tile group header, a coding tree unit (CTU) or a coding unit (CU).

[0332] 46. ​​The method according to claim 5 or 6, wherein M and N are specified by a profile, level or hierarchical layer of a video coding standard.

[0333] 47. A video system comprising a processing device and a non-transitory memory having instructions stored therein, the instructions being implemented by the processing device to cause the processing device to perform a method according to any one of items 1 to 46.

[0334] 48. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for carrying out the method according to any one of items 1 to 46.

[0335] The second set of items describes certain features and aspects of the techniques disclosed in the previous section, including, for example, example implementations 1-4.

[0336] 1. A method for processing video, comprising: generating a prediction block for a current video block of a video using affine mode motion compensation performed at a sub-block level of the current video block; performing gradient calculations on a region of the current video block to fine-tune the prediction block using a Prediction Fine-Tuning with Optical Flow (PROF) procedure, the size of the region (M×N) being different from the size of a sub-block of the current video block, where M and N are positive integers; and converting between the current video block and a coded representation of the video based on the gradient calculations.

[0337] 2. The method according to item 1, wherein the size of the region is greater than the size of the sub-block.

[0338] 3. The method according to item 1 or 2, wherein M and N are predefined positive integers.

[0339] 4. The method according to claim 1 or 2, wherein the size of the region is based on the size of the sub-block.

[0340] 5. The method according to item 1, wherein the padding operation in the PROF procedure is applied to derive gradients at (M×N) region levels.

[0341] 6. The method according to any one of items 1 to 5, wherein M is defined as min(K0, block width) and N is defined as min(K1, block height), and K0 and K1 are integers.

[0342] 7. The method according to item 6, wherein K0 and K1 are equal to 16.

[0343] 8. The method of claim 1, wherein performing the gradient calculation includes obtaining a first gradient for a first sample in a first sub-block of the current image block and a second gradient for a second sample in a second sub-block of the current image block.

[0344] 9. The method of claim 8, wherein the second subblock is adjacent to the first subblock.

[0345] 10. The method of claim 8, wherein the second sample is used to derive the first gradient.

[0346] 11. The method according to any one of items 8 to 10, wherein the size (M×N) of the region is greater than the size of a sub-block of the current image block.

[0347] 12. The method of claim 1, further comprising deriving one or more motion vectors for a padding operation of the region.

[0348] 13. The method of claim 12, further comprising deriving a specific motion vector associated with integer samples in a reference block of the current video block for the padding operation of the region.

[0349] 14. The method according to claim 13, wherein the particular motion vector corresponds to a motion vector of one of the subblocks in the region.

[0350] 15. The method according to claim 13, wherein the particular motion vector is derived by applying an affine model to a particular position of the region.

[0351] 16. The method according to claim 13, wherein the particular motion vector is derived from the motion vectors of all sub-blocks in the region.

[0352] 17. The method of claim 13, wherein the particular motion vector is derived as an average of the motion vectors of all sub-blocks in the region.

[0353] 18. The method according to claim 13, wherein the particular motion vector is derived as an average of motion vectors of at least some of the subblocks in the region that are located near the center of the region.

[0354] 19. The method of claim 13, wherein the particular motion vector is derived as a function of a plurality of motion vectors including control point motion vectors or sub-block motion vectors in the region.

[0355] 20. The method of claim 12, further comprising deriving a plurality of motion vectors associated with integer samples in a reference block of the current video block for the padding operation of the region.

[0356] 21. The method of claim 20, further comprising: determining a position of the integer sample using a first motion vector of a first subblock in the region for padding a first sample adjacent to the first subblock in the region.

[0357] 22. The method according to item 12, wherein the size of the region (M×N) is greater than the size of a sub-block of the current image block.

[0358] 23. The method according to any one of items 1 to 22, wherein the minimum values ​​of M and N are predetermined as Mmin and Nmin, respectively.

[0359] 24. The method according to item 23, wherein Mmin=Nmin=8.

[0360] 25. A method according to any one of items 1 to 24, further comprising performing a padding operation on the region and generating padding samples in a padding region having a size of (M+dM)×(N+dN).

[0361] 26. The method according to claim 25, wherein the samples within the region are derived from motion compensation by interpolation filtering.

[0362] 27. The method of claim 26, wherein the samples are derived from motion compensation of a plurality of sub-blocks in the region.

[0363] 28. The method according to claim 25, wherein the puddling samples are taken along the four outer sides of the region.

[0364] 29. The method of claim 28, wherein the paddling sample copies the intensity of the nearest integer sample in a reference block of the current image block.

[0365] 30. The method according to claim 28, wherein a padding sample copies the intensity of the nearest sample in the unpadded region.

[0366] 31. A method according to any one of claims 1 to 30, wherein performing the transformation includes generating the coded representation from the current video block.

[0367] 32. The method of any one of claims 1 to 30, wherein performing the transformation includes generating the current video block from the encoded representation.

[0368] 33. A method for processing video, comprising: deriving a final predicted block calculated for a current video block of a video having a first size by fine-tuning one or more intermediate predicted blocks using gradient calculations performed on a video region having a second size according to a rule, the fine-tuning using an optical flow procedure; and using the final predicted block to convert between the current video block and a coded representation of the video.

[0369] 34. The method of claim 33, wherein the rule specifies that the second size is a function of the first size.

[0370] 35. The method according to item 34, wherein the rule specifies that the width (M) of the second size and / or the height (N) of the second size depends on the width (W) of the first size and / or the height (H) of the first size.

[0371] 36. The method according to item 35, wherein M=W and N=H.

[0372] 37. The method of claim 35, wherein M=N=2*Wmc, and Wmc is the width or height of a sub-block of the current video block used to obtain one or more intermediate prediction blocks using motion compensation for the current video block.

[0373] 38. The method of item 35, wherein at least one of M and N is signaled in a video parameter set (VPS), a decoding parameter set (DPS), a sequence parameter set (SPS), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, a tile group header, a tile, a coding tree unit, or a coding unit.

[0374] 39. The method according to item 35, wherein at least one of M and N is defined in a profile, level or tier of the rule.

[0375] 40. The method according to item 35, wherein M=Min(W,T1) and N=Min(H,T2), and T1 and T2 are positive integers.

[0376] 41. The method of claim 40, wherein at least one of T1 and T2 is signaled in a video parameter set (VPS), a decoding parameter set (DPS), a sequence parameter set (SPS), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, a tile group header, a tile, a coding tree unit, or a coding unit.

[0377] 42. The method according to item 40, wherein at least one of T1 and T2 is defined in a profile, level or tier of rules.

[0378] 43. The method according to any one of items 35 to 42, wherein the minimum values ​​of M and N are predetermined as Mmin and Nmin, respectively.

[0379] 44. The method according to item 43, wherein Mmin=Nmin=8.

[0380] 45. A method according to any one of items 33 to 44, further comprising performing a padding operation on the region and generating padding samples in a padding region having a size of (M+dM)×(N+dN).

[0381] 46. ​​The method according to claim 45, wherein the samples within the region are derived from motion compensation by interpolation filtering.

[0382] 47. The method of claim 46, wherein the samples are derived from motion compensation of a plurality of sub-blocks in the region.

[0383] 48. The method according to claim 45, wherein the puddling samples are taken along the four outer sides of the region.

[0384] 49. The method of claim 48, wherein the paddling sample copies the intensity of the nearest integer sample in a reference block of the current image block.

[0385] 50. The method according to claim 48, wherein a padding sample copies the intensity of the nearest sample in the unpadded region.

[0386] 51. The method according to claim 33, wherein the optical flow procedure includes an optical flow with prediction refinement (PROF) procedure, in which one or more intermediate prediction blocks are calculated at a sub-block level and refined using the optical flow calculations.

[0387] 52. The method according to claim 33, wherein the optical flow procedure includes a bidirectional optical flow (BDOF) procedure that uses spatial and temporal gradients to fine-tune the one or more intermediate prediction blocks.

[0388] 53. The method of any of claims 33 to 52, wherein performing the transformation includes generating the encoded representation from the current video block.

[0389] 54. The method of any of claims 33 to 52, wherein performing the transformation includes generating the current video block from the encoded representation.

[0390] 55. A method of video processing comprising: deriving motion information for a current video block of a video using bidirectional optical flow (BDOF) or prediction refinement with optical flow (PROF); performing gradient calculations on samples in a region of the current video block such that at least one sample in the region is omitted from the gradient calculation; and converting between the current video block and a coded representation of a video including the current video block based on the gradient calculations, wherein one or more initial predictions for the current video block are calculated at a sub-block level and refined using optical flow calculations in the PROF, or the one or more initial predictions are refined using spatial and temporal gradients in the BDOF.

[0391] 56. The method according to item 55, wherein in the BIO and PROF, a gradient associated with a sample at a given coordinate is used.

[0392] 57. The method of claim 55, wherein the samples are modified prior to performing the gradient calculation, and the modified samples are used during performing the gradient calculation.

[0393] 58. The method of any of claims 55 to 57, wherein performing the transformation includes generating the encoded representation from the current video block.

[0394] 59. The method of any of items 55 to 57, wherein performing the transformation includes generating the current video block from the encoded representation.

[0395] The third set of items describes certain features and aspects of the techniques disclosed in the previous section, including, for example, example implementations 5 and 6.

[0396] 1. A video processing method comprising: for a current video block of a video, determining a final predicted block for the current video block by using an optical flow calculation to fine-tune one or more initial predictions for the current video block based on gradients of initial prediction samples in accordance with accuracy rules; and using the final predicted block to convert between the current video block and an encoded representation, wherein the optical flow calculation includes prediction fine-tuning by an optical flow with prediction fine-tuning (PROF) procedure or a bidirectional optical flow (BDOF) procedure, and the accuracy rules specify that the same precision be used to represent the gradients for both the PROF procedure and the BDOF procedure.

[0397] 2. The method according to item 1, wherein in the PROF procedure, one or more initial predictions are calculated at the sub-block level during affine motion compensation prediction and are fine-tuned using optical flow calculations.

[0398] 3. The method according to item 1, wherein in the BDOF procedure, the one or more initial predictions are fine-tuned using at least one gradient value obtained during optical flow calculation.

[0399] 4. The method of claim 1, wherein horizontal and vertical gradients, represented as gradientH and gradientV, respectively, are calculated as one of the following:

[0400] i) gradientH[x][y]=(predSamples[x+1][y]-predSample[x-1][y])>>Shift0,gradientV[x][y]=(predSample[x][y+1]-predSample[x][y-1])>>Shift1,

[0401] ii) gradientH[x][y]=Shift((predSamples[x+1][y]-predSample[x-1][y]),Shift0),gradientV[x][y]=Shift((predSample[x][y+1]-predSample[x][y-1]),Shift1),

[0402] iii) gradientH[x][y]=SatShift((predSamples[x+1][y]-predSample[x-1][y]),Shift0),gradientV[x][y]=SatShift((predSample[x][y+1]-predSample[x][y-1]),Shift1),or

[0403] iv) gradientH[x][y]=Shift(predSamples[x+1][y],Shift0)-Shift(predSample[x-1][y],Shift0),gradientV[x][y]=Shift(predSample[x][y+1],Shift0)-predSample[x][y-1],Shift1),and

[0404] Here, Shift(x,n) is defined as Shift(x,n)=(x+offset0)>>n, and Satshift(x,n) is defined as:

[0405]

number

[0406] 5. The method of claim 1, wherein horizontal and vertical gradients, represented as gradientH and gradientV, respectively, are calculated as one of the following:

[0407] i) gradientH[x][y]=(predSamples[x][y]*2-predSamples[x+1][y]-predSample[x-1][y])>>Shift0,gradientV[x][y]=(predSamples[x][y]*2-predSample[x][y+1]-predSample[x][y-1])>>Shift1,

[0408] ii) gradientH[x][y]=Shift((predSamples[x][y]*2-predSamples[x+1][y]-predSample[x-1][y]),Shift0),gradientV[x][y]=Shift((predSamples[x][y]*2-predSample[x][y+1]-predSample[x][y-1]),Shift1),or

[0409] iii) gradientH[x][y]=SatShift((predSamples[x][y]*2-predSamples[x+1][y]-predSample[x-1][y]),Shift0),g radientV[x][y]=SatShift((predSamples[x][y]*2-predSample[x][y+1]-predSample[x][y-1]),Shift1),and

[0410] Here, Shift(x,n) is defined as Shift(x,n)=(x+offset0)>>n, and Satshift(x,n) is defined as follows:

[0411]

number

[0412] 6. The method according to item 4 or 5, wherein Shift0 and / or Shift1 are set to Max(2, (14-BitDepth)), and BitDepth represents the bit depth of the reconstructed samples or input samples.

[0413] 7. The method according to any one of items 1 to 6, wherein Shift0 and / or Shift1 used for PROF and BDOF are the same.

[0414] 8. The method of any one of items 1 to 7, wherein performing the transformation includes generating the encoded representation from the current video block.

[0415] 9. A method according to any one of items 1 to 7, wherein performing the transformation includes generating the current video block from the encoded representation.

[0416] 10. A video processing method comprising: for converting between a current video block of a video and an encoded representation of the video, determining a final predicted block for the current video block by fine-tuning one or more initial predictions for the current video block using an optical flow calculation; and performing the conversion using the final predicted block, wherein the optical flow calculation includes prediction fine-tuning by an optical flow with prediction fine-tuning (PROF) procedure and / or a bidirectional optical flow (BDOF) procedure, and wherein the optical flow calculation further includes a padding operation that can be applied to at least one of the PROF procedure or the BDOF procedure according to a rule to derive padding samples.

[0417] 11. The method according to claim 10, wherein in the PROF procedure, one or more initial predictions are calculated at a sub-block level during affine motion compensation prediction and are fine-tuned using optical flow calculations.

[0418] 12. The method according to item 10, wherein in the BDOF procedure, the one or more initial predictions are fine-tuned using at least one gradient value obtained during optical flow calculation.

[0419] 13. The method according to claim 10, wherein the rules stipulate that the same padding operation is used for the PROF procedure and the BDOF procedure.

[0420] 14. The method of claim 10, wherein the rules specify that the padding samples are derived from integer samples at integer positions in a reference picture of the current video block.

[0421] 15. The method according to item 14, wherein the integer samples are positioned around the padding samples with a difference of MV(Mvx, Mvy) rounded to an integer MV.

[0422] 16. The method according to item 14, rounding MV(Mvx, MvY) to a floor integer (IntX, IntY).

[0423] 17. The method according to item 14, rounding MV(Mvx, MvY) to the nearest integer MV(IntX, IntY).

[0424] 18. Round MV(Mvx,MvY) to integer MV(IntX,IntY) such that IntX=SatShift(MvX,P) and IntY=SatShift(MvY,P), where P denotes the precision of MV.

[0425]

number

[0426] 19. The method according to any of items 15 to 18, further comprising rounding the MV based on a precision of the MV that depends on the color format and / or color components of the current image block.

[0427] 20. The method of claim 10, further comprising signaling information about how to perform the padding operation in a video parameter set (VPS), a decoding parameter set (DPS), a sequence parameter set (SPS), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, a tile group header, a tile, a coding tree unit, or a coding unit.

[0428] 21. The method of claim 10, wherein the information on how to perform the padding operation is in a profile, level, or tier of the rule.

[0429] 22. The method of claim 10, wherein the information on how to perform the padding operation depends on a dimension of the current image block.

[0430] 23. A method according to any one of claims 10 to 22, wherein performing the transformation includes generating the encoded representation from the current video block.

[0431] 24. The method of any of claims 10 to 22, wherein performing the transformation includes generating the current video block from the encoded representation.

[0432] 25. A video system comprising a processing device and a non-transitory memory having instructions stored therein, the instructions being implemented by the processing device to cause the processing device to perform a method according to any one of items 10 to 22.

[0433] 26. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing the method according to any one of items 10 to 22.

[0434] 27. A video encoding method comprising: determining a final predicted block for a current video block of a video by fine-tuning one or more initial predictions for the current video block using an optical flow calculation based on motion gradients according to accuracy rules to encode the current video block into an encoded representation of the video; and encoding the current video block into an encoded representation using the final predicted block, wherein the optical flow calculation includes an optical flow with prediction fine-tuning (PROF) procedure, in which one or more initial predictions are calculated at a sub-block level and fine-tuned using the optical flow calculation, and / or a bidirectional optical flow (BDOF) procedure, in which the one or more initial predictions are fine-tuned using spatial and temporal gradients, and wherein the accuracy rules specify using the same accuracy to represent the gradients for both the PROF procedure and the BDOF procedure.

[0435] 28. A video decoding method comprising: determining a final predicted block for a current video block of a video from an encoded representation of the video by fine-tuning one or more initial predictions for the current video block using an optical flow calculation based on motion gradients in accordance with precision rules; and generating a current video block from the encoded representation using the final predicted block, wherein the optical flow calculation includes a prediction fine-tuning with optical flow (PROF) procedure, in which one or more initial predictions are calculated at a sub-block level and fine-tuned using the optical flow calculation, and / or a prediction fine-tuning with bidirectional optical flow (BDOF) procedure, in which the one or more initial predictions are fine-tuned using spatial and temporal gradients, and wherein the precision rules specify using the same precision to represent the gradients for both the PROF procedure and the BDOF procedure.

[0436] 29. A video encoding method comprising: determining a final predicted block for a current video block of a video into an encoded representation of the video by fine-tuning one or more initial predictions for the current video block using an optical flow calculation; and encoding the current video block into an encoded representation using the final predicted block, wherein the optical flow calculation includes a prediction fine-tuning with optical flow (PROF) procedure, in which one or more initial predictions are calculated at a sub-block level and fine-tuned using the optical flow calculation, and / or a prediction fine-tuning with bidirectional optical flow (BDOF) procedure, in which the one or more initial predictions are fine-tuned using spatial and temporal gradients, and wherein the optical flow calculation further includes a padding operation that can be applied to at least one of the PROF procedure or the BDOF procedure according to a rule to derive padding samples.

[0437] 30. A video decoding method comprising: determining a final predicted block for a current video block of a video from an encoded representation of the video by fine-tuning one or more initial predictions for the current video block using an optical flow calculation; and generating a decoded current video block from the encoded representation using the final predicted block, wherein the optical flow calculation includes a prediction fine-tuning with optical flow (PROF) procedure, in which one or more initial predictions are calculated at a sub-block level and fine-tuned using the optical flow calculation, and / or a prediction fine-tuning with bidirectional optical flow (BDOF) procedure, in which the one or more initial predictions are fine-tuned using spatial and temporal gradients, and wherein the optical flow calculation further includes a padding operation applicable to at least one of the PROF procedure or the BDOF procedure according to a rule to derive padding samples.

[0438] While specific embodiments of the disclosed technology have been described above for purposes of illustration, it will be understood that various modifications may be made without departing from the scope of the invention. Accordingly, the disclosed technology is not to be limited except as by the appended claims.

[0439] Implementations of the subject matter and functional operations described in this patent specification may be implemented in various systems, digital electronic circuits, or computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. Implementations of the subject matter described herein may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible, non-transportable computer-readable medium for execution by or for controlling the operation of a data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter that provides a machine-readable propagated signal, or one or more combinations thereof. The term "data processing unit" or "data processing device" includes all apparatus, devices, and machines for processing data, including, for example, a programmable processing device, a computer, or multiple processing devices or computers. In addition to hardware, the apparatus may include code that creates an environment for the execution of the computer program, such as code that constitutes a processing device firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0440] A computer program (also called a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be recorded as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), may be stored in a single file dedicated to the program, or may be stored in multiple coordinating files (e.g., a file that stores one or more modules, subprograms, or code portions). A computer program can be deployed to run on one computer located at one site, or on multiple computers distributed across multiple sites and interconnected by a communication network.

[0441] The processes and logic flows described herein may be performed by one or more programmable processing devices executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and devices may be implemented as, special purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0442] Processing devices suitable for executing computer programs include, for example, both general purpose and special purpose microprocessors, as well as any one or more processors of any kind of digital computer. Typically, a processing device receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Typically, a computer may include one or more mass storage devices, e.g., magnetic, magneto-optical, or optical disks, for storing data, or may be operatively coupled to receive data from or transfer data to these mass storage devices. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all types of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, flash memory devices, and the like. The processing device and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0443] This specification, together with the drawings, are for illustrative purposes only, and illustrative is intended to mean example. In this specification, the use of "or" is intended to include "and / or" unless the context clearly indicates otherwise.

[0444] Although this patent specification contains many details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather as descriptions of features that may be specific to certain embodiments of a particular invention. Certain features described in this patent document in the context of separate embodiments may be implemented in combination in one example. Conversely, various features described in the context of one example may be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in a particular combination and initially claimed as such, one or more features from a claimed combination may, in some cases, be extracted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0445] Similarly, although operations are shown in a particular order in the figures, this should not be understood as requiring that such operations be performed in the particular order or sequential order shown, or that all of the operations shown be performed, to achieve desired results, nor should the separation of various system components in the examples described in this patent specification be understood as requiring such separation in all embodiments.

[0446] Only certain implementations and examples have been described; other embodiments, extensions and variations are possible based on the content described and illustrated in this patent document.

Claims

1. A method for processing a video, comprising: determining that the coding mode of a current video block is an affine mode; determining, for the current video block, applicability of the optical flow-based prediction refinement procedure in the affine mode according to rules defining conditions for applying the optical flow-based prediction refinement procedure; and performing a conversion between the current video block and the video bitstream based on the determination; one or more initial predictions for the current video block are computed at a sub-block level and refined using optical flow computations during the optical flow prediction refinement procedure; the rules specify that the applicability of the optical flow-based prediction refinement procedure corresponds to a motion vector at a control point of the current video block and is based on a control point motion vector derived for the current video block. method.

2. A motion vector at a control point of the current video block is derived for the current video block. The method of claim 1.

3. the applicability of the optical flow-based prediction refinement procedure is determined based on a width (W) and / or a height (H) of the current video block; The method according to claim 1 or 2.

4. the rules govern the applicability of the optical flow-based prediction refinement procedure based on an affine parameter and / or a number of affine parameters used in affine motion compensation prediction for the current video block. The method according to claim 1 or 2.

5. A delta motion vector is used to calculate a motion vector for the current video block, and the rule specifies the applicability based on a comparison of one of: i) a maximum absolute value of the delta motion vectors; ii) a minimum absolute value of the delta motion vectors; iii) all delta motion vectors; and iv) at least one of the delta motion vectors to a threshold. The method according to claim 1 or 2.

6. The rule specifies that the optical flow-based prediction refinement procedure is applied to a sample position based on a comparison of the absolute value of the corresponding delta motion vector with a threshold. The method according to any one of claims 1 to 5.

7. the rules define the condition based on at least one of a reference picture structure, a type of slice or picture that contains the current video block, or a temporal layer. The method according to any one of claims 1 to 6.

8. determining that the optical flow-based prediction refinement procedure is enabled for a video block of a video image such that a coding mode for coding the current video block into the bitstream is not permitted; The method according to any one of claims 1 to 7.

9. the coding mode that is prohibited when coding the current video block into the bitstream is a generalized bi-prediction mode in which different weights are applied to list 0 prediction and list 1 prediction of the current video block. The method according to claim 8.

10. performing the transformation includes decoding the current video block from the bitstream. The method according to any one of claims 1 to 9.

11. performing the transformation includes encoding the current video block into the bitstream. The method according to any one of claims 1 to 9.

12. 1. An apparatus for processing video data, comprising: a processor; and a non-transitory memory having instructions stored thereon, the apparatus comprising: The instructions, when executed by the processor, cause the processor to: determining that the coding mode of a current video block is an affine mode; determining, for the current video block, applicability of the optical flow-based prediction refinement procedure in the affine mode according to a rule defining a condition for applying the optical flow-based prediction refinement procedure; performing a conversion between the current video block and the video bitstream based on the determination; one or more initial predictions for the current video block are computed at a sub-block level and refined using optical flow computations during the optical flow prediction refinement procedure; the rules specify that the applicability of the optical flow-based prediction refinement procedure corresponds to a motion vector at a control point of the current video block and is based on a control point motion vector derived for the current video block. Device.

13. A non-transitory computer-readable storage medium having stored thereon instructions for causing a processor to: determining that the coding mode of a current video block is an affine mode; determining, for the current video block, applicability of the optical flow-based prediction refinement procedure in the affine mode according to a rule defining a condition for applying the optical flow-based prediction refinement procedure; performing a conversion between the current video block and the video bitstream based on the determination; one or more initial predictions for the current video block are computed at a sub-block level and refined using optical flow computations during the optical flow prediction refinement procedure; the rules specify that the applicability of the optical flow-based prediction refinement procedure corresponds to a motion vector at a control point of the current video block and is based on a control point motion vector derived for the current video block. A non-transitory computer-readable storage medium.

14. 1. A method for storing a video bitstream, comprising the steps of: determining that the coding mode of a current video block is an affine mode; determining, for the current video block, applicability of the optical flow-based prediction refinement procedure in the affine mode according to a rule defining a condition for applying the optical flow-based prediction refinement procedure; generating the bitstream based on said determination; storing the bitstream on a non-transitory computer readable storage medium; one or more initial predictions for the current video block are computed at a sub-block level and refined using optical flow computations during the optical flow prediction refinement procedure; the rules specify that the applicability of the optical flow-based prediction refinement procedure corresponds to a motion vector at a control point of the current video block and is based on a control point motion vector derived for the current video block. method.

Images