Weighted Prediction in Video Codecs

Abstract

A video processing method includes generating a motion vector for a current block of video for conversion between a current block of video and a bitstream representation of the video; calling a weighted prediction processing tool or a second processing tool in a mutually exclusive manner; and performing the conversion according to the calling. Calling the second processing tool includes refining the motion vector.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] In accordance with applicable patent law, this application is intended to promptly claim priority and benefits to International Patent No. PCT / CN2019 / 071759 filed on January 15, 2019. The entire disclosure of the foregoing application is incorporated by reference as part of the disclosure of this application. Technical Field

[0003] This patent document relates to the field of video coding. Background Art

[0004] Currently, efforts are being made to improve the performance of current video codec technologies to provide better compression rates or to provide video encoding and decoding schemes that allow for lower complexity or parallel implementations. Several new video codec tools have recently been proposed by industry experts and are currently being tested to determine their effectiveness. Summary of the invention

[0005] This document provides techniques for incorporating local illumination compensation in embodiments of a video encoder or decoder.

[0006] In one example aspect, a method of video processing is disclosed. The method includes: generating a motion vector of a current block of video for conversion between a current block of video and a bitstream representation of the video; calling a weighted prediction processing tool or a second processing tool in a mutually exclusive manner; and performing the conversion according to the calling. Calling the second processing tool includes refining the motion vector.

[0007] In another example aspect, a method of video processing is disclosed. The method includes: in converting between a video block and a bitstream representation of the video block, determining that the video block is a boundary block of a codec tree unit (CTU) in which the video block is located, and thus enabling a local illumination compensation (LIC) codec tool for the video block; deriving parameters of a local illumination compensation (LIC) for the video block based on determining that the LIC codec tool is enabled for the video block; and performing the conversion by adjusting pixel values ​​of the video block using the LIC.

[0008] In another example aspect, a method of video processing is disclosed. The method includes: in converting between a video block and a bitstream representation of the video block, determining that the video block is an internal block of a codec tree unit (CTU) in which the video block is located, and thus disabling a local illumination compensation (LIC) codec tool for the video block; inheriting LIC parameters of the video block; and performing the conversion by adjusting pixel values ​​of the video block using the LIC.

[0009] In yet another example aspect, another method of video processing is disclosed. The method includes: in converting between a video block and a bitstream representation of the video block, determining that both local illumination compensation and intra block copy codec tools are enabled for use by a current block; and performing the conversion by performing local illumination compensation (LIC) and intra block copy operations on the video block.

[0010] In yet another example aspect, another method of video processing is disclosed. The method includes: performing the conversion by performing local illumination compensation (LIC) and intra block copy operations on the video block; and performing conversion between a current block and a corresponding bitstream representation of the current block.

[0011] In yet another example aspect, another method of video processing is disclosed. The method includes: during conversion between a video block of a video and a bitstream representation of the video, determining local illumination compensation (LIC) parameters for the video block using at least some samples of neighboring blocks of the video block; and performing conversion between the video block and the bitstream representation by performing the LIC using the determined parameters.

[0012] In yet another example aspect, another method of video processing is disclosed. The method includes performing conversion between a video and a bitstream representation of the video, wherein the video is represented as a video frame including video blocks, and local illumination compensation (LIC) is enabled only for the video blocks using a geometric prediction structure including a triangle prediction mode.

[0013] In yet another example aspect, another method of video processing is disclosed. The method includes: performing conversion between a video and a bitstream representation of the video, wherein the video is represented as a video frame including video blocks, and in the conversion to its corresponding bitstream representation, for less than all pixels of a current block, performing local illumination compensation (LIC).

[0014] In yet another example aspect, another method of video processing is disclosed. The method includes: in converting between a video block and a bitstream representation of the video block, determining that both local illumination compensation (LIC) and generalized bidirectional prediction (GBi) or multi-hypothesis inter-frame prediction codec tools are enabled for use by a current block; and performing the conversion by performing LIC and GBi or multi-hypothesis inter-frame prediction operations on the video block.

[0015] In yet another example aspect, another video processing method is disclosed. The method includes: in converting between a video block and a bitstream representation of the video block, determining that both local illumination compensation (LIC) and combined inter-intra prediction (CIIP) codec tools are enabled for use by a current block; and performing the conversion by performing LIC and CIIP operations on the video block.

[0016] In yet another representative aspect, the various techniques described herein may be embodied as a computer program product stored on a non-transitory computer-readable medium. The computer program product includes a program code for executing the methods described herein.

[0017] In yet another representative aspect, a video decoder device may implement the methods described herein.

[0018] The details of one or more embodiments are set forth in the accompanying drawings, drawings, and description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 An example of the derivation process for Merge candidate list construction is shown.

[0020] Figure 2 Example locations of spatial merge candidates are shown.

[0021] Figure 3 An example of candidate pairs considered for redundancy checking of spatial merge candidates is shown.

[0022] Figure 4 Example locations of the second PU (Prediction Unit) for N×2N and 2N×N partitions are shown.

[0023] Figure 5 It is a diagram of the motion vector scaling of the temporal merge candidate.

[0024] Figure 6 An example of candidate positions of the time-domain Merge candidates C0 and C1 is shown.

[0025] Figure 7 An example of a combined bi-predictive Merge candidate is shown.

[0026] Figure 8 An example of a derivation process of motion vector prediction candidates is shown.

[0027] Fig. 9 is an example illustration of motion vector scaling of spatial motion vector candidates.

[0028] Fig.10 An example of an advanced temporal motion vector predictor (ATMVP) for a coding unit (CU) is shown.

[0029] Fig.11An example of one CU having four sub-blocks (AD) and their neighboring blocks (ad) is shown.

[0030] Fig.12 An example of a planar motion vector prediction process is shown.

[0031] Fig.13 is a flowchart of an example of encoding with different motion vector (MV) precisions.

[0032] Fig.14 is an example illustration of a sub-block in which OBMC is applied.

[0033] Fig.15 An example of neighboring sample points used to derive IC parameters is shown.

[0034] Fig.16 is a diagram of dividing a codec unit (CU) into two triangular prediction units.

[0035] Fig.17 An example of the positions of neighboring blocks is shown.

[0036] Fig.18 An example in which a CU applies a first weighting factor group is shown.

[0037] Fig.19 An example of a motion vector storage implementation is shown.

[0038] Fig. 20 An example of a simplified affine motion model is shown.

[0039] Fig.21 An example of an affine MVF (Motion Vector Field) for each sub-block is shown.

[0040] Fig. 22 Examples of (a) a 4-parameter affine model and (b) a 6-parameter affine model are shown.

[0041] Fig.23 An example of a motion vector prediction value (MV) for the AF_INTER mode is shown.

[0042] Figure 24A-24B Examples of candidates for the AF_MERGE mode are shown.

[0043] Fig.25 Candidate positions for the affine merge mode are shown.

[0044] Fig.26 An example process of bilateral matching is shown.

[0045] Fig. 27An example process of template matching is shown.

[0046] Fig.28 An implementation of unilateral motion estimation (ME) in frame rate upconversion (FRUC) is shown.

[0047] Fig.29 An embodiment of an Ultimate Motion Vector Expression (UMVE) search process is shown.

[0048] Fig.30 An example of a UMVE search point is shown.

[0049] Fig.31 An example of a distance index and distance offset mapping is shown.

[0050] Fig.32 An example of an optical flow trajectory is shown.

[0051] Figure 33A-Figure 33B An example of Bi-directional Optical flow (BIO) without block extension is shown: a) access locations outside the block; b) padding is used to avoid extra memory accesses and computations.

[0052] Fig.34 An example of using decoder-side motion vector refinement (DMVR) based on bilateral template matching is shown.

[0053] Fig.35 An example of neighboring samples used in a bilateral filter is shown.

[0054] Fig.36 An example of a window covering two samples used in the weight calculation is shown.

[0055] Fig.37 An example of a decoding flow using the proposed history based motion vector prediction (HMVP) method is shown.

[0056] Fig.38 An example of updating a table in the proposed HMVP method is shown.

[0057] Fig.39 is a block diagram of a hardware platform for implementing the video encoding or decoding techniques described in this document.

[0058] Fig.40 An example of a hardware platform for implementing the methods and techniques described in this document is shown.

[0059] Fig.41 is a flow chart of an example method of video processing.

[0060] Fig.42 is a flow chart of an example method of video processing according to the present disclosure. DETAILED DESCRIPTION

[0061] This document provides several techniques that can be embodied in digital video encoders and decoders. For clear understanding, section headings are used in this document, and the scope of the techniques and embodiments disclosed in each section is not limited to only that section.

[0062] 1. Overview

[0063] This patent document relates to video codec technology. Specifically, it relates to local illumination compensation (LIC) in video codec. It can be applied to existing video codec standards, such as HEVC, or standards to be finalized (e.g., Versatile Video Codec (VVC)). It can also be applied to future video codec standards or video codecs.

[0064] 2. Examples of video encoding / decoding techniques

[0065] Video codec standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T developed H.261 and H.263, ISO / IEC developed MPEG-1 and MPEG-4 Visual, and the two organizations jointly developed the H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Coding (AVC) and H.265 / HEVC standards. Since H.262, video codec standards have been based on a hybrid video codec structure, where temporal prediction plus transform codec is used. In order to explore future video codec technologies beyond HEVC, VCEG and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and put them into reference software called the Joint Exploration Model (JEM). In April 2018, the Joint Video Experts Group (JVET) between VCEG (Q6 / 16) and ISO / IEC JTC1 SC29 / WG11 (MPEG) was established to work on the VVC standard with the goal of a 50% bitrate reduction compared to HEVC.

[0066] 2.1 Inter-frame prediction of HEVC / H.265

[0067] Each inter-predicted PU has motion parameters for one or two reference picture lists. The motion parameters include motion vectors and reference picture indices. The use of one of the two reference picture lists may also be signaled using inter_pred_idc. The motion vector may be explicitly encoded as a delta relative to the prediction value.

[0068] When a CU is encoded in skip mode, a PU is associated with the CU and there are no significant residual coefficients, no encoded motion vector increments or reference picture indices. A Merge mode is specified, whereby the motion parameters of the current PU are obtained from neighboring PUs including spatial and temporal candidates. Merge mode can be applied to any inter-predicted PU, not just for skip mode. An alternative to Merge mode is explicit transmission of motion parameters, where each PU explicitly signals the motion vector (more accurately, the motion vector difference compared to the motion vector predictor), the corresponding reference picture index for each reference picture list, and the use of the reference picture list. In the present disclosure, this mode is referred to as Advanced Motion Vector Prediction (AMVP).

[0069] When signaling indicates that one of the two reference picture lists is to be used, a PU is generated from a block of samples. This is called "unidirectional prediction". Unidirectional prediction is available for P slices and B slices.

[0070] When signaling indicates that both reference picture lists are to be used, a PU is generated from two sample blocks. This is called "bi-prediction". Bi-prediction is only available for B slices.

[0071] The following text provides detailed information about the inter prediction modes specified in HEVC. The description will start with the Merge mode.

[0072] 2.1.1Merge mode

[0073] 2.1.1.1 Merge Mode Candidate Derivation

[0074] When predicting a PU using Merge mode, an index pointing to an entry in the Merge candidate list is parsed from the bitstream and used to retrieve motion information. The construction of this list is specified in the HEVC standard and can be summarized in the following sequence of steps:

[0075] Step 1: Initial candidate derivation

[0076] oStep 1.1: Spatial Candidate Derivation

[0077] oStep 1.2: Redundancy check of spatial domain candidates

[0078] oStep 1.3: Time Domain Candidate Derivation

[0079] Step 2: Additional candidate insertions

[0080] oStep 2.1: Create bidirectional prediction candidates

[0081] oStep 2.2: Insert zero motion candidates

[0082] These steps are also schematically depicted in Figure 1 For spatial Merge candidate derivation, a maximum of four Merge candidates are selected from candidates located at five different positions. For temporal Merge candidate derivation, a maximum of one Merge candidate is selected from two candidates. Since a constant number of candidates for each PU is assumed at the decoder, additional candidates are generated when the number of candidates obtained from step 1 does not reach the maximum number of Merge candidates (MaxNumMergeCand) signaled in the slice header. Since the number of candidates is constant, the index of the best Merge candidate is encoded using Truncated Unarybinarization (TU). If the size of the CU is equal to 8, all PUs of the current CU share a single Merge candidate list, which is the same as the Merge candidate list of the 2N×2N prediction unit.

[0083] Hereinafter, operations associated with the above-mentioned steps will be described in detail.

[0084] 2.1.1.2 Spatial Candidate Derivation

[0085] In the derivation of spatial merge candidates, Figure 2 Up to four Merge candidates are selected from the candidates at the positions depicted in . The order of derivation is A1, B1, B0, A0, and B2. Position B2 is considered only when any PU at position A1, B1, B0, A0 is unavailable (for example, because it belongs to another strip or slice) or is intra-coded. After adding the candidate at position A1, a redundancy check is performed on the addition of the remaining candidates, which ensures that candidates with the same motion information are excluded from the list, thereby improving coding efficiency. Figure 3 An example of candidate pairs considered for redundant check of spatial merge candidates is shown. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundant check. Instead, only candidate pairs with Figure 3The arrows in the table link pairs, and candidates are only added to the list if the corresponding candidates for redundancy checking do not have the same motion information. Another source of repeated motion information is a "second PU" associated with a partition other than 2N×2N. As an example, Figure 4 The second PU is depicted for both N×2N and 2N×N cases. When the current PU is partitioned as N×2N, the candidate at position A1 is not considered for list construction. In fact, adding this candidate would result in two prediction units having the same motion information, which is redundant for having only one PU in the codec unit. Similarly, position B1 is not considered when the current PU is partitioned as 2N×N.

[0086] 2.1.1.3 Time Domain Candidate Derivation

[0087] In this step, only one candidate is added to the list. Specifically, in the derivation of the temporal Merge candidate, the scaled motion vector is derived based on the co-located PU belonging to the picture with the minimum POC (Picture Order Count) difference with the current picture in a given reference picture list. The reference picture list to be used for the derivation of the co-located PU is explicitly signaled in the slice header. Figure 5 is a diagram of the motion vector scaling of the temporal Merge candidate. Figure 5 As shown by the dotted line in , a scaled motion vector for the temporal Merge candidate is obtained, which is scaled from the motion vector of the collocated PU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the collocated picture and the collocated picture. The reference picture index of the temporal Merge candidate is set equal to zero. The actual implementation of the scaling process is described in the HEVC specification. For B slices, two motion vectors (one for reference picture list 0 and the other for reference picture list 1) are obtained and combined to generate a bidirectional prediction Merge candidate.

[0088] In the collocated PU(Y) belonging to the reference frame, the position of the temporal candidate is selected between candidates C0 and C1, such as Figure 6 If the PU at position C0 is not available, is intra-coded, or is outside the current CTU row, position C1 is used. Otherwise, position C0 is used in the derivation of the time-domain Merge candidate.

[0089] 2.1.1.4 Additional candidate insertion

[0090] In addition to the spatial and temporal Merge candidates, there are two additional types of Merge candidates: combined bidirectional prediction Merge candidates and zero Merge candidates. The combined bidirectional prediction Merge candidate is generated by utilizing the spatial and temporal Merge candidates. The combined bidirectional prediction Merge candidate is only used for B slices. The combined bidirectional prediction candidate is generated by combining the first reference picture list motion parameters of an initial candidate with the second reference picture list motion parameters of another initial candidate. If the two tuples provide different motion hypotheses, they will form a new bidirectional prediction candidate. As an example, Figure 7 Shown is when two candidates in the original list (on the left) (which have mvL0 and refIdxL0 or mvL1 and refIdxL1) are used to create a combined bi-predictive Merge candidate that is added to the final list (on the right). There are many rules for combining that are considered to generate these additional Merge candidates.

[0091] Zero motion candidates are inserted to fill the remaining entries in the Merge candidate list and thus reach the MaxNumMergeCand capacity. These candidates have zero spatial displacement and a reference picture index that starts at zero and increases each time a new zero motion candidate is added to the list. The number of reference frames used by these candidates is 1 and 2 for unidirectional prediction and bidirectional prediction, respectively. Finally, no redundancy check is performed on these candidates.

[0092] 2.1.1.5 Motion Estimation Regions for Parallel Processing

[0093] To speed up the encoding process, motion estimation can be performed in parallel, thereby deriving motion vectors for all prediction units within a given region at the same time. Deriving Merge candidates from spatial neighbors may interfere with parallel processing because a prediction unit cannot derive motion parameters from neighboring PUs until its associated motion estimation is completed. In order to ease the trade-off between coding efficiency and processing latency, HEVC defines a Motion Estimation Region (MER), and uses the "log2_parallel_merge_level_minus2" syntax element to signal the size of the MER in the picture parameter set. When a MER is defined, Merge candidates that fall into the same region are marked as unavailable and are therefore not considered in list construction.

[0094] 2.1.2AMVP

[0095] AMVP exploits the spatiotemporal correlation of motion vectors with neighboring PUs, which is used for explicit transmission of motion parameters. For each reference picture list, a motion vector candidate list is constructed by first checking the availability of the PU positions in the temporal domain adjacent to the left and above, removing redundant candidates and adding zero vectors to make the candidate list a constant length. The encoder can then select the best predictor from the candidate list and send the corresponding index indicating the selected candidate. Similar to Merge index signaling, the index of the best motion vector candidate is encoded using truncated unary. In this case, the maximum value to be encoded is 2 (see Figure 8 ). In the following sections, details about the derivation process of motion vector prediction candidates will be provided.

[0096] 2.1.2.1 Derivation of AMVP Candidates

[0097] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. For spatial motion vector candidate derivation, the final result is based on the location of Figure 8 The motion vectors of each PU at the five different positions depicted are used to derive two motion vector candidates.

[0098] For temporal motion vector candidate derivation, one motion vector candidate is selected from two candidates derived based on two different collocated positions. After generating the first spatiotemporal candidate list, duplicate motion vector candidates in the list are removed. If the number of potential candidates is greater than two, motion vector candidates whose reference picture index within the associated reference picture list is greater than 1 are removed from the list. If the number of spatiotemporal motion vector candidates is less than two, an additional zero motion vector candidate is added to the list.

[0099] 2.1.2.2 Spatial Motion Vector Candidates

[0100] In the derivation of spatial motion vector candidates, the Figure 2 Consider up to two candidates out of the five potential candidates derived in the PU of the depicted position, and those positions are the same as the positions of the motion Merge. The derivation order on the left side of the current PU is defined as A0, A1, and scaling A0, scaling A1. The derivation order on the upper side of the current PU is defined as B0, B1, B2, scaling B0, scaling B1, scaling B2. Therefore, for each side, there are four cases that can be used as motion vector candidates, two of which do not require the use of spatial domain scaling, and two use spatial domain scaling. The four different cases are summarized as follows:

[0101] No airspace scaling

[0102] -(1) Same reference picture list, and same reference picture index (same POC) -(2) Different reference picture list, but same reference picture (same POC)

[0103] Airspace scaling

[0104] -(3) Same reference picture list, but different reference pictures (different POC) -(4) Different reference picture list, and different reference pictures (different POC)

[0105] First check the non-spatial scaling case, then the spatial scaling. Spatial scaling is considered when the POC is different between the reference pictures of the neighboring PU and the reference pictures of the current PU regardless of the reference picture list. If all PUs of the left candidate are unavailable or are intra-coded, then scaling for the upper motion vector is allowed to help parallel derivation of the left and upper MV candidates. Otherwise, spatial scaling is not allowed for the upper motion vector.

[0106] like Fig. 9 As depicted, in the spatial scaling process, the motion vectors of neighboring PUs are scaled in a similar manner as in the temporal scaling. The main difference is that the reference picture list and the index of the current PU are given as input; the actual scaling process is the same as that of the temporal scaling process.

[0107] 2.1.2.3 Temporal Motion Vector Candidates

[0108] Except for the reference picture index derivation, all the processes used to derive the temporal merge candidate are the same as those used to derive the spatial motion vector candidate (see Figure 6 ). The reference picture index is signaled to the decoder.

[0109] 2.2 New inter-frame prediction method in JEM

[0110] 2.2.1 Motion vector prediction based on sub-CU

[0111] In JEM with QTBT (QuadTrees plus Binary Trees), each CU can have at most one motion parameter set for each prediction direction. Two sub-CU level motion vector prediction methods are considered in the encoder by dividing the large CU into sub-CUs and deriving the motion information of all sub-CUs of the large CU. The optional temporal motion vector prediction (ATMVP) method allows each CU to extract multiple sets of motion information from multiple blocks smaller than the current CU in the collocated reference picture. In the Spatial-Temporal Motion Vector Prediction (STMVP) method, the motion vector of the sub-CU is recursively derived by using the temporal motion vector predictor and the spatial neighboring motion vector.

[0112] In order to maintain a more accurate motion field for sub-CU motion prediction, motion compression of reference frames is currently disabled.

[0113] 2.2.1.1 Optional temporal motion vector prediction

[0114] In the optional temporal motion vector prediction (ATMVP) method, the temporal motion vector prediction (TMVP) is modified by extracting multiple sets of motion information (including motion vectors and reference indices) from blocks smaller than the current CU. Fig.10 As shown, a sub-CU is a square N×N block (N is set to 4 by default).

[0115] ATMVP predicts the motion vectors of sub-CUs within a CU in two steps. The first step is to use the so-called time domain vector to identify the corresponding block in the reference picture. The reference picture is also called the motion source picture. The second step is to divide the current CU into sub-CUs and obtain the motion vector and the reference index of each sub-CU from the blocks corresponding to each sub-CU, such as Fig.10 shown.

[0116] In the first step, the reference picture and the corresponding block are determined by the motion information of the spatial neighboring blocks of the current CU. In order to avoid the repeated scanning process of the neighboring blocks, the first Merge candidate in the Merge candidate list of the current CU is used. The first available motion vector and its associated reference index are set to the index of the temporal vector and the motion source picture. In this way, in ATMVP, the corresponding block can be identified more accurately than in TMVP, where the corresponding block (sometimes called the collocated block) is always in the lower right or center position relative to the current CU.

[0117] In the second step, the corresponding block of the sub-CU is identified by the time domain vector in the motion source picture by adding the time domain vector to the coordinates of the current CU. For each sub-CU, the motion information of its corresponding block (the minimum motion grid covering the center sample point) is used to derive the motion information of the sub-CU. After identifying the motion information of the corresponding N×N block, it is converted into the motion vector and reference index of the current sub-CU in the same way as the TMVP of HEVC, where motion scaling and other processes apply. For example, the decoder checks whether the low latency condition is met (that is, the POC of all reference pictures of the current picture is less than the POC of the current picture), and may use the motion vector MV x (e.g., the motion vector corresponding to the reference picture list X) is used to predict the motion vector MV of each sub-CU y (eg, where X equals 0 or 1, and Y equals 1-X).

[0118] 2.2.1.2 Spatiotemporal Motion Vector Prediction (STMVP)

[0119] In this method, the motion vector of a sub-CU is recursively derived in raster scan order. Fig.11 This concept is illustrated in Figure 1. Let us consider an 8×8 CU, which contains four 4×4 sub-CUs: A, B, C, and D. The neighboring 4×4 blocks in the current frame are labeled a, b, c, and d.

[0120] The motion derivation of sub-CU A starts by identifying its two spatial neighbors. The first neighbor is the N×N block (block c) above the sub-CUA. If this block c is not available or is intra-coded, check the other N×N blocks above the sub-CU A (from left to right, starting with block c). The second neighbor is the block on the left of sub-CU A (block b). If block b is not available or is intra-coded, check the other blocks on the left of sub-CU A (from top to bottom, starting with block b). The motion information obtained from the adjacent blocks in each list is scaled to the first reference frame of the given list. Next, the temporal motion vector predictor (TMVP) of sub-block A is derived by following the same process of TMVP derivation as specified by HEVC. The motion information of the collocated block at position D is extracted and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors (up to 3) are averaged separately for each reference list. The average motion vector is assigned as the motion vector of the current sub-CU.

[0121] 2.2.1.3 Sub-CU motion prediction mode signaling

[0122] Sub-CU modes are enabled as additional Merge candidates, and no additional syntax elements are required to signal these modes. Two additional Merge candidates are added to the Merge candidate list of each CU to represent ATMVP mode and STMVP mode. If the sequence parameter set indicates that ATMVP and STMVP are enabled, up to seven Merge candidates can be used. The encoding logic of the additional Merge candidates is the same as that of the Merge candidates in HM, which means that for each CU in a P slice or B slice, two RD checks may be required for the two additional Merge candidates.

[0123] In JEM, all bins of the Merge index are context-coded by CABAC, whereas in HEVC, only the first bin is context-coded, and the remaining bins are context-bypass-coded.

[0124] 2.2.2 Pairwise Average Candidates

[0125] The pairwise average candidate is generated by averaging the predefined candidate pairs in the current Merge candidate list, and the predefined pairs are defined as {(0,1),(0,2),(1,2),(0,3),(1,3),(2,3)}, where these numbers represent the Merge index of the Merge candidate list. The average motion vector is calculated separately for each reference list. If two motion vectors are available in one list, they are averaged even if they point to different reference pictures; if only one motion vector is available, the motion vector is used directly; if no motion vector is available, the list is kept invalid. The pairwise average candidate replaces the combined candidate in the HEVC standard.

[0126] The complexity analysis of the pairwise average candidates is summarized in Table 1. For the additional computation for averaging (last column in Table 1), each pair requires 4 additions and 4 shifts (MVx and MVy in L0 and L1), and the worst case requires 4 reference index comparisons per pair (refIdx0 is valid in L0 and refIdx1 is valid). There are 6 pairs, 24 additions, 24 shifts, and 24 reference index comparisons. The combined candidate pairs in the HEVC standard use 2 reference index comparisons per pair (refIdx0 is valid in L0 and refIdx1 is valid in L1), and there are 12 pairs, 24 reference index comparisons.

[0127] Table 1: Operational analysis of pairwise average candidates

[0128]

[0129] 2.2.3. Planar motion vector prediction

[0130] In JVET-K0135, planar motion vector prediction is proposed.

[0131] In order to generate a smooth fine-grained motion field, Fig.12 A brief description of the planar motion vector prediction process is given.

[0132] Planar motion vector prediction is achieved by averaging horizontal and vertical linear interpolation on a 4×4 block basis as shown below.

[0133] P(x,y)=(H×P h (x,y)+W×P v (x,y)+H×W) / (2×H×W) Equation (1)

[0134] W and H represent the width and height of the block. (x,y) is the coordinate of the current sub-block relative to the upper left sub-block. All distances are expressed as pixel distance divided by 4. P(x,y) is the motion vector of the current sub-block.

[0135] The horizontal prediction P at position (x,y) h (x,y) and vertical prediction P v (x,y) is calculated as follows:

[0136] P h (x,y)=(W-1-x)×L(-1,y)+(x+1)×R(W,y) Equation (2)

[0137] P v (x,y)=(H-1-y)×A(x,-1)+(y+1)×B(x,H) Equation (3)

[0138] Where L(-1,y) and R(W,y) are the motion vectors of the 4×4 blocks to the left and right of the current block. A(x,-1) and B(x,H) are the motion vectors from the 4×4 blocks to the top and bottom of the current block.

[0139] The reference motion information of the left column and the upper row neighboring blocks are derived from the spatial neighboring blocks of the current block.

[0140] The reference motion information of neighboring blocks in the right column and bottom row is derived as follows.

[0141] (1) Derivation of the motion information of the 4×4 block adjacent to the lower right corner in the temporal domain

[0142] (2) Using the derived motion information of the lower right adjacent 4×4 block and the motion information of the upper right adjacent 4×4 block, the motion vector of the right column adjacent 4×4 block is calculated as described in equation (4).

[0143] (3) Using the derived motion information of the lower right neighboring 4×4 block and the motion information of the lower left neighboring 4×4 block, calculate the motion vector of the bottom row neighboring 4×4 block as described in equation (5).

[0144] R(W,y)=((Hy-1)×AR+(y+1)×BR) / H Equation (4)

[0145] B(x,H)=((Wx-1)×BL+(x+1)×BR) / W Equation (5)

[0146] Where AR is the motion vector of the upper right spatially adjacent 4×4 block, BR is the motion vector of the lower right temporally adjacent 4×4 block, and BL is the motion vector of the lower left spatially adjacent 4×4 block.

[0147] The motion information obtained from neighboring blocks of each list is scaled to the first reference picture of the given list.

[0148] 2.2.4 Adaptive Motion Vector Difference Resolution

[0149] In HEVC, when use_integer_mv_flag in the slice header is equal to 0, the motion vector difference (MVD) (between the PU's motion vector and the predicted motion vector) is signaled in units of quarter luma samples. In JEM, Locally Adaptive Motion Vector Resolution (LAMVR) is introduced. In JEM, MVD can be encoded and decoded in units of quarter luma samples, integer luma samples, or four luma samples. The MVD resolution is controlled at the codec unit (CU) level, and the MVD resolution flag is conditionally signaled for each CU with at least one non-zero MVD component.

[0150] For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter luma sample MV precision is used in the CU. When the first flag (equal to 1) indicates that quarter luma sample MV precision is not used, another flag is signaled to indicate whether integer luma sample MV precision or quad luma sample MV precision is used.

[0151] When the first MVD resolution flag of a CU is zero or is not coded for the CU (meaning that all MVDs in the CU are zero), a quarter luma sample MV resolution is used for the CU. When the CU uses integer luma sample MV precision or four luma sample MV precision, the MVP in the AMVP candidate list of the CU is rounded to the corresponding precision.

[0152] In the encoder, the RD check at the CU level is used to determine which MVD resolution is to be used for the CU. That is, for each MVD resolution, the RD check at the CU level is performed three times. To speed up the encoder, the following coding scheme is applied in JEM:

[0153] · During the RD check of a CU with a normal quarter-luminance sample MVD resolution, the motion information of the current CU (integer luminance sample accuracy) is stored. The stored motion information (after rounding) is used as the starting point for further small-range motion vector refinement during the RD check for the same CU with integer luminance samples and 4-luminance sample MVD resolution, so that the time-consuming motion estimation process is not repeated three times.

[0154] · Conditionally call the RD check of a CU with 4-luminance sample MVD resolution. For a CU, when the RD cost of the integer luminance sample MVD resolution is much greater than the RD cost of the quarter-luminance sample MVD resolution, skip the RD check of the 4-luminance sample MVD resolution for the CU.

[0155] The encoding process is as Fig.13 shown. First, test the 1 / 4 pixel MV, calculate the RD cost and denote it as RDCost0, then test the integer MV, and the RD cost is denoted as RDCost1. If RDCost1 < th * RDCost0 (where th is a positive value), then test the 4-pixel MV; otherwise, skip the 4-pixel MV. Basically, when checking the integer or 4-pixel MV, the motion information and RD cost, etc. for the 1 / 4 pixel MV are known, and the motion information and RD cost can be reused to accelerate the encoding process of the integer or 4-pixel MV.

[0156] 2.2.5. Higher motion vector storage accuracy

[0157] In HEVC, the motion vector accuracy is a quarter pixel (for 4:2:0 video, a quarter luminance sample and an eighth chrominance sample). In JEM, the accuracy of the internal motion vector storage and Merge candidates is increased to 1 / 16 pixel. The higher motion vector accuracy (1 / 16 pixel) is used for motion-compensated inter prediction of CUs encoded / decoded in the skip / Merge mode. As described in Section 2.2.2, for CUs encoded / decoded in the normal AMVP mode, integer pixel or quarter pixel motion is used.

[0158] The SHVC upsampling interpolation filter with the same filter length and normalization factor as the HEVC motion compensation interpolation filter is used as the motion compensation interpolation filter for the additional fractional pixel positions. The chroma component motion vector precision is 1 / 32 samples in JEM, and the additional interpolation filter for the 1 / 32 pixel fractional position is derived by averaging two adjacent filters at 1 / 16 pixel fractional positions.

[0159] 2.2.6. Overlapped Block Motion Compensation

[0160] Overlapped block motion compensation (OBMC) was previously used in H.263. In JEM, unlike H.263, OBMC can be turned on and off using CU-level syntax. When OBMC is used in JEM, OBMC is performed for all motion compensation (MC) block boundaries except the right and bottom boundaries of the CU. In addition, it is applied to both luminance and chrominance components. In JEM, an MC block corresponds to a codec block. When a CU is encoded and decoded with sub-CU modes, including Sub-CU Merge, Affine, and FRUC (Frame Rate Up Conversion) modes, each sub-block of the CU is an MC block. In order to handle CU boundaries in a unified manner, OBMC is performed for all MC block boundaries at the sub-block level, where the sub-block size is set equal to 4×4, as Fig.14 shown.

[0161] When OBMC is applied to the current sub-block, in addition to the current motion vector, the motion vectors of four connected neighboring sub-blocks (if available and different from the current motion vector) are also used to derive the prediction block of the current sub-block. These multiple prediction blocks based on multiple motion vectors are combined to generate the final prediction signal of the current sub-block.

[0162] The prediction block based on the motion vector of the neighboring sub-block is denoted as P N , where N represents the index for the adjacent upper, lower, left and right sub-blocks, and the prediction block based on the motion vector of the current sub-block is represented as P C When P N When the motion information contains the same motion information of the neighboring sub-block as the current sub-block, OBMC is not N Otherwise, P N Each sample point is added to P C In the same point in the same N Four rows / columns are added to P C The weighting factors {1 / 4, 1 / 8, 1 / 16, 1 / 32} are used for P N, and weighting factors {3 / 4, 7 / 8, 15 / 16, 31 / 32} are used for P C The exception is small MC blocks (i.e. when the height or width of the codec block is equal to 4 or the CU is coded in sub-CU mode), for which only P N Two rows / columns are added to P C In this case, the weighting factors {1 / 4, 1 / 8} are used for P N , and weighting factors {3 / 4, 7 / 8} are used for P C For P generated based on the motion vector of the vertically (horizontally) adjacent sub-block N , P N The samples in the same row (column) of P are added to P with the same weighting factor. C .

[0163] In JEM, for CUs with a size less than or equal to 256 luma samples, a CU level flag is signaled to indicate whether OBMC is applied for the current CU. For CUs with a size greater than 256 luma samples or not encoded in AMVP mode, OBMC is applied by default. At the encoder, when OBMC is applied to a CU, its impact is taken into account during the motion estimation stage. The prediction signal formed by OBMC using the motion information of the upper and left neighboring blocks is used to compensate the upper and left boundaries of the original signal of the current CU, and then the normal motion estimation process is applied.

[0164] 2.2.7. Local illumination compensation

[0165] Local illumination compensation (LIC) is based on a linear model of illumination variation using a scaling factor a and an offset b, and is adaptively enabled or disabled for each codec unit (CU) of inter-mode coding.

[0166] When LIC is applied to a CU, the least square error method is used to derive parameters a and b by using the neighboring samples of the current CU and its corresponding reference samples. More specifically, Fig.19 As shown, the subsampling (2:1 subsampling) neighboring samples and corresponding samples (identified by the motion information of the current CU or sub-CU) in the reference picture are used. The IC parameters are derived and applied to each prediction direction separately.

[0167] When the CU is encoded and decoded in Merge mode, the LIC flag is copied from the adjacent block in a manner similar to the motion information copying in Merge mode; otherwise, the LIC flag is signaled for the CU to indicate whether LIC is applicable.

[0168] When LIC is enabled for a picture, an additional CU-level RD check is required to determine whether LIC is applied to the CU. When LIC is enabled for a CU, the Mean-Removed Sum of Absolute Difference (MR-SAD) and the Mean-Removed Sum of AbsoluteHadamard-Transformed Difference (MR-SATD) (instead of SAD and SATD) are used for integer pixel motion search and fractional pixel motion search, respectively.

[0169] To reduce coding complexity, the following coding scheme is applied in JEM. When there is no significant illumination change between the current picture and its reference pictures, LIC is disabled for the entire picture. To identify this situation, the histogram of the current picture and each reference picture of the current picture is calculated at the encoder. If the histogram difference between the current picture and each reference picture of the current picture is less than a given threshold, LIC is disabled for the current picture; otherwise, LIC is enabled for the current picture.

[0170] 2.2.8. Hybrid Intra- and Inter-Prediction

[0171] In JVET-L0100, multi-hypothesis prediction is proposed, where hybrid intra-frame and inter-frame prediction is a way to generate multiple hypotheses.

[0172] When multi-hypothesis prediction is applied to improve intra mode, multi-hypothesis prediction combines one intra prediction and one Merge index prediction. In the Merge CU, a flag is signaled for the Merge mode to select the intra mode from the intra candidate list when the flag is true. For the luma component, the intra candidate list is derived from 4 intra prediction modes including DC mode, planar mode, horizontal mode and vertical mode, and the size of the intra candidate list can be 3 or 4, depending on the block shape. When the CU width is greater than twice the CU height, the horizontal mode does not include the intra mode list, and when the CU height is greater than twice the CU width, the vertical mode is removed from the intra mode list. Weighted averaging is used to combine one intra prediction mode selected by the intra mode index and one Merge index prediction selected by the Merge index. For chroma components, DM is always applied without additional signaling. The weights used for combined prediction are described as follows. When DC mode or planar mode is selected, or the CB width or height is less than 4, equal weights are applied. For those CBs whose CB width and height are greater than or equal to 4, when horizontal / vertical mode is selected, a CB is first divided vertically / horizontally into four equal-area regions. Each weight set, denoted as (w_intra i,w_inter i ), where i is 1 to 4, and (w_intra1, w_inter1) = (6, 2), (w_intra2, w_inter2) = (5, 3), (w_intra3, w_inter3) = (3, 5) and (w_intra4, w_inter4) = (2, 6) will be applied to the corresponding areas. (w_intra1, w_inter1) is used for the area closest to the reference sample, and (w_intra4, w_inter4) is used for the area farthest from the reference sample. Then, the combined prediction can be calculated by adding the two weighted predictions and shifting right by 3 bits. In addition, the intra prediction mode of the intra-frame hypothesis of the prediction value can be saved for reference by subsequent adjacent CUs.

[0173] 2.2.9. Triangle Prediction Unit Mode

[0174] The concept of triangle prediction unit mode is to introduce a new triangle partitioning for motion compensated prediction. Fig.16 As shown, it divides the CU into two triangular prediction units (PUs) along the diagonal or opposite diagonal directions. Each triangular prediction unit in the CU is inter-predicted using its own unidirectional prediction motion vector and reference frame index, which are derived from a single unidirectional prediction candidate list. After predicting the triangular prediction unit, an adaptive weighting process is performed on the diagonal edges. Then, the transformation and quantization process is applied to the entire CU. Note that this mode is only applicable to Skip mode and Merge mode.

[0175] One-way prediction candidate list

[0176] The unidirectional prediction candidate list includes five unidirectional prediction motion vector candidates. Fig.17 As shown, it is derived from seven neighboring blocks of the CU, including five spatial neighboring blocks (1 to 5) and two temporal collocated blocks (6 to 7). The motion vectors of the seven neighboring blocks are collected and placed in the unidirectional prediction candidate list in the order of unidirectional prediction motion vector, L0 motion vector of bidirectional prediction motion vector, L1 motion vector of bidirectional prediction motion vector, and average motion vector of L0 motion vector and L1 motion vector of bidirectional prediction motion vector. If the number of candidates is less than five, the zero motion vector is added to the list.

[0177] Adaptive weighting process

[0178] After predicting each triangle prediction unit, an adaptive weighting process is applied to the diagonal edges between two triangle prediction units to derive the final prediction for the entire CU. The two weighting factor groups are listed below:

[0179] The first weighting factor group: {7 / 8, 6 / 8, 4 / 8, 2 / 8, 1 / 8} and {7 / 8, 4 / 8, 1 / 8} are used for luma and chroma samples respectively;

[0180] The second weighting factor group: {7 / 8, 6 / 8, 5 / 8, 4 / 8, 3 / 8, 2 / 8, 1 / 8} and {6 / 8, 4 / 8, 2 / 8} are used for luma and chroma samples respectively.

[0181] A weighting factor set is selected based on the comparison of the motion vectors of the two triangle prediction units. When the reference pictures of the two triangle prediction units are different from each other or the difference of their motion vectors is greater than 16 pixels, the second weighting factor set is used. Otherwise, the first weighting factor set is used. For example Fig.18 shown.

[0182] Motion vector storage

[0183] Motion vector of triangle prediction unit ( Fig.19 Mv1 and Mv2 in CU are stored in 4×4 grids. For each 4×4 grid, a unidirectional prediction motion vector or a bidirectional prediction motion vector is stored, depending on the position of the 4×4 grid in the CU. Fig.19 As shown, a unidirectional prediction motion vector Mv1 or Mv2 is stored for a 4×4 grid located in a non-weighted area. On the other hand, a bidirectional prediction motion vector is stored for a 4×4 grid located in a weighted area. The bidirectional prediction motion vector is derived from Mv1 and Mv2 according to the following rule:

[0184] 1. In the case where Mv1 and Mv2 have motion vectors from different directions (L0 or L1), Mv1 and Mv2 are simply combined to form a bidirectional prediction motion vector.

[0185] 2. In the case where both Mv1 and Mv2 come from the same L0 (or L1) direction,

[0186] 2.a. If the reference picture of Mv2 is the same as a picture in the L1 (or L0) reference picture list, then Mv2 is scaled to that picture. Mv1 and scaled Mv2 are combined to form a bidirectional prediction motion vector.

[0187] 2.b. If the reference picture of Mv1 is the same as a picture in the L1 (or L0) reference picture list, then Mv1 is scaled to that picture. The scaled Mv1 and Mv2 are combined to form a bidirectional prediction motion vector.

[0188] 2.c. Otherwise, only store Mv1 for the weighted region.

[0189] 2.2.10. Affine Motion Compensated Prediction

[0190] In HEVC, only the translational motion model is used for motion compensation prediction (MCP). In the real world, there are many kinds of motion, such as zooming in / out, rotation, perspective motion, and other irregular motions. In JEM, a simplified affine transformation motion compensation prediction is applied. Fig. 20 As shown, the affine motion field of a block is described by two control point motion vectors.

[0191] The motion vector field (MVF) of a block is described by the following equation:

[0192]

[0193] Where (v 0x ,v 0y ) is the motion vector of the upper left control point, and (v 1x ,v 1y ) is the motion vector of the upper right control point.

[0194] To further simplify motion compensated prediction, sub-block based affine transformation prediction is applied. The sub-block size M×N is derived from equation (7), where MvPre is the fractional precision of the motion vector (1 / 16 in JEM), (v 2x ,v 2y ) is the motion vector of the lower left control point calculated according to equation (6).

[0195]

[0196] After being derived from equation (7), M and N should be adjusted downward, if necessary, to be divisors of w and h, respectively.

[0197] In order to derive the motion vector of each M×N sub-block, Fig.21 As shown, the motion vector of the center sample of each sub-block is calculated according to Equation 1 and rounded to a fractional precision of 1 / 16. Then, the motion compensation interpolation filter mentioned in Section 2.2.5 is applied to generate a prediction for each sub-block using the derived motion vector.

[0198] After MCP, the high-precision motion vector of each sub-block is rounded and saved with the same precision as the normal motion vector.

[0199] 2.2.10.1.AF_INTER mode

[0200] In JEM, there are two affine motion modes: AF_INTER mode and AF_MERGE mode. For CUs with width and height greater than 8, AF_INTER mode can be applied. The affine flag at the CU level is signaled in the bitstream to indicate whether AF_INTER mode is used. In this mode, neighboring blocks are used to construct a pair of motion vectors {(v0, v1)|v0={v A , v B , v c},v1={v D , v E}} candidate list. Fig.23 As shown, v0 is selected from the motion vector of block A, block B or block C. The motion vector from the adjacent block is scaled according to the reference list and the relationship between the POC of the reference of the adjacent block, the POC of the reference of the current block and the POC of the current CU. And the method of selecting v1 from adjacent blocks D and E is similar. If the number of candidate lists is less than 2, the list is filled with motion vector pairs composed by copying each AMVP candidate. When the candidate list is greater than 2, the candidates are first sorted according to the consistency of the adjacent motion vectors (the similarity of the two motion vectors in a pair of candidates), and only the first two candidates are retained. The RD cost check is used to determine which motion vector pair candidate is selected as the control point motion vector prediction (CPMVP) of the current CU. And the index indicating the CPMVP position in the candidate list is signaled in the bitstream. After determining the CPMVP of the current affine CU, affine motion estimation is applied and the control point motion vector (CPMV) is found. Then, the difference between CPMV and CPMVP is signaled in the bitstream.

[0201] In AF_INTER mode, when using the 4 / 6 parameter affine mode, 2 / 3 control points are required, and therefore 2 / 3 MVDs need to be encoded and decoded for these control points, such as Fig. 22 In JVET-K0337, it is proposed to derive MV as follows, that is, predict mvd1 and mvd2 from mvd0.

[0202]

[0203]

[0204]

[0205] in mvd iand mv1 are the predicted motion vector, motion vector difference and motion vector of the upper left pixel (i=0), upper right pixel (i=1) or lower left pixel (i=2), respectively. Fig. 22 Note that the addition of two motion vectors (e.g., mvA(xA, yA) and mvB(xB, yB)) is equal to the separate summation of the two components, i.e., newMV = mvA + mvB, and the two components of newMV are set to (xA + xB) and (yA + yB), respectively.

[0206] 2.2.10.2. Fast Affine ME Algorithm in AF_INTER Mode

[0207] In affine mode, the MVs of 2 or 3 control points need to be determined jointly. Direct joint search of multiple MVs is computationally complex. A fast affine ME algorithm is proposed and applied to VTM / BMS.

[0208] The fast affine ME algorithm is described for a 4-parameter affine model, and the idea can be extended to a 6-parameter affine model.

[0209]

[0210]

[0211] Replacing (a-1) with a', the motion vector can be rewritten as:

[0212]

[0213] Assuming the motion vectors of the two control points (0,0) and (0,w) are known, we can derive the affine parameters from equation (13),

[0214]

[0215] The motion vector can be rewritten in vector form as:

[0216]

[0217] in

[0218]

[0219]

[0220] P = (x, y) is the pixel position.

[0221] At the encoder, the MVD of AF_INTER is derived iteratively. i (P) is the MV derived in the i-th iteration at position P, and dMV Ci It is represented as MV in the i-th iteration C Then, in the (i+1)th iteration,

[0222]

[0223] Pic ref Indicated as a reference picture, and Pic cur Represents the current picture, and represents Q = P + MV i (P). Assuming we use MSE as the matching criterion, we need to minimize:

[0224]

[0225] Assumptions Small enough, we can rewrite it approximately using the first-order Taylor expansion as follows.

[0226]

[0227] in, Indicates E i+1 (P)=Pic cur (P)-Pic ref (Q),

[0228]

[0229]

[0230]

[0231]

[0232]

[0233] To calculate the incremental MV of the control points (0,0) and (0,w).

[0234] Assuming that such an MVD derivation process is iterated n times, the final MVD is calculated as follows:

[0235]

[0236]

[0237]

[0238]

[0239] Using JVET-K0337, the incremental MV of the control point (0, w) represented by mvd1 is predicted from the incremental MV of the control point (0, 0) represented by mvd0. Now, only mvd1 is actually encoded.

[0240] AF_MERGE Mode

[0241] When applying CU in AF_MERGE mode, it obtains the first block encoded and decoded in affine mode from the valid adjacent reconstructed blocks. And the selection order of candidate blocks is from left, top, top right, bottom left to top left, such as Fig.24A As shown, the motion vectors v2, v3 and v4 of the upper left corner, upper right corner and lower left corner of the CU containing block A are derived. And the motion vector v0 of the upper left corner of the current CU is calculated based on v2, v3 and v4. Secondly, the motion vector v1 of the upper right corner of the current CU is calculated.

[0242] After deriving the CPMV v0 and v1 of the current CU, the MVF of the current CU is generated according to the simplified affine motion model equation 1. In order to identify whether the current CU is encoded or decoded in AF_MERGE mode, when there is at least one neighboring block encoded or decoded in affine mode, the affine flag is signaled in the bitstream.

[0243] Fig.24A and Fig. 24B Example showing AF_MERGE candidates

[0244] In JVET-L0366, which is planned to be adopted into VTM 3.0, the affine merge candidate list is constructed by the following steps:

[0245] (1) Insert inherited affine candidates

[0246] An inherited affine candidate is one that is derived from the affine motion model of its valid neighboring affine codec blocks. In a common basis, such as Fig.25 As shown, the scanning order of candidate positions is: A1, B1, B0, A0 and B2.

[0247] After the candidates are derived, a full pruning process is performed to check if the same candidate has been inserted into the list. If there is an identical candidate, the derived candidate is discarded.

[0248] (2) Insert the constructed affine candidate

[0249] If the number of candidates in the affine merge candidate list is less than MaxNumAffineCand (set to 5 in this draft), the constructed affine candidate is inserted into the candidate list. The constructed affine candidate refers to a candidate constructed by combining the neighboring motion information of each control point.

[0250] The motion information of the control point is first obtained from Fig.25 CPk(k=1,2,3,4) represents the kth control point. A0, A1, A2, B0, B1, B2 and B3 are the spatial locations of predicted CPk(k=1,2,3); T is the temporal location of predicted CP4.

[0251] The coordinates of CP1, CP2, CP3 and CP4 are (0,0), (W,0), (H,0) and (W,H), respectively, where W and H are the width and height of the current block.

[0252] The motion information for each control point is obtained according to the following priority order:

[0253] For CP1, the priority of checking is B2->B3->A2. If B2 is available, use B2. Otherwise, if B2 is available, use B3. If neither B2 nor B3 is available, use A2. If all three candidates are unavailable, the motion information of CP1 cannot be obtained.

[0254] For CP2, the checking priority is B1->B0.

[0255] For CP3, the checking priority is A1->A0.

[0256] For CP4, use T.

[0257] Second, affine merge candidates are constructed using combinations of control points.

[0258] Constructing a 6-parameter affine candidate requires motion information of three control points. The three control points can be selected from one of the following four combinations ({CP1, CP2, CP4}, {CP1, CP2, CP3}, {CP2, CP3, CP4}, {CP1, CP3, CP4}). The combination {CP1, CP2, CP3}, {CP2, CP3, CP4}, {CP1, CP3, CP4} will be converted into a 6-parameter motion model represented by the top left, top right, and bottom left control points.

[0259] Constructing a 4-parameter affine candidate requires motion information of two control points. These two control points can be selected from one of the following six combinations ({CP1,CP4},{CP2,CP3},{CP1,CP2},{CP2,CP4},{CP1,CP3},{CP3,CP4}). The combination {CP1,CP4},{CP2,CP3},{CP2,CP4},{CP1,CP3},{CP3,CP4} will be converted into a 4-parameter motion model represented by the top left and top right control points.

[0260] The combinations of constructed affine candidates are inserted into the candidate list in the following order: {CP1,CP2,CP3}, {CP1,CP2,CP4}, {CP1,CP3,CP4}, {CP2,CP3,CP4}, {CP1,CP2}, {CP1,CP3}, {CP2,CP3}, {CP1,CP4}, {CP2,CP4}, {CP3,CP4}.

[0261] For a combined reference list X (X is 0 or 1), the reference index with the highest usage in the control point is selected as the reference index of list X, and the motion vector pointing to the difference reference picture will be scaled.

[0262] After the candidates are derived, a full pruning process is performed to check if the same candidate has been inserted into the list. If there is an identical candidate, the derived candidate will be discarded.

[0263] (3) Fill with zero motion vectors

[0264] If the number of candidates in the affine merge candidate list is less than 5, a zero motion vector with a zero reference index is inserted into the candidate list until the list is full.

[0265] 2.2.11. Pattern matching motion vector derivation

[0266] The Pattern Matched Motion Vector Derivation (PMMVD) mode is a special Merge mode based on the Frame Rate Up Conversion (FRUC) technology. In this mode, the motion information of the block is derived at the decoder side instead of signaling the motion information of the block.

[0267] When the Merge flag of the CU is true, the FRUC flag is signaled to the CU. When the FRUC flag is false, the Merge index is signaled and the normal Merge mode is used. When the FRUC flag is true, an additional FRUC mode flag is signaled to indicate which method (bilateral matching or template matching) will be used to derive the motion information of the block.

[0268] On the encoder side, the decision on whether to use FRUC Merge mode for a CU is based on the RD cost selection made for normal Merge candidates. That is, both matching modes (bilateral matching and template matching) of the CU are checked by using RD cost selection. The matching mode that causes the minimum cost is further compared with other CU modes. If the FRUC matching mode is the most effective mode, the FRUC flag is set to true for the CU and the relevant matching mode is used.

[0269] The motion derivation process in FRUC Merge mode has two steps: first, a CU-level motion search is performed, followed by sub-CU-level motion refinement. At the CU level, the initial motion vector of the entire CU is derived based on bilateral matching or template matching. First, a list of MV candidates is generated, and the candidate that causes the minimum matching cost is selected as the starting point for further CU-level refinement. Then, a local search based on bilateral matching or template matching is performed near the starting point, and the MV result of the minimum matching cost is used as the MV of the entire CU. Subsequently, the motion information is further refined at the sub-CU level, taking the derived CU motion vector as the starting point.

[0270] For example, the following derivation process is performed for the W×H CU motion information derivation. In the first stage, the MV of the entire W×HCU is derived. In the second stage, the CU is further divided into M×M sub-CUs. The value of M is calculated as shown in (16), and D is a predefined partition depth, which is set to 3 by default in JEM. Then the MV of each sub-CU is derived.

[0271]

[0272] like Fig.26 As shown, bilateral matching is used to derive the motion information of the current CU by finding the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. Under the assumption of continuous motion trajectory, the motion vectors MV0 and MV1 pointing to the two reference blocks should be proportional to the temporal distance between the current picture and the two reference pictures (i.e., TD0 and TD1). As a special case, when the current picture is temporally between the two reference pictures and the temporal distance from the current picture to the two reference pictures is the same, bilateral matching becomes mirror-based bidirectional MV.

[0273] like Fig. 27 As shown, template matching is used to derive the motion information of the current CU by finding the closest match between the template in the current picture (the top and / or left adjacent blocks of the current CU) and the block in the reference picture (the same size as the template). In addition to the above-mentioned FRUC Merge mode, template matching is also applied to AMVP mode. In JEM, as done in HEVC, AMVP has two candidates. New candidates are derived through the template matching method. If the candidate newly derived by template matching is different from the first existing AMVP candidate, it is inserted at the beginning of the AMVP candidate list, and then the list size is set to 2 (meaning removing the second existing AMVP candidate). When applied to AMVP mode, only CU level search is applied.

[0274] 2.2.11.1.CU-level MV candidate set

[0275] The MV candidates set at the CU level include:

[0276] (i) If the current CU is in AMVP mode, it is the original AMVP candidate,

[0277] (ii) All Merge candidates,

[0278] (iii) Several MVs in the interpolated MV field introduced in Section 2.2.11.3.

[0279] (iv) Top and left neighboring motion vectors

[0280] When bilateral matching is used, each valid MV of the Merge candidate is used as input to generate an MV pair assuming bilateral matching. For example, in reference list A, one valid MV of the Merge candidate is (MVa, refa). Then, find the reference picture refb of its paired bilateral MV in other reference list B, so that refa and refb are on different sides of the current picture in time. If such refb is not available in reference list B, refb is determined to be a reference different from refa, and its temporal distance to the current picture is the minimum in list B. After determining refb, MVb is derived by scaling MVa based on the temporal distance between the current picture refa and refb.

[0281] The four MVs from the interpolated MV field are also added to the CU level candidate list. More specifically, the interpolated MVs at positions (0,0), (W / 2,0), (0,H / 2), and (W / 2,H / 2) of the current CU are added.

[0282] When FRUC is applied in AMVP mode, the original AMVP candidates are also added to the CU-level MV candidate set.

[0283] At the CU level, for an AMVP CU, up to 15 MVs are added to the candidate list, and for a Merge CU, up to 13 MVs are added to the candidate list.

[0284] 2.2.11.2. Sub-CU level MV candidate set

[0285] The MV candidates set at the sub-CU level include:

[0286] (i) Search the determined MV from the CU level,

[0287] (ii) neighboring MVs at the top, left, upper-left, and upper-right corners,

[0288] (iii) a scaled version of the collocated MV from the reference image,

[0289] (iv) Up to 4 ATMVP candidates,

[0290] (v) A maximum of 4 STMVP candidates.

[0291] The scaled MV from the reference picture is derived as follows: Traverse all reference pictures in both lists. The MV at the collocated position of the sub-CU in the reference picture is scaled to the reference of the starting CU level MV.

[0292] ATMVP and STMVP candidates are limited to the top four.

[0293] At the sub-CU level, up to 17 MVs are added to the candidate list.

[0294] 2.2.11.3. Generation of interpolated MV fields

[0295] Before encoding and decoding the frame, an interpolated motion field is generated for the entire picture based on single-sided ME. The motion field can then be used as a CU-level or sub-CU-level MV candidate later.

[0296] First, the motion fields of each reference picture in both reference lists are traversed at the 4×4 block level. For each 4×4 block, if the motion associated with the block passes through a 4×4 block in the current picture (e.g. Fig.28 As shown) and the block is not assigned any interpolated motion, the motion of the reference block is scaled to the current picture according to the temporal distances TD0 and TD1 (in the same way as the MV scaling of TMVP in HEVC), and the scaled motion is assigned to the block in the current frame. If no scaled MV is assigned to the 4×4 block, the motion of the block is marked as unavailable in the interpolated motion field.

[0297] 2.2.11.4. Interpolation and matching costs

[0298] When the motion vector points to a fractional sample location, motion compensated interpolation is required. To reduce complexity, bilinear interpolation is used for bilateral matching and template matching instead of conventional 8-tap HEVC interpolation.

[0299] The calculation of the matching cost is a little different at different steps. When selecting a candidate from the candidate set at the CU level, the matching cost is the absolute sum difference (SAD) of the bilateral match or template match. After determining the starting MV, the matching cost C of the bilateral match of the sub-CU level search is calculated as follows:

[0300]

[0301] where w is a weighting factor set to 4 based on experience, MV and MV sIndicates the current MV and the starting MV respectively. SAD is still used as the matching cost of template matching for sub-CU level search.

[0302] In FRUC mode, MV is derived by using luma samples only. The derived motion will be used for both luma and chroma for MC inter prediction. After determining MV, final MC is performed using 8-tap interpolation filter for luma and 4-tap interpolation filter for chroma.

[0303] 2.2.11.5.MV refinement

[0304] MV refinement is a pattern-based MV search with bilateral matching cost or template matching cost as the criterion. In JEM, two search modes are supported—Unrestricted Center-Biased Diamond Search (UCBDS) and Adaptive Cross Search—for MV refinement at CU level and sub-CU level, respectively. For both CU and sub-CU level MV refinement, MVs are searched directly with quarter-luminance sample MV precision, and followed by one-eighth-luminance sample MV refinement. The search range for MV refinement for CU and sub-CU steps is set equal to 8 luminance samples.

[0305] 2.2.11.6. Selection of prediction direction in template matching FRUC Merge mode

[0306] In bilateral matching Merge mode, bidirectional prediction is always applied because the motion information of the CU is derived based on the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. There is no such restriction for template matching Merge mode. In template matching Merge mode, the encoder can choose from unidirectional prediction in list 0, unidirectional prediction in list 1, or bidirectional prediction for the CU. The selection is based on the template matching cost as follows:

[0307] If costBi <= factor * min (cost0, cost1)

[0308] Bidirectional prediction is used;

[0309] Otherwise, if cost0 <= cost1

[0310] Then use the one-way prediction in list 0;

[0311] otherwise,

[0312] Use the one-way prediction in Listing 1;

[0313] Where cost0 is the SAD of template matching for list 0, cost1 is the SAD of template matching for list 1, and costBi is the SAD of template matching for bidirectional prediction. The value of factor is equal to 1.25, which means that the selection process is biased towards bidirectional prediction.

[0314] The inter prediction direction selection is only applied to the CU level template matching process.

[0315] 2.2.12. Generalized Bidirectional Prediction

[0316] In traditional bidirectional prediction, the prediction values ​​from L0 and L1 are averaged to generate the final prediction value using equal weight 0.5. The prediction value generation formula is shown in equation (32).

[0317] P TraditionalBiPred =(P L0 +P L1 +RoundingOffset)>>shiftNum Equation (32)

[0318] In equation (32), the final prediction value of the traditional bidirectional prediction is P TraditionalBiPred , P L0 and P L1 are the predicted values ​​from L0 and L1 respectively, and RoundingOffset and shiftNum are used to normalize the final predicted value.

[0319] Generalized Bi-prediction (GBI) is proposed to allow different weights to be applied to the prediction values ​​from L0 and L1. The generation of the prediction value is shown in equation (33).

[0320] P GBi =((1-w1)*P L0 +w1*P L1 +RoundingOffset GBi )>>shiftNum GBi Equation (33)

[0321] In equation (33), P GBi is the final prediction value of GBi, (1-w1) and w1 are the selected GBI weights applied to the prediction values ​​of L0 and L1 respectively. RoundingOffset GBi and shiftNum GBi It is used to normalize the final prediction value in GBi.

[0322] The supported weights for w1 are {-1 / 4, 3 / 8, 1 / 2, 5 / 8, 5 / 4}. One equal weight set and four unequal weight sets are supported. For the equal weight case, the process of generating the final prediction value is exactly the same as in the traditional bidirectional prediction mode. For the true bidirectional prediction case under random access (RA) conditions, the number of candidate weight sets is reduced to three.

[0323] For Advanced Motion Vector Prediction (AMVP) mode, if the CU is bi-predictive coded, the weight selection in GBI is explicitly signaled at the CU level. For Merge mode, the weight selection is inherited from the Merge candidate. In the proposed scheme, GBI supports weighted averaging of DMVR generated templates and the final prediction value of BMS-1.0.

[0324] 2.2.13. Multi-hypothesis inter-frame prediction

[0325] In the multi-hypothesis inter-frame prediction mode, in addition to the traditional unidirectional / bidirectional prediction signal, one or more additional prediction signals are signaled. The resulting overall prediction signal is obtained by weighted superposition per sample point. uni / bi and the first additional inter-frame prediction signal / hypothesis h3, the resulting prediction signal p3 is obtained as follows:

[0326] p3=(1-α)p uni / bi +αh3 Equation (34)

[0327] The changes to the prediction unit syntax are shown in bold in the following table:

[0328]

[0329] The weighting factor α is specified by the syntax element add_hyp_weight_idx according to the following mapping:

[0330] add_hyp_weight_idx α 0 1 / 4 1 -1 / 8

[0331] Note that for the additional prediction signal, the concept of prediction list 0 / list 1 is abolished and a combined list is used instead. The combined list is generated by alternately inserting reference frames from list 0 and list 1 with increasing reference indices, omitting reference frames that have already been inserted, thus avoiding double entries.

[0332] Similar to above, more than one additional prediction signal may be used. The resulting overall prediction signal is iteratively accumulated with each additional prediction signal.

[0333] p n+1 =(1-α n+1 ) n +α n+1h n+1 Equation (35)

[0334] The resulting overall prediction signal is obtained as the final p n (i.e. p n has the largest index n).

[0335] Note that also for inter-prediction blocks using MERGE mode (rather than skip mode), additional inter-prediction signals can be specified. Also note that in the case of MERGE, not only uni / bi-prediction parameters, but also additional prediction parameters of the selected Merge candidate can be used for the current block.

[0336] 2.2.14.Multiple Hypothesis Prediction of One-Way Prediction in AMVP Model

[0337] When multi-hypothesis prediction is applied to improve the unidirectional prediction of AMVP mode, a flag is signaled to enable or disable multi-hypothesis prediction for inter_dir equal to 1 or 2, where 1, 2, and 3 represent list 0, list 1, and bidirectional prediction, respectively. In addition, when the flag is true, a Merge index is signaled. In this way, multi-hypothesis prediction turns unidirectional prediction into bidirectional prediction, where one motion is obtained using the original syntax elements in AMVP mode and the other motion is obtained using the Merge scheme. The final prediction combines the two predictions into a bidirectional prediction using a 1:1 weight. The Merge candidate list is first derived from the Merge mode, excluding the sub-CU candidates (e.g., affine, alternative temporal motion vector prediction (ATMVP)). Next, it is separated into two separate lists, one for List 0 (L0) containing all L0 motions from the candidates, and the other for List 1 (L1) containing all L1 motions. After removing redundancy and filling gaps, two Merge lists are generated for L0 and L1, respectively. There are two constraints when applying multi-hypothesis prediction to improve AMVP mode. First, multi-hypothesis prediction is enabled for those CUs whose luma codec block (CB) area is greater than or equal to 64. Second, multi-hypothesis prediction is only applicable to L1 when in low-latency B pictures.

[0338] 2.2.15. Multi-hypothesis prediction in skip / merge mode

[0339] When multi-hypothesis prediction is applied to skip or merge mode, whether multi-hypothesis prediction is enabled is explicitly signaled. In addition to the original prediction, an additional Merge index prediction is selected. Therefore, each candidate of multi-hypothesis prediction implies a Merge candidate pair, in which one Merge candidate is used for the first Merge index prediction and the other Merge candidate is used for the second Merge index prediction. However, in each pair, the Merge candidate of the second Merge index prediction is implicitly derived as the subsequent Merge candidate (that is, the Merge index already signaled plus 1) without signaling any additional Merge index. After removing redundancy by excluding pairs containing similar Merge candidates and filling in gaps, a candidate list for multi-hypothesis prediction is formed. Then, the motion from the pair of two Merge candidates is obtained to generate the final prediction, where 5:3 weights are applied to the first Merge index prediction and the second Merge index prediction, respectively. In addition, a Merge or skip CU with multi-hypothesis prediction enabled can save motion information of additional hypotheses in addition to the motion information of existing hypotheses for reference by subsequent adjacent CUs. Note that sub-CU candidates (e.g., affine, ATMVP) are excluded from the candidate list, and multi-hypothesis prediction is not applicable to skip mode for low-latency B pictures. In addition, when multi-hypothesis prediction is applied to Merge or Skip mode, for those CUs whose CU width or CU height is less than 16, or those CUs whose CU width and CU height are both equal to 16, bilinear interpolation filters are used for multi-hypothesis motion compensation. Therefore, the worst-case bandwidth (access samples required for each sample point) of each Merge or Skip CU with multi-hypothesis prediction enabled is calculated in Table 1, and each number is less than half of the worst-case bandwidth of each 4×4 CU with multi-hypothesis prediction disabled.

[0340] 2.2.16. Final motion vector representation

[0341] The Ultimate Motion Vector Expression (UMVE) is proposed. UMVE is used in Skip or Merge mode using the proposed motion vector expression method.

[0342] UMVE reuses the same Merge candidates as in VVC. Among the Merge candidates, candidates can be selected and further extended by the proposed motion vector representation method.

[0343] UMVE provides a new motion vector expression with simplified signaling. The expression method includes the starting point, motion amplitude and motion direction.

[0344] Fig.29 An example of a UMVE search process is shown.

[0345] Fig.30An example of a UMVE search point is shown.

[0346] The proposed technique uses the Merge candidate list as is. However, for the extension of UMVE, only candidates of the default Merge type (MRG_TYPE_DEFAULT_N) are considered.

[0347] The base candidate index defines the starting point. The base candidate index indicates the best candidate among the candidates in Table 2, as shown below.

[0348] Table 2. Example base candidate IDX

[0349] Basic Candidate IDX 0 1 2 3 The Nth MVP First MVP Second MVP Third MVP Fourth MVP

[0350] If the number of basic candidates is equal to 1, the basic candidate IDX is not signaled.

[0351] The distance index is the motion amplitude information. The distance index indicates the predefined distance from the starting point information. Example predefined distances are shown in Table 3 as follows:

[0352] Table 3. Example distances IDX

[0353]

[0354] The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate four directions as shown in Table 4 below.

[0355] Table 4. Example Direction IDX

[0356] Directions to IDX 00 01 10 11 x-axis + – N / A N / A y-axis N / A N / A + –

[0357] The UMVE flag is signaled immediately after the Skip flag and the Merge flag are sent. If the Skip and Merge flags are true, the UMVE flag is parsed. If the UMVE flag is equal to 1, the UMVE syntax is parsed. However, if it is not 1, the AFFINE flag is parsed. If the AFFINE flag is equal to 1, this is AFFINE mode, but if it is not 1, the Skip / Merge index is parsed for the Skip / Merge mode of the VTM.

[0358] No additional line buffering is needed for UMVE candidates, because software's Skip / Merge candidates are used directly as base candidates. The complement of the MV is decided before motion compensation, using the input UMVE index. No need to keep a long line buffer for this.

[0359] 2.2.17. Affine Merge Mode with Prediction Offset

[0360] UMVE is extended to the affine merge mode, which will also be referred to as the UMVE affine mode hereinafter. The proposed method selects the first available affine merge candidate as the base prediction value. Then, it applies the motion vector offset to the motion vector value of each control point from the base prediction value. If there is no available affine merge candidate, the proposed method is not used.

[0361] The inter prediction direction of the selected basic prediction value and the reference index for each direction are used without change.

[0362] In the current implementation, the affine model of the current block is assumed to be a 4-parameter model, and only 2 control points need to be derived. Therefore, only the first 2 control points of the basic prediction value will be used as the control point prediction value.

[0363] For each control point, the zero_MVD flag is used to indicate whether the control point of the current block has the same MV value as the corresponding control point prediction value. If the zero_MVD flag is true, no other signaling is required for the control point. Otherwise, the distance index and offset direction index are signaled for the control point.

[0364] A distance offset table of size 5 is used, as shown in the following table. The distance index is signaled to indicate which distance offset is used. The mapping of the distance index and the distance offset value is shown in reference 5.

[0365] Table 5 - Example distance offsets

[0366] Distance DX 0 1 2 3 4 Distance Offset 1 / 2 pixel 1 pixel 2 pixels 4 pixels 8 pixels

[0367] Fig.31 An example of a distance index and distance offset mapping is shown.

[0368] The direction index can represent the four directions shown below, of which only the x or y direction may be

[0369] Offset direction IDX 00 01 10 11 x-dir-factor +1 –1 0 0 y-dir-factor 0 0 +1 –1

[0370] The MV is bad, but not in both directions.

[0371] If inter prediction is unidirectional, the signaled distance offset is applied to the offset direction of the prediction value for each control point. The result will be the MV value for each control point.

[0372] For example, when the basic prediction value is unidirectional, and the motion vector value of the control point is MVP (v px ,v py ). When the distance offset and direction index are signaled, the motion vector of the corresponding control point of the current block will be calculated as follows.

[0373] MV(v x,v y )=MVP(v px ,v py )+MV(x-dir-factor*distance-offset,y-dir-factor*distance-offset) Equation (36)

[0374] If inter prediction is bidirectional, the signaled distance offset is applied in the signaled offset direction of the L0 motion vector of the control point prediction value; and the same distance offset with the opposite direction is applied to the L1 motion vector of the control point prediction value. The result will be the MV value of each control point in each inter prediction direction.

[0375] For example, when the base prediction value is unidirectional, and the motion vector value of the control point on L0 is MVP L0 (v 0px ,v 0py ), and the motion vector value of the control point on L1 is MVP L1 (v 1px ,v 1py ). When the distance offset and direction index are signaled, the motion vector of the corresponding control point of the current block will be calculated as follows.

[0376] MV L0 (v 0x ,v 0y )=MVP L0 (v 0px ,v 0py )+MV(x-dir-factor*distance-offset,y-dir-factor*distance-offset) Equation (37)

[0377] MV L1 (v 0x ,v 0y )=MVP L1 (v 0px ,v 0py )+MV(-x-dir-factor*distance-offset,-y-dir-factor*distance-offset) Equation (38)

[0378] 2.2.18. Bidirectional optical flow

[0379] Bi-directional optical flow (BIO) is a sample-level motion refinement that is performed on the basis of block-wise motion compensation for bidirectional prediction. Sample-level motion refinement does not use signaling.

[0380] Hypothesis I (k) is the luminance value from reference k (k=0,1) after block motion compensation, and I (k) The horizontal and vertical components of the gradient. Assuming that the optical flow is valid, the motion vector field (v x ,v y ) is given by the equation:

[0381]

[0382] Combine the optical flow equation with Hermite interpolation to obtain the motion trajectory of each sample point, and finally get the function value I (k) and derivatives The only third-order polynomial that matches is the BIO prediction:

[0383]

[0384] Here, τ0 and τ1 represent the distance to the reference frame, as Fig.31 As shown. The distances τ0 and τ1 are calculated based on the POC of Ref0 and Ref1: τ0 = POC(current) - POC(Ref0), τ1 = POC(Ref1) - POC(current). If the two predictions are from the same temporal direction (either both from the past or both from the future), the signals are different (i.e. τ0·τ1 < 0). In this case, BIO is only applied when the predictions are not from the same moment (i.e. τ0≠τ1), both reference regions have non-zero motion (MVx0, MVy0, MVx1, MVy1≠0), and the block motion vector is proportional to the temporal distance (MVx0 / MVx1 = MVy0 / MVy1 = -τ0 / τ1).

[0385] By minimizing the distance between points A and B ( Fig.32 The motion vector field (v x ,v y ). The model uses only the first linear term of the local Taylor expansion for Δ:

[0386]

[0387] All values ​​in equation (41) depend on the sample point position (i′, j′), which has been ignored from the sign so far. Assuming that the motion is consistent in the local surrounding area, we minimize Δ within a (2M+1)×(2M+1) square window Ω centered at the current prediction point (i, j), where M equals 2:

[0388]

[0389] For this optimization problem, JEM uses a simplified approach, first minimizing in the vertical direction and then minimizing in the horizontal direction. This yields:

[0390]

[0391]

[0392] in,

[0393]

[0394]

[0395]

[0396] To avoid division by zero or very small values, regularization parameters r and m are introduced into equations (43) and (44).

[0397] r=500·4 d-8 Equation (46)

[0398] m=700·4 d-8 Equation (47)

[0399] Here d is the bit depth of the video samples.

[0400] To keep the memory access of BIO the same as for regular bi-predictive motion compensation, all predictions and gradient values are only calculated for the positions inside the current block. In equation (45), the (2M+1)×(2M+1) square window Ω centered at the current prediction point on the boundary of the prediction block needs to access the positions outside the block (such as Fig.33A In JEM, the block outside The value of is set equal to the nearest available value inside the block. For example, this can be implemented as padding, such as Fig.33B shown.

[0401] Using BIO, it is possible to refine the motion field for each sample. To reduce the computational complexity, a block-based BIO design is used in JEM. The motion refinement is calculated based on a 4×4 block. In the block-based BIO, s in Equation 30 is aggregated over all samples in the 4×4 block. n value, then s n The aggregated value of is used to derive the BIO motion vector offset for the 4×4 block. More specifically, the following formula is used for block-based BIO derivation:

[0402]

[0403] where b k represents the sample set of the kth 4×4 block belonging to the prediction block. n ((s n,bk )>>4) instead to derive the associated motion vector offset.

[0404] In some cases, the MV refinement of BIO may be unreliable due to noise or irregular motion. Therefore, in BIO, the amplitude of MV refinement is clipped to a threshold thBIO. The threshold is determined based on whether the reference pictures of the current picture are all from one direction. If all the reference pictures of the current picture are from one direction, the threshold is set to 12×2 14-d ; otherwise, it is set to 12×2 13-d .

[0405] The gradient of the BIO is calculated simultaneously with the motion compensated interpolation using an operation consistent with the HEVC motion compensation process (2D separable FIR). The input to this 2D separable FIR is the same reference frame samples as the motion compensation process and the fractional positions (fracX, fracY) according to the fractional part of the block motion vector. In the case of , the signal is first interpolated vertically using BIOfilterS corresponding to the fractional position fracY with a descaling offset d-8, and then a gradient filter BIOfilterG corresponding to the fractional position fracX with a descaling offset 18-d is applied in the horizontal direction. In the case of , the gradient filter is first applied vertically using BIOfilterG corresponding to the fractional position fracY with the descaling offset d-8, and then the signal displacement is performed in the horizontal direction using BIOfilterS corresponding to the fractional position fracX with the descaling offset 18-d. The length of the interpolation filter BIOfilterG and the signal displacement BIOfilterF used for the gradient calculation is short (6 taps) in order to maintain a reasonable complexity. Table 6 shows the filters for the gradient calculation of different fractional positions of the block motion vectors in the BIO. Table 7 shows the interpolation filters used for the prediction signal generation in the BIO.

[0406] Table 6: Example filters for gradient calculation in BIO

[0407] Fractional pixel position Gradient interpolation filter (BIOfilterG) 0 {8,-39,-3,46,-17,5} 1 / 16 {8,-32,-13,50,-18,5} 1 / 8 {7,-27,-20,54,-19,5} 3 / 16 {6,-21,-29,57,-18,5} 1 / 4 {4,-17,-36,60,-15,4} 5 / 16 {3,-9,-44,61,-15,4} 3 / 8 {1,-4,-48,61,-13,3} 7 / 16 {0,1,-54,60,-9,2} 1 / 2 {-1,4,-57,57,-4,1}

[0408] Table 7: Example interpolation filters for prediction signal generation in BIO

[0409] Fractional pixel position Interpolation filter for prediction signal (BIOfilterS) 0 {0,0,64,0,0,0} 1 / 16 {1,-3,64,4,-2,0} 1 / 8 {1,-6,62,9,-3,1} 3 / 16 {2,-8,60,14,-5,1} 1 / 4 {2,-9,57,19,-7,2} 5 / 16 {3,-10,53,24,-8,2} 3 / 8 {3,-11,50,29,-9,2} 7 / 16 {3,-11,44,35,-10,3} 1 / 2 {3,-10,35,44,-11,3}

[0410] In JEM, BIO is applied to all bidirectional prediction blocks when the two predictions come from different reference pictures. BIO is disabled when LIC is enabled for a CU.

[0411] In JEM, OBMC is applied to the block after the normal MC process. To reduce computational complexity, BIO is not applied during the OBMC process. This means that BIO is only applied to the MC process of a block when using its own MV, and is not applied to the MC process when using the MV of a neighboring block during the OBMC process.

[0412] 2.2.19. Decoder-side motion vector refinement

[0413] In the bidirectional prediction operation, for the prediction of a block region, two prediction blocks formed using the motion vectors (MVs) of list 0 and the MVs of list 1, respectively, are combined to form a single prediction signal. In the decoder-side motion vector refinement (DMVR) method, the two motion vectors of the bidirectional prediction are further refined by a bilateral template matching process. Bilateral template matching is applied in the decoder to perform a distortion-based search between the bilateral template and the reconstructed samples in the reference picture to obtain the refined MV without sending additional motion information.

[0414] In DMVR, bilateral templates are generated from the initial MV0 of list 0 and MV1 of list 1 as a weighted combination (i.e., average) of the two prediction blocks, as shown in Fig.34 As shown. The template matching operation includes calculating a cost metric between the generated template and the sample area (around the initial prediction block) in the reference picture. For each of the two reference pictures, the MV that produces the minimum template cost is considered to be the updated MV in the list to replace the original MV. In JEM, 9 MV candidates are searched for each list. The 9 candidate MVs include the original MV and 8 surrounding MVs, in which one luminance sample is offset relative to the original MV in the horizontal or vertical direction or both. Finally, two new MVs (i.e., MV0′ and MV1′ as shown in Figure 33) are used to generate the final bidirectional prediction result. The sum of absolute differences (SAD) is used as a cost metric. Please note that when calculating the cost of a prediction block generated by a surrounding MV, the rounded MV (rounded to integer pixels) is actually used to obtain the prediction block, not the actual MV.

[0415] DMVR is applied to the Merge mode of bi-directional prediction, where one MV comes from the past reference picture and the other comes from the future reference picture, without transmitting additional syntax elements. In JEM, DMVR is not applied when LIC, affine motion, FRUC or sub-CU Merge candidate is enabled for a CU.

[0416] 3. Related tools

[0417] 3.1.1. Diffusion filter

[0418] In JVET-L0157, a diffusion filter is proposed, where the intra / inter prediction signal of a CU can be further modified by a diffusion filter.

[0419] 3.1.1.1 Uniform diffusion filter

[0420] The uniform diffusion filter is constructed by dividing the prediction signal by a given value h I or IV This is achieved by convolving with a fixed mask (defined below).

[0421] In addition to the prediction signal itself, a row of reconstructed samples to the left and above the block is used as input to the filtered signal, whereby the use of these reconstructed samples in inter-blocks can be avoided.

[0422] Let pred be the prediction signal on a given block obtained by intra-frame or motion compensated prediction. In order to handle the boundary points of the filter, the prediction signal needs to be expanded to the prediction signal pred ext This extended prediction can be formed in two ways: either, as an intermediate step, a row of reconstructed samples to the left and above the block is added to the prediction signal and the resulting signal is then mirrored in all directions. Or only the prediction signal itself is mirrored in all directions. The latter extension is used for inter blocks. In this case, only the prediction signal itself consists of the extended prediction signal pred ext Input.

[0423] If you want to use the filter h I , then it is proposed to use h I *pred replaces the prediction signal pred.

[0424] Use the aforementioned boundary extension. Here, the filter mask h I Given as follows:

[0425]

[0426] If you want to use the filter h IV , then it is proposed to use h IV *pred replaces the prediction signal pred.

[0427] Here, the filter h IV Given as follows:

[0428] h IV =h I *h I *h I *h I . Equation (50)

[0429] 3.1.1.2. Directional diffusion filter

[0430] Instead of using a signal adaptive diffusion filter, a directional filter and a horizontal filter h are used. hor and the vertical filter h ver , which still has a fixed mask. More precisely, the corresponding mask h in the previous section I The uniform diffusion filtering is simply restricted to be applied only in the vertical direction or in the horizontal direction. The vertical filter is applied by applying a fixed filter mask to

[0431]

[0432] is applied to the prediction signal, and the horizontal filter is implemented by using the transposed mask to achieve.

[0433] 3.1.2. Bilateral filter

[0434] A bilateral filter was proposed in JVET-L0406 and is always applied to luma blocks with non-zero transform coefficients and slice quantization parameters greater than 17. Therefore, no signaling is required to use the bilateral filter. Bilateral filtering, if applied, is performed on the decoded samples immediately after the inverse transform. In addition, the filter parameters (i.e., weights) are explicitly derived from the codec information.

[0435] The filtering process is defined as:

[0436]

[0437] Where P 0,0 is the intensity of the current sample point and P0′ ,0 is the modified intensity of the current sample point, P k,0 and W k are the intensity and weight parameters of the kth neighboring sample point respectively. Fig.35 An example of a current sample point and its four neighboring sample points (ie, K=4) is depicted in FIG.

[0438] More specifically, the weight W associated with the kth neighboring sample point k (x) is defined as follows:

[0439] Wk (x) = Distance k ×Range k (x) Equation (52)

[0440] in

[0441]

[0442]

[0443] σ d Depends on the codec mode and codec block size. The described filtering process is applied to intra-codec blocks and to inter-codec blocks when the TU is further partitioned to enable parallel processing.

[0444] In order to better capture the statistical characteristics of the video signal and improve the performance of the filter, the weight function generated by equation (52) is given by the parameters σ listed in Table 8 d Adjust the parameter σ d Depends on the codec mode and block segmentation parameters (minimum size).

[0445] Table 8 σ for different block sizes and coding modes d Example value of

[0446] Min(block width,block height) Intra mode Interframe mode 4 82 62 8 72 52 other 52 32

[0447] To further improve the codec performance, for inter-frame codec blocks when TUs are not segmented, the intensity difference between the current sample and one of its neighboring samples is replaced by the representative intensity difference between two windows covering the current sample and the neighboring samples. Therefore, the filtering process equation is modified as follows:

[0448]

[0449] Where P k,m and P 0,m Indicated by P k,0 and P 0,0 The mth sample value in the window centered at . In the proposed scheme, the window size is set to 3×3. Fig.36 The coverage P is depicted in 2,0 and P 0,0 Example of two windows.

[0450] 3.1.3. Intra-frame block copy

[0451] Decoder:

[0452] In some cases, the currently (partially) decoded picture is considered as a reference picture. The current picture is placed at the last position of reference picture list 0. Therefore, for slices that use the current picture as the only reference picture, its slice type is considered to be a P slice. The bitstream syntax in this approach follows the same syntax structure used for inter-frame coding and the decoding process is unified with inter-frame coding. The only prominent difference is that the block vector (which is a motion vector pointing to the current picture) always uses integer pixel resolution.

[0453] The changes with the block level CPR_flag method are:

[0454] 1. When searching for this pattern in the encoder, the block width and height are both less than or equal to 16.

[0455] 2. When the luma block vector is odd, chroma interpolation is enabled.

[0456] 3. When the SPS flag is turned on, the adaptive motion vector resolution (AMVP) of CPR mode is enabled.

[0457] In this case, when using AMVR, the block vector can switch between 1-pixel integer resolution and 4-pixel integer resolution at the block level.

[0458] Encoder:

[0459] The encoder performs RD checks on blocks with width or height no greater than 16. For non-Merge mode, a block vector search is first performed using a hash-based search. If no valid candidate is found in the hash search, a local search based on block matching is performed.

[0460] In hash-based search, the hash key matching (32-bit CRC) between the current block and the reference blocks is extended to all allowed block sizes. The hash key calculation for each position in the current picture is based on 4×4 blocks. For a current block of larger size, a hash key match with a reference block occurs when all of its 4×4 blocks match the hash keys in the corresponding reference positions. If multiple reference blocks are found to match the current block with the same hash key, the block vector cost of each candidate is calculated and the one with the smallest cost is selected.

[0461] In block matching search, the search range is set to 64 pixels to the left and top of the current block.

[0462] 3.1.4. History-based motion vector prediction

[0463] PCT Application No. PCT / CN2018 / 093987, filed on July 2, 2018, and entitled “MOTION VECTOR PREDICTION BASED ON LOOK-UPTABLES,” the contents of which are incorporated herein by reference, describes one or more lookup tables in which at least one motion candidate is stored to predict motion information of a block.

[0464] A history-based MVP (HMVP) method is proposed, where the HMVP candidate is defined as the motion information of the previous coded block. During the encoding / decoding process, a table with multiple HMVP candidates is maintained. When a new slice is encountered, the table is cleared. Whenever there is an inter-frame coded block, the associated motion information is added to the last entry of the table as a new HMVP candidate. The entire encoding and decoding process is as follows Fig.37 In one example, the table size is set to 1 (eg, L=16 or 6, or 44), which indicates that up to L HMVP candidates can be added to the table.

[0465] (1) In one embodiment, if there are more than L HMVP candidates from the previously coded block, a First-In-First-Out (FIFO) rule is applied so that the table always contains the latest L previously coded motion candidates. Fig.38 An example of applying the FIFO rule to remove HMVP candidates and add new candidates to the table used in the proposed method is depicted.

[0466] (2) In another embodiment, whenever a new motion candidate is added (such as when the current block is inter-coded and in non-affine mode), a redundancy check process is first applied to identify whether there is the same or similar motion candidate in the LUT.

[0467] 3.2.JVET-M0101

[0468] 3.2.1. Add HEVC-style weighted prediction

[0469] The following table shows an example of syntax elements. To implement HEVC-style weighted prediction, the following syntax changes shown in bold may be made to the PPS and slice header:

[0470]

[0471]

[0472]

[0473]

[0474]

[0475] 3.2.2. Calling (bi-prediction with weighted averaging, BWA, also known as GBI) and WP in a mutually exclusive manner

[0476] Although both BWA and WP apply weights to motion compensated prediction signals, the specific operations are different. In the case of bidirectional prediction, WP applies linear weights and constant offsets to each prediction signal, depending on the (weight, offset) parameters associated with the ref_idx used to generate the prediction signal. Although there are some range restrictions on these (weight, offset) values, the allowed range is relatively large and there are no normalization restrictions on the parameter values. WP parameters are signaled at the picture level (in the slice header). WP has been shown to provide very large codec gains for fading sequences, and it is generally desirable to enable WP for such content.

[0477] BWA uses a CU-level index to indicate how to combine two prediction signals in the case of bidirectional prediction. BWA uses a weighted average, i.e., the sum of the two weights equals 1. BWA provides a codec gain of about 0.66% for the CTC RA configuration, but its efficiency is much lower than WP (JVET-L0201) for fading content.

[0478] Method #1 :

[0479] The PPS syntax element weight_bipred_flag is checked to determine whether gbi_idx is signaled at the current CU. The syntax signaling is modified in bold as follows:

[0480]

[0481] If the current slice refers to a PPS with weight_bipred_flag set to 1, the first option will disable BWA for all reference pictures of the current slice.

[0482] Method #2 :

[0483] BWA is disabled only when both reference pictures used in bidirectional prediction have weighted prediction turned on, i.e. the (weights, offsets) parameters of these reference pictures have non-default values. This allows bidirectionally predicted CUs using reference pictures with default WP parameters (i.e. WP is not invoked for these CUs) to still use BWA. The syntax signaling is modified in bold as follows:

[0484]

[0485]

[0486] 4. Question

[0487] In LIC, two parameters including a scaling parameter a and an offset b need to be derived by using adjacent reconstructed pixels, which may cause a delay problem.

[0488] Hybrid intra-frame and inter-frame prediction, diffusion filter, bilateral filter, OBMC and LIC need to further modify the inter-frame prediction signal in different ways, and they all have delay problems.

[0489] 5. Some Example Embodiments and Techniques

[0490] The following detailed inventions should be considered as examples to explain the general concept. These inventions should not be interpreted narrowly. In addition, these inventions can be combined in any way.

[0491] Example 1:

[0492] In some embodiments, LIC is performed only on blocks located at the boundary of CTU (referred to as boundary blocks), and the LIC parameters can be derived using adjacent reconstructed samples of the current CTU. In this case, for blocks not located at the boundary of CTU (referred to as internal blocks), LIC is always disabled without being signaled.

[0493] In some embodiments, the selection of adjacent reconstructed samples outside the current CTU may depend on the position of the block relative to the CTU covering the current block.

[0494] In some embodiments, for blocks at the left boundary of a CTU, only the reconstructed neighboring samples to the left of the CU may be used to derive the LIC parameters.

[0495] In some embodiments, for a block at the top boundary of a CTU, only the reconstructed neighboring samples on the upper side of the block may be used to derive the LIC parameters.

[0496] In some embodiments, for a block at the top left corner of a CTU, the LIC parameters may be derived using reconstructed neighboring samples to the left and / or top of the block.

[0497] In some embodiments, for a block at the upper right corner of a CTU, the reconstructed neighboring samples of the upper right and / or upper edge of the block are used to derive the LIC parameters.

[0498] In some embodiments, for a block at the bottom left corner of a CTU, the LIC parameters are derived using reconstructed neighboring samples to the bottom left and / or left of the block.

[0499] Example 2:

[0500] In some embodiments, LIC parameter sets may be derived only for blocks located at a CTU boundary, and internal blocks of the CTU may inherit from one or more of these LIC parameter sets.

[0501] In some embodiments, a LIC parameter lookup table is maintained for each CTU, and each derived LIC parameter set is inserted into the LIC parameter table. The LIC parameter lookup table may be maintained using the method described in PCT Application No. PCT / CN2018 / 093987, filed on July 2, 2018, entitled “MOTION VECTORPREDICTION BASED ON LOOK-UP TABLES” (the contents of which are incorporated herein by reference).

[0502] In some embodiments, such a lookup table is maintained for each reference picture, and when deriving LIC parameters, LIC parameters are derived for all reference pictures.

[0503] In some embodiments, such a lookup table is cleared at the beginning of each CTU or CTU row or slice or slice group or picture.

[0504] In some embodiments, for intra blocks coded in AMVP mode or affine inter mode, if the LIC flag is true, the LIC parameter set used is explicitly signaled. For example, an index is signaled to indicate which entry of the lookup table is used for each reference picture. In some embodiments, for unidirectionally predicted blocks, the LIC flag will be false if there is no valid entry for the reference picture in the lookup table. In some embodiments, for bidirectionally predicted blocks, the LIC flag will be false if there is no valid entry for any reference picture in the lookup table. In some embodiments, for bidirectionally predicted blocks, the LIC flag may be true if there is a valid entry for at least one reference picture in the lookup table. For each reference picture with valid LIC parameters, one LIC parameter index is signaled.

[0505] In some embodiments, more than one lookup table is maintained. In some embodiments, when encoding / decoding the current CTU, it only uses parameters from the lookup table generated by some previously encoded / decoded CTU, and the lookup table generated by the current CTU is used for some later CTUs. In some embodiments, each lookup table may correspond to one reference picture / one reference picture list / a specific range of block sizes / a specific codec mode / a specific block shape, etc.

[0506] In some embodiments, if the internal block is encoded and decoded in Merge or Affine Merge mode, for spatial and / or temporal Merge candidates, the LIC flag and LIC parameters are inherited from the corresponding neighboring blocks.

[0507] In some embodiments, the LIC flag is inherited and the LIC parameters may be explicitly signaled. In some embodiments, the LIC flag is explicitly signaled and the LIC parameters may be inherited. In some embodiments, both the LIC flag and the LIC parameters are explicitly signaled. In some embodiments, the difference between the LIP parameters of a block and the inherited parameters is signaled.

[0508] In some embodiments, for boundary blocks coded in AMVP mode or affine inter mode, if the LIC flag is true, the LIC parameters used are implicitly derived using the same method as in 2.2.7.

[0509] In some embodiments, if the boundary block is encoded and decoded in Merge or Affine Merge mode, for spatial domain or / and temporal domain Merge candidates, the LIC flag and LIC parameters are inherited from the corresponding neighboring blocks.

[0510] In some embodiments, the LIC flag is inherited and the LIC parameters may be explicitly signaled. In some embodiments, the LIC flag is explicitly signaled and the LIC parameters may be inherited. In some embodiments, both the LIC flag and the LIC parameters are explicitly signaled.

[0511] In some embodiments, if a block is coded in Merge mode, LIC is always disabled for a combined Merge candidate or an average Merge candidate.

[0512] In some embodiments, if the LIC flag of any of the two Merge candidates used to generate the combined Merge candidate or the average Merge candidate is true, the LIC flag is set to true. In some embodiments, for an internal block, if both Merge candidates used to generate the combined Merge candidate or the average Merge candidate use LIC, it can inherit LIC parameters from either of them. In some embodiments, the LIC flag is inherited and the LIC parameters can be explicitly signaled. In some embodiments, the LIC flag is explicitly signaled and the LIC parameters can be inherited. In some embodiments, both the LIC flag and the LIC parameters are explicitly signaled.

[0513] In some embodiments, if a block is encoded in Merge mode from an HMVP Merge candidate, the LIC is always disabled. In some embodiments, if a block is encoded in Merge mode from an HMVP Merge candidate, the LIC is always enabled. In some embodiments, if the LIC flag of the HMVP Merge candidate is true, the LIC flag is set to true. For example, the LIC flag can be explicitly signaled. In some embodiments, if the LIC flag of the HMVP Merge candidate is true, the LIC flag is set to true and the LIP parameters are inherited from the HMVP Merge candidate. In some embodiments, the LIC parameters are explicitly signaled. In some embodiments, the LIC parameters are implicitly derived.

[0514] Example 3:

[0515] LIC can be used with the intra block copy (IBC, or current picture reference) mode.

[0516] In some embodiments, if the block is encoded in intra block copy mode, an indication of LIC usage (e.g., LIC flag) may be further signaled. In some embodiments, in Merge mode, if the IBC encoded block inherits motion information from a neighboring block, it may also inherit the LIC flag. In some embodiments, it may also inherit LIC parameters.

[0517] In some embodiments, if the LIC flag is true, the LIC parameters are implicitly derived using the same method as 2.2.7. In some embodiments, the method described in Example 1 and / or Example 2 may be used. In some embodiments, if LIC is enabled to encode and decode a block, an indication of IBC usage may be further signaled.

[0518] Example 4:

[0519] In some embodiments, hybrid intra and inter prediction, diffusion filters, bilateral filters, OBMC, LIC or any other tool that modifies the inter prediction signal or modifies the reconstructed blocks from motion compensation (eg, causing latency issues) is used exclusively.

[0520] In some embodiments, if LIC is enabled, all other tools are implicitly disabled. If the LIC flag is true (explicitly signaled or implicitly derived), the on / off flags of other tools (if any) are not signaled but are implicitly derived to be off.

[0521] In some embodiments, if hybrid intra and inter prediction is enabled, all other tools are implicitly disabled. If the hybrid intra and inter prediction flag is true (explicitly signaled or implicitly derived), the on / off flags of other tools (if any) are not signaled, but are implicitly derived to be off.

[0522] In some embodiments, if the diffusion filter is enabled, all other tools are implicitly disabled. If the diffusion filter flag is true (explicitly signaled or implicitly derived), the on / off flags of other tools (if any) are not signaled but are implicitly derived to be off.

[0523] In some embodiments, if OBMC is enabled, all other tools are implicitly disabled. If the OBMC flag is true (explicitly signaled or implicitly derived), the on / off flags of other tools (if any) are not signaled but implicitly derived to be off.

[0524] In some embodiments, if the bilateral filter is enabled, all other tools are implicitly disabled. If the bilateral filter flag is true (explicitly signaled or implicitly derived), the on / off flags of other tools (if any) are not signaled, but are implicitly derived to be off.

[0525] In some embodiments, different tools may be checked in sequence. Furthermore, this checking process terminates when a decision is made to enable one of the above tools. In some embodiments, the checking order is LIC → diffusion filter → hybrid intra-frame and inter-frame prediction → OBMC → bilateral filter. In some embodiments, the checking order is LIC → diffusion filter → bilateral filter → hybrid intra-frame and inter-frame prediction → OBMC.

[0526] In some embodiments, the check order is LIC→OBMC→diffusion filter→bilateral filter→hybrid intra-frame and inter-frame prediction. In some embodiments, the check order is LIC→OBMC→diffusion filter→hybrid intra-frame and inter-frame prediction→bilateral filter. In some embodiments, the order can be adaptively changed based on information of previous codecs and / or based on codec information of the current block (e.g., block size / reference picture / MV information / low delay check flag / slice / picture / slice type), such as based on the mode of neighboring blocks.

[0527] Example 5:

[0528] In some embodiments, LIC is used exclusively with GBI or multi-hypothesis inter prediction.

[0529] In some embodiments, the GBI information may be signaled after the LIC information. In some embodiments, whether the GBI information is signaled may depend on the signaled / inferred LIC information. In addition, whether the GBI information is signaled may depend on the LIC information and weighted prediction information associated with at least one reference picture of the current block or the current slice / slice group / picture containing the current block. In some embodiments, this information may be signaled in the SPS / PPS / slice header / slice group header / slice / CTU / CU.

[0530] In some embodiments, the LIC information may be signaled after the GBI information. In some embodiments, whether the LIC information is signaled may depend on the signaled / inferred GBI information. In addition, whether the LIC information is signaled may depend on the GBI information and weighted prediction information associated with at least one reference picture of the current block or the current slice / slice group / picture containing the current block. In some embodiments, this information may be signaled in the SPS / PPS / slice header / slice group header / slice / CTU / CU.

[0531] In some embodiments, if the LIC flag is true, the syntax elements required for GBI or multi-hypothesis inter prediction are not signaled.

[0532] In some embodiments, if the GBI flag is true (eg, unequal weights are applied to two or more reference pictures), syntax elements required for LIC or multi-hypothesis inter prediction are not signaled.

[0533] In some embodiments, if GBI is enabled for blocks of two or more reference pictures with unequal weights, the syntax elements required for LIC or multi-hypothesis inter prediction are not signaled.

[0534] In some embodiments, LIC is used exclusively with sub-block techniques (e.g., affine mode). In some embodiments, LIC is used exclusively with triangle prediction mode. In some embodiments, LIC is always disabled when triangle prediction mode is enabled for a block. In some embodiments, the LIC flag of a TPM Merge candidate may be inherited from spatial or temporal blocks or other kinds of motion candidates (e.g., HMVP candidates). In some embodiments, the LIC flag of a TPM Merge candidate may be inherited from some spatial or temporal blocks (e.g., only A1, B1). In some embodiments, the LIC flag of a TPMMerge candidate may always be set to false.

[0535] In some embodiments, if multi-hypothesis inter prediction is enabled, the syntax elements required for LIC or GBI are not signaled.

[0536] In some embodiments, different tools may be checked in a certain order. In addition, when it is determined that one of the above tools is enabled, this checking process terminates. In some embodiments, the checking order is LIC->GBI->multiple hypothesis inter-frame prediction. In some embodiments, the checking order is LIC->multiple hypothesis inter-frame prediction->GBI. In some embodiments, the checking order is GBI->LIC->multiple hypothesis inter-frame prediction.

[0537] In some embodiments, the check order is GBI->multi-hypothesis inter prediction->LIC. In some embodiments, the order can be adaptively changed based on information of previous codecs and / or based on information of codecs of the current block (e.g., block size / reference map / MV information / low delay check flag / slice / picture / slice type) (such as a mode based on neighboring blocks).

[0538] In some embodiments, the above method may be applied only when the current block is uni-predicted.

[0539] Example 6:

[0540] In some embodiments, LIC is used exclusively with combined inter-intra prediction (CIIP). In some embodiments, CIIP information may be signaled after LIC information. In some embodiments, whether CIIP information is signaled may depend on signaled / inferred LIC information. In addition, whether CIIP information is signaled may depend on LIC information and weighted prediction information associated with at least one reference picture of the current block or the current slice / slice group / picture containing the current block. In some embodiments, this information may be signaled in SPS / PPS / slice header / slice group header / slice / CTU / CU.

[0541] In some embodiments, the LIC information may be signaled after the CIIP information.

[0542] In some embodiments, whether to signal the LIC information may depend on the signaled / inferred CIIP information. In addition, whether to signal the LIC information may depend on the CIIP information and weighted prediction information associated with at least one reference picture of the current block or the current slice / slice group / picture containing the current block. In some embodiments, this information may be signaled in the SPS / PPS / slice header / slice group header / slice / CTU / CU.

[0543] In some embodiments, if the LIC flag is true, the syntax elements required by CIIP are not signaled. In some embodiments, if the CIIP flag is true, the syntax elements required by LIC are not signaled. In some embodiments, if CIIP is enabled, the syntax elements required by LIC are not signaled.

[0544] Example 7:

[0545] In some embodiments, when one or some of the following codec tools are applied (except weighted prediction), weighted prediction cannot be applied:

[0546] a.BIO (also known as BDOF)

[0547] b.CIIP

[0548] c. Affine prediction

[0549] d. Overlapped Block Motion Compensation (OBMC)

[0550] e. Decoder lateral motion vector refinement (DMVR)

[0551] In some embodiments, since weighted prediction is enabled / disabled, some codec tools are disabled / enabled accordingly. In some embodiments, since some codec tools are enabled / disabled, weighted prediction is disabled / enabled accordingly.

[0552] In some embodiments, if the codec tool and weighted prediction are mutually exclusive, then when weighted prediction is applied, information (e.g., a flag) indicating whether the codec tool is used is not signaled. Such information (e.g., a flag) can be inferred to be zero. In some embodiments, this information can still be explicitly signaled in the SPS / PPS / slice header / slice group header / slice / CTU / CU.

[0553] Example 8:

[0554] In some embodiments, LIC may be used exclusively with weighted prediction at the block level.

[0555] In some embodiments, when the current block is bi-directionally predicted, the signaling of LIC information may depend on weighted_bipred_flag.

[0556] In some embodiments, when the current block is uni-directionally predicted, the signaling of LIC information may depend on weighted_pred_flag.

[0557] In some embodiments, signaling of LIC information may depend on weighted prediction parameters associated with one or all reference pictures associated with the current block.

[0558] In some embodiments, if weighted prediction is enabled for some or all reference pictures of a block, LIC may be disabled for the block and syntax elements related to LIC are not signaled.

[0559] In some embodiments, even if weighted prediction is enabled for some or all reference pictures of a block, LIC may still be applied and weighted prediction may be disabled for the block. In some embodiments, LIC may be applied to reference pictures to which weighted prediction is not applied and LIC may be disabled on reference pictures to which weighted prediction is applied.

[0560] In some embodiments, LIC may be used together with weighted prediction for a block.

[0561] Example 9:

[0562] In some embodiments, LIC may be used exclusively with weighted prediction at picture level / slice level / slice group level / CTU group level.

[0563] In some embodiments, if weighted prediction is enabled for some or all reference pictures of a picture / slice / slice group / CTU group, LIC is disabled and all related syntax elements are not signaled.

[0564] In some embodiments, if LIC is enabled for a picture / slice / slice group / CTU group, weighted prediction is disabled for all its reference pictures and all related syntax elements are not signaled.

[0565] In some embodiments, LIC may be used together with weighted prediction.

[0566] Example 10:

[0567] In some embodiments, LIC may be used exclusively with CPR mode. In some embodiments, when CPR mode is enabled for a block, signaling of an indication of LIC usage and / or side information may be skipped. In some embodiments, when LIC mode is enabled for a block, signaling of an indication of CPR usage and / or side information may be skipped.

[0568] Example 11:

[0569] In some embodiments, LIC may be used with multi-hypothesis prediction (as described in 2.2.13, 2.2.14, 2.2.15).

[0570] In some embodiments, the LIC flag is explicitly signaled for multi-hypothesis AMVP and Merge modes (e.g., as described in 2.2.14). In some embodiments, the explicitly signaled LIC flag is applied to both AMVP mode and Merge mode. In some embodiments, the explicitly signaled LIC flag is applied only to AMVP mode, while the LIC flag for Merge mode is inherited from the corresponding Merge candidate. Different LIC flags may be used for AMVP mode and Merge mode. At the same time, different LIC parameters may be derived / inherited for AMVP mode and Merge mode. In some embodiments, Merge mode always disables LIC. In some embodiments, the LIC flag is not signaled, and LIC is always disabled for AMVP mode. However, for Merge mode, the LIC flag or / and LIC parameters may be inherited or derived.

[0571] In some embodiments, the LIC flag is inherited from the corresponding Merge candidate of the multi-hypothesis Merge mode (e.g., as described in 2.2.15). In some embodiments, the LIC flag is inherited for each of the two selected Merge candidates, so different LIC flags can be inherited for the two selected Merge candidates. At the same time, different LIC parameters can be derived / inherited for the two selected Merge candidates. In some embodiments, the LIC flag is inherited only for the first selected Merge candidate, and LIC is always disabled for the second selected Merge candidate.

[0572] In some embodiments, the LIC flag is explicitly signaled for the multi-hypothesis inter prediction mode (e.g., as described in 2.2.13). In some embodiments, if the block is predicted in Merge mode (or UMVE mode) and additional motion information, the explicitly signaled LIC flag may be applied to the Merge mode (or UMVE mode) and the additional motion information. In some embodiments, if the block is predicted in Merge mode (or UMVE mode) and additional motion information, the explicitly signaled LIC flag may be applied to the additional motion information. For Merge mode, the LIC flag and / or LIC parameters may be inherited or derived. In some embodiments, LIC is always disabled for Merge mode.

[0573] In some embodiments, if a block is predicted with Merge mode (or UMVE mode) and additional motion information, the LIC flag is not signaled and is disabled for the additional motion information. For Merge mode, the LIC flag and / or LIC parameters may be inherited or derived. In some embodiments, if a block is predicted with AMVP mode and additional motion information, the explicitly signaled LIC flag may be applied to AMVP mode and additional motion information. In some embodiments, LIC is always disabled for additional motion information. In some embodiments, different LIC parameters may be derived / inherited for Merge mode (or UMVE mode) / AMVP mode and additional motion information.

[0574] In some embodiments, when multiple hypotheses are applied to a block, illumination compensation may be applied to some prediction signals, but not all prediction signals. In some embodiments, when multiple hypotheses are applied to a block, more than one flag may be signaled / derived to indicate the use of illumination compensation for the prediction signal.

[0575] Example 12:

[0576] In some embodiments, the LIC flag may be inherited from the base Merge candidate in UMVE mode. In some embodiments, the LIC parameters are derived implicitly as described in 2.2.7. In some embodiments, for boundary blocks encoded and decoded in UMVE mode, the LIC parameters are derived implicitly as described in 2.2.7. In some embodiments, for internal blocks encoded and decoded in UMVE mode, the LIC parameters are inherited from the base Merge candidate. In some embodiments, the LIC flag may be explicitly signaled in UMVE mode.

[0577] Example 13:

[0578] The above proposed method or LIC may be applied under certain conditions, such as block size, slice / picture / slice type or motion information.

[0579] In some embodiments, when the block size contains less than M*H samples, eg, 16 or 32 or 64 luma samples, the proposed method or LIC is not allowed.

[0580] In some embodiments, when the minimum size of the width or / and height of the block is less than or not greater than X, the proposed method or LIC is not allowed. In one example, X is set to 8.

[0581] In some embodiments, when the width of the block > th1 or >= th1 and / or the height of the block > th2 or >= th2, the proposed method or LIC is not allowed. In one example, th1 and / or th2 is set to 8.

[0582] In some embodiments, when the width of the block < th1 or <= th1 and / or the height of the block < th2 or <= th2, the proposed method or LIC is not allowed. In one example, th1 and / or th2 are set to 8.

[0583] In some embodiments, LIC is disabled for the affine inter prediction mode or / and the affine Merge mode.

[0584] In some embodiments, LIC is disabled for sub-block coding tools (such as ATMVP or / and STMVP or / and the planar motion vector prediction mode).

[0585] In some embodiments, LIC is only applied to certain components. For example, LIC is applied to the luminance component. Alternatively, LIC is applied to the chrominance component.

[0586] In some embodiments, if the LIC flag is true, then BIO or / and DMVR is disabled.

[0587] In some embodiments, LIC is disabled for bi-directionally predicted blocks.

[0588] In some embodiments, LIC is disabled for intra blocks coded with the AMVP mode.

[0589] In some embodiments, LIC is only allowed for uni-directionally predicted blocks.

[0590] Example 14:

[0591] In some embodiments, the selection of neighboring samples used to derive LIC parameters may depend on coding information, block shape, etc.

[0592] In some embodiments, if the width >= height or the width > height, then only the upper neighboring pixels are used to derive LIC parameters.

[0593] In some embodiments, if the width < height, then only the left neighboring pixels are used to derive LIC parameters.

[0594] Example 15:

[0595] In some embodiments, only one LIC flag is signaled for a block with a geometric partitioning structure (such as the triangular prediction mode). In this case, all partitions of the block (all PUs) share the same value of the LIC enable flag.

[0596] In some embodiments, for some PUs, LIC may always be disabled regardless of whether the LIC flag is signaled.

[0597] In some embodiments, if the block is partitioned from the top right corner to the bottom left corner, one LIC parameter set is derived and used for both PUs. In some embodiments, LIC is always disabled for the bottom PU. In some embodiments, if the top PU is coded in Merge mode, the LIC flag is not signaled.

[0598] In some embodiments, if the block is partitioned from the top left corner to the bottom right corner, LIC parameters are derived for each PU. In some embodiments, the top neighboring samples of the block are used to derive the LIC parameters of the top PU, and the left neighboring samples of the block are used to derive the LIC parameters of the left PU. In some embodiments, one LIC parameter set is derived and used for both PUs.

[0599] In some embodiments, if both PUs are coded in Merge mode, the LIC flag is not signaled and may be inherited from the Merge candidate. LIC parameters may be derived or inherited.

[0600] In some embodiments, if one PU is coded in AMVP mode and the other PU is coded in Merge mode, the signaled LIC flag may only be applied to the PU coded in AMVP mode. For the PU coded in Merge mode, the LIC flag and / or LIC parameters are inherited. In some embodiments, the LIC flag is not signaled and is disabled for the PU coded in AMVP mode. For the PU coded in Merge mode, the LIC flag and / or LIC parameters are inherited. In some embodiments, the LIC is disabled for the PU coded in Merge mode.

[0601] In some embodiments, if both PUs are coded in AMVP mode, one LIC flag may be signaled for each PU.

[0602] In some embodiments, one PU may utilize reconstructed samples from another PU within the current block that have already been reconstructed to derive LIC parameters.

[0603] Example 16:

[0604] In some embodiments, LIC may be performed on a portion of pixels instead of the entire block.

[0605] In some embodiments, LIC may be performed only on pixels around the block boundary, while LIC is not performed on other pixels within the block.

[0606] In some embodiments, LIC is performed on the top W×N rows or N×H columns, where N is an integer and W and H represent the width and height of the block. For example, N is equal to 4.

[0607] In some embodiments, LIC is performed on the upper left (Wm)×(Hn) region, where m and n are integers, and W and H represent the width and height of the block. For example, m and n are equal to 4.

[0608] Example 17:

[0609] In some embodiments, the LIC flag may be used to update the HMVP table.

[0610] In some embodiments, the HMVP candidate may include a LIC flag in addition to the motion vector and other information.

[0611] In some embodiments, the LIC flag is not used to update the HMVP table. For example, a default LIC flag value is set for each HMVP candidate.

[0612] Example 18:

[0613] In some embodiments, if LIC is used to encode and decode a block, the updating of the HMVP table can be skipped. In some embodiments, the LIC encoding and decoding block can also be used to update the HMVP table.

[0614] Example 19:

[0615] In some embodiments, candidate motions may be reordered according to the LIC flags.

[0616] In some embodiments, in a Merge / AMVP or other motion candidate list, LIC-enabled candidates may be placed before all or part of LIC-disabled candidates.

[0617] In some embodiments, LIC-disabled candidates are placed before all or part of LIC-enabled candidates in a Merge / AMVP or other motion candidate list.

[0618] In some embodiments, the Merge / AMVP or other motion candidate list construction process may be different for LIC codec blocks and non-LIC codec blocks.

[0619] In some embodiments, for LIC coded blocks, the Merge / AMVP or other motion candidate lists may not contain motion candidates derived from non-LIC coded spatial / temporal neighboring blocks or non-neighboring blocks or HMVP candidates with LIC flag equal to false.

[0620] In some embodiments, for non-LIC coded blocks, the Merge / AMVP or other motion candidate lists may not contain motion candidates derived from LIC coded spatial / temporal neighboring blocks or non-neighboring blocks or HMVP candidates with LIC flag equal to true.

[0621] Example 20:

[0622] In some embodiments, whether to enable or disable the above method can be notified by signaling such as SPS / PPS / VPS / sequence header / picture header / slice header / slice group header / CTU group. In some embodiments, which method to use can be notified by signaling such as SPS / PPS / VPS / sequence header / picture header / slice header / slice group header / CTU group. In some embodiments, whether to enable or disable the above method and / or which method to apply can depend on block dimension, video processing data unit (VPDU), picture type, low delay check flag, current block (such as reference picture, unidirectional prediction or bidirectional prediction) or previously coded block codec information.

[0623] 6. Additional Embodiments

[0624] Method #1 :

[0625] The PPS syntax element weight_bipred_flag is checked to determine whether the LIC_enabled_flag is signaled at the current CU. The syntax signaling is modified in bold as follows:

[0626]

[0627] The first option is to disable LIC for all reference pictures of the current slice / slice / slice group, for example,

[0628] 1. If the current slice references a PPS with weighted_bipred_flag set to 1 and performs bidirectional prediction on the current block;

[0629] 2. If the current slice references a PPS with weighted_pred_flag set to 1 and performs unidirectional prediction on the current block.

[0630] Method #2 :

[0631] For the bidirectional prediction case:

[0632] LIC is disabled only when both reference pictures used in bidirectional prediction have weighted prediction turned on, i.e. the (weight, offset) parameters of these reference pictures have non-default values. This allows bidirectional prediction CUs using reference pictures with default WP parameters (i.e. WP is not invoked for these CUs) to still use LIC.

[0633] For the one-way prediction case:

[0634] LIC is disabled only when the reference pictures used in unidirectional prediction have weighted prediction turned on, i.e. the (weights, offsets) parameters of these reference pictures have non-default values. This allows unidirectional prediction CUs using reference pictures with default WP parameters (i.e. WP is not called for these CUs) to still use LIC. The syntax signaling is modified in bold as follows:

[0635]

[0636] In the above table, X represents the reference picture list (X is 0 or 1).

[0637] For the above example, the following might apply:

[0638] 1. The sps_LIC_enabled_flag that controls the use of LIC in each sequence may be replaced by an indication of the use of LIC per picture / view / slice / slice group / CTU row / region / multiple CTUs / CTUs.

[0639] 2. The control of the signaling of LIC_enabled_flag may also depend on the block dimension.

[0640] 3. Control of the signaling of LIC_enabled_flag may additionally depend on the GBI index.

[0641] 4. The control of the signaling of the GBI index may also depend on the LIC_enabled_flag.

[0642] Fig.39 2600 is a block diagram showing an example of the architecture of a computer system or other control device 2600 that can be used to implement various portions of the presently disclosed technology. Fig.39In the embodiment, computer system 2600 includes one or more processors 2605 and memory 2610 connected via interconnect 2625. Interconnect 2625 can represent any one or more separate physical buses, point-to-point connections, or both connected through appropriate bridges, adapters, or controllers. Thus, interconnect 2625 can include, for example, a system bus, a peripheral component interconnect (PCI) bus, a hypertransport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), an IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) standard 674 bus, sometimes referred to as "FireWire".

[0643] The processor(s) 2605 may include a central processing unit (CPU) to control the overall operation of, for example, a host computer. In some embodiments, the processor(s) 2605 do this by executing software or firmware stored in the memory 2610. The processor(s) 2605 may be or may include one or more programmable general or special purpose microprocessors, digital signal processors (DSPs), programmable controllers, application specific integrated circuits (ASICs), programmable logic devices (PLDs), etc., or a combination of these devices.

[0644] The memory 2610 may be or include the main memory of the computer system. The memory 2610 represents any suitable form of random access memory (RAM), read-only memory (ROM), flash memory, etc., or a combination of these devices. When used, the memory 2610 may include a machine instruction set, which, when executed by the processor 1405, causes the processor 2605 to perform operations to implement embodiments of the technology disclosed herein.

[0645] And, an (optional) network adapter 2615 is also connected to the processor(s) 2605 via the interconnect 2625. The network adapter 2615 provides the computer system 2600 with the ability to communicate with remote devices (such as storage clients and / or other storage servers) and may be, for example, an Ethernet adapter or a Fibre Channel adapter.

[0646] Fig.40A block diagram of an exemplary embodiment of a device 2700 that can be used to implement various parts of the technology disclosed herein is shown. The mobile device 2700 can be a laptop, a smartphone, a tablet, a camcorder, or other types of devices capable of processing video. The mobile device 2700 includes a processor or controller 2701 that processes data, and a memory 2702 that communicates with the processor 2701 to store and / or buffer data. For example, the processor 2701 may include a central processing unit (CPU) or a microcontroller unit (MCU). In some embodiments, the processor 2701 may include a field-programmable gate array (FPGA). In some embodiments, the mobile device 2700 includes a graphics processing unit (GPU), a video processing unit (VPU), and / or a wireless communication unit, or communicates with a GPU, a VPU, and / or a wireless communication unit for various visual and / or communication data processing functions of a smartphone device. For example, the memory 2702 may include and store processor executable code that, when executed by the processor 2701, configures the mobile device 2700 to perform various operations, such as, for example, receiving information, commands, and / or data, processing information and data, and sending or providing the processed information / data to another device, such as an actuator or an external display. To support various functions of the mobile device 2700, the memory 2702 may store information and data, such as instructions, software, values, images, and other data processed or referenced by the processor 2701. For example, various types of random access memory (RAM) devices, read-only memory (RAM) devices, flash memory devices, and other suitable storage media may be used to implement the storage function of the memory 2702. In some embodiments, the mobile device 2700 includes an input / output (I / O) unit 2703 to couple the processor 2701 and / or the memory 2702 to other modules, units, or devices. For example, the I / O unit 1503 can engage the processor 2701 and the memory 2702 to utilize various types of wireless interfaces compatible with typical data communication standards, such as between one or more computers in the cloud and user devices. In some embodiments, the mobile device 2700 can be engaged with other devices using a wired connection via the I / O unit 2703. The mobile device 2700 can also engage with other external interfaces such as data storage and / or visual or audio display devices 2704 to retrieve and transmit data and information that can be processed by the processor, stored in the memory, or displayed on the output unit of the display device 2704 or an external device. For example, the display device 2704 can display video frames according to the disclosed technology.

[0647] Fig.41 is a flowchart representation of a method 4100 for video processing. The method 4100 includes, in converting between a video block and a bitstream representation of the video block, determining (4102) that the video block is a boundary block of a codec tree unit (CTU) in which the video block is located, and thus enabling a local illumination compensation (LIC) codec tool for the video block, deriving (4104) parameters of a local illumination compensation (LIC) for the video block based on the determination that the LIC codec tool is enabled for the video block, and performing (4106) the conversion by adjusting pixel values ​​of the video block using the LIC. For example, item 1 of Section 5 discloses some example variations and embodiments of the method 4100.

[0648] The following clause-based list describes certain features and aspects of the disclosed technology listed in Section 5.

[0649] 1. A video processing method, comprising:

[0650] In converting between the video block and a bitstream representation of the video block, determining that the video block is a boundary block of a codec tree unit (CTU) in which the video block is located, and thus enabling a local illumination compensation (LIC) codec tool for the video block;

[0651] deriving parameters of a local illumination compensation (LIC) for the video block based on determining that the LIC codec is enabled for the video block; and

[0652] The conversion is performed by adjusting the pixel values ​​of the video block using the LIC.

[0653] 2. The method of clause 1, wherein the conversion generates video blocks from a bitstream representation.

[0654] 3. The method of clause 1, wherein the conversion generates a bitstream representation from the video blocks.

[0655] 4. A method according to any of clauses 1 to 3, wherein the derivation uses samples of neighbouring blocks depending on the position of the current block within the CTU.

[0656] 5. A method according to clause 4, wherein (1) when the current block position is located at the left boundary of a CTU, the derivation uses only the reconstructed neighboring samples on the left, or (2) when the current block position is located at the upper boundary of a CTU, the derivation uses only the reconstructed neighboring samples on the upper side, or (3) when the current block position is located at the upper left corner of a CTU, the derivation uses the reconstructed neighboring samples on the left and / or upper side, or (4) when the current block position is located at the upper right corner of a CTU, the derivation uses the reconstructed neighboring samples on the right and / or upper side, or (5) when the current block position is located at the lower left corner of a CTU, the derivation uses the reconstructed neighboring samples on the left and / or lower side.

[0657] 6. A video processing method, comprising:

[0658] In converting between a video block and a bitstream representation of the video block, determining that the video block is an internal block of a codec tree unit (CTU) in which the video block is located, and thus disabling a local illumination compensation (LIC) codec tool for the video block;

[0659] Inherit the parameters of the LIC of the video block; and

[0660] The conversion is performed by adjusting the pixel values ​​of the video block using the LIC.

[0661] 7. The method according to clause 6, wherein the inheritance comprises:

[0662] Maintaining a lookup table (LUT) of LIC parameters previously used for other blocks; and

[0663] LIC parameters for the video block are derived based on one or more LIC parameters from a lookup table.

[0664] 8. A method as described in clause 7, wherein the other blocks include blocks from a CTU or blocks from a reference picture for the current block.

[0665] 9. A method according to any of clauses 7 to 9, wherein the lookup table is cleared and rebuilt for each CTU or row of CTUs or slice or slice group or picture of a video block.

[0666] 10. A method according to clause 7, wherein the current block uses Advanced Motion Vector Prediction (AMVP) mode or Affine Inter mode, then the bitstream representation is configured to include a flag indicating which LIC parameters are used for the current block.

[0667] 11. The method of clause 10, wherein the flag indicates an index to an entry in the LUT.

[0668] 12. A method according to any of clauses 7 to 11, wherein multiple LUTs are used for a CTU.

[0669] 13. A method according to clause 12, wherein at least one LUT is maintained for each reference picture or each reference picture list used in the transformation of the current block.

[0670] 14. The method of clause 7, wherein the current block uses a Merge mode or an Affine Merge mode for a spatial or temporal Merge candidate motion vector, and wherein the inheritance comprises inheriting one or both of a LIC flag and a LIC parameter of a neighboring block.

[0671] 15. A method according to clause 7, wherein the current block uses a Merge mode or an Affine Merge mode for a spatial or temporal Merge candidate motion vector, and wherein the inheritance comprises configuring the bitstream with one or more difference values, and calculating the LIC flag or the LIC parameter based on the one or more difference values ​​and one or both of the LIC flag and the LIC parameter of the neighboring block.

[0672] 16. A method according to clause 1, wherein the current block is encoded using Advanced Motion Vector Prediction (AMVP) or Affine Inter mode, and wherein the parameters are derived using least squares error minimization, in which a sub-set of neighboring samples is used for error minimization.

[0673] 17. A video processing method, comprising:

[0674] In converting between a video block and a bitstream representation of the video block, determining that both local illumination compensation and intra-block copy codec tools are enabled for use with the current block; and

[0675] The conversion is performed by performing local illumination compensation (LIC) and intra block copy operations on the video blocks.

[0676] 18. The method of clause 17, wherein a LIC flag in the bitstream is configured to indicate that LIC is enabled for the current block.

[0677] 19. The method of clause 18, wherein the converting generates video blocks from the bitstream representation.

[0678] 20. The method of clause 18, wherein the converting generates a bitstream representation from the video blocks.

[0679] 21. The method of clause 17, wherein the current block uses Merge mode, and wherein the current block inherits the LIC flag value of the neighboring block.

[0680] 22. The method of clause 17, wherein the current block uses Merge mode, and wherein the current block inherits LIC parameters of a neighboring block.

[0681] 23. A video processing method, comprising:

[0682] During conversion between bitstreams of a video including multiple pictures having multiple blocks, a local illumination compensation (LIC) mode of a current block of the video is determined based on a LIC mode rule; and conversion between the current block and a corresponding bitstream representation of the current block is performed.

[0683] 24. The method of clause 23, wherein the LIC mode rule specifies use of the LIC mode excluding inter prediction, diffusion filter, bilateral filter, overlapped block motion compensation, or tools to modify the inter prediction signal of the current block.

[0684] 25. A method according to clause 24, wherein the LIC mode rules are explicitly signalled in the bitstream.

[0685] 26. The method of clause 23, wherein the LIC mode rule specifies that the LIC mode is to be used only if the current block does not use a generalized bi-prediction (GBI) codec mode.

[0686] 27. A method according to any of clauses 23 or 26, wherein the LIC mode rules are explicitly signalled in the bitstream and the GBI indication is omitted from the bitstream.

[0687] 28. The method of clause 23, wherein the LIC mode rule specifies that the LIC mode is to be used only if the current block does not use the multiple hypothesis coding mode.

[0688] 29. A method according to any of clauses 23 or 28, wherein the LIC mode rules are signalled in the bitstream and the multi-hypothesis coding mode is implicitly disabled.

[0689] 30. A method according to any of clauses 28 to 29, wherein the LIC mode rule specifies that LIC is to be applied to a selected subset of the prediction signal of the multiple hypothesis mode.

[0690] 31. The method of clause 23, wherein the LIC mode rule specifies inheritance of the LIC flag from a base Merge candidate of the current block, the base Merge candidate also using the Final Motion Vector Expression (UMVE) mode.

[0691] 32. A method according to clause 31, wherein the current block inherits its LIC flag from a base Merge candidate of UMVE mode.

[0692] 33. A method according to clause 32, wherein the current block is inherited using the calculation in section 2.2.7.

[0693] 34. A method according to any of clauses 1 to 33, wherein the LIC tool is used only if the current block satisfies a condition related to the size of the current block or the slice type or the slice type or the picture type or the type of motion information associated with the current block.

[0694] 35. The method of clause 24, wherein the condition does not include a block size that is smaller than M*H samples, where M and H are pre-specified integer values.

[0695] 36. A method according to clause 35, wherein M*H is equal to 16 or 32 or 64.

[0696] 37. A method according to clause 34, wherein the condition specifies that the width or height of the current block is less than or not greater than X, where X is an integer.

[0697] 38. The method of clause 37, wherein X=8.

[0698] 39. A method according to clause 34, wherein the condition specifies that the current block corresponds to a luma sample.

[0699] 40. The method of clause 34, wherein the condition specifies that the current block corresponds to a chroma sample.

[0700] 41. A video processing method, comprising:

[0701] During conversion between a video block of video and a bitstream representation of the video, determining local illumination compensation (LIC) parameters for the video block using at least some samples of neighboring blocks of the video block; and

[0702] The conversion between the video block and the bitstream representation is performed by performing LIC using the determined parameters.

[0703] 42. The method of clause 41, wherein the converting generates video blocks from the bitstream representation.

[0704] 43. The method of clause 41, wherein the conversion generates a bitstream representation from the video blocks.

[0705] 44. A method according to any of clauses 41 to 43, wherein identifying at least some samples of neighbouring blocks depends on codec information of the current block or on the shape of the current block.

[0706] 45. A method according to clause 44, wherein the width of the current block is greater than or equal to the height of the current block and thus only upper neighboring pixels are used to derive the LIC parameters.

[0707] 46. ​​A video processing method, comprising:

[0708] Conversion between video and a bitstream representation of the video is performed, wherein the video is represented as video frames including video blocks, and local illumination compensation (LIC) is enabled only for the video blocks using a geometric prediction structure including a triangle prediction mode.

[0709] 47. The method of clause 46, wherein the converting generates video blocks from the bitstream representation.

[0710] 48. The method of clause 46, wherein the conversion generates a bitstream representation from the video blocks.

[0711] 49. A method according to any of clauses 46 to 48, wherein, for a current block with LIC enabled, all prediction unit partitions share the same LIC flag value.

[0712] 50. A method according to any of clauses 46 to 48, wherein, for a current block partitioned from the upper right corner to the lower left corner, a single LIC parameter set is used for both partitions of the current block.

[0713] 51. A method according to any of clauses 46 to 48, wherein, for a current block partitioned from the upper left corner to the lower right corner, a single LIC parameter set is used for each partition of the current block.

[0714] 52. A video processing method, comprising:

[0715] Conversion between a video and a bitstream representation of the video is performed, wherein the video is represented as a video frame including video blocks, and in the conversion to its corresponding bitstream representation, local illumination compensation (LIC) is performed for less than all pixels of a current block.

[0716] 53. The method of clause 52, wherein the converting generates video blocks from the bitstream representation.

[0717] 54. The method of clause 52, wherein the conversion generates a bitstream representation from the video blocks.

[0718] 55. A method according to any of clauses 52 to 54, wherein LIC is performed only on pixels on the boundary of the current block.

[0719] 56. A method according to any of clauses 52 to 54, wherein LIC is performed only on pixels in a top portion of the current block.

[0720] 57. A method according to any of clauses 52 to 54, wherein LIC is performed only on pixels in the right part of the current block.

[0721] 58. A method for video processing, comprising:

[0722] In converting between a video block and a bitstream representation of the video block, determining that both local illumination compensation (LIC) and generalized bi-prediction (GBi) or multi-hypothesis inter prediction codecs are enabled for use with the current block; and

[0723] The conversion is performed by performing LIC and GBi or multi-hypothesis inter prediction operations on the video blocks.

[0724] 59. A method as described in clause 58, wherein information associated with the GBi codec tool is signalled subsequent to information associated with the LIC codec tool.

[0725] 60. A method as described in clause 58, wherein information associated with the LIC codec tool is signalled subsequent to information associated with the GBi codec tool.

[0726] 61. A method according to clause 59 or 60, wherein the information is signaled in a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a slice group header, a slice, a codec unit (CU) or a codec tree unit (CTU).

[0727] 62. The method of clause 58, wherein the LIC flag is true, and wherein the bitstream representation does not include one or more syntax elements associated with a GBi or multiple hypothesis inter prediction codec.

[0728] 63. The method of clause 58, wherein the GBi flag is true, and wherein the bitstream representation does not include one or more syntax elements associated with a LIC or a multiple hypothesis inter prediction codec.

[0729] 64. The method of clause 58, wherein a multi-hypothesis inter prediction (LIC) flag is true, and wherein the bitstream representation does not include one or more syntax elements associated with a LIC or GBi codec.

[0730] 65. A method for video processing, comprising:

[0731] In converting between a video block and a bitstream representation of the video block, determining that both local illumination compensation (LIC) and combined inter-intra prediction (CIIP) codec tools are enabled for use with the current block; and

[0732] The conversion is performed by performing LIC and CIIP operations on the video blocks.

[0733] 66. A method as described in clause 65, wherein information associated with the CIIP codec tool is signalled subsequent to information associated with the LIC codec tool.

[0734] 67. A method as described in clause 65, wherein the information associated with the LIC codec tool is signaled after the information associated with the CIIP codec tool.

[0735] 68. A method according to clause 66 or 67, wherein the information is signaled in a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a slice group header, a slice, a codec unit (CU) or a codec tree unit (CTU).

[0736] 69. The method of clause 65, wherein the LIC flag is true, and wherein the bitstream representation does not include one or more syntax elements associated with the CIIP codec tool.

[0737] 70. The method of clause 65, wherein the CIIP flag is true, and wherein the bitstream representation does not include one or more syntax elements associated with a LIC codec tool.

[0738] 71. A method according to any of clauses 1 to 70, wherein the bitstream is configured with a bit field that controls the conversion operation.

[0739] 72. A method according to clause 71, wherein the bit field is included at a sequence level, a picture level, a video level, a sequence header level, a picture header level, a slice header level, a slice level or a codec tree unit group level.

[0740] 73. A video encoding apparatus comprising a processor configured to implement the method according to any one or more of clauses 1 to 72.

[0741] 74. A video decoding apparatus comprising a processor configured to implement the method according to any one or more of clauses 1 to 72.

[0742] 75. A computer readable medium having stored thereon code which, when executed, causes a processor to perform the method according to any one or more of clauses 1 to 72.

[0743] Fig.42is a flow chart representation of a method 4200 for video processing. The method 4200 includes generating (4210) a motion vector for a current block of video for conversion between a current block of video and a bitstream representation of the video. The method 4200 includes calling (4220) a weighted prediction processing tool or a second processing tool in a mutually exclusive manner. Here, calling the second processing tool includes refining the motion vector. The method 4200 also includes performing (4230) the conversion in accordance with the call.

[0744] In some embodiments, the conversion generates the current block from a bitstream representation. In some embodiments, wherein the conversion generates the bitstream representation from the current block.

[0745] In some embodiments, invoking the weighted prediction processing tool includes applying linear weights and offsets to derive a final prediction.

[0746] In some embodiments, the call includes determining to enable the second processing tool for the current block if the weighted prediction processing tool is disabled for the current block. In some embodiments, the call includes determining to disable the second processing tool for the current block if the weighted prediction processing tool is enabled for the current block. In some embodiments, the call includes determining to enable the weighted prediction processing tool for the current block if the second processing tool is disabled for the current block. In some embodiments, the call includes determining to disable the weighted prediction processing tool for the current block if the second processing tool is enabled for the current block.

[0747] In some embodiments, refining the motion vector comprises applying a motion vector offset to refine the motion vector. The motion vector offset is determined based on a sum of absolute differences derived from different reference lists for the current block. In some embodiments, the sum of absolute differences is calculated based on partial positions derived from different reference lists for the current block. In some embodiments, the second processing tool comprises a decoder-side motion vector refinement (DMVR) process.

[0748] In some embodiments, the calling further comprises calling the weighted prediction processing tool or the third processing tool in a mutually exclusive manner. The third processing tool applies a prediction offset to refine the final prediction. The prediction offset is determined based on at least one of a horizontal or vertical gradient derived from different reference lists for the current block. In some embodiments, the third processing tool comprises a bidirectional optical flow prediction (BDOF) process.

[0749] In some embodiments, the calling further includes calling a weighted prediction processing tool or a fourth processing tool in a mutually exclusive manner. The fourth processing tool includes one of the following: a combined inter-intra prediction (CIIP) processing tool that generates a final prediction based on inter prediction and intra prediction, an affine processing tool that generates a final prediction based on a control point motion vector, or an overlapped block motion compensation (OBMC) processing tool that generates a final prediction based on a motion vector of a sub-block and a spatially adjacent sub-block.

[0750] In some embodiments, in the case where the weighted prediction processing tool is applied to the current block, the information indicating whether the second processing tool is applied is defaulted to have a zero value. In some embodiments, in the case where the weighted prediction processing tool is applied to the current block, the information indicating whether the second processing tool is applied is signaled in a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a slice group header, a slice, a codec unit (CU), or a codec tree unit (CTU).

[0751] The disclosed and other embodiments, modules and functional operations described in this document may be implemented in digital electronic circuits, or in computer software, firmware or hardware, including the structures disclosed in this document and their structural equivalents, or in a combination of one or more of them. The disclosed and other embodiments may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium, for execution by a data processing device or for controlling the operation of the data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material combination that implements a machine-readable propagation signal, or a combination of one or more of them. The term "data processing device" encompasses all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device may also include code that creates an operating environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagation signal is an artificially generated signal, for example, a machine-generated electrical signal, an optical signal, or an electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[0752] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one or more computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0753] The processes and logic flows described herein may be performed by one or more programmable processors running one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0754] For example, processors suitable for executing computer programs include general and special purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operably coupled to receive data from one or more mass storage devices or to transfer data to them or both. However, a computer does not require such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, for example, including semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by or incorporated into a dedicated logic circuit.

[0755] Although this patent document contains many details, these should not be interpreted as limitations on any invention or the scope of the claimed protection, but rather as descriptions of features unique to specific embodiments of specific inventions. Certain features described in this patent document in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although the above-mentioned features may be described as working in certain combinations and even initially claimed to be protected in this way, in some cases, one or more features in the combination may be deleted from the claimed combination, and the claimed combination may be directed to a sub-combination or a variation of a sub-combination.

[0756] Similarly, although operations are described in a particular order in the drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in sequence, or that all of the illustrated operations be performed, in order to achieve the desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0757] Only a few implementations and examples are described, and other implementations, enhancements, and variations can be made based on what is described and illustrated in this patent document.

Claims

1. A video processing method, comprising: generating a motion vector for a current block of a video for conversion between a current block of the video and a bitstream of the video; Invoking a first weighted prediction processing tool or a second processing tool in a mutually exclusive manner, wherein the second processing tool is a decoder-side motion vector refinement (DMVR) tool for refining the motion vector; and performing said conversion in accordance with said calling, wherein invoking the first weighted prediction processing tool comprises applying a linear weight and offset to at least one prediction block of the current block to derive a final prediction of the current block, wherein whether to signal the index associated with the fifth processing tool in the bitstream is based on whether the first weighted prediction processing tool is enabled, the size of the current block, and whether the current block is encoded and decoded in a bi-directional prediction mode, wherein the index indicates whether the fifth processing tool is enabled or disabled for the current block, the fifth processing tool being a bidirectional prediction with CU-level weights; and wherein, in response to enabling the fifth processing tool for the current block, different weights are applied to the first prediction block and the second prediction block of the current block to obtain a final prediction of the current block.

2. The method according to claim 1, wherein: The conversion generates the current block from the bitstream.

3. The method according to claim 1, wherein: The converting generates the bitstream based on the current block.

4. The method according to claim 1, wherein: The call includes: In case the first weighted prediction processing tool is enabled for the current block, it is determined to disable the second processing tool for the current block.

5. The method according to claim 1, wherein: The call includes: Based on disabling the first weighted prediction processing tool for the current block, determining to enable the second processing tool for the current block.

6. The method according to claim 1, wherein: Refining the motion vector includes applying a motion vector offset to refine the motion vector, wherein the motion vector offset is determined based on a sum of absolute differences derived from a first reference block in reference list 0 and a second reference block in reference list 1 of the current block.

7. The method according to claim 6, wherein: The sum of absolute differences is calculated based on partial positions of the first reference block and the second reference block.

8. The method according to claim 1, wherein: In case that the first weighted prediction processing tool is applied to the current block, information indicating whether the second processing tool is applied to the current block is defaulted to have a zero value.

9. The method according to claim 1, wherein: The information indicating whether the second processing tool is applied is signaled in at least one of a sequence parameter set (SPS) or a picture parameter set (PPS).

10. The method according to claim 1, wherein: The calling also includes calling the first weighted prediction processing tool or the third processing tool in a mutually exclusive manner, wherein the third processing tool applies a prediction offset to a prediction sample in a bidirectional prediction direction to derive a final prediction of the current block, wherein the prediction offset is determined based on multiple gradients in different directions.

11. The method according to claim 10, wherein: The third processing tool includes a bidirectional optical flow prediction process.

12. The method according to claim 10, wherein: In case that the first weighted prediction processing tool is applied to the current block, information indicating whether the third processing tool is applied to the current block is defaulted to have a zero value.

13. The method according to claim 10, wherein: The information indicating whether the third processing tool is applied is signaled in at least one of a sequence parameter set (SPS) or a picture parameter set (PPS).

14. The method according to claim 1, wherein: The calling also includes calling the first weighted prediction processing tool or the fourth processing tool in a mutually exclusive manner, and the fourth processing tool includes one of the following: a combined inter-frame intra-frame prediction processing tool that generates a final prediction based on inter-frame prediction and intra-frame prediction, an affine processing tool that generates a final prediction based on a control point motion vector, or an overlapped block motion compensation processing tool that generates a final prediction based on a motion vector of a sub-block and a spatially adjacent sub-block.

15. The method according to claim 1, wherein: The index is signaled in the bitstream if at least one condition is met, wherein the at least one condition comprises: The current block is encoded and decoded in a bidirectional prediction mode; disabling the first weighted prediction processing tool for the luma component and the chroma component of the current block; and The product of the width and height of the current block is equal to or greater than 256.

16. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions stored thereon, wherein: When the instructions are executed by a processor, the processor: generating a motion vector for a current block of a video for conversion between a current block of the video and a bitstream of the video; Invoking a first weighted prediction processing tool or a second processing tool in a mutually exclusive manner, wherein the second processing tool is a decoder-side motion vector refinement (DMVR) tool for refining the motion vector; and performing said conversion in accordance with said calling, wherein invoking the first weighted prediction processing tool comprises applying a linear weight and offset to at least one prediction block of the current block to derive a final prediction of the current block, wherein whether to signal the index associated with the fifth processing tool in the bitstream is based on whether the first weighted prediction processing tool is enabled, the size of the current block, and whether the current block is encoded and decoded in a bi-directional prediction mode, wherein the index indicates whether the fifth processing tool is enabled or disabled for the current block, the fifth processing tool being a bidirectional prediction with CU-level weights; and wherein, in response to enabling the fifth processing tool for the current block, different weights are applied to the first prediction block and the second prediction block of the current block to obtain a final prediction of the current block.

17. The device according to claim 16, wherein: The call includes: In case the first weighted prediction processing tool is enabled for the current block, it is determined to disable the second processing tool for the current block.

18. A non-transitory computer-readable storage medium storing instructions that cause a processor to: generating a motion vector for a current block of a video for conversion between a current block of the video and a bitstream of the video; The first weighted prediction processing tool or the second processing tool is called in a mutually exclusive manner, wherein: The second processing tool is a decoder-side motion vector refinement (DMVR) tool for refining the motion vectors; as well as performing said conversion in accordance with said calling, wherein invoking the first weighted prediction processing tool comprises applying a linear weight and offset to at least one prediction block of the current block to derive a final prediction of the current block, wherein whether to signal the index associated with the fifth processing tool in the bitstream is based on whether the first weighted prediction processing tool is enabled, the size of the current block, and whether the current block is encoded and decoded in a bi-directional prediction mode, wherein the index indicates whether the fifth processing tool is enabled or disabled for the current block, the fifth processing tool being a bidirectional prediction with CU-level weights; and wherein, in response to enabling the fifth processing tool for the current block, different weights are applied to the first prediction block and the second prediction block of the current block to obtain a final prediction of the current block.

19. The medium of claim 18, wherein the calling comprises: In case the first weighted prediction processing tool is enabled for the current block, it is determined to disable the second processing tool for the current block.

20. A method for storing a bit stream of a video, comprising: Generate a motion vector for the current block; Invoking a first weighted prediction processing tool or a second processing tool in a mutually exclusive manner, wherein the second processing tool invoked is a decoder-side motion vector refinement (DMVR) tool for refining the motion vector; and generating the bitstream according to the call; storing the bitstream in a non-transitory computer-readable recording medium, wherein invoking the first weighted prediction processing tool comprises applying a linear weight and offset to at least one prediction block of the current block to derive a final prediction of the current block, wherein whether to signal the index associated with the fifth processing tool in the bitstream is based on whether the first weighted prediction processing tool is enabled, the size of the current block, and whether the current block is encoded and decoded in a bi-directional prediction mode, wherein the index indicates whether the fifth processing tool is enabled or disabled for the current block, the fifth processing tool being a bidirectional prediction with CU-level weights; and wherein, in response to enabling the fifth processing tool for the current block, different weights are applied to the first prediction block and the second prediction block of the current block to obtain a final prediction of the current block.

21. A video processing method, comprising: generating a motion vector for a current block of a video for conversion between the current block of the video and a bitstream of the video; Invoking the weighted prediction processing tool or the second processing tool in a mutually exclusive manner, wherein the second processing tool is a decoder-side motion vector refinement (DMVR) tool for refining the motion vector; and performing said conversion in accordance with said calling, wherein whether to signal an index associated with the fifth processing tool in the bitstream is based on whether the weighted prediction processing tool is enabled, the size of the current block, and whether the current block is encoded and decoded in a bi-directional prediction mode, wherein the index indicates whether the fifth processing tool is enabled or disabled for the current block, the fifth processing tool being a bidirectional prediction with CU-level weights; and wherein, in response to enabling the fifth processing tool for the current block, different weights are applied to the first prediction block and the second prediction block of the current block to obtain a final prediction of the current block.

22. The method according to claim 21, wherein: The conversion generates the current block from the bitstream.

23. The method according to claim 21, wherein: The converting generates the bitstream from the current block.

24. The method according to claim 21, wherein: Invoking the weighted prediction processing tool includes applying linear weights and offsets to derive a final prediction.

25. The method according to any one of claims 21 to 24, wherein: The call includes: In case the weighted prediction processing tool is disabled for the current block, it is determined to enable a second processing tool for the current block.

26. The method according to any one of claims 21 to 24, wherein: The call includes: In case the weighted prediction processing tool is enabled for the current block, it is determined to disable a second processing tool for the current block.

27. The method according to any one of claims 21 to 24, wherein: The call includes: In case the second processing tool is disabled for the current block, it is determined to enable the weighted prediction processing tool for the current block.

28. The method according to any one of claims 21 to 24, wherein: The call includes: In case the second processing tool is enabled for the current block, it is determined to disable the weighted prediction processing tool for the current block.

29. The method according to any one of claims 21 to 24, wherein: Refining the motion vector includes applying a motion vector offset to refine the motion vector, wherein the motion vector offset is determined based on a sum of absolute differences derived from different reference lists for the current block.

30. The method of claim 29, wherein: The sum of absolute differences is calculated based on partial positions derived from different reference lists of the current block.

31. The method according to any one of claims 21 to 24, wherein: The calling also includes calling the weighted prediction processing tool or a third processing tool in a mutually exclusive manner, wherein the third processing tool applies a prediction offset to refine a final prediction, wherein the prediction offset is determined based on at least one of a horizontal or vertical gradient derived from a different reference list of the current block.

32. The method according to claim 31, wherein: The third processing tool includes a bidirectional optical flow prediction (BDOF) process.

33. The method according to any one of claims 21 to 24, wherein: The calling also includes calling a weighted prediction processing tool or a fourth processing tool in a mutually exclusive manner, wherein the fourth processing tool includes one of the following: a combined inter-frame-intra-frame prediction (CIIP) processing tool that generates a final prediction based on inter-frame prediction and intra-frame prediction, an affine processing tool that generates a final prediction based on a control point motion vector, or an overlapped block motion compensation (OBMC) processing tool that generates a final prediction based on a motion vector of a sub-block and a spatially adjacent sub-block.

34. The method according to any one of claims 21 to 24, wherein: In case the weighted prediction processing tool is applied to the current block, information indicating whether the second processing tool is applied is defaulted to have a zero value.

35. The method according to any one of claims 21 to 24, wherein: In the case where the weighted prediction processing tool is applied to the current block, information indicating whether the second processing tool is applied is signaled in a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a slice group header, a slice, a codec unit (CU) or a codec tree unit (CTU).

36. A video processing apparatus comprising a processor configured to implement the method according to any one of claims 21 to 35.

37. A computer readable medium having stored thereon instructions which, when executed, cause a processor to implement the method of any one of claims 21 to 35.

Images