Method and apparatus for multiple hypothesis prediction in video coding systems

Abstract

A method and apparatus for predicting coding are disclosed. According to the method, combined prediction members are determined, wherein each combined prediction member includes a weighted sum of a first prediction candidate and a second prediction candidate using a target weight selected from a weight set. Boundary matching costs associated with the combined prediction members are determined, wherein each boundary matching cost is determined by using prediction samples of a current block based on the combined prediction member with the target weight and neighboring reconstructed samples of the current block for the combined prediction member with the target weight. A final combined prediction based on at least one boundary matching cost decision is then used to encode or decode the current block.

Description

[0001] Cross Reference To

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 294,424, filed December 29, 2021, which is hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0003] The present invention relates to video coding systems. In particular, the present invention relates to multiple-hypothesis prediction coding. BACKGROUND

[0004] Versatile Video Coding (VVC) is the latest international video coding standard developed by the Joint Video Expert Team (JVET) of ITU-T Video Coding Experts Group (VCEG) and ISO / IEC Moving Picture Experts Group (MPEG). The standard has been published as an ISO standard: ISO / IEC 23090-3:2021, Information technology — Coding of audio-visual

[0005] Figure 1A An exemplary adaptive inter / intra video coding system incorporating in-loop processing is illustrated. For intra prediction 110, prediction data is derived from previously coded video data in the current picture. For inter prediction 112, motion estimation (ME) is performed at the encoder side and motion compensation (MC) is performed based on the results of ME to provide prediction data derived from other pictures and motion data. A switch 114 selects either intra prediction 110 or inter prediction 112 and the selected prediction data is provided to a summer 116 to form prediction error, also referred to as residual. The prediction error is then processed by transform (T) 118 and subsequent quantization (Q) 120. The transformed and quantized residual is then coded by an entropy coder 122 to be included in a video bitstream corresponding to compressed video data. The bitstream associated with the transform coefficients is then packed with side information, such as motion and coding modes associated with intra and inter prediction, as well as other information, such as parameters associated with in-loop filters applied to regions of the underlying picture. Side information associated with intra prediction 110, inter prediction 112, and in-loop filters 130 is provided to entropy coder 122, as is Figure 1AAs shown. When using inter-frame prediction mode, one or more reference images must also be reconstructed at the encoder end. Therefore, the transformed and quantized residuals are processed by inverse quantization (IQ) 124 and inverse transform (IT) 126 to recover the residuals. Then, at reconstruction (REC) 128, the residuals are added back to the prediction data 136 to reconstruct the video data. The reconstructed video data can be stored in the reference image buffer 134 and used to predict other frames.

[0006] like Figure 1A As shown, the input video data undergoes a series of processing steps in the encoding system. Due to these processing steps, the reconstructed video data from REC128 may suffer various forms of degradation. Therefore, loop filters 130 are often applied to the reconstructed video data before it is stored in the reference picture buffer 134 to improve video quality. For example, deblocking filters (DF), sample adaptive offset (SAO), and adaptive loop filters (ALF) can be used. It may be necessary to incorporate loop filter information into the bitstream so that the decoder can correctly recover the required information. Therefore, loop filter information is also provided to the entropy encoder 122 for incorporation into the bitstream. Figure 1A In this process, the loop filter 130 is applied to the reconstructed video before the reconstructed samples are stored in the reference image buffer 134. Figure 1A The system described herein is intended to illustrate an exemplary architecture of a typical video encoder. It may correspond to a High Efficiency Video Coding (HEVC) system, VP8, VP9, ​​H.264, or VVC.

[0007] like Figure 1B As shown, the decoder can use similar or identical function blocks to the encoder, except for transform 118 and quantization 120, since the decoder only needs inverse quantization 124 and inverse transform 126. Instead of entropy encoder 122, the decoder uses entropy decoder 140 to decode the video bitstream into quantized transform coefficients and the required coding information (e.g., ILPF information, intra-frame prediction information, and inter-frame prediction information). Intra-frame prediction 150 on the decoder side does not require mode search. Instead, the decoder only needs to generate intra-frame predictions based on the intra-frame prediction information received from entropy decoder 140. Furthermore, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) based on the inter-frame prediction information received from entropy decoder 140 without motion estimation.

[0008] According to VVC, similar to HEVC, the input image is divided into non-overlapping square block regions called CTUs (Coding Tree Units). Each CTU can be further divided into one or more smaller coding units (CUs). The resulting CU partitions can be squares or rectangles. Furthermore, VVC divides the CTUs into prediction units (PUs) as units for applying prediction processes, such as inter-frame prediction, intra-frame prediction, etc.

[0009] The VVC standard incorporates various new coding tools to further improve coding efficiency over the HEVC standard. Among the various new coding tools, some of the coding tools relevant to the present invention are summarized as follows.

[0010] Inter prediction overview

[0011] According to Section 3.4 of JVET-T2002 (Jianle Chen et al., “Algorithm description of VVC test model 11 (VTM 11)”, Joint Video Team (JVT) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 20th Meeting, Teleconference, 7-16 Oct 2020, Doc: JVET-T2002), for each inter prediction CU, the motion parameters include the motion vector, the reference picture index and the reference picture list usage flag, as well as the additional information needed for inter prediction sample generation. The motion parameters can be signaled in an explicit or implicit manner. When a CU is coded in skip mode, the CU is associated with one PU and has no significant residual coefficients, no coded motion vector difference or reference picture index. Merge mode is specified, where the motion parameters of the current CU are obtained from neighboring CUs, including spatial and temporal candidates, as well as additional scheduling introduced in VVC. Merge mode can be applied to any inter prediction CU, not only in skip mode. An alternative to merge mode is the explicit transmission of motion parameters, where the motion vector, the corresponding reference picture index for each reference picture list and the reference picture list usage flag, as well as other needed information, are explicitly sent per CU.

[0012] In addition to the inter coding functionality in HEVC, VVC includes many new and improved inter prediction coding tools, listed as follows:

[0013] Extended merge prediction

[0014] Merge mode with MVD (MMVD)

[0015] Symmetric MVD (SMVD) signalling

[0016] Affine motion compensated prediction

[0017] - Subblock-based temporal motion vector prediction (SbTMVP)

[0018] - Adaptive motion vector resolution (AMVR)

[0019] - Motion field storage: 1 / 16 luma sample MV storage and 8x8 motion field compression

[0020] - Bi-prediction with CU-level weight (BCW)

[0021] - Bi-directional optical flow (BDOF)

[0022] - Decoder side motion vector refinement (DMVR)

[0023] - Geometric partitioning mode (GPM)

[0024] - Combined inter and intra prediction (CIIP)

[0025] The following description provides details of those inter prediction methods specified in VVC.

[0026] Extended merge prediction

[0027] In VVC, the merge candidate list is constructed by including the following five categories of candidates in turn:

[0028] 1) Spatial MVP from spatial neighboring CUs

[0029] 2) Temporal MVP from collocated CUs

[0030] 3) History-based MVP from FIFO table

[0031] 4) Pairwise average MVP

[0032] 5) Zero MV.

[0033] The size of the merge list is signaled in the sequence parameter set (SPS) header and the maximum allowed size of the merge list is 6. For each CU coded in merge mode, the index of the best merge candidate is coded using truncated unary (TU). The first bin of the merge index is coded using context coding, and bypass coding is used for the remaining bins.

[0034] This section provides the derivation process of each category of merge candidate. As done in HEVC, VVC also supports deriving the merge candidate list (or called merge candidate list) in parallel for all CUs within a certain size of region.

[0035] Spatial candidate derivation

[0036] The derivation of spatial merge candidate in VVC is the same as in HEVC except that the positions of the first two merge candidates are swapped. In the positions of Figure 2 The top four merge candidates (B0, A0, B1, and A1) of the current CU 210 are selected among the candidates in the positions shown. The order of derivation is B0, A0, B1, A1, and B2. The position B2 is only considered when one or more neighboring CUs of the positions B0, A0, B1, A1 are not available (e.g., belong to another slice or tile) or are intra coded. After adding the candidate of position A1, a redundancy check is performed for the addition of the remaining candidates, which guarantees that candidates with the same motion information are excluded from the list, thus improving the coding efficiency. To reduce the computational complexity, all possible pairs of candidates are not considered in the mentioned redundancy check. Instead, only the pairs linked with arrows in Figure 3 are considered, and a candidate is only added to the list when the corresponding candidate used for the redundancy check does not have the same motion information.

[0037] Temporal candidate derivation

[0038] In this step, only one candidate is added to the list. Specifically, in the derivation of this temporal merge candidate for the current CU 410, a scaled motion vector is derived based on a collocated CU 420 belonging to a collocated reference picture as shown. Figure 4 The reference picture list and the reference index of the collocated CU are explicitly signaled in the slice header. As shown in Figure 4A scaled motion vector 430 for a temporal merge candidate is derived as shown by the dashed line in FIG. 4, using the POC (Picture Order Count) distances tb and td from the motion vector 440 of the co-located CU, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal merge candidate is set equal to zero.

[0039] The position of the temporal candidate is selected between the positions C0 and C1 as shown in FIG. 5. Figure 5 If the CU at position C0 is not available, intra coded or outside the current CTU row, then the position C1 is used. Otherwise, the position C0 is used to derive the temporal merge candidate.

[0040] History-based merge candidate derivation

[0041] History-based MVP (HMVP) merge candidates are added to the merge list after the spatial MVPs and TMVPs. In this method, the motion information of previously coded blocks is stored in a table and used as MVPs for the current CU. The table with multiple HMVP candidates is maintained during the encoding / decoding process. The table will be reset (emptied) when a new CTU row is encountered. Whenever there is a non-subblock inter coded CU, the associated motion information is added as a new HMVP candidate to the last entry of the table.

[0042] The HMVP table size S is set to 6, which indicates that up to 5 history-based MVP (HMVP) candidates can be added to the table. When a new motion candidate is inserted into the table, a constrained first-in-first-out (FIFO) rule is used, where a redundancy check is first applied to find if there is a same HMVP in the table. If found, the same HMVP is removed from the table and all the HMVP candidates after it are moved forward, and the same HMVP is inserted to the last entry of the table.

[0043] The HMVP candidates can be used in the merge candidate list construction process. The latest few HMVP candidates in the table are checked in turn and inserted into the candidate list after the TMVP candidates. A redundancy check is applied to the HMVP candidates to spatial or temporal merge candidates.

[0044] To reduce the number of redundancy check operations, the following simplifications are introduced:

[0045] 1. The last two entries in the table are checked for redundancy to the A1 and B1 spatial candidates, respectively.

[0046] 2. The merge candidate list construction process of HMVP is terminated once the total number of available merge candidates reaches the maximum allowed number of merge candidates - 1.

[0047] Pairwise average merge candidate derivation

[0048] A pairwise average candidate is generated by averaging a predefined pair of candidates in the existing merge candidate list using the first two merge candidates. The first merge candidate is defined as p0Cand, and the second merge candidate can be defined as p1Cand, respectively. An average motion vector is calculated for each reference list according to the availability of the motion vectors of p0Cand and p1Cand, respectively. If both motion vectors are available in one list, they are averaged even if they point to different reference pictures, and their reference picture is set to one of p0Cand and p0Cand; if only one motion vector is available, it is used directly; if no motion vector is available, this list is kept invalid. In addition, if the half-pel interpolation filter indices of p0Cand and p1Cand are different, they are set to 0.

[0049] When the merge list is not full after adding the pairwise average merge candidate, zero MVPs are inserted at the end until the maximum number of merge candidates is met.

[0050] Merge estimation region

[0051] Merge estimation region (MER) allows independent derivation of the merge candidate list for CUs in the same MER. The candidate blocks that are within the same MER as the current CU are not included in the generation of the merge candidate list of the current CU. In addition, only when (xCb + cbWidth) » Log2ParMrgLevel is greater than xCb » Log2ParMrgLevel and (yCb + cbHeight) » Log2ParMrgLevel is greater than (yCb » Log2ParMrgLevel), where (xCb, yCb) is the top-left luma sample position of the current CU in the picture and (cbWidth, cbHeight) is the CU size. The MER size is selected at the encoder side and signaled in the sequence parameter set (SPS) as log2_parallel_merge_level_minus2.

[0052] Merge mode with MVD (MMVD)

[0053] In addition to the merge mode where the implicitly derived motion information is directly used for the prediction sample generation of the current CU, a merge mode with motion vector difference (MMVD) is introduced in VVC. The MMVD flag is signaled immediately after the regular merge flag to specify whether the MMVD mode is used for the CU.

[0054] In MMVD, after a merge candidate (referred to as a base merge candidate in this disclosure) is selected, it is further refined by the signaled MVD information. The further information includes a merge candidate identification, an index for specifying the motion magnitude, and an index for indicating the motion direction. In MMVD mode, one of the first two candidates in the merge list is selected to be used as the MV base. The MMVD candidate identification is signaled to specify which one is used between the first and second merge candidates.

[0055] The distance index specifies the motion magnitude information and indicates a predefined offset from the starting point (612 and 622) of the L0 reference block 610 and the L1 reference block 620. As shown in Figure 6 the offset is added to the horizontal component or the vertical component of the starting MV, where different patterns of small circles correspond to different offset amounts from the center. The relationship of the distance index and the predefined offset amounts is specified in Table 1.

[0056] Table 1 - Relationship of distance index and predefined offset amounts

[0057]

[0058] The direction index represents the direction of the MVD relative to the starting point. The direction index can represent four directions as shown in Table 2. It is noted that the meaning of the MVD sign can change depending on the information of the starting MV. When the starting MV is a non-predicted MV or a bi-predicted MV, both lists point to the same side of the current picture (i.e., both reference POCs are greater than the POC of the current picture or both are less than the POC of the current picture), the sign in Table 2 specifies the sign of the MV offset added to the MV component of the starting MV. When the starting MV is a bi-predicted MV, the two MVs point to different sides of the current picture (i.e., one reference POC is greater than the POC of the current picture and the other reference POC is less than the POC of the current picture), and the difference of the POCs in list 0 (can be simply referred to as list 0 or L0) is greater than the POCs in list 1 (can be simply referred to as list 1 or L1), the sign in Table 2 specifies that the sign of the MV offset added to the MV component of the list 0 of the starting MV and the sign of the MV of the list 1 have opposite values. Otherwise, if the difference of the POCs in list 1 is greater than list 0, the sign in Table 2 specifies that the sign of the MV offset added to the MV component of the list 1 of the starting MV and the sign of the MV of the list 0 have opposite values.

[0059] The MVD is scaled according to the difference of the POCs in each direction. If the difference of the POCs in the two lists is the same, no scaling is needed. Otherwise, if the difference of the POCs in list 0 is greater than the difference in list 1, the MVD of list 1 is scaled by defining the POC difference of L0 as td and the POC difference of L1 as tb, as Figure 5The MVDs of List 0 are scaled in the same way if the POC difference of L1 is greater than L0. If the starting MV is uni-predicted, the MVD is added to the available MV.

[0060] Table 2 - MV offset sign specified by direction index

[0061] Direction Index 00 01 10 11 X-axis + - N / A N / A Y-axis N / A N / A + -

[0062] Affine motion compensation prediction

[0063] In HEVC, only translational motion model is applied for motion compensated prediction (MCP). While in real world, there are many kinds of motions, such as zoom-in / zoom-out, rotation, perspective motion and other irregular motions. In VVC, block-based affine transform motion compensated prediction is applied. As shown in the figure. In Figure 7A In -B, the affine motion field of block 710 is described by the motion information of two control points (4 parameters) in or Figure 7A three control point motion vectors (6 parameters) in. Figure 7B For 4-parameter affine motion model, the motion vector at sample position (x, y) in the block is derived as:

[0064]

[0065] For 6-parameter affine motion model, the motion vector at sample position (x, y) in the block is derived as:

[0066]

[0067] where (mv 0x , mv 0y ) is the motion vector of the top-left control point, (mv 1x , mv 1y ) is the motion vector of the top-right control point, and (mv 2x , mv 2y ) is the motion vector of the bottom-left control point.

[0068] To simplify the motion compensated prediction, block-based affine transform prediction is applied. To derive the motion vector of each 4x4 luma sub-block, the motion vector of the center sample of each sub-block is calculated according to the above equation and rounded to 1 / 16 fractional precision. Then, a motion compensated interpolation filter is applied to generate the prediction of each sub-block with the derived motion vector. The sub-block size of chroma components is also set to 4x4. The MV of 4x4 chroma sub-block is calculated as the average of the MVs of the top-left and bottom-right luma sub-blocks in the 8x8 luma region.

[0069] To simplify the motion compensated prediction, block-based affine transform prediction is applied. To derive the motion vector of each 4x4 luma sub-block, the motion vector of the center sample of each sub-block is calculated according to the above equation and rounded to 1 / 16 fractional precision. Then, a motion compensated interpolation filter is applied to generate the prediction of each sub-block with the derived motion vector. The sub-block size of chroma components is also set to 4x4. The MV of 4x4 chroma sub-block is calculated as the average of the MVs of the top-left and bottom-right luma sub-blocks in the 8x8 luma region. Figure 8

[0070] ​For translational motion inter-frame prediction, there are also two affine motion inter-frame prediction modes: affine merging mode and affine AMVP mode.

[0071] Affine Merge Prediction (AF_MERGE)

[0072] The AF_MERGE mode can be applied to CUs with a width and height greater than or equal to 8. In this mode, the CPMV (Control Point MV) of the current CU is generated based on the motion information of spatially adjacent CUs. There can be up to five CPMV prediction candidates, and an index is signaled to indicate which one should be used for the current CU. The following three types of CPMV candidates are used to construct the affine merging candidate list:

[0073] – Inherited affine merge candidates inferred from the CPMV of adjacent CUs

[0074] – Constructing affine merge candidate CPMVPs using translational MV derivations of adjacent CUs

[0075] -Zero MV

[0076] In VVC, there are at most two inherited affine candidates, which come from the affine motion models of neighboring blocks: one from the left-neighboring CU and one from the top-neighboring CU. Candidate blocks and Figure 2 The same applies. For the left predictor, the scan order is A0->A1, and for the upper predictor, the scan order is B0->B1->B2. Only the first inherited candidate for each side is selected. No pruning check is performed between two inherited candidates. When identifying adjacent affine CUs, their control point motion vectors are used to derive CPMVP candidates from the affine merge list of the current CU. Figure 9 As shown, if the lower-left adjacent block A of the current block 910 is encoded in affine mode, then the motion vectors v2, v3, and v4 of the upper-left corner, upper-right corner, and lower-left corner containing block A of CU 920 are obtained. When block A is encoded using a 4-parameter affine model, the two CPMVs (i.e., v0 and v1) of the current CU are calculated based on v2 and v3. When block A is encoded using a 6-parameter affine model, the three CPMVs of the current CU are calculated based on v2, v3, and v4.

[0077] Constructing affine candidates refers to building candidates by combining the translational motion information of adjacent control points. For example... Figure 10As shown, the motion information of the control points is derived from the specified spatial and temporal neighbors of the current block 1010. CPMVk(k = 1, 2, 3, 4) denotes the k-th control point. For CPMV1, the B2->B3->A2 blocks are checked and the MV of the first available block is used. For CPMV2, the Bl->B0 blocks are checked, for CPMV3, the Al->A0 blocks are checked. If TMVP is available, it is used as CPMV4.

[0078] After obtaining the MVs of the four control points, affine merge candidates are constructed based on the motion information. The following combinations of control point MVs are used to construct in order:

[0079] {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4},

[0080] {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}

[0081] The combinations of 3 CPMVs construct 6-parameter affine merge candidates and the combinations of 2 CPMVs construct 4-parameter affine merge candidates. To avoid the motion scaling process, if the reference indices of the control points are different, the related combinations of control point MVs are discarded.

[0082] After checking the inherited affine merge candidates and the constructed affine merge candidates, if the list is still not full, zero MVs are inserted at the end of the list.

[0083] Affine AMVP prediction

[0084] The affine AMVP mode can be applied to a CU whose width and height are both greater than or equal to 16. An affine flag at CU level is signaled in the bitstream to indicate whether the affine AMVP mode is used or not, and then another flag is signaled to indicate whether 4-parameter affine or 6-parameter affine is used. In this mode, the difference between the CPMVs of the current CU and their prediction variables CPMVPs are signaled in the bitstream. The affine AMVP candidate list size is 2, which is generated in order by the following four kinds of CPVM candidates:

[0085] - inherited affine AMVP candidate inferred from the CPMVs of neighboring CUs

[0086] - constructed affine AMVP candidate CPMVP derived using the translation MVs of neighboring CUs

[0087] - translation MVs from neighboring CUs

[0088] - zero MVs

[0089] The checking order of inherited affine AMVP candidates is the same as the checking order of inherited affine merge candidates. The only difference is that for AMVP candidates, only affine CUs with the same reference picture as the current block are considered. When inserting inherited affine motion predictors into the candidate list, no clipping process is applied.

[0090] Constructed AMVP candidates are derived from the designated spatial neighbors as shown in Figure 10 The same checking order as in the affine merge candidate construction is used. In addition, the reference picture index of the neighboring blocks is checked. In the checking order, the first block that is inter coded and has the same reference picture as in the current CU is used. When the current CU is coded using 4-parameter affine mode and both mv0 and mv1 are available, they are added as one candidate to the affine AMVP list. When the current CU is coded using 6-parameter affine mode and all three CPMVs are available, they are added as one candidate to the affine AMVP list. Otherwise, the constructed AMVP candidate is set to be unavailable.

[0091] If after inserting the valid inherited affine AMVP candidates and constructed AMVP candidates, the number of affine AMVP candidates is still less than 2, mv0, mv1 and mv2 are added as translational MVs to predict all control point MVs of the current CU when available. Finally, if the affine AMVP list is still not full, it is padded with zero MVs.

[0092] Affine motion information is stored

[0093] In VVC, the CPMVs of an affine CU are stored in a separate buffer. The stored CPMVs are only used to generate inherited CPMVs for the recently coded CU in affine merge mode and affine AMVP mode. The subblock MVs derived from the CPMVs are used for motion compensation, MV derivation for the merge / AMVP list of translational MVs and deblocking.

[0094] To avoid an extra picture row buffer for the CPMVs, the affine motion data inherited from the CUs of the above CTU are treated differently for inheritance from normal neighboring CUs. If the candidate CU for affine motion data inheritance is in the above CTU row, the left-bottom and right-bottom subblock MVs in the row buffer are used for affine MVP derivation instead of the CPMVs. In this way, the CPMVs are only stored in the local buffer. If the candidate CU is 6-parameter affine coded, the affine model is degenerated to the 4-parameter model. As shown in Figure 11 along the top CTU boundary, the left-bottom and right-bottom subblock motion vectors of a CU are used for affine inheritance of the CU in the bottom CTU. In Figure 11In the middle, lines 1110 and 1112 represent the x and y coordinates of the picture with the origin (0, 0) at the top-left corner. Legend 1120 shows the meaning of various motion vectors, where arrow 1122 represents the CPMV in the local buffer for affine inheritance, arrow 1124 represents the MC / merge / skip MVP / deblock / TMVP and inheritance in the subblock vector line buffer for MC / merge / skip MVP / deblock / TMVP and affine in the local buffer, and arrow 1126 represents the subblock vector for MC / merge / skip MVP / deblock / TMVP.

[0095] Adaptive Motion Vector Resolution (AMVR)

[0096] In HEVC, when use_integer_mv_flag in slice header is equal to 0, the motion vector difference (MVD) (between the motion vector and the prediction motion vector of a CU) is signaled in quarter luma sample unit. In VVC, a CU-level adaptive motion vector resolution (AMVR) scheme is introduced. AMVR allows the MVD of a CU to be coded with different precision. According to the mode of the current CU (normal AMVP mode or affine AMVP mode), the MVD of the current CU can be adaptively selected as follows:

[0097] - Normal AMVP mode: quarter luma sample, half luma sample, integer luma sample, or four luma sample.

[0098] - Affine AMVP mode: quarter luma sample, integer luma sample, or 1 / 16 luma sample.

[0099] The CU-level MVD resolution indication is conditionally signaled if the current CU has at least one non-zero MVD component. If all MVD components (i.e., horizontal and vertical MVDs for reference list L0 and reference list L1) are zero, quarter luma sample MVD resolution is inferred.

[0100] For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter luma sample MVD precision is used for the CU. If the first flag is 0, no further signaling is needed and quarter luma sample MVD precision is used for the current CU. Otherwise, a second flag is signaled to indicate whether half luma sample or other MVD precision (integer or quarter luma sample) is used for normal AMVP CUs. In the case of half luma sample, the half luma sample position uses a 6-tap interpolation filter instead of the default 8-tap interpolation filter. Otherwise, a third flag is signaled to indicate whether integer luma sample or quarter luma sample MVD precision is used for normal AMVP CUs. In the case of affine AMVP CUs, the second flag is used to indicate whether integer luma sample or 1 / 16 luma sample MVD precision is used. To ensure that the reconstructed MVs have the expected precision (quarter luma sample, half luma sample, integer luma sample or quarter luma sample), the motion vector predictors of the CU are rounded to the same MVD precision before the MVDs are added. The motion vector predictor is rounded to zero (i.e., negative motion vector predictor is rounded to positive infinity and positive motion vector predictor is rounded to negative infinity).

[0101] The encoder uses the RD check to determine the motion vector resolution for the current CU. To avoid always performing four CU-level RD checks for each MVD resolution, the RD check for MVD precisions other than quarter luma sample is conditionally invoked in VTM11 only. For normal AVMP mode, the RD cost of quarter luma sample MVD precision and integer luma sample MV precision are first calculated. Then, the RD cost of integer luma sample MVD precision is compared with the RD cost of quarter luma sample MVD precision to decide whether it is necessary to further check the RD cost of quarter luma sample MVD precision. When the RD cost of quarter luma sample MVD precision is much smaller than the RD cost of integer luma sample MVD precision, the RD check of quarter luma sample MVD precision is skipped. Then, the check of half luma sample MVD precision is skipped if the RD cost of integer luma sample MVD precision is significantly larger than the best RD cost of the previously tested MVD precisions. For affine AMVP mode, if the affine inter mode, affine AMVP mode is not selected after checking the rate distortion cost of affine merge / skip mode, merge / skip mode, quarter luma sample MVD precision normal AMVP mode and quarter luma sample MVD precision, the 1 / 16 luma sample MV precision and 1-pixel MV precision affine inter mode are not checked. Furthermore, the affine parameters obtained in the quarter luma sample MV precision affine inter mode are used as the starting search points for the 1 / 16 luma sample and quarter luma sample MV precision affine inter mode.

[0102] Bi-prediction with CU-level weights (BCW)

[0103] In HEVC, bi-predicted signal P bi-pred is generated by averaging two prediction signals P0 and P1 obtained from two different reference pictures and / or using two different motion vectors. In VVC, the bi-prediction mode is extended beyond simple averaging to allow a weighted average of the two prediction signals.

[0104] P bi-pred = ((8-w)*P0 + w*P1 + 4) » 3 (3)

[0105] The weighted average bi-prediction allows five weights, w e {-2, 3, 4, 5, 10}. For each bi-predicted CU, the weight w is determined by one of the following two ways: 1) for non-merge CUs, the weight index is signaled after the motion vector difference; 2) for merge CUs, the weight index is inferred from the merge candidate index from neighboring blocks. BCW is only applied to CUs with 256 or more luma samples (i.e., CU width times CU height is greater than or equal to 256). For low-delay pictures, all 5 weights are used. For non-low-delay pictures, only 3 weights (w e {3, 4, 5}) are used. At the encoder, fast search algorithms are applied to find the weight index without significantly increasing the complexity of the encoder. These algorithms are summarized as follows. Detailed information is disclosed in the VTM software and document JVET-L0646 (Yu-Chi Su et al., “CE4-related: Generalized bi-prediction improvements incorporating JVET-L0197 and JVET-L0296,” Joint Video Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, Twelfth Meeting: Macao, P.R. China, 3-12 October 2018, File: JVET-L0646).

[0106] When combined with AMVR, if the current picture is a low-delay picture, unequal weights are conditionally checked only for 1-pixel and 4-pixel motion vector precisions.

[0107] When combined with affine, affine ME is performed for unequal weights only when and if the affine mode is selected as the current best mode.

[0108] When the two reference pictures in bi-prediction are the same, unequal weights are conditionally checked only.

[0109] Unequal weights are not searched when certain conditions are met, depending on the POC distance between the current picture and its reference pictures, the coding QP, and the temporal level.

[0110] The BCW weight index is coded using one context-coded bin and one bypass-coded bin. The first context-coded bin indicates whether equal weights are used; if unequal weights are used, a bypass-coded bin is used to signal which unequal weights are used.

[0111] Weighted prediction (WP) is an encoding tool supported by H.264 / AVC and HEVC standards, which is effective in encoding video content with fading. Support of WP is also added to the VVC standard. WP allows sending weight parameters (weight and offset) for each reference picture in each reference picture list L0 and L1. Then, during motion compensation, the weight and offset of the corresponding reference picture are applied. WP and BCW are designed for different types of video content. To avoid the interaction between WP and BCW, which complicates the VVC decoder design, if a CU uses WP, the BCW weight index is not signaled, and the weight w is inferred to be 4 (i.e., equal weights are applied). For merge CUs, the weight index is inferred from the merge candidate index from neighboring blocks. This can be applied to both normal merge mode and inherited affine merge mode. For constructed affine merge mode, the affine motion information is constructed based on the motion information of up to 3 blocks. The BCW index for a CU using constructed affine merge mode is simply set to be equal to the BCW index of the first control point MV.

[0112] In VVC, CIIP and BCW cannot be jointly applied for a CU. When a CU is coded using CIIP mode, the BCW index of the current CU is set to 2, i.e., equal weights w = 4. Equal weights imply the default value of the BCW index.

[0113] Combined inter / intra prediction (CIIP)

[0114] In VVC, when a CU is coded in merge mode, if the CU contains at least 64 luma samples (i.e., the CU width times the CU height is equal to or larger than 64), and if both the CU width and the CU height are smaller than 128 luma samples, an extra flag is signaled to indicate whether the combined inter / intra prediction (CIIP) mode is applied to the current CU. As the name suggests, CIIP prediction combines an inter prediction signal with an intra prediction signal. The inter prediction signal in CIIP mode P inter is derived using the same inter prediction process applied to regular merge mode; and the intra prediction signal P intra is derived after the regular intra prediction process with the planar mode. Then, the intra and inter prediction signals are combined using a weighted average, where the weight value wtis calculated according to the coding modes of the top and left neighboring blocks of the current CU 1210 (as shown in Figure 12 .

[0115] If the top neighbor is available and intra-frame encoding is used, set isIntraTop to 1; otherwise, set isIntraTop to 0.

[0116] If the left neighbor is available and intra-frame coding is used, set isIntraLeft to 1; otherwise, set isIntraLeft to 0.

[0117] If (isIntraLeft+isIntraTop) equals 2, then wt is set to 3;

[0118] Otherwise, if (isIntraLeft+isIntraTop) equals 1, then wt is set to 2;

[0119] Otherwise, set wt to 1.

[0120] The CIIP predictions are as follows:

[0121] P CIIP =((4-wt))*P inter +wt*P intra +2)>>2 (4)

[0122] CCLM (Cross-Component Linear Model)

[0123] The main idea behind CCLM mode (sometimes abbreviated as LM mode) is that there is often some correlation between the color components (e.g., Y / Cb / Cr, YUV, and RGB) of a color image. These colors may be referred to as the first color, the second color, and the third color in this disclosure. CCLM technology utilizes this correlation by predicting the chromaticity components of a block from juxtaposed reconstructed luminance samples using a linear model whose parameters are derived from the reconstructed luminance and chromaticity samples adjacent to the block.

[0124] In VVC, the CCLM mode utilizes inter-channel dependencies by predicting chromaticity samples from reconstructed luminance samples. This prediction is performed using a linear model of the form P(i,j)=a·rec′. L (i, j) + b (5)

[0125] Here, P(i,j) represents the predicted chromaticity sample in CU, while rec′ L (i, j) represent reconstructed luminance samples from the same CU, which are downsampled for non-4:4:4 color formats. Model parameters a and b are derived from adjacent luminance and chrominance samples reconstructed at the encoder and decoder ends, without explicit signaling.

[0126] Three CCLM modes, namely CCLM LT, CCLM L and CCLM T, are specified in VVC. These three modes differ in the location of the reference samples used for model parameter derivation. Only samples from the top boundary are involved for CCLM T mode, only samples from the left boundary are involved for CCLM L mode. In CCLM LT mode, samples from both the top boundary and the left boundary are used.

[0127] Overall, the prediction process of CCLM mode includes three steps:

[0128] 1) downsampling of the luma block and its neighboring reconstructed samples to match the size of the corresponding chroma block,

[0129] 2) model parameter derivation based on the reconstructed neighboring samples, and

[0130] 3) applying the model equation (1) to generate the chroma intra prediction samples.

[0131] Downsampling of luma component: To match the chroma sample positions of 4:2:0 or 4:2:2 color format video sequences, two types of downsampling filters can be applied to the luma samples, both with a 2-to-1 downsampling ratio in horizontal and vertical directions. These two filters correspond to “type-0” and “type-2” 4:2:0 chroma format content, respectively, given by

[0132]

[0133] Based on the SPS-level identification information, a two-dimensional 6th order (i.e., f2) or 5th order (i.e., fl) filter is applied to the luma samples within the current block and their neighboring luma samples. SPS-level refers to Sequence Parameter Set level. An exception occurs if the above row of the current block is a CTU boundary. In this case, a one-dimensional filter [1, 2, 1] / 4 is applied to the above neighboring luma samples to avoid using multiple luma queues above the CTU boundary.

[0134] Model parameter derivation process: The model parameters a and b from equation (5) are derived based on the reconstructed neighboring luma and chroma samples at the encoder and decoder side to avoid the need for any signaling overhead. In the originally adopted version of CCLM mode, a linear minimum mean square error (LMMSE) estimator is used for the derivation of the parameters. However, in the final design, only four samples are involved to reduce the computational complexity. Figure 13 An MxN chroma block 1310, the corresponding 2Mx2N luma block 1320 and the relative sample positions of their neighboring samples (shown as solid circles and triangles) of “type 0” content are shown.

[0135] InFigure 13 In the example of FIG. 6, four samples used in the CCLM LT mode are shown, which are marked with triangles. They are located at the positions of the upper boundary M / 4 and M*3 / 4, and the left boundary N / 4 and N*3 / 4. In the CCLM T and CCLM L modes, the upper and left boundaries are extended to the size of (M+N) samples, and the four samples for model parameter derivation are located at (M+N) / 8, (M+N)*3 / 8, (M+N)*5 / 8 and (M+N)*7 / 8.

[0136] Once the four samples are selected, four comparison operations are used to determine the two smallest and two largest luma sample values. Let X l denote the average of the two largest luma sample values, and let X s denote the average of the two smallest luma sample values. Similarly, let Y l and Y s denote the average of the corresponding chroma sample values. Then, the linear model parameters are obtained according to the following equations:

[0137]

[0138] b = Y s-a * X s . (7)

[0139] In this equation, the division operation for computing the parameter a is implemented by a look-up table. In order to reduce the memory required to store this table, the diff value, i.e., the difference between the maximum and minimum values, as well as the parameter a are represented in exponential notation. Here, the value of diff is approximated with a 4-bit significand and an exponent. Thus, the table for diff contains only 16 elements. This has the advantage of both reducing the complexity of the computation and reducing the size of the memory required to store the table.

[0140] MMLM Overview

[0141] As the name suggests, the original CCLM mode employs a linear model to predict the chroma samples of the luma samples of the entire CU, while in MMLM (Multiple Model CCLM), there can be two models. In MMLM, the neighboring luma samples and the neighboring chroma samples of the current block are divided into two groups, and each group is used as a training set to derive a linear model (i.e., a specific a and b is derived for a specific group). In addition, the samples of the current luma block are also classified based on the same rule as the classification of the neighboring luma samples.

[0142] The threshold (Threshold) is the average of the neighboring reconstructed luma samples. The neighboring samples with Rec' L[x, y] <= Threshold are classified into the 1st group; while the neighboring samples with Rec' L[x, y] > Threshold are classified into the 2nd group.

[0143] Accordingly, the prediction of chroma is obtained using a linear model:

[0144]

[0145] Chroma Intra Mode Coding

[0146] For chroma intra mode coding, a total of 8 intra modes are allowed for chroma intra mode coding. These modes include five traditional intra modes and three cross-component linear model modes (CCLM, LM_A and LM_L). The chroma mode signaling and derivation process is shown in Table 3. The chroma mode coding directly depends on the intra prediction mode of the corresponding luma block. Since separate block partitioning structures for luma and chroma components are enabled in I slices, one chroma block can correspond to multiple luma blocks. Therefore, for the chroma (Chroma) derived mode (DM) mode, the intra prediction mode of the corresponding luma block covering the center position of the current chroma block is directly inherited.

[0147] Table 3. Derivation of chroma prediction mode from luma mode when CCLM is enabled

[0148]

[0149]

[0150] As shown in Table 4, a single binarization table is used regardless of the value of sps_cclm_enabled_flag.

[0151] Table 4. Unified binarization table for chroma prediction mode

[0152]

[0153] The first bin indicates whether it is a regular mode (i.e., 0) or an LM mode (i.e., 1). If it is an LM mode, the next bin indicates whether it is LM_CHROMA (i.e., 0) or not (i.e., 1). If not LM_CHROMA, the next bin indicates whether it is LM_L (i.e., 0) or LM_A (i.e., 1). For this case, the first bin of the binarization table for the corresponding intra_chroma_pred_mode can be ignored before entropy coding when sps_cclm_enabled_flag is 0. Or, in other words, the first bin is inferred to be 0 and thus not coded. This single binarization table is used for the case when sps_cclm_enabled_flag is equal to 0 and 1. The first two bins are context coded using their own context model and the rest of the bins are bypass coded.

[0154] Multiple Hypothesis Prediction (MHP)

[0155] In multi-hypothesis inter prediction mode (JVET-M0425), one or more additional motion-compensated prediction signals are emitted in addition to the traditional bi-prediction signal. The final overall prediction signal is obtained by sample-wise superposition. With the bi-prediction signal p bi and the first additional inter prediction signal / hypothesis h3, the resulting prediction signal p3 is as follows:

[0156] p3 = (1 - a)p bi + a h3 (8)

[0157] The weight factor a is specified by the new syntax element add_hyp_weight_idx according to the following mapping (Table 5):

[0158] Table 5. Mapping of a to add_hyp_weight_idx

[0159] add_hyp_weight_idx α 0 1 / 4 1 -1 / 8

[0160] Similar to the above, more than one additional prediction signal can be used. The resulting overall prediction signal is cumulatively iterated with each additional prediction signal.

[0161] p n+1 = (1 - a n+1 )p n + a n+1 h n+1 (9)

[0162] The resulting overall prediction signal is obtained as the last p n (i.e., p n with the largest index n). For example, up to two additional prediction signals can be used (i.e., n is limited to 2).

[0163] The motion parameters of each additional prediction hypothesis can be explicitly signaled by specifying a reference index, a motion vector predictor index, and a motion vector difference, or implicitly signaled by specifying a merge index. A separate multi-hypothesis merge flag distinguishes between the two signaling modes.

[0164] For the inter AMVP mode, MHP is only applied when non-equal weights in BCW are selected in bi-prediction mode. Details of MHP for VVC can be found in JVET-W2025 (Muhammed Coban et al., “Algorithm description of enhancement compression model 2 (ECM 2),” Joint Video Team (JVT) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 23rd Meeting, Teleconference, 7-16 July 2021, Doc.: JVET-W2025).

[0165] A combination of MHP and BDOF is possible, but BDOF is applied only to bi-predicted signal part of the prediction signal (i.e. the normal first two hypotheses). SUMMARY

[0166] A method and apparatus for prediction coding are disclosed. According to the method, input data associated with a current block is received, wherein the input data comprises pixel data of the current block to be encoded at an encoder side or encoded data associated with the current block to be decoded at a decoder side. Combined prediction members are determined, wherein each combined prediction member comprises a weighted sum of a first prediction candidate and a second prediction candidate from a prediction candidate group using a target weight selected from a weight set. Boundary matching costs associated with the combined prediction members are determined, wherein each boundary matching cost is determined by using prediction samples of the current block based on the combined prediction member with the target weight and neighboring reconstructed samples of the current block for the combined prediction member with the target weight. The current block is then encoded or decoded using a final combined prediction decided based on at least one boundary matching cost.

[0167] In one embodiment, a weight value of a weight index is assigned according to a boundary matching cost associated with a combined prediction member with a target weight, wherein a smaller boundary matching cost assigns a smaller weight value and a larger boundary matching cost assigns a larger weight value.

[0168] In one embodiment, a weight index associated with the final combined prediction is determined implicitly at the encoder side and the decoder side. The implicitly determined weight index corresponds to the combined prediction member with the target weight that results in the smallest boundary matching cost.

[0169] In one embodiment, the first prediction candidate and the second prediction candidate are from a prediction candidate group, and the prediction candidate group of the current block comprises a first bi-prediction candidate and a second bi-prediction candidate, and the first prediction candidate corresponds to the first bi-prediction candidate and the second prediction candidate corresponds to the second bi-prediction candidate.

[0170] In one embodiment, the first prediction candidate corresponds to an existing prediction and the second prediction candidate is from a prediction candidate group comprising one or more prediction hypotheses. In one embodiment, the one or more prediction hypotheses comprises at least two prediction hypotheses, and the final combined prediction is updated by setting the first prediction candidate to a previous final combined prediction and setting the second prediction candidate to a next prediction hypothesis of the prediction candidate group. In one embodiment, the target weight selected from the weight set is determined implicitly according to the boundary matching cost.

[0171] In one embodiment, the prediction samples of the current block correspond to a subset of the prediction block of the current block. For example, the subset of the prediction block of the current block can include the top n1 lines and the left m1 columns of the prediction block, and the neighboring reconstructed samples of the current block include the n2 lines above and the m2 columns to the left of the current block, where n1, m1, n2 and m2 are integers greater than or equal to 0. In one example, n1 and m1 are equal to 2, and n2 and m2 are equal to 2. In another example, n1 and m1 are equal to 2, and n2 and m2 are equal to 1. In one embodiment, one or more syntaxes for n1, m1, n2 and m2 are signaled or parsed at a CU, CB, PU, PB, TU, TB, CTU, CTB, slice level, picture level, SPS (Sequence Parameter Set) level, PPS (Picture Parameter Set) level, or a combination thereof.

[0172] According to another method, input data associated with a current block is received, wherein the input data includes pixel data of the current block to be encoded at an encoder side or encoded data associated with the current block to be decoded at a decoder side. Prediction members associated with an existing prediction are determined, wherein each prediction member corresponds to the existing prediction and one weight sum of one target combined prediction candidate from a set of combined prediction candidates, and wherein each combined prediction candidate corresponds to a combination of one target motion vector and one target weight for combining the existing prediction and the one target combined prediction candidate, the one target motion candidate from a set of m motion candidates and the one target weight from a set of n weights, m and n being positive integers. Boundary matching costs associated with the prediction members are determined, wherein each boundary matching cost for the each prediction member is determined by based on prediction samples of the current block of the each prediction member and neighboring reconstructed samples of the current block. The current block is encoded or decoded using a final prediction decided based on at least one boundary matching cost. BRIEF DESCRIPTION OF DRAWINGS

[0173] Figure 1A An exemplary adaptive inter / intra video coding system incorporating a loop process is illustrated.

[0174] Figure 1B A corresponding decoder of the encoder in Figure 1A is illustrated.

[0175] Figure 2 A neighboring block used to derive a spatial merge candidate of VVC is illustrated.

[0176] Figure 3 Possible pairs of candidates considered for redundancy check in VVC are illustrated.

[0177] Figure 4An example of temporal candidate derivation is illustrated, where a scaled motion vector is derived according to the POC (Picture Order Count) distance.

[0178] Figure 5 An example of the position of a temporal candidate selected between candidates C0 and C1 is illustrated.

[0179] Figure 6 An example of distance offset in horizontal and vertical directions from the starting MV according to the merge mode with MVD (MMVD) is illustrated.

[0180] Figure 7A An example of an affine motion field of a block described by motion information of two control points (4 parameters) is shown.

[0181] Figure 7B An example of an affine motion field of a block described by motion information of three control point motion vectors (6 parameters) is illustrated.

[0182] Figure 8 An example of block-based affine transform prediction is illustrated, where the motion vector of each 4x4 luma sub-block is derived from control point MVs.

[0183] Figure 9 An example of a control point MV derivation inheriting affine candidates based on neighboring blocks is illustrated.

[0184] Figure 10 An example of constructing an affine candidate by combining translational motion information from each control point of spatial neighbors and temporal is shown.

[0185] Figure 11 An example of affine motion information storage for motion information inheritance is illustrated.

[0186] Figure 12 An example of weight value derivation for combined inter and intra prediction (CIIP) according to the coding mode of the top and left neighboring blocks is illustrated.

[0187] Figure 13 An example of model parameter derivation for CCLM (cross component linear model) using neighboring chroma samples and neighboring luma samples is illustrated.

[0188] Figure 14 An example of boundary samples and neighboring samples for boundary matching cost derivation for a candidate mode is illustrated.

[0189] Figure 15 An example of an exemplary multiple prediction hypothesis is shown, where the multiple prediction hypothesis includes an existing hypothesis and two hypotheses of prediction h1 and prediction h2.

[0190] Figure 16A flow diagram illustrating an exemplary video coding system utilizing multiple prediction hypotheses according to an embodiment of the application is shown.

[0191] Figure 17 A flow diagram illustrating another exemplary video coding system utilizing multiple prediction hypotheses according to an embodiment of the application is shown. DETAILED DESCRIPTION

[0192] It is readily understood that the components of the application, as generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations. Thus, the following more detailed description of the embodiments of the system and method of the application, as represented in the figures, is not intended to limit the scope of the application, as claimed, but is merely representative of selected embodiments of the application. Reference throughout this specification to "one embodiment," "an embodiment," or similar language means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. Thus, appearances of the phrase "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment.

[0193] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. One skilled in the relevant art will recognize, however, that the application can be practiced without one or more of the specific details, or with other methods, components, etc. In other instances, well-known structures or operations are not shown or described in detail. The detailed description is not intended to limit the scope of the application unless otherwise specifically stated. The described embodiments are best understood by reference to the drawings, wherein like elements are numbered alike, as follows:

[0194] As video coding is improved, more coding tools are developed. However, the coding gain of new coding tools is not additive. Possible reasons are

[0195] - Not all new coding modes can be a candidate mode for a block considering the syntax overhead.

[0196] - With the increase of candidate modes for a block, longer codewords are needed to indicate the coding mode from multiple candidate modes.

[0197] Take merge mode as an example. Compared with HEVC, new merge candidates (e.g., pair-wise average merge candidate, HMVP merge candidate, etc.) are suggested to be added to the merge candidate list. As mentioned before, the index of the best merge candidate is coded / decoded to indicate the selected merge candidate for the current block. However,

[0198] - The number of merge candidates in the merge candidate list is limited to a predefined number, so not all merge candidates can be added to the merge candidate list.

[0199] - If the number of merge candidates in the merge candidate list increases, the codeword length of the best merge candidate index becomes longer.

[0200] Therefore, a scheme is proposed to reorder the candidate modes adaptively. The general rule is shown as follows.

[0201] - The cost of each candidate mode is calculated, which can be (but not limited to) the merge candidate mode and / or can be extended to any other tool's candidate mode

[0202] - Next, the priority order of these candidate modes is decided according to the cost.

[0203] In one embodiment, those candidate modes with smaller cost get higher priority.

[0204] In another embodiment, those candidate modes with smaller cost get lower priority.

[0205] - Next, the candidate modes are reordered according to the priority order.

[0206] The index with smaller value indicates the candidate mode with higher priority.

[0207] In other words, the index refers to the reordered candidate mode.

[0208] Initially, the value of the index refers to the index of the candidate mode.

[0209] After applying the proposed reordering scheme, the value of the index refers to the reordered index of the candidate mode.

[0210] In another embodiment, for those candidate modes with higher priority, shorter codeword is used for encoding / decoding. In another embodiment, for the candidate mode with the highest priority, it is implicitly set as the coding mode of the current block. In the present invention, the scheme to determine the priority order includes boundary matching. For each candidate mode, the boundary matching cost is calculated. The priority order depends on the boundary matching cost of these candidate modes.

[0211] Boundary matching cost of a candidate mode

[0212] The boundary matching cost of a candidate mode refers to the current prediction ( Figure 14The boundary matching cost of a candidate mode represents a measure of discontinuity between the current prediction and the neighboring reconstruction. The top boundary matching refers to a comparison between the current top prediction samples and the neighboring top reconstructed samples 1420, and the left boundary matching refers to a comparison between the current left prediction samples and the neighboring left reconstructed samples 1430.

[0213] In one embodiment, the boundary matching cost is calculated using a predefined subset of the current prediction. The n1 rows of the top boundary within the current block and / or the m1 columns of the left boundary within the current block are used, where n1 and m1 are integers greater than or equal to 0. In addition, the n2 rows of the top neighboring reconstruction and / or the m2 columns of the left neighboring reconstruction are used, where n2 and m2 are integers greater than or equal to 0.

[0214] Here is an example of calculating the boundary matching cost with n1 = 2, m1 = 2, n2 = 2, and m2 = 2.

[0215]

[0216] In the above equations, the weights (a, b, c, d, e, f, g, h, i, j, k, l) can be any positive integer, for example a = 2, b = 1, c = 1, d = 2, e = 1, f = 1, g = 2, h = 1, i = 1, j = 2, k = 1, l = 1.

[0217] Here is another example of calculating the boundary matching cost with n1 = 2, m1 = 2, n2 = 1, and m2 = 1.

[0218]

[0219] In the above equations, the weights (a, b, c, g, h, i) can be any positive integer, for example a = 2, b = 1, c = 1, g = 2, h = 1, i = 1.

[0220] Here is another example of calculating the boundary matching cost with n1 = 2, m1 = 1, n2 = 2, and m2 = 2.

[0221]

[0222] In the above equations, the weights (d, e, f, j, k, l) can be any positive integer, for example d = 2, e = 1, f = 1, j = 2, k = 1, l = 1.

[0223] Here is another example of calculating the boundary matching cost with n1 = 1, m1 = 1, n2 = 1, and m2 = 1.

[0224]

[0225] In the above equation, the weights (a,c,g,i) can be any positive integers, such as a=1, c=1, g=1, i=1.

[0226] This is another example of calculating the boundary matching cost for n1=2, m1=1, n2=2, and m2=1.

[0227]

[0228] In the above equation, the weights (a,b,c,d,e,f,g,i) can be any positive integers, such as a=2, b=1, c=1, d=2, e=1, f=1, g=1, i=1.

[0229] This is another example of calculating the boundary matching cost for n1=1, m1=2, n2=1, and m2=2.

[0230]

[0231] In the above equation, the weights (a,c,g,h,i,j,k,l) ​​can be any positive integers, such as a=1, c=1, g=2, h=1, i=1, j=2, k=1, l=1.

[0232] The examples below for n1 and m1 can also be applied to n2 and m2.

[0233] For example, nl can be any positive integer, such as 1, 2, 3, 4, etc.

[0234] For example, m1 can be any positive integer, such as 1, 2, 3, 4, etc.

[0235] For example, n1 and / or m1 vary with the block width, height, or area.

[0236] As an example, for larger blocks (e.g., area > threshold), a larger m1 is used.

[0237] Threshold = 64, 128 or 256.

[0238] When the area exceeds the threshold, m1 increases to 2. (Initially, m1 is 1.)

[0239] When the area exceeds the threshold, m1 increases to 4. (Initially, m1 is 1 or 2.)

[0240] According to another example, for taller blocks (e.g., height > threshold * width), use a larger m1 and / or a smaller n1.

[0241] Threshold = 1, 2 or 4.

[0242] When height > threshold * width, m1 is increased to 2. (Originally m1 was 1.)

[0243] When height > threshold * width, m1 is increased to 4. (Originally m1 was 1 or 2.)

[0244] According to another example, for larger blocks (e.g., area > threshold), a larger n1 is used.

[0245] Threshold = 64, 128, or 256.

[0246] When area > threshold, n1 is increased to 2. (Initially, n1 is 1.)

[0247] When area > threshold, n1 is increased to 4. (Initially, n1 is 1 or 2.)

[0248] According to another example, for wider blocks (e.g., width > threshold * height), a larger n1 is used and / or a smaller m1 is used.

[0249] Threshold = 1, 2, or 4.

[0250] When width > threshold * height, n1 is increased to 2. (Originally n1 was 1.)

[0251] When width > threshold * height, n1 is increased to 4. (Originally n1 was 1 or 2.)

[0252] For another example, nl and / or ml can be defined in a standard or depend on signaling / parsing of syntax from CU / CB, PU / PB, TU / TB, CTU / CTB, slice level, picture level, SPS (Sequence Parameter Set) level, and / or PPS (Picture Parameter Set) level.

[0253] In another embodiment, when the current block is located at the top boundary within a CTU row, top boundary matching is not used and / or only left boundary matching is used. (No neighboring reconstructed samples across CTU rows are used.)

[0254] In another embodiment, when the current block is located at the left boundary within a CTU, left boundary matching is not used and / or only top boundary matching is used.

[0255] In another embodiment, when the current block is taller (e.g., height > threshold * width), only left boundary matching is used.

[0256] In another embodiment, when the current block is wider (e.g., width > threshold * height), only top boundary matching is used.

[0257] In another embodiment, the top-left neighboring reconstructed sample is available for boundary matching. For example, the boundary matching cost is added with the following:

[0258] |reco -1，-1 -pred 0，0 !.

[0259] merge candidates as candidate modes

[0260] The proposed scheme can be used to reorder the merge candidates. Originally, the index of the best merge candidate refers to the order of the merge candidate in the merge candidate list. For the proposed scheme, the index of the best merge candidate (index_best_merge) refers to the priority order based on the boundary matching cost.

[0261] For example, assume that the merge candidate list consists of {cand0, cand1, cand2, cand3, cand4, cand5}. These prediction candidates are referred to as a prediction candidate group in this disclosure.

[0262] Original:

[0263] index_best_merge 0 refers to cand0

[0264] index_best_merge 1 refers to cand1

[0265] index_best_merge 2 refers to cand2

[0266] ...

[0267] Proposed:

[0268] Compute the boundary matching cost for each merge candidate {cost_cand0, cost_cand1, cost_cand2,...}

[0269] cost_cand0 is the boundary matching cost of cand0

[0270] cost_cand1 is the boundary matching cost of cand1

[0271] cost_cand2 is the boundary matching cost of cand2

[0272] ...

[0273] Reorder {cand0, cand1, cand2,...} according to the boundary matching cost

[0274] In one example,

[0275] If cost candO > cost candl > cost cand2 > cost cand3 > cost cand4 > cost cand5, the reordered merge candidates form {cand5, cand4, cand3, cand2, candl, candO}.

[0276] After reordering,

[0277] index best merge 0 refers to cand5 (the merge candidate with the smallest cost is signaled with the shortest codeword).

[0278] index best merge 1 refers to cand4

[0279] index best merge 2 refers to cand3

[0280] ...

[0281] Otherwise, if cost candO < cost candl < cost cand2 < cost cand3 < cost cand4 < cost cand5, the order of the merge candidates remains the same as the original order.

[0282] In another example,

[0283] If cost candO < cost candl < cost cand2 < cost cand3 < cost cand4 < cost cand5, the reordered merge candidates form {cand5, cand4, cand3, cand2, candl, candO}.

[0284] After reordering,

[0285] index best merge 0 refers to cand5 (the merge candidate with the largest cost is signaled with the shortest codeword).

[0286] index best merge 1 refers to cand4

[0287] index best merge 2 refers to cand3

[0288] ...

[0289] Otherwise, if cost candO > cost candl > cost cand2 > cost cand3 > cost cand4 > cost cand5, the order of the merge candidates is kept as original.

[0290] In another embodiment, index best merge with smaller value is encoded as shorter codeword length. For example, index best merge is encoded with truncated unary codeword.

[0291] In another embodiment, reordering is applied to a subset of the merge candidate list.

[0292] For example, the subset refers to the first n candidates candO, candl, cand2 in original. Then index best mergeO / l / 2 refers to the priority order based on boundary match, and index best merge 3 / 4 / 5 refers to cand3, cand4 and cand5 in original.

[0293] For another example, the subset refers to the last n candidates cand3, cand4, cand5 in original. Then index best merge 3 / 4 / 5 refers to the priority order based on boundary match, and index best mergeO / l / 2 refers to candO, candl and cand2 in original.

[0294] For yet another example, the subset refers to spatial merge candidates.

[0295] In another embodiment, the best merge candidate is inferred as the merge candidate with the smallest boundary match cost among all the merge candidates. Therefore, index best merge does not need to be signaled / parsed by the encoder / decoder, and can be inferred as 0.

[0296] In another embodiment, the best merge candidate is inferred as the merge candidate with the largest boundary match cost among all the merge candidates. Therefore, index best merge does not need to be signaled / parsed by the encoder / decoder, and can be inferred as 0.

[0297] In another embodiment, the merge candidates in this section refer to the merge candidates including one or more of the following:

[0298] 1) Spatial MVP from spatial neighboring CUs

[0299] 2) Temporal MVP from collocated CUs

[0300] 3) History-based MVP from FIFO table

[0301] 4) Pairwise average MVP

[0302] 5) Zero MV.

[0303] In another embodiment, the merge candidate in this section refers to a merge candidate of CIIP. The prediction samples within the current block are generated according to the CIIP process.

[0304] In another embodiment, the merge candidate in this section refers to a merge candidate of subblock merge candidate, e.g., affine merge candidate. The prediction samples within the current block are generated according to the affine process.

[0305] BCW as a candidate mode

[0306] The proposed scheme can be used to reorder the BCW candidates to improve the syntax of bcw_idx. The two bi-prediction candidates are referred to as the prediction candidate group in this disclosure. The following is the syntax table of BCW in the VVC standard.

[0307]

[0308] bcw_idx[x0][y0] specifies the weight index of bi-prediction with CU weights. The array indices x0, y0 specify the position (x0, y0) of the top-left luma sample of the considered coding block relative to the top-left luma sample of the picture.

[0309] An example of BCW as a candidate mode is shown as follows:

[0310] - Calculate the boundary matching cost of each BCW candidate weight

[0311] - In one embodiment, after reordering according to the cost,

[0312] bcw_idx equal to 0 indicates the BCW candidate weight is the smallest, and bcw_idx equal to 4 indicates the BCW candidate weight is the largest.

[0313] In another way, bcw_idx is implicit and the BCW candidate weight with the smallest cost is used.

[0314] - In another embodiment, after reordering according to the cost,

[0315] bcw_idx equal to 0 indicates the BCW candidate weight is the largest, and bcw_idx equal to 4 indicates the BCW candidate weight is the smallest.

[0316] In another way, bcw_idx is implicit and the BCW candidate weight with the largest cost is used.

[0317] In the embodiment where BCW is the candidate mode, the boundary matching cost is evaluated for a set of possible weights used to select the final combined prediction. The prediction associated with each combined BCW prediction (i.e., the selected weight) is referred to as a combined prediction member. The total number of combined prediction members corresponds to the total number of weights. bcw_idx is signaled / parsed or implicitly referenced for the final combined prediction. The ordered boundary matching costs should result in a more efficient representation of bcw_idx, i.e., lower bit rate.

[0318] MHP as a candidate mode

[0319] For a block where MHP is applied, one or more prediction hypotheses (i.e., prediction signals) are combined with the existing prediction hypothesis to form the final (resultant) prediction for the current block. In other words, the final prediction is iteratively accumulated with each additional hypothesis of the prediction signal. An example of the accumulation is shown below.

[0320] p n+1 = (1 - a n+1 ) p n + a n+1 h n+1

[0321] The resultant prediction signal is taken as the last p n (i.e., the p n with the largest index n). p0 is the first (existing) prediction for the current block. In the case where MHP is the candidate mode, the existing prediction and the weight of the prediction hypothesis are initially formed into a combined prediction. When there are two or more prediction hypotheses (i.e., n > 1), the current combined prediction is formed as the weight sum of the previous combined prediction and the next prediction hypothesis. For example, if MHP is applied to a merge candidate, p0 is indicated by the existing merge index. The additional prediction is denoted as h, and will be further combined with the previously accumulated prediction by a weight a.

[0322] Thus, for each additional hypothesis of the prediction, a weight index is signaled / parsed to indicate the weight and / or for each hypothesis of the prediction, an inter index is signaled / parsed to indicate the motion candidate (used to generate the prediction samples for that hypothesis). The syntax overhead of MHP can be an issue in certain cases. For example, the syntax overhead of MHP can become large when the number of additional hypotheses of the prediction increases. As another example, the syntax overhead of MHP can become large when the number of candidate weights for each additional hypothesis of the prediction increases. Thus, to make the signaling of MHP weights and / or the signaling of motion candidates more efficient, a reordering scheme is proposed.

[0323] In one embodiment, the boundary matching cost of each MHP candidate weight is calculated for each additional prediction hypothesis. For the case of two candidate weights with a prediction hypothesis, an example of the proposed process is shown below:

[0324] Step 0: For the prediction additional hypothesis, compute cost_w0 and cost_w1 as the cost of the first and second candidate weights, respectively.

[0325] Step 1 : For the prediction hypothesis, the candidate weights are reordered according to the cost.

[0326] In one sub-embodiment, the candidate weights with smaller cost get higher priority.

[0327] If cost_w0 > cost_w1, weight index 0 refers to w1 and weight index 1 refers to w0.

[0328] Otherwise, no reordering is used, and weight index 0 and 1 refer to the original w0 and w1.

[0329] In another sub-embodiment, the candidate weights with larger cost get higher priority.

[0330] If cost_w0 < cost_w1, weight index 0 refers to w1 and weight index 1 refers to w0.

[0331] Otherwise, no reordering is used, and weight index 0 and 1 refer to the original w0 and w1.

[0332] In another sub-embodiment, the candidate weight with the smallest cost is used for the current additional prediction hypothesis. In this case, the weight index of the current additional prediction hypothesis is inferred.

[0333] In another sub-embodiment, the weight of each hypothesis or any subset of hypotheses is implicitly set according to the cost. The weight can be a scaled value of the cost or a scaled value of the multiplicative inverse of the cost. For example, if the cost equals 2, the multiplicative inverse of the cost equals 1 / 2.

[0334] In another sub-embodiment, the candidate weight with the largest cost is used for the current additional prediction hypothesis. In this case, the weight index of the current additional prediction hypothesis is inferred.

[0335] Steps 0 and 1 are repeated for each additional prediction hypothesis, and the meaning of each weight index for each additional prediction hypothesis is obtained.

[0336] In another embodiment, the boundary matching cost is computed for each MHP motion candidate. The following sub-embodiments are exemplified for the merge mode of MHP. MHP can be applied to other inter modes, such as inter AMVP and / or affine, and when inter AMVP or affine is used, the "merge" in the following examples will be replaced by the naming of this inter mode.

[0337] In one sub-embodiment, the boundary match cost of each motion candidate is computed. For example, the candidate pattern includes candO to cand4. Initially, index 0 (shorter codeword) refers to candO and index 4 (longer codeword) refers to cand4. Using the proposed method, the meaning of the index follows the priority. If the priority order (based on the boundary match cost) specifies that cand4 has the highest priority, then index 0 maps to cand4.

[0338] In another sub-embodiment, the boundary match cost of each motion candidate is computed. For example, the candidate pattern includes candO to cand4. Initially, index 0 (shorter codeword) refers to candO and index 4 (longer codeword) refers to cand4. Using the proposed method, the meaning of the index follows the priority. If the priority order (based on the boundary match cost) specifies that cand4 has the highest priority, then the index is not signaled / resolved and the selected motion candidate is inferred to be the motion candidate with the highest priority.

[0339] In another sub-embodiment, for each hypothesis, the current prediction of a motion candidate (used to compute the boundary match cost) is the motion compensated result produced by that motion candidate.

[0340] An example combining the above two sub-embodiments is shown below. For example, if the number of additive hypotheses is equal to 2, the first three motion candidates with higher priority are used to form the resulting prediction of the current MHP block. For another example, if the number of additive hypotheses is equal to 2, the existing hypotheses are preserved, the two motion candidates with higher priority are used to form the prediction of the additional hypotheses, and the resulting prediction is formed by the existing hypotheses and the additional hypotheses.

[0341] In another sub-embodiment, the current prediction of a motion candidate (used to compute the boundary match cost) is a combined prediction (i.e., a weighted average of the prediction from the motion candidate and the existing prediction (pO)). The weights are indicated by a signaled / resolved weight index.

[0342] In another sub-embodiment, for each hypothesis, the current prediction of a motion candidate (used to compute the boundary match cost) is a combined prediction (a weighted average of the prediction from the motion candidate and the previously accumulated prediction). The weights are indicated by a signaled / resolved weight index. Figure 15 An example is shown in FIG. 15, where prediction pO 1510 corresponds to the existing hypothesis and predictions h1 1520 and h2 1530 correspond to the additional hypotheses. When the proposed method is applied to reorder the motion candidate signals of h2, h2 has 4 motion candidates (including candO to cand3).

[0343] - The cost of candn is computed as a weighted average of the pO prediction, the h1 prediction, and the candn prediction

[0344] - That is,

[0345] o For h2, the current prediction of candidate 0 = the weighted average of (p0 prediction, h1 prediction, prediction of candidate 0)

[0346] o For h2, the current prediction of candidate 1 = the weighted average of (p0 prediction, h1 prediction, prediction of candidate 1)

[0347] o...

[0348] - When the proposed method is applied to reorder the signaling of motion candidates of hl and hl has 4 motion candidates (including candO to cand3),

[0349] o For h1, the current prediction of candidate 0 = the weighted average of (prediction of p0, prediction of candidate 0)

[0350] o For h1, the current prediction of candidate 1 = the weighted average of (prediction of p0, prediction of candidate 1)

[0351] o...

[0352] In another embodiment, the boundary matching cost is computed for each MHP combination (motion candidate and weight) for each prediction hypothesis. The following sub-embodiment takes MHP in merge mode as an example, MHP can be applied to other inter modes, e.g., inter AMVP and / or affine. When inter AMVP or affine is used, “merge” in the following example will be replaced by the naming of this inter mode.

[0353] In one sub-embodiment, a combination refers to a motion candidate and a weight. If there are m motion candidates and n weights for each motion candidate, the number of combinations is m*n.

[0354] In another sub-embodiment, the current prediction of a combination (used to compute the boundary matching cost) is the combination prediction (the weighted average of the prediction from the motion candidate and the existing prediction (p0)). This method jointly determines the merge index and the weight representing the additional prediction hypothesis. For example, the combination with the highest priority is the selected MHP combination. There is no need to signal / resolve the merge index and the weight to indicate the additional prediction hypothesis. In another example, a joint index is signaled / resolved to determine the MHP combination. The number of additional hypotheses can be fixed in this sub-embodiment.

[0355] In another embodiment, the boundary matching cost is computed for each MHP motion candidate. The following takes MHP in merge mode as an example, MHP can be applied to other inter modes, e.g., inter AMVP and / or affine. When inter AMVP or affine is used, “merge” in the following will be replaced by the naming of this inter mode. Thus,

[0356] - merge index (to indicate the motion candidate for each hypothesis).

[0357] o For example, according to the order of the merge candidates in the merge candidate list

[0358] merge candO for hypothesis 0, merge candl for hypothesis 1,...

[0359] o For another example, depending on the cost.

[0360] First use the merge candidate with smaller cost.

[0361] o For another example, according to the predefined number of merge candidates

[0362] If the number of hypotheses is 4, use the 4 merge candidates in the merge candidate list as the motion candidate for each hypothesis

[0363] Use the first 4 merge candidates, or

[0364] Use any 4 merge candidates in the merge candidate list.

[0365] - The weights (to combine the prediction hypotheses) are implicit according to the cost.

[0366] - The MHP prediction is given by (weightO)*(hypothesisO) + (weightl)*(hypothesisl) + (weight2)*(hypothesis2) +...

[0367] In one sub-embodiment, a fixed number of prediction hypotheses is used. In other words, a fixed number of hypotheses is mixed and the matching cost is used to derive the weights implicitly.

[0368] In another sub-embodiment, the weights of the motion candidates (or hypotheses) with higher priority are larger than the weights of the motion candidates with lower priority.

[0369] In another sub-embodiment, the current prediction (to calculate the boundary matching cost) of a motion candidate is the motion compensated result produced by this motion candidate.

[0370] In another sub-embodiment, the first n motion candidates in the merge candidate list are used to generate the prediction hypotheses. With this proposed method, no merge index signal is sent for MHP.

[0371] In another sub-embodiment, all the motion candidates in the merge candidate list are used to generate the prediction hypotheses. The weights can decide whether a motion candidate is useful or not. If its weight is zero, this motion candidate is not actually used. With this proposed method, no merge index signal is sent for MHP.

[0372] In another sub-embodiment, the weight of a motion candidate with a higher priority is greater than the weight of a motion candidate with a lower priority.

[0373] For example, the weight follows the cost ratio of different motion candidates. If there are two motion candidates and cost candO = 2 * cost candl, then weight candO = 2 * weight candl or weight candO = 1 / 2 * weight candl.

[0374] For another example, the cost of each motion candidate is first normalized to the interval [MIN_VALUE, MAX_VALUE]. MAX_VALUE is predefined, for example, the number of prediction hypotheses. MIN_VALUE is predefined, for example, 0. For example, (MAX_VALUE - normalized cost) can be the weight of the motion candidate or the normalized cost can be the weight of the motion candidate.

[0375] For another example, the weight is a scaling value of the cost or a scaling value of the multiplicative inverse of the cost. For example, if the cost is equal to 2, then the multiplicative inverse of the cost is equal to 1 / 2.

[0376] - a scaling value represents a scaling factor * an original value.

[0377] - if the scaling factor = 1, then there is no scaling.

[0378] In another sub-embodiment, the weight and the merge index are implicit for the proposed method.

[0379] The generation of the current prediction of the method can refer to any other method proposed in the present invention.

[0380] In another embodiment, the proposed scheme is applied to a subset of all additional prediction hypotheses. In other words, steps 0 and 1 above are repeated for a subset of all additional prediction hypotheses. For example, only the candidate weights of the first additional prediction hypotheses combined with the existing prediction hypotheses are reordered with the proposed scheme.

[0381] In a sub-embodiment, the subset is predefined in the standard.

[0382] In another sub-embodiment, the subset depends on the width, height or area of the current block. For example, for blocks with an area greater (or lower) than a threshold, the subset includes more prediction hypotheses.

[0383] In another sub-embodiment, the reordered results from the subset can be reused for the remaining additional prediction hypotheses. For example,

[0384] - Based on the reordering result of the first prediction hypothesis, weight indices 0 and 1 refer to wl and w0, respectively.

[0385] - For the following prediction hypotheses, weight indices 0 and 1 refer to wl and w0, respectively, as well.

[0386] In another embodiment, the proposed scheme is applied to a subset of all candidate weights for additional prediction hypotheses. In other words, steps 0 and 1 above are repeated for a subset of all candidate weights for additional prediction hypotheses. Take an example where the number of candidate weights (for additional hypotheses of prediction) is equal to 4.

[0387] - For additional prediction hypotheses, only the first two (or the last) candidate weights are reordered using the proposed scheme.

[0388] In one sub-embodiment, the subset is predefined in the standard.

[0389] In another sub-embodiment, the subset depends on the width, height or area of the current block. For example, for blocks with block area larger (or smaller) than a threshold, the subset includes more candidate weights.

[0390] In another embodiment, the prediction hypotheses can be prediction signals from single or bi-prediction motion compensation results.

[0391] The proposed reordering scheme can be unified for different tools (not limited to those in the following examples). For example, the proposed reordering scheme for MHP, LM, BCW, MMVD and / or merge candidates is unified with the same rule for computing boundary matching cost.

[0392] The proposed method in this invention can be enabled and / or disabled according to implicit rules (e.g. block width, height or area) or according to explicit rules (e.g. syntax on block, slice, picture, SPS or PPS level). For example, the proposed reordering is applied when the block area is smaller than a threshold.

[0393] The term "block" in this invention can refer to TU / TB, CU / CB, PU / PB or CTU / CTB.

[0394] The term "LM" in this invention can be considered as a CCLM / MMLM mode or any other extension / variant of CCLM (e.g. the CCLM extension / variant proposed in this invention). The variations here mean that when the block indication refers to using one of the cross component modes (e.g. CCLM LT, MMLM LT, CCLM L, CCLM T, MMLM L, MMLM T and / or intra prediction mode), some optional modes can be selected which are not one of the conventional DC, planar and angular modes for the current block. An example of incorporating a convolutional cross component mode (CCCM) as an optional mode is shown below. When this optional mode is applied to the current block, the cross component information (including the non-linear term) of the model is used to generate the chroma prediction. The optional mode can follow the template selection of CCLM, so the CCCM family includes CCCM LT, CCCM L and / or CCCM T.

[0395] The methods proposed in this invention (for CCLM) can be used for any other LM mode.

[0396] Any combination of the methods proposed in this invention can be applied.

[0397] Any of the aforementioned proposed methods can be implemented in an encoder and / or a decoder. For example, any of the proposed methods can be implemented in an intra / inter coding module (e.g. inter prediction 112 in Figure 1A , a motion compensation module (e.g. MC 152 in Figure 1B , a merge decoder's candidate derivation module. Alternatively, any of the proposed methods can be implemented as a circuit coupled to the intra / inter coding module and / or the motion compensation module of an encoder, the merge candidate derivation module of a decoder.

[0398] Figure 16A flowchart of an exemplary video coding system utilizing multiple prediction hypotheses according to embodiments of the application is illustrated. The steps shown in the flowchart can be implemented as program code executable on one or more processors (e.g., one or more CPUs) at the encoder side. The steps shown in the flowchart can also be implemented based on hardware, such as one or more electronic devices or processors arranged to perform the steps in the flowchart. According to the method, input data associated with a current block is received in step 1610, where the input data comprises pixel data of the current block to be encoded at the encoder side or encoded data associated with the current block to be decoded at the decoder side. A combined prediction member is determined in step 1620, where each combined prediction member comprises a weight sum of a first prediction candidate and a second prediction candidate using a target weight selected from a set of weights. A boundary matching cost associated with the combined prediction member is determined in step 1630, where each boundary matching cost for the combined prediction member with the target weight is determined by using predicted samples of the current block based on the combined prediction member with the target weight and neighboring reconstructed samples. The current block is encoded or decoded using a final combined prediction decided based on at least one boundary matching cost in step 1640.

[0399] Figure 17 A flowchart of another exemplary video coding system utilizing multiple prediction hypotheses according to embodiments of the application is illustrated. According to the method, input data associated with a current block is received in step 1710, where the input data comprises pixel data of the current block to be encoded at the encoder side or encoded data associated with the current block to be decoded at the decoder side. A prediction member associated with an existing prediction is determined in step 1720, where each prediction member corresponds to a weight sum of the existing prediction and a target combined prediction candidate from a set of combined prediction candidates, and where each combined prediction candidate corresponds to a combination of a target motion candidate and a target weight, for combining the existing prediction and the target combined prediction candidate, the target motion candidate from a set of m motion candidates and the target weight from a set of n weights, and m, n are positive integers. A boundary matching cost associated with the prediction member is determined in step 1730, where each boundary matching cost for the each prediction member is determined by using predicted samples of the current block based on the each prediction member and neighboring reconstructed samples of the current block. The current block is encoded or decoded using a final prediction decided based on at least one boundary matching cost in step 1740.

[0400] The illustrated flowchart is intended to explain an example of video encoding according to the present application. Each step can be modified, rearranged, split, or combined by those skilled in the art without departing from the spirit of the present application to implement the present application. In the present disclosure, specific syntax and semantics have been used to explain examples to implement embodiments of the present application. Those skilled in the art can implement the present application by replacing the syntax and semantics with equivalent syntax and semantics without departing from the spirit of the present application.

[0401] The foregoing description is provided to enable any person skilled in the art to practice the present application provided in the context of particular applications and its requirements. Various modifications to the described embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments. Thus, the present application is not intended to be limited to the particular embodiments described, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details are exemplified in order to provide a thorough understanding of the present application. However, those skilled in the art will understand that the present application can be practiced without the specific details.

[0402] Embodiments of the present application as described above can be implemented in various hardware, software code, or a combination of both. For example, one embodiment of the present application can be one or more circuits integrated into a video compression chip or program codes integrated into video compression software to perform the processes described herein. Embodiments of the present application can also be program codes to be executed on a digital signal processor (DSP) to perform the processes described herein. The present application can also relate to a number of functions to be performed by computer processors, digital signal processors, microprocessor, or field programmable gate arrays (FPGAs). These processors can be configured to perform particular tasks according to the software code or firmware code defined by the specific method embodied in the application. The software code or firmware code can be developed in different programming languages and different formats or styles. The software code can also be compiled into different formats or styles for different target platforms. However, different code formats, styles, and languages of the software code and other means of configuring the code to perform the tasks according to the present application will not depart from the spirit and scope of the present application.

[0403] The present application can be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The described examples are to be considered in all respects only as illustrative and not restrictive. The scope of the application is, therefore, indicated by the appended claims, rather than by the foregoing description. All changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope.

Claims

1. A method of predictive coding, the method comprising: receiving input data associated with a current block, wherein the input data comprises pixel data of the current block to be encoded at an encoder side or encoded data associated with a current block to be decoded at a decoder side; determining a combined prediction member, wherein each combined prediction member comprises a weighted sum of a first prediction candidate and a second prediction candidate using a target weight selected from a set of weights; determining a boundary matching cost associated with the combined prediction member, wherein each boundary matching cost for the combined prediction member with the target weight is determined by using predicted samples of the current block based on the combined prediction member with the target weight and neighboring reconstructed samples of the current block; and encoding or decoding the current block using a final combined prediction decided based on at least one boundary matching cost.

2. The method of claim 1, wherein, a weight index is used to indicate the target weight selected from the set of weights.

3. The method of claim 2, wherein, a weight value of the weight index is assigned according to a boundary matching cost associated with a combined prediction member with a target weight, and wherein a smaller weight value is assigned for a smaller boundary matching cost and a larger weight value is assigned for a larger boundary matching cost.

4. The method of claim 2, wherein, the weight index associated with the final combined prediction is signaled in a bitstream at an encoder side or parsed from a bitstream at a decoder side.

5. The method of claim 2, wherein, the weight index associated with the final combined prediction is implicitly determined at the encoder side and the decoder side.

6. The method of claim 5, wherein, the implicitly determined weight index corresponds to a combined prediction member with a target weight that results in a minimum boundary matching cost.

7. The method of claim 1, wherein, the first prediction candidate and the second prediction candidate are from a prediction candidate group, and the prediction candidate group for the current block comprises a first bi-prediction candidate and a second bi-prediction candidate, and wherein the first prediction candidate corresponds to the first bi-prediction candidate and the second prediction candidate corresponds to the second bi-prediction candidate.

8. The method of claim 1, wherein, the first prediction candidate corresponds to an existing prediction and the second prediction candidate is from a prediction candidate group comprising one or more prediction hypotheses.

9. The method of claim 8, wherein, the one or more prediction hypotheses comprise at least two prediction hypotheses, and by setting the first prediction candidate to a previous final combined prediction and the second prediction candidate to a next prediction hypothesis in the prediction candidate group.

10. The method of claim 9, wherein, the target weight selected from the set of weights is implicitly determined according to the boundary matching cost.

11. The method of claim 1, wherein, the predicted samples of the current block correspond to a subset of a predicted block of the current block.

12. The method of claim 11, wherein, the subset of the predicted block of the current block comprises n1 top rows and m1 left rows of the predicted block, and the neighboring reconstructed samples of the current block comprise n2 rows above the current block and m2 rows left of the current block, wherein n1, m1, n2 and m2 are integers greater than or equal to 0.

13. The method of claim 12, wherein, n1 and m1 are equal to 2, and n2 and m2 are equal to 2.

14. The method of claim 12, wherein, n1 and m1 are equal to 2, and n2 and m2 are equal to 1.

15. The method of claim 12, wherein, one or more syntaxes for n1, m1, n2 and m2 are signaled or parsed at a CU, CB, PU, PB, TU, TB, CTU, CTB, tile level, picture level, sequence parameter set (SPS) level, picture parameter set (PPS) level or a combination thereof.

16. An apparatus for predictive encoding / decoding, the apparatus comprising one or more electronic devices or processors arranged to: receiving input data associated with a current block, wherein, The input data includes pixel data of the current block to be encoded on the encoder side or encoded data associated with the current block to be decoded on the decoder side; Determine the combined prediction members, wherein each combined prediction member includes the weights of a first prediction candidate and a second prediction candidate using target weights selected from the weight set; Determine the boundary matching cost associated with the combined prediction member, wherein each boundary matching cost is determined for the combined prediction member with the target weight by using prediction samples of the current block of the combined prediction member with the target weight and neighboring reconstruction samples of the current block; and The current block is encoded or decoded using a final combined prediction based on at least one boundary matching cost decision.

17. A predictive encoding / decoding method, the method comprising: Receive input data associated with the current block, wherein the input data includes pixel data of the current block to be encoded on the encoder side or encoded data associated with the current block to be decoded on the decoder side; Determine prediction members associated with existing predictions, wherein each prediction member corresponds to a weighted sum of an existing prediction and a target combined prediction candidate from a set of combined prediction candidates, and wherein each combined prediction candidate corresponds to a combination of a target motion candidate and a target weight for combining the existing prediction and the target combined prediction candidate, wherein the target motion candidate is from a set of m motion candidates and the target weight is from a set of n weights, and m and n are positive integers; Determine the boundary matching cost associated with each predicted member, wherein for each predicted member, each boundary matching cost is determined by using predicted samples of the current block and neighboring reconstructed samples of the current block based on each predicted member; and The current block is encoded or decoded using a final prediction based on the cost of at least one boundary matching.

18. The coding method of claim 17, wherein, The joint weights and motion candidate indices associated with the target motion candidate and the final predicted target weights are either signaled in the bitstream on the encoder side or parsed from the bitstream on the decoder side.

19. The coding method of claim 17, wherein, Implicitly determine the joint weights and motion candidate indices associated with the final predicted target motion candidate and the target weight.

Images