Method and apparatus for hybrid prediction in video coding system

Abstract

Methods and apparatus for video coding and decoding are provided. The current block is divided into a first region and a second region along a partition line. A mixing region around the partition line is determined, where the mixing region includes a first mixing region having first pixels in a first region defined by the partition line and distance1 and a second mixing region having second pixels in a second region defined by the partition line and distance2. The distance1 and the distance2 correspond to two non-negative values, and the two non-negative values can be different. A first predictor and a second predictor are determined for the first region and the second region, respectively. A hybrid predictor of the hybrid region is generated by mixing the first predictor and the second predictor. A current block is encoded or decoded by using prediction data including a hybrid predictor.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure is a part of a non-provisional application claiming priority to U.S. Provisional Patent Application No. 63 / 334,327 filed on April 25, 2022. The contents of the above-listed applications are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure generally relates to video coding systems. In particular, the present invention relates to using adaptive mixing regions to mix predictors to improve coding efficiency. Background Art

[0004] Versatile video coding (VVC) is the latest international video coding standard developed by the Joint Video Experts Team (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG). The standard was published as an ISO standard in February 2021: ISO / IEC23090-3:2021, Information technology - Codec representation of immersive media - Part 3: Versatile video coding. VVC is based on its previous generation High Efficiency Video Coding (HEVC) by adding more codec tools to improve codec efficiency and handle various types of video sources including three-dimensional (3D) video signals.

[0005] Figure 1AAn example adaptive inter / intra video codec system incorporating loop processing is shown. For intra prediction, prediction data is derived based on previously encoded video data in the current picture. For inter prediction 112, motion estimation (ME) is performed at the encoder end and motion compensation (MC) is performed based on the results of ME to provide prediction data derived from other pictures and motion data. Switch 114 selects intra prediction 110 or inter prediction 112, and the selected prediction data is provided to adder 116 to form a prediction error, also known as a residual. The prediction error is then processed by a transform (T) 118 followed by a quantization (Q) 120. The transformed and quantized residual is then encoded by an entropy encoder 122 to be included in a video bitstream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packaged with auxiliary information (such as auxiliary information such as motion and coding modes associated with intra prediction and inter prediction) and other information (parameters associated with loop filters applied to underlying image areas, etc.). As Figure 1A As shown, auxiliary information associated with intra prediction 110, inter prediction 112 and loop filter 130 is provided to entropy encoder 122. When inter prediction mode is used, one or more reference pictures must also be reconstructed at the encoder end. Therefore, the transformed and quantized residual is processed by inverse quantization (Inverse Quantization, referred to as IQ) 124 and inverse transformation (Inverse Transformation, referred to as IT) 126 to restore the residual. The residual is then added back to the prediction data 136 in reconstruction (REC) 128 to reconstruct the video data. The reconstructed video data can be stored in the reference picture buffer 134 and used for prediction of other frames.

[0006] like Figure 1A As shown, the input video data undergoes a series of processes in the encoding system. Due to the series of processes, the reconstructed video data from REC128 may be subject to various damages. Therefore, before the reconstructed video data is stored in the reference picture buffer 134, a loop filter 130 is generally applied to the reconstructed video data to improve the video quality. For example, a deblocking filter (DF for short), a sample adaptive offset (SAO for short), and an adaptive loop filter (ALF for short) may be used. The loop filter information may need to be incorporated into the bitstream so that the decoder can correctly recover the required information. Therefore, the loop filter information is also provided to the entropy encoder 122 to be incorporated into the bitstream. In Figure 1A In , a loop filter 130 is applied to the reconstructed video before the reconstructed samples are stored in a reference picture buffer 134 . Figure 1A The system in is intended to illustrate an example structure of a typical video encoder. It may correspond to a High Efficiency Video Coding (HEVC) system, VP8, VP9, ​​H.264, or VVC.

[0007] like Figure 1B The decoder shown may use similar or partially identical functional blocks as the encoder, except for transform 118 and quantization 120, since the decoder only needs inverse quantization 124 and inverse transform 126. The decoder uses an entropy decoder 140 instead of an entropy encoder 122 to decode the video bitstream into quantized transform coefficients and required codec information (e.g., ILPF information, intra-frame prediction information, and inter-frame prediction information). The intra-frame prediction 150 at the decoder end does not need to perform a pattern search. Instead, the decoder only needs to generate an intra-frame prediction based on the intra-frame prediction information received from the entropy decoder 140. In addition, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) based on the intra-frame prediction information received from the entropy decoder 140 without motion estimation.

[0008] According to VVC, the input picture is divided into non-overlapping square block areas called coding tree units (CTUs), similar to HEVC. Each CTU can be divided into one or more smaller-sized coding units (CUs). The generated CU partitions can be square or rectangular. In addition, VVC divides CTUs into prediction units (PUs) as a unit to apply prediction processing such as inter-frame prediction, intra-frame prediction, etc.

[0009] The VVC standard incorporates various new codec tools to further improve the codec efficiency beyond the HEVC standard. Among the various new codec tools, some codec tools related to the present invention are summarized as follows.

[0010] Inter-frame prediction overview

[0011] According to Section 3.4 of JVET-T2002 (Jianle Chen, et.al., "Algorithm description for Versatile Video Coding and Test Model 11 (VTM 11)", Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 20th Meeting, bytleconference, 7-16 October 2020, Document: JVET-T2002), for each inter-prediction CU, motion parameters consisting of motion vector, reference picture index and reference picture list usage index and additional information are used for the generation of inter-prediction samples. Motion parameters can be sent explicitly or implicitly. When the CU is encoded and decoded in skip mode, the CU is associated with one PU and has no significant residual coefficients, no coded motion vector increment or reference picture index. Merge mode means that the motion parameters of the current CU are obtained from neighboring CUs, including spatial and temporal candidates, as well as additional scheduling introduced in VVC. Merge mode can be used for any inter-predicted CU. An alternative to merge mode is explicit transmission of motion parameters, where the motion vector of each CU, the corresponding reference picture index and reference picture list usage flag of each reference picture list and other required information are explicitly sent.

[0012] In addition to the inter-frame coding and decoding functions in HEVC, VVC also includes many new and improved inter-frame prediction coding and decoding tools, as listed below:

[0013] - Extended Merger Forecast

[0014] -Merge mode with MVD (MMVD)

[0015] -Symmetric MVD (SMVD) transmission

[0016] Affine Motion Compensated Prediction

[0017] -Subblock-based temporal motion vector prediction (SbTMVP)

[0018] -Adaptive motion vector resolution (AMVR)

[0019] - Motion field storage: 1 / 16 luminance sample MV storage and 8x8 motion field compression

[0020] - Bi-prediction with CU-level weight (BCW)

[0021] -Bi-directional optical flow (BDOF)

[0022] -Decoder side motion vector refinement (DMVR)

[0023] -Geometric partitioning mode (GPM)

[0024] - Combined inter and intra prediction (CIIP)

[0025] - The following description provides details of those inter prediction methods specified in VVC.

[0026] Expanded Consolidated Forecast

[0027] In VVC, the merge candidate list is constructed by including the following five categories of candidates in sequence:

[0028] 1) Spatial MVP from spatially adjacent CUs

[0029] 2) Time MVP from the same CU

[0030] 3) History-based MVP from FIFO table

[0031] 4) Paired Average MVP

[0032] 5) Zero MV.

[0033] The size of the merge list is sent in the sequence parameter set (SPS) header and the maximum allowed size of the merge list is 6. For each CU encoded in merge mode, the index of the best merge candidate is encoded using truncated unary binarization. The first bin of the merge index is encoded and decoded using context, and bypass encoding and decoding is used for the remaining bins.

[0034] This section provides the derivation process of merge candidates for each category. As done in HEVC, VVC also supports parallel derivation of merge candidate lists (or merge candidate lists) for all CUs in a certain size region.

[0035] Spatial candidate derivation

[0036] The derivation of spatial merge candidates in VVC is the same as in HEVC, except that the positions of the first two merge candidates are swapped. Figure 2 A maximum of four merge candidates (B0, A0, B1 and A1) of the current CU 210 are selected from the candidates at the positions shown. The derived order is B0, A0, B1, A1 and B2. Position B2 is only considered when one or more neighboring CUs at positions B0, A0, B1 and A1 are not available (for example, belong to another fragment or tile) or are intra-encoded. After the candidate at position A1 is added, a redundancy check is performed on the addition of the remaining candidates to ensure that candidates with the same motion information are excluded from the list, thereby improving the encoding and decoding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only the possible candidate pairs are considered. Figure 3 The pairs are connected by arrows in , and candidates are added to the list only if the corresponding candidates used for redundancy checking do not have the same motion information.

[0037] Time candidate derivation

[0038] In this step, only one candidate is added to the list. Specifically, in the derivation of this temporal merge candidate for the current CU 410, the scaled motion vector is based on the candidate belonging to Figure 4 The co-located CU 420 of the co-located reference picture shown is derived. The reference picture list and the reference index used to derive the co-located CU are explicitly sent in the slice header. Figure 4 As shown by the dashed line in , the scaled motion vector 430 of the temporal merge candidate is obtained, which is scaled from the motion vector 440 of the co-located CU using the Picture Order Count (POC) distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal merge candidate is set equal to zero.

[0039] The position of the temporal candidate is selected between candidates C-0 and C1, such as Figure 5 As shown. If the CU at position C0 is not available, is intra-coded or is outside the current CTU row, then position C1 is used. Otherwise, position C0 is used to derive the temporal merge candidate.

[0040] History-based Merge Candidate Derivation

[0041] History-based MVP (HMVP) merge candidates are added to the merge list after spatial MVP and TMVP. In this method, the motion information of the previous codec block is stored in the table and used as the MVP of the current CU. During the encoding / decoding process, a table with multiple HMVP candidates is retained. When a new CTU row is encountered, the table will be reset (cleared). Whenever there is a non-sub-block inter-coded CU, the associated motion information will be added to the last entry of the table as a new HMVP candidate.

[0042] The HMVP table size S is set to 6, which indicates that a maximum of 5 history-based MVP (HMVP) candidates can be added to the table. When inserting a new motion candidate into the table, a constrained first-in-first-out (FIFO) rule is used, where first a redundancy check is applied to find if there is an identical HMVP in the table. If found, the identical HMVP is removed from the table and all subsequent HMVP candidates are moved forward, and the identical HMVP is inserted into the last entry of the table.

[0043] HMVP candidates can be used in the merge candidate list construction process. The latest few HMVP candidates in the table are checked in turn and inserted into the candidate list after the TMVP candidate. Redundancy check is applied to HMVP candidates to spatial or temporal merge candidates.

[0044] To reduce the number of redundant checking operations, the following simplifications are introduced:

[0045] 1. The last two entries in the table are checked for redundancy with respect to the A1 and B1 spatial candidates respectively.

[0046] 2. Once the total number of available merge candidates reaches the maximum allowed merge candidate - 1, the merge candidate list construction process of HMVP is terminated.

[0047] Pairwise average merge candidate derivation

[0048] A pairwise average candidate is generated by averaging a predetermined candidate pair in an existing merge candidate list using the first two merge candidates. The first merge candidate is defined as p0Cand, and the second merge candidate may be defined as p1Cand. The average motion vector is calculated for each reference list according to the availability of the motion vectors of p0Cand and p1Cand. If two motion vectors are available in one list, they are averaged and their reference pictures are set to one of p0C and p0C even if they point to different reference pictures; if only one motion vector is available, this motion vector is used directly; if no motion vector is available, this list is kept invalid. In addition, if the half-pixel interpolation filter index of p0Cand and p1Cand is different, it is set to 0.

[0049] When the merge list is not full after adding pairwise average merge candidates, zero MVPs are inserted at the end until the maximum number of merge candidates is reached.

[0050] Merge estimated areas

[0051] The merge estimation region (MER) allows independent derivation of merge candidate lists for CUs in the same merge estimation region (MER). Candidate blocks in the same MER as the current CU are not included in the generation of the merge candidate list for the current CU. In addition, the history-based update of the motion vector predictor candidate list is updated only if (xCb+cbWidth)>>Log2ParMrgLevel is greater than xCb>>Log2ParMrgLevel and (yCb+cbHeight)>>Log2ParMrgLevel is greater than (yCb>>Log2ParMrgLevel), where (xCb, yCb) is the top-left luma sample position of the current CU in the picture and (cbWidth, cbHeight) is the CU size. The MER size is selected at the encoder side and is signaled in the Sequance Parameter Set (SPS) as log2_parallel_merge_level_minus2.

[0052] Merge Mode with MVD (MMVD)

[0053] In addition to the merge mode in which the implicitly derived motion information is directly used for the prediction sample generation of the current CU, a merge mode with motion vector difference (MMVD) is introduced in VVC. The MMVD flag is sent immediately after the regular merge flag is sent to specify whether the MMVD mode is used for the CU.

[0054] In MMVD, after a merge candidate (referred to in this disclosure as a basic merge candidate) is selected, it is further refined by the MVD information sent. Further information includes a merge candidate flag, an index for specifying the magnitude of motion, and an index for indicating the direction of motion. In MMVD mode, one of the first two candidates in the merge list is selected for use as the MV basis. The MMVD candidate flag is sent to specify which one to use between the first and second merge candidates.

[0055] The distance index specifies the motion magnitude information and indicates a predetermined offset from the start points (612 and 622) of the L0 reference block 610 and the L1 reference block 620. Figure 6 As shown, the offset is added to the horizontal component or vertical component of the starting MV, where the small circles of different styles correspond to different offsets from the center. The relationship between the distance index and the predetermined offset is specified in Table 1

[0056] Table 1 - Relationship between distance index and predetermined offset

[0057]

[0058] The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate four directions as shown in Table 2. It should be noted that the meaning of the MVD symbol can change according to the information of the starting MV. When the starting MV is a unidirectional prediction MV or a bidirectional prediction MV, where both lists point to the same side of the current picture (i.e., the POCs of both reference pictures are greater than the POC of the current picture, or both are less than the POC of the current picture), the symbol in Table 2 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectional prediction MV, the two MVs point to different sides of the current picture (i.e., the POC of one reference picture is greater than the POC of the current picture, and the POC of the other reference picture is less than the POC of the current picture), and the difference of the POC in List 0 is greater than the POC in List 1, the symbol in Table 2 specifies that the sign of the MV offset added to the List 0 MV component of the starting MV has the opposite value to the sign of the List 1 MV. Otherwise, if the difference of the POC in List 1 is greater than that in List 0, the symbol in Table 2 specifies that the sign of the MV offset added to the List 1 MV component of the starting MV has the opposite value to the sign of the List 0 MV.

[0059] The MVD is scaled based on the difference in POC in each direction. If the difference in POC in both lists is the same, no scaling is required. Otherwise, if the POC difference in list 0 is greater than that in list 1, the MVD of list 1 is scaled by defining the POC difference of L0 as td and the POC difference of L1 as tb, as Figure 4 If the POC difference of L1 is greater than L0, the MVD of list 0 is scaled in the same way. If the starting MV is unidirectionally predicted, the MVD is added to the available MVs.

[0060] Table 2 - MV offset symbols specified by direction index

[0061]

[0062]

[0063] Affine Motion Compensated Prediction

[0064] In HEVC, only the translational motion model is applied to motion compensation prediction (MCP). In the real world, there are many kinds of motion, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, block-based affine transformation motion compensation prediction is applied. Fig. 7A As shown in FIG. 1B , the affine motion field of block 710 is given by Fig. 7A The motion information of the two control points (4 parameters) in Figure 7B The three control point motion vectors (6 parameters) are described in .

[0065] For the 4-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as:

[0066]

[0067] For the 6-parameter affine motion model, the motion vector at a sample position (x, y) in the block is derived as:

[0068]

[0069] Among them, (mv0x, mv0y) is the motion vector of the upper left control point, (mv1x, mv1y) is the motion vector of the upper right control point, and (mv2x, mv2y) is the motion vector of the lower left control point.

[0070] To simplify motion compensated prediction, block-based affine transform prediction is applied. To derive the motion vector for each 4×4 luminance sub-block, the motion vector of the center sample of each sub-block is Figure 8As shown, it is calculated according to the above equation and rounded to 1 / 16 fractional precision. Then, the motion compensated interpolation filter is applied to generate a prediction for each subblock with a derived motion vector. The subblock size of the chroma component is also set to 4×4. The MV of the 4×4 chroma subblock is calculated as the average of the MVs of the top left and bottom right luminance subblocks in the same 8x8 luminance region.

[0071] For translational motion inter prediction, there are also two affine motion inter prediction modes: affine merge mode and affine AMVP mode.

[0072] Affine Merge Predictions

[0073] AF_MERGE mode can be applied to CUs with width and height greater than or equal to 8. In this mode, the control point MV (CPMV) of the current CU is generated based on the motion information of the spatially adjacent CUs. There can be up to five CPMVP (CPMV prediction) candidates, and the index is sent to indicate which candidate to use for the current CU. The following three types of CPVM candidates are used to form the affine merge candidate list:

[0074] - Inherited affine merge candidates inferred from the CPMV of neighboring CUs

[0075] -Constructed affine meraecandidate (CPMVP) derived from the translation MV of the adjacent CU

[0076] -Zero MV

[0077] In VVC, there are at most two inherited affine candidates, which come from the affine motion models of neighboring blocks, one from the left neighboring CU and one from the upper neighboring CU. Figure 2 The blocks shown are the same. For the left predictor, the scanning order is A0->A1, and for the upper predictor, the scanning order is B0->B1->B2. Only the first inherited candidate on each side is selected. No pruning check is performed between two inherited candidates. When a neighboring affine CU is identified, its control point motion vectors are used to derive the CPMVP candidate in the affine merge list of the current CU. Fig. 9 As shown, if the lower left neighboring block A of the current block 910 is encoded and decoded in affine mode, the motion vectors v2, v3 and v4 of the upper left corner, upper right corner and lower left corner of the CU 920 containing block A are obtained. When block A is encoded and decoded using a 4-parameter affine model, the two CPMVs (i.e., v0 and v1) of the current CU are calculated based on v2 and v3. When block A is encoded and decoded using a 6-parameter affine model, the three CPMVs of the current CU are calculated based on v2, v3 and v4.

[0078] Constructing affine candidates means constructing candidates by combining the adjacent translation motion information of each control point. Fig.10 As shown, the motion information of the control point is derived from the specified spatial and temporal neighboring blocks of the current block 1010. CPMVk (k=1, 2, 3, 4) represents the kth control point. For CPMV1, B2->B3->A2 blocks are checked and the MV of the first available block is used. For CPMV2, B1->B0 blocks are checked, and for CPMV3, A1->A0 blocks are checked. If TMVP is available, it is used as CPMV4.

[0079] After obtaining the MVs of the four control points, an affine merge candidate is constructed based on the motion information. The following combination of control point MVs is used to construct in order:

[0080] {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3,CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}

[0081] Combinations of 3 CPMVs construct 6-parameter affine merge candidates and combinations of 2 CPMVs construct 4-parameter affine merge candidates. To avoid motion scaling processing, the relevant combination of control point MVs is discarded if the reference indexes of the control points are different.

[0082] After the inherited affine merge candidates and the constructed affine merge candidates are checked, if the list is still not full, a zero MV is inserted at the end of the list.

[0083] Affine AMVP prediction

[0084] Affine AMVP mode can be applied to CUs with width and height greater than or equal to 16. A CU-level affine flag is sent in the bitstream to indicate whether affine AMVP mode is used, and then another flag is sent to indicate whether 4-parameter affine or 6-parameter affine is used. In this mode, the difference between the CPMV of the current CU and its predicted sub-CPMVP is sent in the bitstream. The affine AVMP candidate list size is 2, which is generated by the following four CPVM candidates in sequence:

[0085] - Inherited affine AMVP candidates inferred from the CPMV of neighboring CUs

[0086] -The constructed affine AMVP candidate CPMVP is derived using the translation MV of the adjacent CU

[0087] -Translated MV from neighboring CU

[0088] -Zero MV

[0089] The order in which inherited affine AMVP candidates are checked is the same as that of inherited affine merge candidates. The only difference is that for AVMP candidates, only affine CUs with the same reference picture as the current block are considered. When an inherited affine motion predictor is inserted into the candidate list, the pruning process is not applied.

[0090] The AMVP candidates constructed from Fig.10 The specified spatial neighboring blocks shown in are derived. The same check order as in the affine merge candidate construction is used. In addition, the reference picture index of the neighboring blocks is also checked. In the checking order, the first block that uses inter-frame coding and has the same reference picture in the current CU is used. When the current CU is encoded using 4-parameter affine mode, and both mv0 and mv1 are available, they are added to the affine AMVP list as a candidate. When the current CU is encoded using 6-parameter affine mode, and all three CPMVs are available, they are added to the affine AMVP list as a candidate. Otherwise, the constructed AMVP candidate is set to unavailable.

[0091] If the number of affine AMVP candidate lists is still less than 2 after inserting valid inherited affine AMVP candidates and constructed AMVP candidates, mv0, mv1, and mv2 are added as translation MVs to predict all control point MVs of the current CU when available. Finally, if the affine AMVP list is still not full, zero MVs are used to fill the affine AMVP list.

[0092] Affine motion information storage

[0093] In VVC, the CPMV of an affine CU is stored in a separate buffer. The stored CPMV is only used to generate inherited CPMVP for the most recently decoded CU in affine merge mode and affine AMVP mode. The sub-block MV derived from the CPMV is used for motion compensation, merged MV derive / translate MV's AMVP list and deblocking.

[0094] To avoid picture line buffers for extra CPMV, affine motion data inherited from CUs of upper CTUs is handled differently than that inherited from regular neighboring CUs. If the candidate CU for affine motion data inheritance is in the upper CTU row, the lower left and lower right sub-block MVs in the line buffer are used for affine MVP derivation instead of the CPMVs. In this way, the CPMVs are stored only in the local buffer. If the candidate CU is a 6-parameter affine codec, the affine model degenerates to a 4-parameter model. Fig.11 As shown, along the top CTU boundary, the bottom left and bottom right sub-block motion vectors of the CU are used for affine inheritance of the CU in the bottom CTU. Fig.11 , columns 1110 and rows 1112 represent the x and y coordinates of the picture with the origin (0, 0) at the upper left corner. Legend 1120 shows the meaning of various motion vectors, where arrow 1122 represents the CPMV for affine inheritance in the local buffer, arrow 1124 represents the sub-block vector for MC / Merge / Skip / AMVP / Deblocking / TMVPs in the local buffer and the sub-block vector for affine inheritance in the row buffer, and arrow 1126 represents the sub-block vector for MC / Merge / Skip / AMVP / Deblocking / TMVPs.

[0095] Adaptive Motion Vector Resolution (AMVR)

[0096] In HEVC, when use_integer_mv_g in the slice header is equal to 0, the motion vector difference (MVD) (between the CU's motion vector and the predicted motion vector) is sent in units of one-quarter brightness samples. In VVC, the CU-level adaptive motion vector resolution (AMVR) scheme is introduced. AMVR allows the CU's MVD to be encoded and decoded with different precisions. Depending on the current CU mode (normal AMVP mode or affine AVMP mode), the current CU's MVD can be adaptively selected as follows:

[0097] - Conventional AMVP modes: quarter brightness samples, half brightness samples, integer brightness samples, or quad brightness samples.

[0098] - Affine AMVP mode: quarter luma samples, integer luma samples, or 1 / 16 luma samples.

[0099] If the current CU has at least one non-zero MVD component, the CU-level MVD resolution indication is conditionally sent.If all MVD components (ie, horizontal and vertical MVD for reference list L0 and reference list L1) are zero, then quarter luma sample MVD resolution is inferred.

[0100] For CUs with at least one non-zero MVD component, a first flag is sent to indicate whether quarter luma sample MVD precision is used for the CU. If the first flag is 0, no further transmission is required, and quarter luma sample MVD precision is used for the current CU. Otherwise, a second flag is sent to indicate that half luma samples or other MVD precision (integer or four luma samples) are used for conventional AMVP CUs. In the case of half luma samples, a 6-tap interpolation filter is used for the half luma sample positions instead of the default 8-tap interpolation filter. Otherwise, a third flag is sent to indicate whether integer luma samples or four luma sample MVD precision is used for conventional AMVP CUs. In the case of affine AMVP CUs, the second flag is used to indicate whether integer luma samples or 1 / 16 luma sample MVD precision is used. To ensure that the reconstructed MV has the expected precision (quarter luma samples, half luma samples, integer luma samples, or four luma samples), the motion vector predictor of the CU will be rounded to the same precision as the MVD before being added to the MVD. The motion vector predictor is rounded towards zero (ie, a negative motion vector predictor is rounded towards positive infinity, and a positive motion vector predictor is rounded towards negative infinity).

[0101] The encoder uses the RD check to determine the motion vector resolution of the current CU. In VTM11, in order to avoid always performing four CU-level RD checks for each MVD resolution, the RD check of MVD precision other than quarter luma samples is only called conditionally. For the conventional AVMP mode, the RD cost of the quarter luma sample MVD precision and the integer luma sample MV precision is first calculated. Then, the RD cost of the integer luma sample MVD precision is compared with the RD cost of the quarter luma sample MVD precision to decide whether it is necessary to further check the RD cost of the four luma sample MVD precision. When the RD cost of the quarter luma sample MVD precision is much smaller than the RD cost of the integer luma sample MVD precision, the RD cost check of the four luma sample MVD precision is skipped. Then, if the RD cost of the integer luma sample MVD precision is significantly greater than the best RD cost of the previously tested MVD precision, the check of the half luma sample MVD precision is skipped. For the affine AMVP mode, if the affine inter mode is not selected after checking the rate-distortion costs of the affine merge / skip mode, merge / skip mode, quarter luma sample MVD accuracy regular AMVP mode, and quarter luma sample MVD accuracy affine AMVP mode, the 1 / 16 luma sample MV accuracy and 1 pixel MV accuracy affine inter modes are not checked. In addition, the affine parameters obtained in the quarter luma sample MV accuracy affine inter mode are used as the starting search points for the 1 / 16 luma sample and quarter luma sample MV accuracy affine inter modes.

[0102] Bi-Prediction with CU-level Weight (BCW)

[0103] In HEVC, the bidirectional prediction signal P bi-pred Generated by averaging two prediction signals P0 and P1 obtained from two different reference pictures and / or using two different motion vectors. In VVC, the bidirectional prediction mode is extended beyond simple averaging, allowing a weighted average of the two prediction signals.

[0104] P bi-pred =((8-w)*P0+w*P1+4)>>3 (3)

[0105] Weighted average bidirectional prediction allows five weights, w∈{-2,3,4,5,10}. For each bidirectionally predicted CU, the weight w is determined in one of two ways: 1) For non-merged CUs, the weight index is sent after the motion vector difference; 2) For merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is only applicable to CUs with 256 or more luma samples (i.e., CU width multiplied by CU height is greater than or equal to 256). For low-latency pictures, all 5 weights are used. For non-low-latency pictures, only 3 weights (w∈{3,4,5}) are used. At the encoder, a fast search algorithm is used to find the weight index without significantly increasing the complexity of the encoder. These algorithms are summarized below. The detailed information is disclosed in the VTM software and the document JVET-L0646 (Yu-Chi Su, et.al., "CE4-related: Generalized bi-prediction improvements combined from JVET-L0197 and JVET-L0296", Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 12th Meeting: Macao, CN, 3-12 Oct. 2018, Document: JVET-L0646).

[0106] - When combined with AMVR, unequal weights are only conditionally checked for 1-pixel and 4-pixel motion vector precision if the current picture is a low-latency picture.

[0107] - When combined with affine, affine ME will be performed for unequal weights if and only if the affine mode is selected as the current best mode.

[0108] - Unequal weights are only checked conditionally when the two reference pictures in bidirectional prediction are the same.

[0109] - Unequal weights are not searched when certain conditions are met, which depend on the POC distance between the current picture and its reference pictures, the codec QP and the temporal level.

[0110] The BCW weight index is encoded using a bin for context coding and a subsequent bypass coding bin. The first context coding bin indicates whether equal weights are used; if unequal weights are used, an additional bin is sent using bypass coding to indicate which unequal weights are used.

[0111] Weighted prediction (WP for short) is a codec tool supported by the H.264 / AVC and HEVC standards for efficient encoding and decoding of video content with fading. Support for WP has also been added to the VVC standard. WP allows weighting parameters (weights and offsets) to be sent for each reference picture in each reference picture list L0 and L1. Then, during motion compensation, the weights and offsets of the corresponding reference pictures are applied. WP and BCW are designed for different types of video content. In order to avoid the interaction between WP and BCW, which complicates the VVC decoder design, if the CU uses WP, the BCW weight index is not sent, and the weight w is inferred to be 4 (ie, equal weights are applied). For the merged CU, the weight index is inferred from the neighboring blocks based on the merge candidate index. This can be applied to both the normal merge mode and the inherited affine merge mode. For the constructed affine merge mode, the affine motion information is constructed based on the motion information of up to 3 blocks. The BCW index of the CU using the constructed affine merge mode is simply set equal to the BCW index of the first control point MV.

[0112] In VVC, CIIP and BCW cannot be used jointly for a CU. When a CU is coded in CIIP mode, the BCW index of the current CU is set to 2, (ie, equal weight of w=4). Equal weight means the default value of the BCW index.

[0113] Combined Inter and Intra Prediction (CIIP)

[0114] In VVC, when a CU is encoded and decoded in merge mode, if the CU contains at least 64 luma samples (i.e., the CU width multiplied by the CU height is equal to or greater than 64), and if both the CU width and the CU height are less than 128 luma samples, an additional flag is sent to indicate whether the combined inter / intra prediction (CIIP) mode is applied to the current CU. As the name suggests, CIIP prediction combines the inter prediction signal with the intra prediction signal. The inter prediction signal in the CIIP mode Pinter is derived using the same inter prediction process as applied to the regular merge mode; and the intra prediction signal Pintra is derived following the regular intra prediction process with the planar mode. The intra and inter prediction signals are then combined using weighted averaging, where the weight value wt is calculated based on the top and left neighboring blocks of the current CU 1210 (such as Fig.12 The codec mode is calculated as follows:

[0115] - if the top neighboring block is available and is intra-coded, isIntraTop is set to 1, otherwise isIntraTop is set to 0;

[0116] - if the left neighboring block is available and is intra-coded, isIntraLeft is set to 1, otherwise isIntraLeft is set to 0;

[0117] - If (isIntraLeft + isIntraTop) is equal to 2, then wt is set to 3;

[0118] - Otherwise, if (isIntraLeft+isIntraTop) is equal to 1, wt is set to 2;

[0119] - Otherwise, wt is set to 1.

[0120] The CIIP forecast is formed as follows:

[0121] P CIIP =((4-wt)*P inter +wt*P intra + 2 )>>2 (4)

[0122] Geometric Partitioning Mode (GPM)

[0123] In VVC, the geometric partitioning mode (GPM) can support inter-frame prediction, such as JVET-W2002 (Adrian Browne, et al., Algorithm description for Versatile Video Coding and Test Model 14 (VTM 14), ITU-T / ISO / IEC Joint Video Exploration Team (JVET), 23rd Meeting, by teleconference, 7-16 July 2021, document: document JVET-M2002). The geometric partitioning mode uses a CU-level flag as a merge mode. Other merge modes include regular merge mode, MMVD mode, CIIP mode, and sub-block merge mode. For each possible CU size, the geometric partitioning mode supports a total of 64 partitions, w×h=2m×2n where m,n∈{3…6}, excluding 8x64 and 64x8. The GPM mode can be applied to skip or merge CUs whose size is within the above limits and has at least two regular merge modes.

[0124] When this mode is used, the CU is divided into two parts by a straight line that is geometrically positioned at a certain angle. In VVC, there are a total of 20 angles and 4 offset distances used for GPM, which has been reduced from 24 angles in the early draft. The location of the partition line is mathematically derived based on the angle and offset parameters of the specific partition. In VVC, there are a total of 64 partitions, such as Fig.13 As shown in Figure 1, where partitions are grouped by angle and dotted lines indicate redundant partitions. Each part of the geometric partition in the CU uses its own motion for inter-frame prediction; each partition only allows unidirectional prediction, that is, each part has a motion vector and a reference index. Fig.13 , each line corresponds to the boundary of a partition. Partitions are grouped according to their angles. For example, partition group 1310 consists of three vertical GPM partitions (i.e., 90°). Partition group 1320 consists of four inclined GPM partitions at a small angle to the vertical direction. In addition, partition group 1330 consists of three vertical GPM partitions (i.e., 270°), similar to group 1310, but in the opposite direction. Unidirectional prediction motion constraints are applied to ensure that each CU only requires two motion compensated predictions, the same as traditional bidirectional prediction. The unidirectional prediction motion of each partition is derived using the process described later.

[0125] If the geometry partition mode is used for the current CU, a geometry partition index indicating the selected partition mode (angle and offset) of the geometry partition and two merge indices (one for each partition) are further sent. The number of maximum GPM candidate sizes is explicitly represented in the Sequence Parameter Set (SPS), as well as the syntactic binarization of the GPM merge indices specified. After predicting each part of the geometry partition, the sample values ​​along the edges of the geometry partition are adjusted using a blending process with adaptive weights using the process described later. This is the prediction signal for the entire CU and, like other prediction modes, the transform and quantization process will be applied to the entire CU. Finally, using the process described later, the motion field of the CU predicted using the geometry partition mode is stored.

[0126] One-way prediction candidate list construction

[0127] The unidirectional prediction candidate list is derived directly from the merge candidate list constructed according to the extended merge prediction process. Denote n as the index of the unidirectional prediction motion in the geometric unidirectional prediction candidate list. The LX motion vector (X=0 or 1, i.e. LX=L0 or L1) of the nth extended merge candidate, where X is equal to the parity of n, is used as the nth unidirectional prediction motion vector geometric partition mode. These motion vectors are in Fig.14 If there is no corresponding LX motion vector of the extended merge candidate, the L(1-X) motion vector of the same candidate replaces the unidirectional prediction motion vector of the geometric partition mode.

[0128] Blending along geometric partition edges

[0129] After predicting each part of the geometric partition using its own motion, blending is applied to the two prediction signals to derive samples around the geometric partition edges. The blending weights for each position of the CU are derived based on the distance between the respective position and the partition edge.

[0130] Two integer mixing matrices (WO and W1) are used for the GPM mixing process. The weights in the GPM mixing matrix contain a range of [0, 8] and are derived based on the displacement from the sample position to the GPM partition boundary 1540, such as Fig.15 shown.

[0131] Specifically, the weights are given by a discrete ramp function with a displacement and two thresholds, as Fig.16 As shown in Figure 2, the two endpoints of the slope (i.e., -τ and τ) correspond to Fig.15 in lines 1542 and 1544.

[0132] Here, the threshold τ defines the width of the GPM mixing area and is selected as a fixed value in VVC. That is, as described in JVET-Z0137 (Han Gao, et.al., "Non-EE2: Adaptive Blending for GPM", Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 26th Meeting, bytleconference, 20-29 April 2022, JVET-Z0137), the mixing strength or mixing area width θ is fixed for all different contents.

[0133] The weight values ​​in the blend mask can be given by a ramp function:

[0134]

[0135] With θ=2 pixels fixed in the current ECM (VVC) design, this ramp function can be quantized as:

[0136] ω m，n =Clip3(0,8,(d(m,n)+32+4)>>3) (6)

[0137] The distance from a position (x,y) to the edge of the partition is derived as:

[0138]

[0139]

[0140] where i, j are the angle and offset indices of the geometry partition, which depend on the geometry partition index sent. x，j and ρ y，j The sign of depends on the angle index i.

[0141] Fig.17 An example of GPM mixing according to ECM 4.0 (Muhammed Coban, et.al., "Algorithm description of Enhanced Compression Model 4 (ECM 4)", Joint Video Experts Team (JVET) of ITU-TSG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 26th Meeting, by teleconference, 20-29 April 2022, JVET-Y2025) is shown. Fig.17In , the size of the mixed region on each side of the partition boundary is denoted by θ. The weight of each part of the geometric partition is derived as follows:

[0142] wIdxL(x,y)=partIdx? 32+d(x,y):32-d(x,y) (11)

[0143]

[0144] w1(x, y) = 1 - w0(x, y) (13)

[0145] partIdx depends on the angle index i. An example of weight w0 is Fig.15 As shown, the angle 1510 and offset 1520 indicates the GPM index i, point 1530 corresponds to the center of the block. Row 1540 corresponds to the GPM partition boundary.

[0146] Motion field storage in geometric partitioning mode

[0147] Mv1 from the first part of the geometry partition, Mv2 from the second part of the geometry partition, and the combined MV of Mv1 and Mv2 are stored in the motion field of the CU encoded by the geometry partition mode.

[0148] The type of stored motion vector for each individual position in the motion field is determined as:

[0149] sType=abs(motionIdx)<32?2: (motionIdx≤0?(1-partIdx):partIdx) (14)

[0151] where motionIdx is equal to d(4x+2,4y+2), which is recalculated from equation (7). partIdx depends on the angle index i.

[0152] If sType is equal to 0 or 1, then Mv0 or Mv1 is stored in the corresponding motion field, otherwise if sType is equal to 2, the combined MV from Mv0 and Mv2 is stored. The combined Mv is generated using the following process:

[0153] 1) If Mv1 and Mv2 are from different reference picture lists (one from L0 and the other from L1), then Mv1 and Mv2 are simply combined to form a bi-directional prediction motion vector.

[0154] 2) Otherwise, if Mv1 and Mv2 are from the same list, only the unidirectional predicted motion Mv2 is stored.

[0155] Intra-mode codec with 67 intra-prediction modes

[0156] To capture arbitrary edge directions that occur in natural video, the number of directional intra modes in VVC is extended from the 33 used in HEVC to 65. New directional modes not present in HEVC are indicated by dashed arrows in 14. Planar and DC modes remain unchanged. These denser directional intra prediction modes apply to all block sizes and for luma and chroma intra prediction.

[0157] In VVC, for non-square blocks, several traditional angular intra prediction modes are adaptively replaced with wide-angle intra prediction modes.

[0158] In HEVC, each intra-coded block has a square shape and the length of each of its sides is a power of 2. Therefore, no division operation is required to generate intra predictors using DC mode. In VVC, blocks can have a rectangular shape, which in general requires the use of division operations for each block. To avoid division operations for DC prediction, only the longer sides are used to calculate the average for non-square blocks.

[0159] In order to keep the complexity of most probable mode (MPM) list generation low, the intra mode codec with 6 MPMs is used by considering two available adjacent intra modes. The following three aspects are considered when constructing the MPM list:

[0160] - Default intra mode

[0161] - Adjacent Intra Mode

[0162] - Intra-mode for export

[0163] Regardless of whether the MRL and ISP codecs are applied, a unified 6-MPM list is used for intra blocks. The MPM list is constructed based on the intra modes of the neighboring blocks on the left and above. Assuming that the mode on the left is marked as Left and the mode of the block above is marked as Above, the unified MPM list is constructed as follows:

[0164] - When a neighboring block is not available, its intra mode is set to planar by default.

[0165] -If both Left and Above modes are non-angle modes:

[0166] -MPM List → {plane, DC, V, H, V-4, V+4}

[0167] - If one of Left and Above modes is an angle mode and the other is a non-angle mode:

[0168] - Set mode Max to the larger mode between Left and Above

[0169] -MPM List → {Flat, Max, DC, Max-1, Max+1, Max-2}

[0170] -If Left and Above are both angled and they are different:

[0171] - Set mode Max to the larger mode between Left and Above

[0172] - If the difference between the modes Left and Above is in the range 2 to 62, inclusive

[0173] ●MPM List → {Plane, Left, Above, DC, Max-1, Max+1}

[0174] -otherwise

[0175] ●MPM List → {Plane, Left, Above, DC, Max-2, Max+2}

[0176] -If Left and Above are both angled and they are the same:

[0177] -MPM List → {Plane, Left, Left-1, Left+1, DC, Left-2}

[0178] In addition, the first bin of the MPM index codeword is decoded by the CABAC context. A total of three contexts are used, corresponding to whether the current intra block is MRL-enabled, ISP-enabled, or a normal intra block.

[0179] In the 6 MPM list generation processes, pruning is used to remove repeated patterns so that only unique patterns can be included in the MPM list. For the entropy coding and decoding of the 61 non-MPM modes, Truncated Binary Code (TBC) is used.

[0180] Wide-Angle Intra Prediction for Non-SquareBlocks

[0181] The conventional angular intra prediction direction is defined as clockwise from 45 degrees to -135 degrees. In VVC, several traditional angular intra prediction modes are adaptively replaced with wide-angle intra prediction modes for non-square blocks. The replaced modes are sent using the original mode index, which is remapped to the index of the wide-angle mode after parsing. The total number of intra prediction modes remains unchanged, i.e. 67, and the intra mode encoding and decoding method remains unchanged.

[0182] To support these prediction directions, a top reference of length 2W+1 and a left reference of length 2H+1 are respectively Fig.19A and Fig.19B Definition shown.

[0183] The number of alternative modes in the wide-angle direction mode depends on the aspect ratio of the block. The alternative intra prediction modes are shown in Table 3.

[0184] Table 3 - Intra prediction modes replaced by wide angle mode

[0185] Aspect Ratio Alternative intra prediction modes W / H==16 Mode 12, 13, 14, 15 W / H==8 Mode 12, 13 W / H==4 Mode 2,3,4,5,6,7,8,9,10,11 W / H==2 Mode 2,3,4,5,6,7, W / H==1 none W / H==1 / 2 Mode 61, 62, 63, 64, 65, 66 W / H==1 / 4 Modes 57,58,59,60,61,62,63,64,65,66 W / H==1 / 8 Mode 55, 56 W / H==1 / 16 Modes 53, 54, 55, 56

[0186] In VVC, 4:2:2 and 4:4:4 as well as 4:2:0 chroma formats are supported. The chroma derived mode (DM) derivation table for the 4:2:2 chroma format was originally ported from HEVC, expanding the number of entries from 35 to 67 to be consistent with the expansion of intra prediction modes. Since the HEVC specification does not support prediction angles below -135 degrees and above 45 degrees, the luma intra prediction modes from 2 to 5 are mapped to 2. Therefore, the chroma DM derivation table for the 4:2:2: chroma format is updated by replacing some values ​​of the mapping table entries to more accurately convert the prediction angles of chroma blocks.

[0187] Decoder Side Intra Mode Derivation (DIMD)

[0188] When DIMD is used, two intra modes are derived from reconstructed neighboring samples, and these two predictors are combined with a planar mode predictor with weights derived from the gradients. DIMD mode is used as an alternative prediction mode and is always checked in high complexity RDO mode.

[0189] In order to implicitly derive the intra prediction mode of a block, texture gradient analysis is performed on the encoder and decoder side. This process starts with an empty gradient histogram (HoG) with 65 entries, corresponding to 65 angular modes. The amplitudes of these entries are determined during texture gradient analysis.

[0190] In the first step, DIMD selects T=3 rows and columns of templates from the left and top of the current block, respectively. This area is used as a reference for gradient-based intra prediction mode derivation.

[0191] In the second step, horizontal and vertical Sobel filters are applied to all 3×3 window locations, centered on the pixels in the template midline. At each window location, the Sobel filter computes the pure horizontal and vertical intensities Gx and Gy, respectively. The texture angle of the window is then calculated as:

[0192] angle=arctan(G x / δ y ), (15)

[0193] This can be converted into one of the 65 angular intra prediction modes. Once the intra prediction mode index for the current window is derived as idx, the magnitude of its entry in HoG[idx] is updated by the following addition:

[0194] ampl=|G x |+|G y | (16)

[0195] 20A-C show examples of the HoG calculated after applying the above operations to all pixel positions in the template. Fig. 20A An example of a selected template 2020 for a current block 2010 is shown. The template 2020 includes T horizontal columns above the current block and T vertical rows to the left of the current block. For intra prediction of the current block, the area 2030 above and to the left of the current block corresponds to the reconstructed area, while the area 2040 below and to the right of the block corresponds to the unavailable area. Fig. 20B An example is shown where T=3 and HoG is calculated for pixel 2060 in the middle horizontal column and pixel 2062 in the middle vertical row. For example, for pixel 2052, a 3x3 window 2050 is used. Figure 2 C shows an example of the amplitude (ampl) calculated based on equation (16) for the angular intra prediction mode as determined from equation (15).

[0196] Once the HoG is calculated, the indices with the two highest histogram bins are selected as the two implicitly derived intra prediction modes for the block, and further combined with the planar mode as the prediction for the DIMD mode. Prediction fusion is applied as a weighted average of the three predictors mentioned above. For this purpose, the weight of the plane is fixed to 21 / 64 (~1 / 3). The remaining weight 43 / 64 (~2 / 3) is then shared between the two HoG IPMs, proportional to the amplitude of their HoG bins. Fig.21 An example of mixed processing is shown. Fig.21As shown, two intra modes (M1 2112 and M2 2114) are selected based on the index of the two highest bins with the histogram bin 2110. Three predictors (2140, 2142 and 2144) are used to form a hybrid prediction. The three predictors correspond to applying M1, M2 and planar intra modes (2120, 2122 and 2124, respectively) to the reference pixel 2130 to form the corresponding predictor. The three predictors are weighted by corresponding weighting factors (ω1, ω2 and ω3) 2150. The weighted predictors are summed using adder 2152 to generate a hybrid predictor 2160.

[0197] Furthermore, two implicitly derived intra modes are included in the MPM list so that DIMD processing is performed before building the MPM list. The primary derived intra mode of a DIMD block is stored with the block and used for MPM list construction for neighboring blocks.

[0198] Template-based Intra Mode Derivation (TIMD)

[0199] The template-based intra mode derivation (TIMD) mode uses neighboring templates at the encoder and decoder to implicitly derive the intra prediction mode of the CU, instead of sending the intra prediction mode to the decoder. Fig. 22 As shown, the template prediction samples (2212 and 2214) of the current block 2210 are generated using the template reference samples (2220 and 2222) of each candidate mode. The cost is calculated as the sum of the absolute transformed differences (SATD) between the predicted samples and the reconstructed samples of the template. The intra prediction mode with the lowest cost is selected as the DIMD mode and used for intra prediction of the CU. The candidate mode can be 67 intra prediction modes such as in VVC or extended to 131 intra prediction modes. Typically, MPM can provide clues to indicate the directional information of the CU. Therefore, in order to reduce the intra mode search space and utilize the characteristics of the CU, the intra prediction mode can be implicitly derived from the MPM list.

[0200] For each intra prediction mode in the MPM, the SATD between the predicted and reconstructed samples of the template is calculated. The first two intra prediction modes with the smallest SATD are selected as TIMD modes. These two TIMD modes are fused with weights after applying PDPC processing, and this weighted intra prediction is used to encode and decode the current CU. The position dependent intra prediction combination (PDPC) is included in the derivation of the TIMD mode.

[0201] The costs of the two selection modes are compared to a threshold, and in the test a cost factor of 2 is applied as follows:

[0202] costMode2<2*costMode1.

[0203] If this condition is true, fusion is applied, otherwise only mode 1 is used. The weight of the mode is calculated based on its SATD cost as follows:

[0204] weight1=costMode2 / (costMode1+costMode2)

[0205] weight2=1-weight1.

[0206] Intra Sub-Partitions (ISP)

[0207] Intra Sub-Partition (ISP) divides the luma intra prediction block vertically or horizontally into 2 or 4 sub-partitions depending on the block size. For example, the minimum block size of ISP is 4x8 (or 8x4). If the block size is larger than 4x8 (or 8x4), the corresponding block is divided into 4 sub-partitions. It has been noted that M×128 (M≤64) and 128×N (N≤64) ISP blocks may cause potential problems with the 64×64 Virtual Decoder Pipeline Unit (VDPU). For example, an M×128 CU in a single-tree case has an M×128 luma TB and two corresponding M / 2×64 chroma TBs. If the CU uses ISP, the luma TB will be divided into 4 M×32 TBs (only horizontally divisible), each smaller than a 64×64 block. However, in the current ISP design, the chroma blocks are not divided. Therefore, the size of both chroma components will be larger than 32×32 blocks. Similarly, a 128×N CU using ISP may produce similar situations. Therefore, these two situations are problems of 64×64 decoder pipeline. For this reason, the CU size that can use ISP is limited to a maximum of 64×64. Fig.23A and Fig. 23B An example showing both possibilities. All subpartitions satisfy the condition of having at least 16 samples.

[0208] In ISP, 1xN and 2xN sub-block predictions are not allowed to depend on the reconstructed values ​​of previously decoded 1xN and 2xN sub-blocks of the codec block, making the minimum prediction width of the sub-block four samples. For example, an 8xN (N>4) codec block encoded and decoded using ISP with vertical partitioning is divided into two prediction regions, each of size 4xN, and four transforms of size 2xN. In addition, a 4xN codec block (encoded and decoded using ISP with vertical partitioning) uses the full 4xN block for prediction; four transforms are used, each of which is 1xN. Although 1xN and 2xN transform sizes are allowed, it can be asserted that the transforms of these blocks in a 4xN region can coexist. For example, when a 4xN prediction region contains four 1xN transforms, there is no transform in the horizontal direction; the transform in the vertical direction can be performed as a single 4xN transform in the vertical direction. Similarly, when a 4xN prediction region contains two 2xN transform blocks, the transform operations for the two 2xN blocks in each direction (horizontally and vertically) can be performed in parallel. Therefore, the latency is not increased when processing these smaller blocks compared to processing 4x4 regular codec intra blocks.

[0209] Table 4

[0210] Block size Coefficient group size 1×N, N≥16 1×16 N×1, N≥16 16×1 2×N, N≥8 2×8 N×2, N≥8 8×2 All other possible M×N situations 4×4

[0211] For each subpartition, the reconstructed sample is obtained by adding the residual signal to the prediction signal. Here, the residual signal is generated by processes such as entropy decoding, inverse quantization, and inverse transform. Therefore, the reconstructed sample values ​​of each subpartition can be used to generate a prediction for the next subpartition, and each subpartition is processed consecutively. In addition, the first subpartition to be processed is the subpartition containing the upper left corner sample of the CU, and then continues down (horizontal split) or to the right (vertical split). As a result, the reference samples used to generate the subpartition prediction signal are located only on the left and above the row. All subpartitions share the same intra mode. The following is a summary of the interaction of ISP with other codec tools.

[0212] -Multiple Reference Line (MRL): If the MRL index of a block is not 0, the ISP codec mode will be inferred to be 0, so the ISP mode information will not be sent to the decoder.

[0213] - Entropy Codec Coefficient Group Size: The size of the entropy codec sub-blocks has been modified so that they have 16 samples in all possible cases, as shown in Table 4. Note that the new size only affects ISP-generated blocks where one dimension has less than 4 samples. In all other cases, the coefficient groups maintain 4×4 dimensions.

[0214] -CBF codec: It is assumed that at least one subpartition has a non-zero CBF. Thus, if n is the number of subpartitions, and the first n-1 subpartitions yield zero CBF, the CBF of the nth subpartition is inferred to be 1.

[0215] - Transform size restriction: All ISP transforms with length greater than 16 points use DCT-II.

[0216] -MTS flag: If the CU uses the ISP codec, the MTS CU flag will be set to 0 and will not be sent to the decoder. Therefore, the encoder will not perform RD tests on the different available transforms for each result subpartition. The transform selection for ISP mode will be fixed instead, and selected based on the intra mode, processing order, and block size used. Therefore, signaling is not required. For example, let tH and tV be the horizontal and vertical transforms selected for the w×h subpartition, where w is the width and h is the height. Then the transform can be selected according to the following rules:

[0217] - If w=1 or h=1, there is no horizontal or vertical transformation, respectively.

[0218] - If w≥4 and w≤16, then tH=DST-VII, otherwise, tH=DCT-II

[0219] - If h≥4 and h≤16, then tV=DST-VII, otherwise, tV=DCT-II

[0220] - In ISP mode, all 67 intra modes are allowed. PDPC is also applied if the corresponding width and height are at least 4 samples long. In addition, the reference sample filtering process (reference smoothing) and intra interpolation filter selection conditions no longer exist, and the Cubic (DCT-IF) filter is always applied for fractional position interpolation in ISP mode.

[0221] Multi-Hypothesis Prediction (MHP)

[0222] In the multi-hypothesis inter prediction mode (JVET-M0425), in addition to the traditional bidirectional prediction signal, one or more additional motion compensated prediction signals are sent. The final overall prediction signal is obtained by weighted superposition of samples. Using the bidirectional prediction signal pbi and the first additional inter prediction signal / hypothesis h3, the resulting prediction signal p3 is as follows:

[0223] p3=(1-α)p bi +αh3 (17)

[0224] The weight factor α is specified by a new syntax element add_hyp_weight_idx according to the following mapping (Table 5):

[0225] Table 5. Mapping α to add_hyp_weight_idx

[0226]

[0227]

[0228] Similar to above, more than one additional prediction signal is used. The resulting overall prediction signal is repeatedly accumulated together with each additional prediction signal.

[0229] p n+1 =(1-α n+1 ) n +α n+1 h n+1 (18)

[0230] The resulting overall prediction signal is retrieved as the last pn (ie the pn with the largest index n). For example, at most two additional prediction signals are used (ie n is limited to 2).

[0231] The motion parameters for each additional prediction hypothesis can be signaled explicitly by specifying the reference index, motion vector predictor index and motion vector difference, or implicitly by specifying the merge index. A separate multi-hypothesis merge flag is used to distinguish between these two signaling modes.

[0232] For inter-frame AMVP mode, MHP is applied only when unequal weights in BCW are selected in bidirectional prediction mode. Detailed information on MHP for VVC can be found in JVET-W2025 (Muhammed Coban, et.al., "Algorithm description of Enhanced Compression Model 2 (ECM 2)", Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 23rd Meeting, bytleconference, 7-16 July 2021, Document: JVET-W2025).

[0233] GPM Extensions

[0234] Several variations of the GPM mode (JVET-W0097 (Zhipin Deng, et.al., "AEE2-related: Combination of EE2-3.3, EE2-3.4and EE2-3.5", Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 23rd Meeting, by teleconference, July 16, 2021, Document: JVET-W0097) and JVET-Y0065 (Yoshitaka Kidani, et.al., "EE2-3.1: GPM with inter and intra prediction (JVET-X0166)", Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 25th Meeting, byteleleconference, December 21, 2021, Document: JVET-W0097) 2022, Document: JVET-Y0065)) has proposed to improve the encoding and decoding efficiency of the GPM mode in VVC. These methods are included in the exploration experiment (EE2) for further evaluation. The main technical aspects are described as follows:

[0235] EE2-3.3 Regarding GPM with MMVD (GPM-MMVD): 1) Additional MVDs are added to the existing GPM merge candidates; 2) The signaling method of MVD is the same as MMVD in VVC, that is, one distance index plus one direction index; 3) Two flags are issued to control whether MMVD is applied to each GPM partition separately.

[0236] EE2-3.4-3.5 Regarding GPM with template matching (GPM-TM): 1) Extend template matching to GPM mode by refining the GPM MV based on the upper left neighboring samples of the current CU; 2) Select template samples according to the GPM split direction; 3) Issue a flag to jointly control whether template matching is applied to the MVs of two GPM partitions.

[0237] JVET-W0097 proposes a combination of EE2-3.3, EE2-3.4 and EE2-3.5 to further improve the encoding and decoding efficiency of the GPM mode. Specifically, in the proposed combination, the existing designs in EE2-3.3, EE2-3.4 and EE2-3.5 remain unchanged, while the following modifications are further made to coordinate the two encoding and decoding tools:

[0238] 1) GPM-MMVD and GPM-TM are dedicated to one GPM CU. This is done by first sending the GPM-MMVD syntax. When both GPM-MMVD control flags are equal to false (i.e., GPM-MMVD is disabled for both GPM partitions), the GPM-TM flag is sent to indicate whether template matching is applied to both GPM partitions. Otherwise (at least one GPM-MMVD flag is equal to true), the value of the GPM-TM flag is inferred to be false.

[0239] 2) The GPM merge candidate list generation method in EE2-3.3 and EE2-3.4-3.5 is directly combined with the MV pruning scheme in EE2-3.4-3.5 (where the MV pruning threshold is adjusted according to the current CU size) to replace the default MV pruning scheme applied in EE2-3.3; in addition, as in EE2-3.4-3.5, multiple zero MVs are added until the GPM candidate list is completely filled.

[0240] In JVET-Y0065, in GPM with inter and intra prediction (or named intra-GPM), the final prediction samples are generated by weighting the inter prediction samples and intra prediction samples for each GPM partition area. The inter prediction samples are derived by the same scheme as the GPM in the current ECM, while the intra prediction samples are derived by the intra prediction mode (IPM) candidate list and the index signal sent from the encoder. The IPM candidate list size is predefined as 3. Fig.24A As shown in Figure 1-C, the available IPM candidates are parallel angle mode (parallel mode) relative to the GPM block boundary, vertical angle mode (vertical mode) relative to the GPM block boundary, and planar mode. Fig.24D As shown, GPM with intra and intra prediction is restricted in the proposed method to reduce the signaling overhead of IPM and avoid the increase in the size of intra prediction circuit on the hardware decoder. In addition, direct motion vector and IPM storage are introduced in the GPM hybrid region to further improve the codec performance.

[0241] Space GPM

[0242] Similar to inter-frame GPM, spatial GPM (SGPM) consists of a partition mode and two associated intra-frame prediction modes. If these modes are sent directly in the bitstream, such as Fig.25A As shown in , it will generate a lot of overhead bits. In order to express the necessary partition and prediction information more efficiently in the bitstream, the candidate list is applied and only the candidate index is sent in the bitstream. Fig.25B As shown, each candidate in the list can derive a combination of a partition mode and two intra prediction modes.

[0243] A template is used to generate the candidate list. The shape of the template is as follows Fig.26 For each possible combination of a partition mode and two intra prediction modes, a prediction is generated for the template, and the partition weights are extended to the template, as Fig.26 As shown in Figure 2. These combinations of template-based predictions and reconstructions are arranged in ascending order of their SATD. The length of the candidate list is set to 16, and these candidates are considered to be the most likely SGPM combinations for the current block. Both the encoder and the decoder build the same candidate list based on the template.

[0244] To reduce the complexity of building the candidate list, the number of possible partition modes and the number of possible intra prediction modes are pruned. In the following tests, 26 of the 64 partition modes are used, and only MPM of the 67 intra prediction modes is used.

[0245] Although GPM together with blending in boundary regions has shown improvement in codec performance, adaptive blending prediction needs to be developed to further improve codec performance. Summary of the invention

[0246] A method and apparatus for video encoding and decoding are disclosed. According to the method, data associated with a current block is received at an encoder side or coded data associated with a current block to be decoded is received at a decoder side. The current block is divided into a first region and a second region along a partition line. A mixed region around the partition line is determined, wherein the mixed region includes a first mixed region and a second mixed region, the first mixed region includes a first pixel in a first region defined by a partition line and distance_1, and the second mixed region includes a second pixel in a second region defined by a partition line and distance_2, distance_1 and distance_2 correspond to two non-negative values, and the two non-negative values ​​are allowed to be different. A first predictor and a second predictor are determined for the first region and the second region, respectively. A mixed predictor of the mixed region is generated by mixing the first predictor and the second predictor. The current block is encoded or decoded by using prediction data including the mixed predictor.

[0247] In one embodiment, the hybrid predictor is generated as a weighted sum of the first predictor and the second predictor.

[0248] In one embodiment, a candidate set of sizes of the mixed region is predefined, wherein the size of the mixed region is related to distance_1 and distance_2. In another embodiment, the size of the mixed region of the current block is implicitly determined based on a target member of the candidate set that achieves a minimum template matching cost, and wherein the template matching cost is measured based on the template of the current block and the extended mixed region.

[0249] In one embodiment, an explicit index is sent or parsed to indicate the size of the mixed region selected from the candidate set of the current block. For example, the syntax associated with the explicit index can be sent at the block level, the sequence parameter set (SPS) level, the picture parameter set (PPS) level, the adaptation parameter set (Adaptation Parameter Set) level, the picture header (PH) level, the slice header (SH) level, or a combination thereof. In one embodiment, the candidate set is reordered according to the cost associated with each member of the candidate set. In one embodiment, the cost corresponds to a template matching cost measured based on the template and the extended mixed region, and wherein the template includes the left adjacent region of the current block, the top adjacent region of the current block, or both.

[0250] In one embodiment, the target candidate (distance_1, distance_2) is selected from the candidate set of the current block according to the block size of the current block. The block size may correspond to a block width, a block height, or a block area. In one embodiment, the candidate set includes data pairs corresponding to the combination of distance_1 and distance_2, wherein distance_1 and distance_2 are selected from the value group of 0, 1, 2, 4, and 8, respectively.

[0251] In one embodiment, if the block width or the block height is less than a predetermined threshold, only the smaller value in the value group is used to form the data pair of the candidate set. In another embodiment, if the first area is smaller than the second area, distance_1 is set to a value smaller than distance_2. In yet another embodiment, if the first area is larger than the second area, distance_1 is set to a value greater than distance_2.

[0252] In one embodiment, the first predictor and the second predictor correspond to prediction hypotheses including one or more first predictions from motion having a predetermined direction or bidirectionality, one or more second predictions generated from motion candidates, one or more third predictions from intra-frame candidates, one or more motion candidates, or any combination thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0253] Figure 1A An example adaptive inter / intra video codec system incorporating loop processing is shown.

[0254] Figure 1B Shown with Figure 1A The decoder corresponding to the encoder in .

[0255] Figure 2 Neighboring blocks used to derive VVC spatial merging candidates are shown.

[0256] Figure 3 Possible candidate pairs considered for redundancy checking in VVC are shown.

[0257] Figure 4 An example of temporal candidate derivation is shown, where a scaled motion vector is derived based on the Picture Order Count (POC) distance.

[0258] Figure 5 The position of the temporal candidate selected between candidates C0 and C1 is shown.

[0259] Figure 6The distance offset from the starting MV in the horizontal and vertical directions according to the Merge Mode with MVD (MMVD for short) is shown.

[0260] Fig. 7A An example of an affine motion field of a block described by motion information of two control points (4-parameters) is shown.

[0261] Figure 7B An example of an affine motion field of a block described by motion information of two control points (6-parameters) is shown.

[0262] Figure 8 An example of block-based affine transform prediction is shown, where the motion vector for each 4×4 luma sub-block is derived from a control point MV.

[0263] Fig. 9 An example of derivation of inherited affine candidates based on control point MVs of neighboring blocks is shown.

[0264] Fig.10 An example of affine candidate construction is shown by combining translational motion information of each control point from spatial neighbors and time.

[0265] Fig.11 An example of affine motion information storage for motion information inheritance is shown.

[0266] Fig.12 An example of derivation of weight values ​​for Combined Inter and Intra Prediction (CIIP) according to the coding modes of the top and left neighboring blocks is shown.

[0267] Fig.13 An example of 64 partitions used in the VVC standard is shown, where the partitions are grouped according to their angles, and the dashed lines represent redundant partitions.

[0268] Fig.14 An example of unidirectional prediction MV selection for geometric partitioning mode is shown.

[0269] Fig.15 An example of bending weight ω0 using geometric partitioning mode is shown.

[0270] Fig.16 An example of a GPM blending process based on discrete ramp functions for blending regions around boundaries is shown.

[0271] Fig.17 An example of a GPM mixing process for GPM mixing in ECM 4.0 is shown.

[0272] Fig.18The intra-frame prediction mode adopted by the VVC video coding standard is shown.

[0273] Fig.19A -B shows blocks that are wider than they are tall ( Fig.19A ) and blocks that are taller than they are wide ( Fig.19B ) is an example of wide-angle intra prediction.

[0274] Fig. 20A An example of a template selected for the current block is shown, where the template includes a horizontal column of T above the current block and a vertical row of T on the left side of the current block.

[0275] Fig. 20B An example is shown in which T=3 and a gradient histogram (HoGs) is calculated for pixels in the middle horizontal column and pixels in the middle vertical row.

[0276] Fig. 20C An example of the amplitude (ampl) of the angular intra prediction mode is shown.

[0277] Fig.21 An example of a blending process is shown where two angular intra modes (M1 and M2) are selected based on the index having the two highest bins in the histogram bins.

[0278] Fig. 22 An example of a template-based intra mode derivation (TIMD) mode is shown, where TIMD implicitly derives the intra prediction mode of a CU using neighboring templates at the encoder and decoder.

[0279] Fig.23A An example of intra sub-partition (ISP) is shown, in which a block is divided into two sub-blocks horizontally or vertically.

[0280] Fig. 23B An example of intra sub-partition (ISP) is shown, in which a block is divided into four sub-blocks horizontally or vertically.

[0281] Fig.24A -C show examples of available IPM candidates: parallel angle patterns relative to GPM block boundaries (parallel patterns, Fig.24A ), the vertical angle mode relative to the GPM block boundary (vertical mode, Fig. 24B ) and planar mode ( Fig.24C ).

[0282] Fig.24DAn example of a GPM with intra and intra prediction is shown, where the intra prediction is restricted to reduce the signaling overhead and hardware decoder cost of the IPM.

[0283] Fig.25A An example of syntax encoding and decoding for spatial GPM (SGPM for short) is shown before using the simplified method.

[0284] Fig.25B An example of simplified syntax encoding and decoding of spatial GPM (SGPM for short) is shown.

[0285] Fig.26 An example of a template of a spatial GPM (SGPM for short) is shown.

[0286] Fig. 27 An example of adaptive blending with separate blending sizes for two blending regions according to one embodiment of the present invention is shown.

[0287] Fig.28 An example of determining costs associated with individual blend sizes based on a template and an extended blend region according to one embodiment of the present invention is shown.

[0288] Fig.29 A flowchart of an exemplary video coding and decoding system using a hybrid predictor according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0289] It is readily understood that the components of the present invention as generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of embodiments of the systems and methods of the present invention, as shown in the figures, is not intended to limit the scope of the claimed invention, but is merely representative of selected embodiments of the present invention. References in this specification to "an embodiment," "some embodiments," or similar language mean that specific features, structures, or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present invention. Therefore, the phrases "in an embodiment" or "in some embodiments" appearing in various places throughout this specification do not necessarily all refer to the same embodiment.

[0290] In addition, the described features, structures or characteristics may be combined in any suitable manner in one or more embodiments. However, those skilled in the art will recognize that the present invention may be implemented without one or more specific details or using other methods, components, etc. In other cases, well-known structures or operations are not shown or described in detail to avoid obscuring various aspects of the present invention. The illustrated embodiments of the present invention will be best understood by reference to the accompanying drawings, in which the same components are represented by the same numbers throughout. The following description is by way of example only, and simply describes some selected embodiments of the apparatus and method consistent with the present invention as claimed herein.

[0291] In order to improve the efficiency of video coding and decoding, a mixture of one or more prediction hypotheses with one or more existing prediction hypotheses is used to achieve better prediction accuracy. In one embodiment, the prediction hypothesis means prediction based on motion with a predetermined direction (list 0 or list 1). In another embodiment, the prediction hypothesis refers to a prediction generated from a motion candidate (e.g., a merged candidate or an AMVP candidate). In another embodiment, the prediction hypothesis means prediction or bidirectional prediction based on motion with a predetermined direction (list 0 or list 1). In another embodiment, the prediction hypothesis means prediction from intra-frame candidates, motion candidates, and candidates with a third mode type. For example, all prediction hypotheses come from intra-frame candidates. As another example, all prediction hypotheses come from motion candidates. As another example, one or more prediction hypotheses come from one or more intra-frame candidates, and another one or more prediction hypotheses come from one or more motion candidates. In another embodiment, the prediction hypothesis means prediction from intra-frame candidates. Intra-frame candidates can refer to any intra-frame prediction mode specified in the standard. For example, the intra-frame prediction mode can be DC, plane, angle prediction mode, TIMD, DIMD, chroma intra-frame prediction mode or any subset of the above. The chroma intra prediction mode includes a cross-component prediction mode, which predicts the chroma predictor in the current block by using co-located luminance reconstructed samples with one or more linear and / or non-linear models. For example, the cross-component prediction mode refers to a CCLM mode or any extension or option from CCLM. A candidate of the third mode type may refer to a candidate of any mode type defined in the standard. For example, a candidate of the third mode type refers to an intra-block copy candidate, which uses a block vector to predict the current block by referring to reconstructed samples in the same picture as the current block. Embodiments of intra-frame candidates and / or motion candidates can be used as embodiments of candidates with the third mode type. In another embodiment, a hybrid prediction tool refers to (but is not limited to) any one or more of the tools listed below or any combination of the listed tools.

[0292] - Hybrid prediction tools include bi-predictive motion candidates, which can be merge candidates and / or AMVP candidates.

[0293] - The hybrid prediction tool includes GPM, one or more variants of GPM extension and / or spatial GPM.

[0294] Since more than one prediction hypothesis is used for the current block, a hybrid process is required to form the final prediction of the current block. The current block can belong to a luminance or chrominance component. For a single tree partition, the luminance and chrominance components are in the same tree partition structure, and the proposed method can be used only for the luminance component, only for the chrominance component, or for both luminance and chrominance components. For a double tree partition, the luminance and chrominance are in different tree split structures, and the proposed method can be used only for the luminance component, only for the chrominance component, or for both luminance and chrominance components. In one embodiment, when the proposed method is used for both luminance and chrominance components, the settings of the proposed method are unified for the luminance and chrominance components. Taking the luminance and chrominance components in the same tree partition structure as an example, the luminance and chrominance components in the same codec unit use the unified settings of the proposed method. For another example of luminance and chrominance components in different tree partition structures, the chrominance block follows the settings of the proposed method for one or more co-located luminance blocks.

[0295] In the present invention, an adaptive hybrid process is proposed to improve the weighting scheme used in hybrid prediction. GPM is taken as an example of a hybrid tool below. However, the proposed adaptive hybrid process can be applied to one or more of the mentioned hybrid prediction tools and / or any combination of the mentioned hybrid prediction tools.

[0296] According to an embodiment of the present invention, first a partition line (eg, a GPM partition boundary) is defined to divide the current block into two prediction regions, such as Fig. 27 As shown. The area near the partition line (i.e., theta1 from the partition line and -theta2 from the partition line) is defined as the mixed area. Here, theta1 and theta2 are two non-negative values, and the sign of -theta2 is to indicate that the mixed area 2 is on the other side of the partition line of the mixed area 1. The sizes of the mixed area 1 and the mixed area 2 are called theta1 (or distance_1) and theta2 (or distance_2), respectively. Within the mixed area, multiple (for example, 2, first_hyp_pred and second_hyp_pred) prediction hypotheses are combined with weights (refer to W0[x][y]). Outside the mixed area: for samples located in the first prediction area, the weight of the second prediction hypothesis is zero, and the weight of the first prediction hypothesis is N, where N is a positive integer; for samples located in the second prediction area, the weight of the first prediction hypothesis is zero, and the weight of the second prediction hypothesis is N. Several embodiments are proposed:

[0297] ● In one embodiment, final_pred[x][y]=(first_hyp_pred[x][y]*W0[x][y]+seconf_hyp_pred[x][y]*(N-W0[x][y])+offset1)>>shift1

[0298] ○(x,y) is the sample position in the current block.

[0299] ○ For the sample at (x, y) in the current block, W0[x][y] is the weight of first_hyp_pred, and (N-W0[x][y]) is the weight of second_hyp_pred.

[0300] ●In another embodiment, N is predefined as a fixed positive integer (e.g., 8, 16, 32, or 64) or specified by block level, sequence parameter set (Sequence Parameter Set, for short) level, picture parameter set (Picture Parameter Set, for short) level, adaptability parameter set (Adaptation Parameter Set, for short APS) level, picture header (Picture header, for short PH) level, and / or slice header (Slice header, for short SH) level syntax.

[0301] ● In another embodiment, offset1 and shift1 are determined according to N and / or BitDepth. For the example of N=8,

[0302] ○shift1=Max(5,17-BitDepth)

[0303] ○offset1=1<<(shift1-1)

[0304] ●In another embodiment, if the sample (x, y) is located in the second prediction region (i.e., the distance (x, y) is less than or equal to -theta2), then W0[x][y] is defined to be 0. Alternatively, the definition of W0[x][y] follows the derivation of the mixed region, and the derived value is equal to or close to 0.

[0305] ● In another embodiment, if the sample (x, y) is in the first prediction region (i.e., distance(x, y) is greater than or equal to theta1), then W0[x][y] is defined as N. Alternatively, the definition of W0[x][y] follows the derivation of the mixed region, and the derived value is equal to or close to N.

[0306] In another embodiment, thata1 equals 0 means that there is no mixing in the first prediction region. In this case, W0[x][y] is defined as N in the first prediction region.

[0307] In another embodiment, thata2 is equal to 0 indicating that there is no mixing in the second prediction region. That is, W0[x][y] is defined as 0 in the second prediction region.

[0308] ●In another embodiment, if the sample (x, y) is located in the mixed region (i.e., distance(x, y) is greater than -theta2 and less than theta1), W0[x][y] is defined according to distance, theta1 and / or theta2. For example, W0[x][y] is defined according to the existing GPM weight derivation (e.g., VVC method), and theta (used for GPM weight derivation) is set to theta1 or theta2 we proposed.

[0309] ○ In another sub-embodiment, if the sample (x, y) is located in the mixed region within the first prediction region (i.e., distance(x, y) is greater than 0 and less than theta1), W0[x][y] is defined based on the distance and theta1. For example, W0[x][y] is defined as (N*(distance(x, y)+theta1)) / (2*theta1) or can be simplified by quantization. For example, after quantization, W0[x][y] is defined as ((distance′(x, y)+16*theta1+offset2)>>shift2) and is clipped to [0, N].

[0310] ■distance′ can be wIdxL in the GPM introduction section

[0311] ■offset2=theta1>>1

[0312] ■shift2=log2(theta1)

[0313] ○ To give another example, after quantization, W0[x][y] is defined as ((distance′(x, y)+16*theta1+offset3)>>shift3) and is clipped to [0, N].

[0314] ■distance' can be wIdxL in the GPM introduction section

[0315] ■Offset3 can be right-shifted by 1. Shift3 can be log2(N). For example, if N is 8, offset3 will be 4 and shift3 will be 3.

[0316] ○ In another sub-embodiment, if the sample (x, y) is located in a mixed region within the second prediction region (distance(x, y) is less than 0 and greater than -theta2), then W0[x][y] is defined based on distance and theta2. For example, W0[x][y] is defined as (N*(distance(x, y)+theta2)) / (2*theta2) or can be simplified by quantization. For example, after quantization, W0[x][y] is defined as ((distance′(x, y)+16*theta2+offset2)>>shift2) and is clipped to [0, N].

[0317] ■distance' can be wIdxL in the GPM introduction section

[0318] ■offset2=theta2>>1

[0319] ■shift2=log2(theta2)

[0320] ○ To give another example, after quantization, W0[x][y] is defined as ((distance′(x, y)+16*theta2+offset3)>>shift3) and is clipped to [0, N].

[0321] ■distance' can be wIdxL in the GPM introduction section

[0322] ■Offset3 can be right-shifted by 1. Shift3 can be log2(N). For example, if N is 32, offset 3 will be 16 and shift3 will be 5.

[0323] ○In another sub-embodiment, if the sample (x, y) is located in the mixed area of ​​the partition line (distance(x, y) is equal to 0), then W0[x][y] is defined as the case where "the sample (x, y) is located in the mixed area within the first prediction area", the case where "the sample (x, y) is located in the mixed area within the second prediction area", any proposed embodiment, or is defined as equal weight (N>>1).

[0324] According to the present invention, some embodiments are proposed to determine the values ​​of theta1 and theta2 as follows. In one embodiment, the values ​​of theta1 and theta2 refer to how many pixels or samples there are. For example, a value equal to 2 means 2 pixels. In another embodiment, one value is used as an anchor point for the remaining values. For example, if the value is 2 is the anchor point, the blending size based on a value of 1 is half the blending size based on a value of 2, and the blending size based on a value of 4 is twice the blending size based on a value equal to 2.

[0325] In one embodiment, theta1 is predefined as a fixed value (e.g., 0, 1 / 2, 1 / 4, 1, 2, 4, or 8) or specified by block-level, SPS-level, PPS-level, APS-level, PH-level, and / or SH-level syntax. Similarly, theta2 may be predefined as a fixed value or specified by block-level, SPS-level, PPS-level, APS-level, PH-level, and / or SH-level syntax.

[0326] In another embodiment, theta1 is selected from a candidate set including at least one candidate value. Similarly, theta2 may be selected from a candidate set including at least one candidate value.

[0327] In one sub-embodiment, the candidate set includes at least one of {0, 1 / 2, 1 / 4, 1, 2, 4, or 8} or any combination of the above values.

[0328] In another sub-embodiment, the candidate set includes at least one of {a / b, a, b*a} or any combination of the above values, wherein a and b are set to positive integers. For example, a is equal to 2, and b is equal to 4.

[0329] In another sub-embodiment, the candidate set varies with the block width, block height and / or block area. For example, when the short side of the current block is equal to or less than a predetermined threshold, only the smaller value is included in the candidate set; otherwise, only the larger value is included in the candidate set.

[0330] In another embodiment, theta1 can be the same as or different from theta2. The benefit of allowing different theta1 and theta2 values ​​(i.e., allowing asymmetric theta1 and theta2) is that the best mixing quality for different video sequences may require different mixing areas of the first prediction area and the second prediction area. For example, if the area of ​​the first prediction area is smaller, theta1 should be smaller than theta2. Or conversely, if the area of ​​the first prediction area is larger, theta1 should be smaller than theta2.

[0331] In one sub-embodiment, theta1 and theta2 each have their own candidate sets (e.g., theta1_set and theta2_set). For example, the number of candidates (i.e., the number of candidates in the candidate set) of theta1_set and theta2_set may be different. For another example, one candidate set is a subset of the other candidate set. For another example, the number of candidates of theta1_set and theta2_set is the same.

[0332] In another sub-embodiment, theta1 and theta2 share a single candidate set. For example, theta1 and theta2 are the same. For another example, theta1 and theta2 can be the same or different.

[0333] In another embodiment, the number of candidates in the candidate set is defined as a fixed value (eg, 3 or 5) or specified by block-level, SPS-level, PPS-level, APS-level, PH-level, and / or SH-level syntax.

[0334] In another embodiment, the selection of theta1 and theta2 depends on explicit signaling.

[0335] In one sub-embodiment, two separate syntaxes are sent at the block level, SPS level, PPS level, APS level, PH level, and / or SH level syntax to indicate theta1 and theta2, respectively. For example, theta1 and theta2 are selected from a candidate set containing {0, 1, 2, 4, 8}, respectively. An index (e.g., index_theta1, ranging from 0 to 4) is sent to select a value from the candidate set, and an index (e.g., index_theta2, ranging from 0 to 4) is sent to select a value from the candidate set.

[0336] In one sub-embodiment, syntax is sent at the block level, SPS level, PPS level, APS level, PH level and / or SH level syntax to indicate the combination of theta1 and theta2.

[0337] -Theta1 and theta 2 are selected from a candidate set consisting of {0, 1, 2, 4, 8}. The candidate combination of theta1 and theta2, denoted as (theta1, theta2), can be:

[0338] ○(0,0), (0,1), (0,2), (0,4), (0,8), (1,0), (1,1), (1,2), (1,4), (1,8), (2,0), (2,1), (2,2), (2,4), (2,8), (4,0), (4,1), (4,2), (4,4), (4,8), (8,0), (8,1), (8,2), (8,4), (8,8).

[0339] ○ The number of candidate combinations can be reduced by other methods proposed in the present invention. For example, when the area of ​​the first prediction region is small, theta1 should be smaller than theta2. Therefore, those combinations with larger theta1 can be eliminated. In another example, when the short side of the current block is equal to or smaller than a predetermined threshold, only smaller values ​​are included in the candidate set, thereby reducing the number of candidate combinations. - An indication index (e.g., ranging from 0 to the number of candidate combinations - 1) is sent.

[0340] o In one approach, the transmitted index may be encoded using a truncated unary codec.

[0341] ○ In other words, the index can be the context codec.

[0342] ○ In another way, the candidate combinations are reordered in ascending order by their template costs to form a reordered list. The measure of the template cost can refer to the relevant section of the implicit derivation rule in the present invention. The index sent refers to the position of the used combination in the reordered list. Both the encoder and the decoder build the same reordered list based on the template.

[0343] ■ The candidate combination with the minimum template cost uses the shortest codeword among all candidate combinations.

[0344] In another embodiment, the selection of theta1 and theta2 is subject to implicit derivation.

[0345] In one sub-embodiment, template matching is used as the implicit derivation rule.

[0346] -Step 1: Use the template (or the neighboring area of ​​the current block, which is encoded or decoded before the current block) to measure the cost of each candidate combination. For example, theta1 and theta 2 are selected from the candidate set containing {0, 1, 2, 4, 8}. The candidate combination of theta1 and theta2, denoted as (theta1, theta2), can be:

[0347] ○(0,0),(0,1),(0,2),(0,4),(0,8),(1,0),(1,1),(1,2),(1,4),(1,8),(2,0),(2,1),(2,2),(2,4),(2,8),(4,0),(4,1),(4,2),(4,4),(4,8),(8,0),(8,1),(8,2),(8,4),(8,8). The number of candidate combinations can be reduced by other methods proposed in the present invention. For example, when the area of ​​the first prediction region is small, theta1 should be smaller than theta2. Therefore, those combinations with larger theta1 can be eliminated. In another example, when the short side of the current block is equal to or less than a predetermined threshold, only smaller values ​​are included in the candidate set, thereby reducing the number of candidate combinations.

[0348] - Step 2: For each candidate combination, the template cost is calculated by the distortion between the template's "prediction" and the reconstruction.

[0349] ○ "Prediction" is generated by applying GPM with blending (using candidate combinations) to templates. Fig.28 As shown, the partition line extends to the template 2810. The template includes the left adjacent area of ​​the current block (left template), the upper adjacent area of ​​the current block (upper template), or both. For example, the template selection depends on the information from the partition line. The information of the partition line may refer to the angle of the partition line and / or the offset distance of the partition line. For example, for angles belonging to or close to vertical partitions, (only) the upper template is selected. For example, for angles belonging to or close to horizontal partitions, (only) the left template is selected. For example, for angles belonging to or close to diagonal partitions (not belonging to or close to vertical or horizontal partitions), the left and upper templates are selected. The mixed weights of the template are unified with the mixed weights used to generate the mixed predictor for the current block, or simplified by using fewer weights. For the example of using simplified mixed weights on the template, the weight of the current block can be in the range of 0 to (X-1) and the weight of the template can be less than X weights. The only case is that only 0 and 1 are the weights of the template.

[0350] ○ Distortion can be SATD, Sum of Absolute Difference (SAD), Mean Squared Errors, Sum of Squared Errors, or any distortion measurement equation / metric.

[0351] - Step 3: theta1 and theta2 are implicitly set by the combination with the minimum stencil cost.

[0352] The method proposed in the present invention can be unified with multiple hybrid tools. For example, the proposed methods for GPM, GPM extension and / or spatial GPM are unified.

[0353] The method proposed in the present invention can only be applied to some predetermined partition lines among all candidate partition lines.

[0354] The method proposed in the present invention can be enabled and / or disabled according to implicit rules (such as block width, height or area) or according to explicit rules (such as about block, tile, segment, picture, SPS or PPS level). For example, when the block area is greater than a threshold, the proposed method is applied. For another example, when the longer block edge is greater than or equal to a threshold (such as 2) multiplied by the shorter block edge, the proposed method is applied.

[0355] The term "block" in the present invention may refer to TU / TB, CU / CB, PU / PB, a predetermined area or CTU / CTB.

[0356] The AMVP in the present invention is similar to the "AMVP" in JVET-T2002 (Jianle Chen, et.al., "Algorithm description for Versatile Video Coding and Test Model 11 (VTM 11)", Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 20th Meeting, bytleconference, 7-16 October 2020, Document: JVET-T2002). The AMVP motion comes from the motion candidate with the syntax "merge flag" equal to false. (For example, general_merge_flag in VVC is equal to false).

[0357] Any combination of the methods proposed in the present invention may be applied.

[0358] Any of the above proposed adaptive predictor hybrid methods for codec tools using hybrid predictors can be implemented in the encoder and / or decoder. For example, a hybrid predictor corresponds to a hybrid of two intra predictors or an intra and inter predictor, which can be implemented in an inter / intra / prediction module of the encoder and / or an inter / intra / prediction module of the decoder. For example, on the encoder side, the required processing can be as follows Figure 1A The inter-frame prediction unit 112 or part of the intra-frame prediction unit 110 shown in the figure is implemented. However, the encoder can also use additional processing units to implement the required processing. For the decoder side, the required processing can be as follows Figure 1BThe MC unit 152 and / or a part of the intra-frame prediction 150 shown are implemented. However, the decoder can also use additional processing units to implement the required processing. Alternatively, any of the methods proposed can be implemented as a circuit coupled to the inter-frame / intra-frame / prediction module of the encoder and / or the inter-frame / intra-frame / prediction module of the decoder to provide the information required by the inter-frame / intra-frame / prediction module. The inter-frame prediction 112 and intra-frame prediction 110 on the encoder side and the MC 152 and intra-frame prediction 150 on the decoder side are shown as separate processing units, which can correspond to executable software or firmware codes stored on a medium such as a hard disk or flash memory, for a central processing unit (Central Processing Unit, referred to as CPU) or a programmable device (such as a digital signal processor (DigitalSignal Processor, referred to as DSP) or a field programmable gate array (Field Programmable GateArray, referred to as FPGA)).

[0359] Fig.29 A flowchart of an exemplary video coding system using an adaptive hybrid predictor according to an embodiment of the present invention is shown. The steps shown in the flowchart can be implemented as program codes that can be executed on one or more processors (e.g., one or more CPUs) on the encoder side. The steps shown in the flowchart can also be implemented based on hardware, such as one or more electronic devices or processors arranged to perform the steps in the flowchart. According to the method, in step 2910, data associated with the current block is received at the encoder side, or coded data associated with the current block to be decoded is received at the decoder side. In step 2920, the current block is divided into a first area and a second area along a partition line. In step 2930, a mixed area around the partition line is determined, wherein the mixed area includes a first mixed area and a second mixed area, the first mixed area includes a first pixel in the first area defined by the partition line and distance_1, and the second mixed area includes a second pixel in the second area defined by the partition line and distance_2, wherein distance_1 and distance_2 correspond to two non-negative values ​​and the two non-negative values ​​are allowed to be different values. In step 2940, a first predictor and a second predictor are determined for the first area and the second area, respectively. In step 2950, ​​a hybrid predictor of the mixed region is generated by mixing the first predictor and the second predictor. In step 2960, the current block is encoded or decoded by using prediction data including the hybrid predictor.

[0360] The flowchart shown is intended to illustrate an example of video encoding and decoding according to the present invention. Without departing from the spirit of the present invention, those skilled in the art may modify each step, rearrange the steps, split the steps or combine the steps to implement the present invention. In the present disclosure, specific syntax and semantics are used to illustrate examples to implement embodiments of the present invention. A technician may implement the present invention by replacing the above syntax and semantics with equivalent syntax and semantics without departing from the spirit of the present invention.

[0361] The above description is presented to enable one of ordinary skill in the art to implement the present invention provided in the context of a specific application and its requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the specific embodiments shown and described, but to conform to the widest scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details are described in order to provide a thorough understanding of the present invention. However, it will be understood by those skilled in the art that the present invention may be implemented.

[0362] The embodiments of the present invention as described above can be implemented in various hardware, software codes or a combination of the two. For example, one embodiment of the present invention can be a program code integrated into one or more circuits in a video compression chip or integrated into video compression software to perform the processing described herein. An embodiment of the present invention can also be a program code to be executed on a digital signal processor (Digital Signal Processor, referred to as DSP) to perform the processing described herein. The present invention can also be related to many functions performed by a computer processor, a digital signal processor, a microprocessor or a field programmable gate array (field programmable gate array, referred to as FPGA). These processors can be configured to perform specific tasks according to the present invention by executing machine-readable software code or firmware code that defines the specific method embodied in the present invention. The software code or firmware code can be developed in different programming languages ​​and different formats or styles. The software code can also be compiled for different target platforms. However, the different code formats, styles and languages ​​of the software code and other ways of configuring the code to perform tasks according to the present invention will not depart from the spirit and scope of the present invention.

[0363] The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples are to be considered in all respects as illustrative only and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than by the foregoing description. All changes within the equivalent meaning and scope of the claims should be included within their scope.

Claims

1. A video encoding and decoding method for a plurality of color pictures, the method comprising: Receiving pixel data associated with the current block at the encoding side, or receiving encoded and decoded data associated with the current block to be decoded at the decoding side; Dividing the current block into a first area and a second area along a partition line; determining a blending area around the partition line, wherein the blending area comprises a first blending area and a second blending area, the first blending area comprises a plurality of first pixels in the first area defined by the partition line and distance_1, the second blending area comprises a plurality of second pixels in the second area defined by the partition line and distance_2, and wherein distance_1 and distance_2 correspond to two positive values, and the two positive values ​​are allowed to be different; determining a first predictor and a second predictor for the first region and the second region respectively; generating a mixed predictor for the mixed area by mixing the first predictor and the second predictor; as well as The current block is encoded or decoded by using the prediction data including the hybrid predictor.

2. The video encoding and decoding method according to claim 1, characterized in that: The hybrid predictor is generated as a weighted sum of the first predictor and the second predictor.

3. The video encoding and decoding method according to claim 1, characterized in that: A candidate set of sizes of the mixed region is predefined, wherein the size of the mixed region is related to distance_1 and distance_2.

4. The video encoding and decoding method according to claim 3, characterized in that: An explicit index is sent or parsed to indicate the size of the mixed region selected from the candidate set for the current block.

5. The video encoding and decoding method according to claim 4, characterized in that: The syntax associated with the explicit index is sent at block level, sequence parameter set level, picture parameter set level, adaptability parameter set level, picture header level, slice header level or a combination.

6. The video encoding and decoding method according to claim 4, characterized in that: The candidate set is reordered according to a cost associated with each member of the candidate set.

7. The video encoding and decoding method according to claim 6, characterized in that: The cost corresponds to a template matching cost, the template matching cost is measured based on a template and an extended mixed region, and wherein the template includes a left neighboring region of the current block, a top neighboring region of the current block, or both.

8. The video encoding and decoding method according to claim 3, characterized in that: The size of the mixed region of the current block is implicitly determined based on a target member of the candidate set, the target member of the candidate set achieving a minimum template matching cost, and wherein the template matching cost is measured based on the template of the current block and an extended mixed region.

9. The video encoding and decoding method according to claim 1, characterized in that: The target candidate of (distance_1, distance_2) is selected from the candidate set of the current block according to the block size of the current block.

10. The video encoding and decoding method according to claim 9, characterized in that: The block size corresponds to block width, block height or block area.

11. The video encoding and decoding method according to claim 9, characterized in that: The candidate set includes data pairs corresponding to a combination of distance_1 and distance_2, and wherein each of distance_1 and distance_2 is selected from a value group including 0, 1, 2, 4, and 8.

12. The video encoding and decoding method according to claim 11, characterized in that: If the block width or the block height is smaller than a predetermined threshold, only smaller values ​​of the value group are used to form data pairs of the candidate set.

13. The video encoding and decoding method according to claim 11, characterized in that: If the first area is smaller than the second area, distance_1 is set to a value smaller than distance_2.

14. The video encoding and decoding method according to claim 11, characterized in that: If the first area is larger than the second area, distance_1 is set to a value larger than distance_2.

15. The video encoding and decoding method according to claim 1, characterized in that: If the first predictor and / or the second predictor corresponds to a prediction hypothesis, the prediction hypothesis includes one or more first predictions from a predetermined direction or a bidirectional direction, one or more second predictions generated from a motion candidate, one or more third predictions from an intra-frame candidate, one or more motion candidates, or any combination thereof.

16. An apparatus for video encoding and decoding, the apparatus comprising one or more electronic devices or processors, the devices being arranged to: Receiving pixel data associated with the current block at the encoding side, or receiving encoded and decoded data associated with the current block to be decoded at the decoding side; Dividing the current block into a first area and a second area along a partition line; determining a blending area around the partition line, wherein the blending area comprises a first blending area and a second blending area, the first blending area comprises a plurality of first pixels in the first area defined by the partition line and distance_1, the second blending area comprises a plurality of second pixels in the second area defined by the partition line and distance_2, and wherein distance_1 and distance_2 correspond to two positive values, and the two positive values ​​are allowed to be different; determining a first predictor and a second predictor for the first region and the second region respectively; generating a mixed predictor for the mixed area by mixing the first predictor and the second predictor; and The current block is encoded or decoded by using the prediction data including the hybrid predictor.

Images