Method and apparatus for multi-hypothesis prediction intra block copy for video coding

Abstract

A method and apparatus for hybrid IBC mode. According to the method, a first predictor or a first hypothesis is derived by using a first prediction mode including an Intra Block Copy (IBC) prediction mode or an Intra Template Matching Prediction (IntraTMP) mode. A second predictor is derived. A hybrid predictor is derived by mixing the first predictor or the first hypothesis with the second predictor. And encoding or decoding the current block by using a predictor candidate set including the hybrid predictor. According to another method, for IBC-CIIP (Joint Intra Block Copy and Intra Prediction), behavior between Inter Prediction mode, IBC AMVP mode, and IBC Merge mode remains consistent.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a non-provisional of and claims priority to U.S. Provisional Patent Application No. 63 / 495,590, filed April 12, 2023; U.S. Provisional Patent Application No. 63 / 496,036, filed April 14, 2023; U.S. Provisional Patent Application No. 63 / 497,241, filed April 20, 2023; U.S. Provisional Patent Application No. 63 / 460,979, filed April 21, 2023; and U.S. Provisional Patent Application No. 63 / 503,734, filed May 23, 2023. These U.S. Provisional Patent Applications are incorporated by reference in their entireties. TECHNICAL FIELD

[0003] The present disclosure relates to video coding systems. In particular, the present disclosure relates to Intra Block Copy (IBC) and / or Intra Template Matching Prediction (IntraTMP). BACKGROUND

[0004] Versatile video coding (VVC) is the latest international video coding standard developed by the Joint Video Experts Team (JVET) of the International Telecommunication Union Video Coding Experts Group (ITU-T VCEG) and the International Organization for Standardization / International Electrotechnical Commission Moving Picture Experts Group (ISO / IEC MPEG). The standard has been published as an ISO standard: ISO / IEC 23090-3:2021, Information technology — Coding of immersive media — Part 3: Versatile video coding, published in February 2021. VVC is developed based on its predecessor, High Efficiency Video Coding (HEVC), by adding more coding tools to improve coding efficiency while handling various types of video sources, including three-dimensional (3D) video signals.

[0005] FIG. 1AAn adaptive inter / intra video coding system with in-loop processing is exemplarily illustrated. For intra prediction 110, prediction data is derived based on previously coded video data in the current picture. For inter prediction 112, Motion Estimation (ME) is performed at the encoder side and based on the result of ME, Motion Compensation (MC) is performed to provide prediction data derived from other pictures and motion data. A switch 114 selects either intra prediction 110 or inter prediction 112, and the selected prediction data is provided to a summer 116 to form prediction error, also known as residual. The prediction error is then processed by a Transform (T) 118, followed by Quantization (Q) 120. The transformed and quantized residual is then encoded by an Entropy Encoder 122 and included in a video bitstream corresponding to the compressed video data. The bitstream associated with the transformed coefficients is then packed with other information such as mode and motion related to intra prediction 110 and inter prediction 112, and parameters related to in-loop filters applied to regions of the underlying picture, etc. The side information related to intra prediction 110, inter prediction 112 and in-loop filters 130 is provided to the Entropy Encoder 122 as shown. When inter prediction mode is used, the reference picture or pictures also have to be reconstructed at the encoder side. Therefore, the transformed and quantized residual is processed by Inverse Quantization (IQ) 124 and Inverse Transformation (IT) 126 to recover the residual. The residual is then added back to the prediction data 136 at a reconstruction (REC) 128 to reconstruct the video data. The reconstructed video data can be stored in a Reference Picture Buffer 134 and used for prediction of other frames. FIG. 1A

[0006] As FIG. 1A ​As shown, the input video data undergoes a series of processing in the encoding system. The reconstructed video data from REC 128 can be impaired due to the series of processing. Therefore, the in-loop filters 130 are usually applied to the reconstructed video data before it is stored to the reference picture buffer 134 to improve the video quality. For example, deblocking filter (DF), Sample Adaptive Offset (SAO) and Adaptive Loop Filter (ALF) can be used. The in-loop filter information can need to be included in the bitstream so that the decoder can properly recover the required information. Therefore, the in-loop filter information is also provided to the entropy encoder 122 for inclusion in the bitstream. In FIG. 1A The reconstructed samples are stored to the reference picture buffer 134 before the in-loop filter 130 is applied to the reconstructed video samples. FIG. 1A The system in

[0007] As FIG. 1B shown, the decoder can use the same or part of the same functional modules as the encoder, except for the transform 118 and quantization 120, because the decoder only needs inverse quantization 124 and inverse transform 126. The decoder uses an entropy decoder (Entropy Decoder, 140) instead of the entropy encoder 122 to decode the video bitstream into quantized transform coefficients and the required coding information (e.g. ILPF information, intra prediction information and inter prediction information). The intra prediction 150 at the decoder side does not need to perform mode search. Instead, the decoder only needs to generate the intra prediction according to the intra prediction information received from the entropy decoder 140. Furthermore, for inter prediction, the decoder only needs to perform motion compensation (MC 152) according to the inter prediction information received from the entropy decoder 140, without motion estimation.

[0008] Intra Template Matching Prediction

[0009] Intra Template Matching Prediction (Intra TMP) is a special intra prediction mode that copies the best prediction block from the reconstructed part of the current frame, whose L-shaped template matches the current template. For a predefined search range, the encoder searches for the most similar template to the current template in the reconstructed part of the current frame and uses the corresponding block as the prediction block. The encoder then signals the use of this mode and performs the same prediction operation at the decoder side.

[0010] The prediction signal is generated by matching the L-shaped causal neighborhood of the current block with another block in a predefined search region in the FIG. 2 middle, the search region including:

[0011] R1: current CTU

[0012] R2: top-left CTU

[0013] R3: top CTU

[0014] R4: left CTU

[0015] In FIG. 2 , the current block 210 in R1 is matched with the corresponding block 212 in R2. The templates of the current block and the matched block are shown as the darker colored L-shaped regions. The region 222 corresponds to the reconstructed region in the current picture 220. The sum of absolute differences (SAD) is used as the cost function. Within each region, the decoder searches for the template with the smallest SAD to the current template and uses its corresponding block as the prediction block.

[0016] The size of all regions (SearchRange_w, SearchRange_h) is set proportionally to the block size (BlkW, BlkH) to have a fixed number of SAD comparisons per pixel. That is:

[0017] SearchRange_w = a * BlkW,

[0018] SearchRange_h = a * BlkH,

[0019] where "a" is a constant that controls the gain / complexity trade-off. In practice, "a" is equal to 5.

[0020] To speed up the template matching process, the search range of all search regions is subsampled by a factor of 2. This results in a 4x reduction in the template matching search. After the best match is found, a refinement process is performed. The refinement is done by performing a second template matching search around the best match, with a reduced range. The reduced range is defined as min(BlkW, BlkH) / 2.

[0021] The intra template matching tool is enabled on coding units (CUs) with width and height less than or equal to 64. The maximum CU size for intra template matching is configurable.

[0022] When the current CU does not use decoder-side intra mode derivation (DIMD), the intra template matching prediction mode is signaled at the CU level by a dedicated flag.

[0023] Current image reference

[0024] Motion compensation is a key technique in hybrid video coding and decoding, exploring pixel correlations between adjacent images. It typically assumes that in a video sequence, the pattern corresponding to an object or background in a frame is shifted to form the corresponding object in a subsequent frame, or correlated with other patterns within the current frame. By estimating this shift (e.g., using block matching techniques), the pattern can be largely reproduced without re-encoding it. Similarly, block matching and duplication have been attempted to allow the selection of reference blocks from images identical to the current block. However, when this concept is applied to camera-captured video, inefficiency has been observed. This is partly because texture patterns in spatially adjacent regions may be similar to the current codec block, but often exhibit some spatial gradients. In camera-captured video, it is difficult to find a block in the same image that perfectly matches the current block. Therefore, improvements in codec performance are limited.

[0025] However, the spatial correlation between pixels within the same image differs for screen content. Typical videos, containing text and graphics, often exhibit repeating patterns within the same image. Therefore, intra-frame (image) block compensation has been observed to be very effective. To leverage this property, a novel prediction mode, Intra-Frame Block Copy (IBC) mode, or Current Image Reference (CPR), is introduced for screen content encoding and decoding. In CPR mode, prediction units (PUs) predict from previously reconstructed blocks within the same image. Furthermore, a displacement vector (called a block vector or BV) is used to indicate the relative displacement from the current block's position to the reference block's position. The prediction error is then encoded using transform, quantization, and entropy encoding / decoding. FIG. 3 An example of IBC compensation is shown, where block 312 is the corresponding block of block 3210, and block 322 is the corresponding block of block 320. In this technique, the reference sample corresponds to the reconstructed sample of the current decoded image before the loop filter operation, including deblocking filtering and Sample Adaptive Offset (SAO) filter in HEVC.

[0026] The first version of IBC was proposed in JCTVC-M0350 (Budagavi et al., AHG8: Video Coding with Intra-Frame Motion Compensation, ITU-T SG16 WP3 and ISO / IEC JTC 1 / SC 29 / WG11 Joint Collaborative Team on Video Coding (JCT-VC), 13th Meeting: Incheon, South Korea, April 18-26, 2013, Document: JCTVC-M0350), for the development of HEVC Range Extension (RExt). In this version, IBC compensation was limited to a small local area, with only 1-D block vectors, and only applicable to block sizes of 2Nx2N. Later, more advanced IBC designs were developed during the standardization process of HEVC Screen Content Coding (SCC).

[0027] Reconstructed-Reordered IBC (RR-IBC)

[0028] The Reconstruction-Reordering IBC (RR-IBC) mode is permitted for IBC-encoded blocks. When RR-IBC is applied, samples in the reconstructed block are flipped according to the current block's flip type. At the encoder end, the original block is flipped before motion search and residual computation, while the prediction block is derived without flipping. At the decoder end, the reconstructed block is flipped back to recover the original block.

[0029] For RR-IBC coded blocks, two flipping methods are supported: horizontal flipping and vertical flipping. First, a syntax flag is signaled to the IBC Advanced Motion Vector Prediction (AMVP) coded block to indicate whether the reconstruction is flipped. If flipped, another flag is further signaled to specify the flipping type. For IBC merging, the flipping type is inherited from neighboring blocks without syntax signaling. Considering horizontal or vertical symmetry, the current block and the reference block are typically horizontally or vertically aligned. Therefore, when a horizontal flip is applied, the vertical component of the BV is not signaled and inferred as 0. Similarly, when a vertical flip is applied, the horizontal component of the BV is not signaled and inferred as 0.

[0030] To better utilize the symmetry property, a flip-aware BV adjustment method is applied to refine the block vector candidates. For example, as FIG. 4A and FIG. 4B As shown, (x nbr ,y nbr ) and (x cur ,y cur ) represent the coordinates of the center samples of neighboring blocks and the current block, respectively, BV nbr and BV cur These represent the block value (BV) of the neighboring block and the current block, respectively. Unlike the BV directly inherited from the neighboring block, the BV... cur The horizontal component is transmitted to BV in the case of horizontal flip encoding of neighboring blocks. nbrhorizontal component of BV nbr h ) by adding a motion offset to the vertical component of BV cur h , denoted as BV nbr cur , (as shown in FIG. 4A ) i.e. BV nbr h . Similarly, the vertical component of BV cur is calculated by adding a motion offset to the vertical component of BV nbr , denoted as BV nbr v , (as shown in FIG. 4B ) i.e. BV cur v = 2(y nbr - y cur ) + BV nbr v .

[0031] IBC Merge mode with Block Vector Difference (IBC-MBVD)

[0032] Affine-MMVD and GPM-MMVD have been adopted as extensions of regular MMVD mode in ECM. It is natural to extend MMVD mode to IBC Merge mode.

[0033] In IBC-MBVD, the distance set is {1-pel, 2-pel, 4-pel, 8-pel, 12-pel, 16-pel, 24-pel, 32-pel, 40-pel, 48-pel, 56-pel, 64-pel, 72-pel, 80-pel, 88-pel, 96-pel, 104-pel, 112-pel, 120-pel, 128-pel} and the BVD directions are two horizontal and two vertical directions.

[0034] The base candidate is selected from the top five candidates in the reordered IBC Merge list. Based on the SAD cost of the template (one row above and one column left of the current block) and its reference, all possible MBVD refinement positions (20x4) are reordered for each refinement position of each base candidate. Finally, the top 8 refinement positions with the lowest template SAD cost are kept as available positions, followed by MBVD index coding. The MBVD index is binarized by Rice code with parameter equal to 1.

[0035] IBC-MBVD coded blocks do not inherit the flipping type from the RR-IBC coded neighboring blocks.

[0036] IBC-LIC​

[0037] In JVET-AC0112 (Yang Wang et al., “EE2-3.6: IBC-CIIP, IBC-GPM, and IBC-LIC,” Joint Video Team (JVT) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 29th Meeting, via teleconference, Jan. 11-20, 2023, Document: JVET-AC0112), IBC-LIC is adopted in ECM (Enhanced Compression Model). Intra block copy with local illumination compensation (IBC-LIC) is a coding tool that compensates local illumination changes between a CU coded using IBC and its prediction block by a linear equation. The parameters of the linear equation are derived the same as LIC for inter prediction, except that the reference template is generated using the block vector in IBC-LIC. IBC-LIC can be applied to IBC AMVP mode and IBC merge mode. IBC AMVP (IBC advanced motion vector prediction mode) corresponds to IBC mode with direct signaling of motion vector predictor index and signaling of motion vector difference if necessary. IBC AMVP is similar to the well-known inter AMVP. However, the motion vector refers to the block vector of IBC. The name IBC AMVP is used in this disclosure. For IBC AMVP mode, an IBC-LIC flag is signaled to indicate the usage of IBC-LIC. For IBC merge mode, the IBC-LIC flag is inferred from the merge candidate.

[0038] IBC-GPM

[0039] In JVET-AC0112, IBC-GPM is adopted in ECM. Intra block copy with geometric partition mode (IBC-GPM) is a coding tool that geometrically partitions a CU into two sub-partitions. The prediction signals of the two sub-partitions are generated using IBC and intra prediction. IBC-GPM can be applied to regular IBC merge mode or IBC TM merge mode. The intra prediction mode (IPM) candidate list is constructed for intra prediction using the same method as inter and intra prediction in GPM, and the IPM candidate list size is predefined as 3. There are 48 geometric partition modes in total, which are divided into two geometric partition mode sets as follows:

[0040] Table 1. Geometric partition modes in the first geometric partition mode set

[0041] ibc_gpm_partition_idx 0 1 2 3 4 5 6 7 angleIdx 0 0 8 8 16 16 24 24 distanceIdx 1 3 1 3 1 3 1 3

[0042] Table 2. Geometric partition modes in the second geometric partition mode set

[0043] ibc_gpm_partition_idx 0 1 2 3 4 5 6 7 8 9 angleIdx 2 2 2 3 3 3 4 4 4 5 distanceIdx 0 1 3 0 1 3 0 1 3 0 ibc_gpm_partition_idx 10 11 12 13 14 15 16 17 18 19 angleIdx 5 5 11 11 11 12 12 12 13 13 distanceIdx 1 3 0 1 3 0 1 3 0 1 ibc_gpm_partition_idx 20 21 22 23 24 25 26 27 28 29 angleIdx 13 14 14 14 18 18 19 19 20 20 distanceIdx 3 0 1 3 1 3 1 3 1 3 ibc_gpm_partition_idx 30 31 32 33 34 35 36 37 38 39 angleIdx 21 21 27 27 28 28 29 29 30 30 distanceIdx 1 3 1 3 1 3 1 3 1 3

[0044] When IBC-GPM is used, a signaling IBC-GPM geometry partition mode set flag is signaled to indicate the selection of the first or second geometry partition mode set, followed by a geometry partition mode index. A signaling IBC-GPM intra flag is signaled to indicate whether intra prediction is used for the first sub-partition. When intra prediction is used for the sub-partition, an intra prediction mode index is signaled. When IBC is used for the sub-partition, a merge index is signaled.

[0045] IBC-CIIP

[0046] In JVET-AC0112, IBC-CIIP is adopted in ECM. Intra block copy and intra prediction (IBC-CIIP) is a coding tool that is used to obtain two prediction signals using IBC merge mode and intra prediction for a CU and perform a weighted sum of the two predictions to generate the final prediction. Specifically, if the intra prediction is planar or DC mode, the final prediction is obtained as follows:

[0047] P = (w ibc * P ibc + ((1 « shift) - w ibc ) * P intra + (1 « (shift - 1))) » shift,

[0048] where P ibc and P intra denote the IBC prediction signal and the intra prediction signal, respectively. (w ibc , shift) is set to (13, 4) and (1, 1) for IBC merge mode and IBC AMVP mode.

[0049] Adaptive template matching merge candidate reordering (ARMC-TM)

[0050] Merge candidates are adaptively reordered by template matching (TM). The reordering method is applied to regular merge mode, TM merge mode, and affine merge mode (excluding subblock-based temporal motion vector prediction (SbTMVP) candidates). For TM merge mode, the merge candidates are reordered before the refinement process.

[0051] First, an initial merge candidate list is constructed according to the given merge order, e.g. spatial, TMVPs, non-adjacent, HMVPs, paired, virtual merge candidates. Then the candidates in the initial list are divided into multiple subgroups. For template matching (TM) merge mode, adaptive DMVR mode, each merge candidate in the initial list is first refined by using TM / multi-pass DMVR. The merge candidates in each subgroup are reordered according to the cost values based on template matching to generate a reordered merge candidate list, and the indices of the selected merge candidates in the reordered merge candidate list are signaled to the decoder. For simplicity, the merge candidates in the last subgroup are not reordered. All zero candidates from the ARMC reordering process are excluded when constructing the merge motion vector candidate list. The subgroup size for regular merge mode and TM merge mode is set to 5. The subgroup size for affine merge mode is set to 3.

[0052] Cost calculation

[0053] In the reordering process, the template matching cost of a merge candidate is measured by the SAD between the template samples of the current block and their corresponding reference samples. The template includes a set of neighboring reconstructed samples of the current block. The reference samples of the template are located by the motion information of the merge candidate. When the merge candidate uses bi-prediction, the reference samples of the template of the merge candidate are also generated by bi-prediction, as shown in FIG. 5 .

[0054] In FIG. 5 , block 512 corresponds to the current block in current picture 510, blocks 522 and 532 correspond to the reference blocks in reference pictures 520 and 530 in list 0 and list 1 respectively. Templates 514 and 516 are used for current block 512, templates 524 and 526 are used for reference block 522, and templates 534 and 536 are used for reference block 532. Motion vectors 540, 542 and 544 are the list 0 motion vectors of the merge candidate, and motion vectors 550, 552 and 554 are the list 1 motion vectors of the merge candidate.

[0055] Refinement of the initial merge candidate list

[0056] When multi-pass DMVR is used to derive the refined motion of the initial merge candidate list, only the first pass (i.e. PU level) of multi-pass DMVR is applied in the reordering. When template matching is used to derive the refined motion, the template size is set to 1. When the block width is more than 2 times the height or the height is more than 2 times the width, only the top or left template is used during the TM motion refinement. TM is extended to perform 1 / 16 pixel MVD precision. The first four merge candidates are reordered by the refined motion in TM merge mode.

[0057] Reordering criteria

[0058] During the reordering process, a candidate is considered redundant if the cost difference between the candidate and its predecessor is less than the lambda value, e.g. |D1-D2|<λ, where D1 and D2 are the costs obtained during the first ARMC ordering, and λ is the Lagrangian parameter used at the encoder side in the RD criterion.

[0059] The algorithm according to ARMC-TM is defined as follows:

[0060] - Determine the minimum cost difference between a candidate and its predecessor among all candidates in the list.

[0061] o If the minimum cost difference is greater than or equal to λ, the list is considered sufficiently diversified and the reordering stops.

[0062] o If the minimum cost difference is less than λ, the candidate is considered redundant and is moved to a further position in the list.

[0063] This further position is the first position where the candidate is sufficiently diversified compared to its predecessor.

[0064] - The algorithm stops after a finite number of iterations (if the minimum cost difference is not less than λ).

[0065] This algorithm applies to regular, TM, BM and affine merge modes. Similar algorithms apply to the merge MMVD and sign MVD prediction methods, which also use ARMC for reordering.

[0066] The value of λ is set to the value of λ used at the encoder side for the rate-distortion criterion to select the best merge candidate for low-delay configuration, and to another value of λ corresponding to another QP for random access configuration. A set of λ values corresponding to each signaled QP offset is provided in the sequence parameter set (SPS) or slice header for QP offsets not present in the SPS.

[0067] Extension to AMVP mode

[0068] The ARMC design also applies to AMVP mode, where AMVP candidates are reordered according to TM cost. For the template matching (TM-AMVP) mode of advanced motion vector prediction, an initial AMVP candidate list is first constructed, and then a refined AMVP candidate list is constructed by the refinement of TM. In addition, MVP candidates with TM cost greater than a threshold (equal to five times the cost of the first MVP candidate) are skipped.

[0069] Note that when wrap-around motion compensation is enabled, MV candidates should be clipped considering the wrap-around offset.

[0070] Multiple hypothesis prediction (MHP)

[0071] In multi-hypothesis inter prediction mode (JVET-M0425), one or more additional motion-compensated prediction signals are signalled in addition to the traditional bi-prediction signal. The final overall prediction signal is obtained by sample-wise weighted addition. The final prediction signal p3 is obtained as follows: bi and the first additional inter prediction signal / hypothesis h3, the final prediction signal p3 is obtained as follows:

[0072] p3 = (1 - a)p bi + a h3 (1)

[0073] The weighting factor a is specified by the new syntax element add_hyp_weight_idx according to the following mapping (Table 3):

[0074] Table 3. Mapping a to add_hyp_weight_idx

[0075] add_hyp_weight_idx α 0 1 / 4 1 -1 / 8

[0076] The number of additional predictions / hypotheses can exceed one. For example, the default number of additional hypotheses in the current ECM 4.0 (ECM is a video codec software library for developing the latest video codec algorithms) is two. As the ECM software evolves, the MHP algorithm is also evolving and becoming a superset of the algorithm in JVET-M0425. According to the MHP encoding process of ECM 4.0, a single reference list is first constructed from the reference lists L0 and L1. With this step, only one reference index needs to be transmitted in the bitstream for decoding. Then, the motion estimation best results from inter and affine searches are put into a vector that stores possible candidates with additional hypotheses added. In the next step, in addition to the results of motion search, motion search results inherited from different merge modes are also appended to the candidate vector. The motion information in these merge modes is inherited from the CU that performs the motion search. With the collected candidates, a low complexity cost search can be invoked to find the most promising prediction unit to code. The candidate with the lowest cost is then sent to a high complexity search. The cost here refers to the sum of absolute Hadamard transformed difference (SATD) plus syntax bits multiplied by lambda (lambda is a factor to convert bits to distortion domain, widely used in rate-distortion optimization in video coding). If MHP is the best coding mode for this CU after competing with other coding modes, then MHP will be the final coding mode selected for this CU. There are two types of searches, “merge search” and “AMVP (advanced motion vector prediction) search”, for finding additional hypotheses. In “merge search”, possible candidates are constructed by GPM (geometric partition mode, default number is 10). Each candidate is checked, and if the cost is reduced after mixing the new candidate, the candidate is appended to the output vector. In “AMVP search”, only two candidates are constructed as predictors for the motion vector. Motion estimation is invoked to search for the best motion vector that makes the relevant prediction well match the residual. The residual is calculated as the original pixel minus the latest prediction of MHP (if there are already any additional hypotheses, these hypotheses should be mixed as prediction to form the latest prediction).

[0077] Merge mode

[0078] To improve the coding efficiency of motion vector (MV) coding in HEVC, HEVC has the skip and merge modes. The skip and merge modes derive motion information from spatial neighboring blocks (spatial candidates) or temporally co-located blocks (temporal candidates). When a PU is in the skip or merge mode, the motion information is not coded, but only the index of the selected candidate is coded. For the skip mode, the residual signal is forced to zero and not coded. In HEVC, if a particular block is coded as skip or merge, the candidate index is signaled to indicate the candidate used for merge in the candidate set. Each merged PU reuses the MV, prediction direction, and reference picture index of the selected candidate.

[0079] For the merge mode of HM-4.0 in HEVC, as shown in FIG. 6 four spatial MV candidates are derived from A0, Al, B0, and Bl, and one temporal MV candidate is derived from T BR or T CTR (uses T BR first). If T BR is not available, T CTR is used instead. Note that if any of the four spatial MV candidates is not available, another MV candidate is derived using the position B2 as a substitute. After the derivation of the four spatial MV candidates and one temporal MV candidate, de -redundancy (pruning) is applied to remove redundant MV candidates. If the number of available MV candidates after de-redundancy (pruning) is less than five, three types of additional candidates are derived and added to the candidate set (candidate list). The encoder selects one final candidate from the candidate set for the skip or merge mode according to rate-distortion optimization (RDO) decision and transmits the index to the decoder.

[0080] Hereafter, we refer to the skip and merge modes as the "merge mode", i.e., in the following paragraphs, when we say "merge mode", we mean the skip and merge modes.

[0081] Filtered Intra Block Copy (FIBC)

[0082] In JVET-AD0217 (Hong-Jheng Jhu et al., "Non-EE2: Filtered Intra Block Copy (FIBC)," Joint Video Team (JVT) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 30th Meeting, Antalya, TR, 21-28 April 2023, Doc: JVET-AD0217), a Filtered Intra Block Copy (FIBC) mode is disclosed, in which the prediction samples of IBC are enhanced by applying a regression model filter. The proposed filter consists of five spatial terms and one bias term. The filter coefficients are derived by minimizing the difference between the template samples and their corresponding reference samples based on a regression model.

[0083] Intra template matching prediction with multiple modes

[0084] In JVET-AD0194 (Po-Han Lin et al., "EE2-1.19: IntraTMP with multiple modes," Joint Video Team (JVT) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 30th Meeting, Antalya, TR, 21-28 April 2023, Doc: JVET-AD0194), Intra Template Matching Prediction with multiple modes (IntraTMP) is proposed. When IntraTMP is enabled, multiple template types (L-shaped, left-side, and above) are used to derive template matching candidates, and these candidates can be further fused by regression model combination.

[0085] Cross-component residual model (CCRM) for inter prediction

[0086] In JVET-AD0108 (Pekka Astola et al., "AHG12: Cross-Component Residual Model (CCRM) for Inter- Prediction", ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 Joint Video Expert Team (JVET), 30th Meeting, Antalya, TR, 21st April - 28th April 2023, Doc: JVET-AD0108), a contribution proposes to apply chroma prediction based on the Cross-Component Residual Model (CCRM) to improve the compression efficiency of inter slices in ECM. The proposed method uses an 8-tap convolutional filter to map the reconstructed luma to improved chroma prediction when the CCRM prediction mode is activated by a TU-level flag. The input of the filter includes 6 spatial luma samples, one non-regressive model term and one bias term. The filter coefficients are derived in step 720 for each block separately using the prediction signals (i.e., predY 710, predCb 712 and predCr 714) and applied in step 730 to the reconstructed luma signal. The reconstructed luma signal is formed by using adder 722 combining the luma prediction (PredY) 710 and the residual luma signal (resY). After applying the filter, step 730 generates the filtered predicted Cb 740 and the filtered predicted Cr 750. The reconstructed Cb signal is formed by using adder 742 combining the filtered predicted Cb 740 and the residual Cb signal (i.e., resCb). Likewise, the reconstructed Cr signal is formed by using adder 752 combining the filtered predicted Cr 750 and the residual Cr signal (i.e., resCr).

[0087] IBC architecture in VVC

[0088] In the HEVC SCC extension, the effective reference region of Intra Block Copy (IBC) is almost the entire reconstructed region of the current picture, with some exceptions for parallel processing purposes. One drawback of the HEVC concept is the need for additional memory in the decoded picture buffer (DPB), which is usually implemented using external memory in hardware implementations.

[0089] In VVC, only the CTUs to the left of the current coding tree unit (CTU) are used as reference sample region at the beginning of the reconstruction process of the current CTU. VVC uses a fixed memory, which enables on-chip storage of IBC, significantly reducing the complexity of implementing IBC in hardware architectures. FIG. 8The IBC concept in HEVC and VVC is illustrated, where each square shape corresponds to a coding tree unit (CTU). The gray-shaded area 840 represents the already coded area, while the white area represents the area to be coded. IBC in HEVC allows the use of the gray-shaded area, except for the two CTUs 850 on the top-right of the current CTU to allow wavefront parallel processing (WPP). For example, the current block 810 can point to the reference block 820 based on the BV 822 in HEVC. On the other hand, IBC in VVC only allows the CTUs to the left of the current CTU as the reference area, represented by the dashed box 860. For example, the current block 810 can point to the reference block 830 based on the BV 832 in VVC.

[0090] Syntax and semantics

[0091] The IBC architecture in VVC forms a dedicated coding mode, where the IBC mode is the third prediction mode besides the intra and inter prediction modes. When the block size is 64x64 or smaller, the bitstream carries an IBC syntax element that indicates the IBC mode of the coding unit. Therefore, the maximum CU size that can utilize IBC is 64x64. However, the reference sample addressing mechanism is still the same as in the HEVC SCC extension by representing a two-dimensional offset and reusing the vector coding process of inter prediction. Another special case occurs when the chroma independent tree (CST) is activated, where the compiler cannot derive the chroma BVs from the luma BVs, resulting in the use of IBC only for the luma coding block.

[0092] Reference area and sample memory

[0093] The IBC design in VVC uses a fixed size of 128x128 memory for each color component to store the reference samples. Note that the maximum CTU size in VVC is also 128x128, i.e., when the maximum CTU size configuration is equal to 128x128, the reference sample memory (RSM) can accommodate the samples of a single CTU. Note that when the maximum CTU size is smaller than 128x128, the RSM can contain the samples of multiple left-neighbor CTUs, resulting in the use of multiple left-neighbor CTUs. For example, when the maximum CTU size is equal to 32x32, the RSM can accommodate the samples of 15 left-neighbor CTUs.

[0094] JVET-AC0193 Non-EE2: Copy padding for IBC

[0095] In the IBC mode, a block vector (BV) is used to indicate the displacement from the current block to the reference block, which must have been fully reconstructed within the current picture. The reference block cannot overlap with the current block.

[0096] In JVET-AC0193 (Na Zhang et al., “Non-EE2: Replication padding for IBC”, Joint Video Team (JVT) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 29th Meeting, via teleconference, 11-20 January 2023, Doc: JVET-AC0193), replication padding can be applied in the overlapping region. With replication padding, the unreconstructed samples in the overlapping region can be padded by replicating their predicted samples, as shown in FIG. 9A and FIG. 9B in equation form,

[0097] P'(x, y) = P(x + BVx, y + BVy).

[0098] where P'(x, y) is the padded sample at position (x, y), P(x + BVx, y + BVy) is the predicted sample, and (BVx, BVy) is the block vector (BV) of the current block. FIG. 9A An example is shown, in which the current block 920 overlaps with the reference block 910. The overlapping region is shown as the dot-filled region. In the conventional method, the samples in the overlapping region have not been reconstructed and are considered unavailable. According to JVET-AC0193, the samples in the overlapping region can be padded by replicating the predicted samples to the corresponding positions, as shown in FIG. 9B where the samples a, b, c, e, f, and g in the overlapping region of the reference block are padded by replicating from the corresponding predicted samples of the current block. Replication padding is only performed when the horizontal block vector component is less than or equal to 0 and the vertical block vector component is less than or equal to 0. In the present invention, methods and apparatuses for improving the coding performance of the intra block copy (IBC) mode are disclosed. SUMMARY

[0099] A video coding method and apparatus using IBC mode or IntraTMP mode are disclosed. According to the method, input data associated with a current block is received, wherein the input data comprises pixel data to be encoded at an encoder side or data associated with the current block to be decoded at a decoder side. A first predictor or a first hypothesis is derived by using a first prediction mode comprising an intra block copy (IBC) prediction mode or an intra template matching prediction (IntraTMP) mode, wherein the IBC prediction mode derives the first predictor or the first hypothesis by locating a target reference block in a same picture as the current picture according to a first block vector, or the IntraTMP mode derives the first predictor or the first hypothesis according to a template matching cost between a reference template of the target reference block and a current template of the current block. A second predictor is derived. A mixed predictor is derived by mixing the first predictor or the first hypothesis with the second predictor. The current block is encoded or decoded using a predictor candidate set comprising the mixed predictor.

[0100] In one embodiment, the first predictor or the first hypothesis is generated by an IBC advanced motion vector prediction (AMVP) mode or an IBC merge mode. In another embodiment, the first predictor or the first hypothesis is generated by weighting a first IBC predictor in the IBC AMVP mode and a second IBC predictor in the IBC merge mode.

[0101] In one embodiment, a plurality of first predictors or a plurality of first hypotheses is generated. In one embodiment, a redundancy check is performed when generating the plurality of first predictors or the plurality of first hypotheses.

[0102] In one embodiment, the first prediction mode comprises an IBC merge mode, an IBC AMVP mode, an IBC generalized prediction merge (GPM) mode, an IBC combined inter and intra prediction (IBC-CIIP) mode, a reconstructed reorder intra block copy (RRIBC) mode, an IBC template matching inter (IBC template matching inter) mode, an IBC template matching merge (IBC template matching merge) mode, an IBC regular merge (IBC regular merge) mode, an IBC motion vector difference merge (IBC motion vector difference merge) mode, an IBC local illumination compensation (IBC-LIC) mode, an IBC subblock (IBC subblock) mode, or a combination thereof. In one embodiment, the second predictor is derived by using a second prediction mode, and the second prediction mode comprises the IBC merge mode, the IBC AMVP mode, the IBC GPM mode, the IBC-CIIP mode, the RRIBC mode, the IBC template matching inter mode, the IBC template matching merge mode, the IBC regular merge mode, the IBC motion vector difference merge mode, the IBC-LIC mode, the IBC subblock mode, or a combination thereof.

[0103] In one embodiment, the first prediction mode corresponds to IBC with bi-prediction. In one embodiment, a first block vector (BV) is derived by applying an IBC search algorithm, and the first BV is used to locate a first reference block to derive a search mode for a second BV. In one embodiment, a block vector difference (BVD) and a block vector predictor (BVP) associated with the first BV of the bi-prediction are signaled or parsed from a bitstream including coded data of the current block.

[0104] In one embodiment, a first BVD associated with the first BV of the bi-prediction is signaled or parsed from a bitstream including coded data of the current block, and a second BVD associated with the second BV of the bi-prediction is inferred to be 0.

[0105] In one embodiment, a merge index is signaled or parsed from a bitstream including coded data of the current block to indicate a target merge candidate in an IBC merge list for the bi-prediction.

[0106] In one embodiment, a flag is signaled or parsed from a bitstream including coded data of the current block to indicate whether to use an IBC AMVP mode for the bi-prediction.

[0107] In one embodiment, if the current block is coded in the IBC AMVP mode, all candidates in an AMVP candidate list of the current block are bi-predictive motion.

[0108] In one embodiment, if the current block is coded in the IBC AMVP mode, a dominant BV of the current block is used to derive a propagated intra mode.

[0109] In one embodiment, when a current slice type of the current block is an I slice, the hybrid predictor is generated by an IBC AMVP predictor and an IBC merge predictor. In one embodiment, when the current slice type of the current block is a non-I slice, the hybrid predictor is generated by mixing an IBC AMVP predictor and an inter merge predictor, or by mixing an inter AMVP predictor and an IBC merge predictor.

[0110] According to another method, the first predictor is derived by using an intra block copy (IBC) prediction mode or an inter prediction mode, wherein the IBC prediction mode corresponds to an IBC advanced motion vector prediction (AMVP) mode or an IBC merge mode, and the first predictor is derived by locating a target reference block in the same picture as the current picture according to a first block vector selected from an IBC merge list or by a search mode of the IBC AMVP mode when the IBC prediction mode is used. The second predictor is derived according to an intra prediction mode. The combined predictor is derived by combining the first predictor and the second predictor. The current block is encoded or decoded using a predictor candidate set including the combined predictor, wherein one or more processes and / or one or more conditions associated with deriving the second predictor, deriving the combined predictor, or both are consistent with the inter prediction mode, the IBC AMVP mode, and the IBC merge mode.

[0111] In one embodiment, the one or more processes and / or one or more conditions include intra list construction.

[0112] In one embodiment, the one or more processes and / or one or more conditions include combined weight design.

[0113] In one embodiment, the one or more processes and / or one or more conditions are related to position dependent prediction combination (PDPC).

[0114] In one embodiment, the one or more processes and / or one or more conditions include deblocking filtering.

[0115] In one embodiment, the one or more processes and / or one or more conditions include one or more constraints related to current block size.

[0116] In one embodiment, the one or more processes and / or one or more conditions include motion reference of other blocks. BRIEF DESCRIPTION OF DRAWINGS

[0117] FIG. 1A An adaptive inter / intra video coding system is exemplarily shown, which includes a loop process.

[0118] FIG. 1B A corresponding decoder of the encoder is shown. FIG. 1A

[0119] FIG. 2 A search region example for intra template matching prediction is shown.

[0120] FIG. 3 ​An example of current picture reference (CPR) compensation is shown, where a block is predicted by a corresponding block in the same picture.

[0121] FIG. 4A An example of horizontal flipped block vector (BV) adjustment is shown.

[0122] FIG. 4B An example of vertical flipped block vector (BV) adjustment is shown.

[0123] FIG. 5 An example of a template for a current block and a corresponding reference block is shown to measure matching cost related to a merge candidate.

[0124] FIG. 6 A neighboring block for deriving a HEVC spatial merge candidate is shown.

[0125] FIG. 7 A cross-component residual model (CCRM) system block diagram for inter prediction is shown.

[0126] FIG. 8 An intra block copy (IBC) concept in HEVC and VVC is shown, where each square shape corresponds to a coding tree unit (CTU).

[0127] FIG. 9A An example is shown where a current block overlaps with a reference block, and samples in the overlapping region have not been reconstructed yet, which are considered as unavailable in traditional methods.

[0128] FIG. 9B An example according to JVET-AC0193 is shown, where samples in the overlapping region can be padded by copying the predicted samples at the corresponding positions.

[0129] FIG. 10 A flowchart of an example video coding system using hybrid IBC prediction mode according to an embodiment of the present invention is shown.

[0130] FIG. 11 A flowchart of an example video coding system using IBC-CIIP prediction mode aligned between IBC AMVP mode, IBC merge mode and inter mode according to an embodiment of the present invention is shown.

DETAILED DESCRIPTION

[0131] The components of the application, as generally described and illustrated in the Figures herein, can be arranged and designed in a wide variety of different configurations. Therefore, the following more detailed description of the embodiments of the systems and methods of the application, as represented in the Figures, is not intended to limit the scope of the application, as claimed, but is merely representative of selected embodiments of the application. The described features, structures, or characteristics can be combined in one or more embodiments. However, the described embodiments are not meant to limit the application, as claimed, in any way. The phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting. For example, the use of "including," "comprising," "having," "containing," "involving," "characterized by," "characterized into," and variations thereof herein, are intended to be broad and encompass the selection of

[0132] Furthermore, the described features, structures, or characteristics can be combined in one or more embodiments. However, the described embodiments are not meant to limit the application, as claimed, in any way. The phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting. For example, the use of "including," "comprising," "having," "containing," "involving," "characterized by," "characterized into," and variations thereof herein, are intended to be broad and encompass the selection of

[0133] To improve the performance related to IBC mode, various methods are disclosed.

[0134] Increasing the variation of IBC mixed mode (IBC-CIIP)

[0135] In one embodiment, the combined IBC mode can generate the final predictor by mixing two IBC merge predictors. For example, one predictor can be generated from the BV of the left neighbor. Another predictor can be generated from the history BV. After that, the two predictors are mixed to generate the IBC-CIIP final predictor. In another embodiment, the combined IBC mode can generate the final predictor by mixing one IBC merge predictor and one Intra TMP predictor. In one embodiment, the combined IBC mode can generate the final predictor by mixing one IBC AMVP predictor and one Intra TMP predictor.

[0136] In another embodiment, the mixing weight is designed according to the type of two predictors. For example, the first predictor is derived from the BV of the neighboring block, and the second predictor is derived based on the non-neighbor candidate. In the mixing process, the first predictor will apply a larger weight than the second predictor. In another embodiment, the mixing weight is designed according to the template matching cost. For example, the mixing weight can be larger for the predictor with lower template matching cost.

[0137] In another embodiment, the blending weights are designed according to the BVs and / or prediction modes of neighboring coding units (CUs). For example, if the top and left CUs are coded by IBC mode, the blending weight of the IBC predictor can be larger. On the other hand, if the top and left CUs are coded by intra mode, the blending weight of the intra predictor can be larger.

[0138] In another embodiment, the blending weights are designed according to the DIMD gradient magnitudes in the neighboring regions (e.g., 3 template lines). For example, if the DIMD gradient magnitude of the selected intra prediction mode is the largest, the blending weight of the intra predictor can be larger.

[0139] In another embodiment, the blending weights (denoted as ω0and ω1) are derived by a regression model that minimizes the mean square error (MSE) between the blended template samples and the current template samples.

[0140] In one example, the blended template samples are composed of K0(e.g., 2) spatial gradients, K1(e.g., 2) position terms, one non-linear term, and one bias term. That is,

[0141]

[0142] where,

[0143] Gx= (2W + NW + SW) - (2E + NE + SE),

[0144] Gy= (2N + NW + NE) - (2S + SW + SE),

[0145] P = (P 2 + (1 « (bitDepth - 1))) » bitDepth,

[0146] where X and Y are the vertical and horizontal relative positions, and B is an intermediate value set to (1 « (bitDepth - 1)).

[0147] In another example, the blended template samples are composed of T (e.g., 5) spatial samples, one non-linear term, and one bias term. That is,

[0148]

[0149] P = (P 2 + (1 « (bitDepth - 1))) » bitDepth,

[0150] and B is an intermediate value set to (1 « (bitDepth - 1)).

[0151] Increasing the variation of IBC-GPM mode

[0152] In another embodiment, the final predictor of IBC-GPM mode can be generated by combining two different IBC merge predictors. For example, one predictor can be generated from the block vector (BV) of the left neighbor. Another predictor can be generated from the history BV. Then, the two predictors are combined to generate the IBC-GPM final predictor. In another embodiment, the final predictor of IBC-GPM mode can be generated by combining one IBC AMVP predictor and one intra predictor. In another embodiment, the final predictor of IBC-GPM mode can be generated by combining one IBC AMVP predictor and one IntraTMP predictor.

[0153] In another embodiment, TM-based reordering can be performed to reorder the IBC-GPM partition groups. In this case, the index of more promising candidate can be further reduced to improve the coding efficiency.

[0154] In one embodiment, the final predictor of IBC-GPM mode can be generated by combining one IBC AMVP / merge predictor and one inter predictor. The inter predictor can be regular merge, MMVD (i.e., merge with motion vector difference), TM merge, BM merge, CIIP (i.e., combined inter merge / intra prediction), affine merge / AMVP, SbtMVP, or other inter modes.

[0155] In one embodiment, the mixing width of IBC-GPM can be determined by RD cost and signaled explicitly by bits. Alternatively, the mixing width of IBC-GPM can be determined by TM / BM cost and derived implicitly at the decoder side. The number of mixing width candidates can depend on the CU size, the prediction mode of the two GPM parts, or the mixing width of neighboring GPM CUs. For example, if both GPM parts are IBC or IntraTMP mode, the mixing width should be smaller.

[0156] In one embodiment, the mixing width of the two IBC-GPM parts can be different and determined separately. For example, if one IBC-GPM CU is composed of one IBC predictor and one inter predictor, the mixing width of the IBC / IntraTMP part can be smaller, while the mixing width of the inter part can be larger, and selected from N mixing width candidates.

[0157] In one embodiment, a joint IBC and regular merge candidate list is constructed, which includes original regular merge candidates and IBC candidates. The IBC-GPM predictor is generated by two merge predictors in the joint IBC and regular merge list, or by one merge predictor in the joint list and another predictor in the intra prediction (IPM) candidate list or TM merge candidate list.

[0158] In one embodiment, the IBC-GPM mode can be considered as an inter mode and share the same signaling syntax with inter GPM modes. The sub-block motion vectors of an IBC-GPM CU will be determined according to the non-IBC part. For example, if an IBC-GPM CU is composed of one IBC predictor and one inter predictor, the sub-block motion vectors of the whole CU are set to be the same as the inter predictor.

[0159] Improved IBC-LIC mode

[0160] In one embodiment, the IBC-LIC mode can be enabled together with the RIBC mode. In this case, the template of IBC-LIC should be derived according to the type of RIBC. For example, if horizontal flipping is enabled, the template of IBC-LIC should be the samples right to the reference block and the samples on top of the reference block. For example, if vertical flipping is enabled, the template of IBC-LIC should be the samples left to the reference block and the samples on bottom of the reference block.

[0161] IBC with MHP

[0162] In one embodiment, the IBC predictor can be a prediction / hypothesis mixed with other predictions. The IBC predictor can be generated by IBC AMVP mode or IBC merge mode.

[0163] In one embodiment, the sub-block based IBC mode is restricted from being an additional prediction / hypothesis mixed with other predictions.

[0164] In one embodiment, multiple BVs or motion offsets are considered in the IBC prediction mode or IntraTMP prediction mode. In one embodiment, multiple predictors are considered in the IBC prediction mode or IntraTMP prediction mode. In one embodiment, there can be redundancy in the multiple predictors or multiple BVs. For example, some predictors or BVs can be the same. In one embodiment, a redundancy check will be performed during the generation of the multiple predictors or multiple BVs.

[0165] In one embodiment, the predictor in IBC prediction mode or IntraTMP prediction mode is combined with other predictors with weights, which can be the predictor in IBC prediction mode or IntraTMP prediction mode or the predictor in other inter prediction modes. In one embodiment, the BV in IBC prediction mode or IntraTMP prediction mode is multiplied with other BVs with weights, which can be the BV in IBC prediction mode or IntraTMP prediction mode or the MV in other inter prediction modes.

[0166] In one embodiment, the predictor in IBC merge mode is multiplied with the predictor in IBC AMVP mode with weights. In one embodiment, the predictor in IBC merge mode is multiplied with the predictor in IBC merge mode with weights. In one embodiment, the predictor in IBC AMVP mode is multiplied with the predictor in IBC merge mode with weights. In one embodiment, the predictor in IBC AMVP mode is multiplied with the predictor in IBC AMVP mode with weights.

[0167] In one embodiment, the predictor is one of IBC modes, including but not limited to IBC merge mode, IBC AMVP mode, IBC GPM mode, IBC-CIIP mode, RRIBC mode, IBC template matching inter mode, IBC template matching merge mode, IBC regular merge mode, IBC motion vector difference merge mode, IBC-LIC mode, IBC subblock mode, and the predictor is multiplied with other predictors with weights, which can be the predictor in other IBC modes, including but not limited to IBC merge mode, IBC AMVP mode, IBC GPM mode, IBC-CIIP mode, RRIBC mode, IBC template matching inter mode, IBC template matching merge mode, IBC regular merge mode, IBC motion vector difference merge mode, IBC-LIC mode, IBC subblock mode or a combination thereof.

[0168] In one embodiment, the predictor is one of IBC modes, including but not limited to IBC merge mode, IBC AMVP mode, IBC GPM mode, IBC-CIIP mode, RRIBC mode, IBC template matching inter mode, IBC template matching merge mode, IBC regular merge mode, IBC motion vector difference merge mode, IBC-LIC mode, IBC subblock mode, and the predictor is multiplied with other predictors with weights, which can be the predictor in other non-IBC modes, such as GPM mode, MMVD mode, MHP mode, affine mode, subblock mode or translation inter prediction mode. In one embodiment, when the predictor is in the reconstruction-reordering mode, the predictor can only be multiplied with other reconstruction-reordering predictors.

[0169] In one embodiment, when the predictor is in the reconstruction-reordering mode, the predictor cannot be multiplied with other predictors.

[0170] In one embodiment, the predictor in IBC mode or IntraTMP mode can only be blended with the predictor in IBC mode or IntraTMP mode.

[0171] In one embodiment, the predictor can only be blended with the predictor in the same prediction mode. For example, inter-predictor is blended with inter-predictor, IBC predictor or IntraTMP predictor is blended with IBC predictor or IntraTMP predictor.

[0172] In one embodiment, when there are more predictors, the blending weight will decrease. For example, the first predictor will use the largest weight, the second predictor will use the second largest weight, and so on.

[0173] In one embodiment, when there are more predictors, the blending weight will increase. For example, the first predictor will use the smallest weight, the second predictor will use the second smallest weight, and so on.

[0174] In one embodiment, the blending weight can be some fixed value.

[0175] In one embodiment, for those predictors to be blended, the predictors will first be reordered according to some metrics, such as template matching cost, boundary matching cost, SSE, SATD, SAD, and then the reordered predictors are blended with decreasing or increasing weights.

[0176] IBC and NAMVP

[0177] In one embodiment, non-adjacent spatial candidates can be used in IBC merging. They can only be referenced if the block to be referenced is within the valid reference region. In one embodiment, the position design of non-adjacent candidates for IBC is related to the boundary position of the valid reference region for IBC.

[0178] In one embodiment, a similarity check can be applied to NAMVP list derivation to increase the diversity of NAMVP. The similarity check threshold on the NAMVP list can be different from the IBC merging list. For example, a strict threshold will be used during NAMVP list generation (e.g., only when the candidate to be inserted is N pixels away from any other candidate in the list, it can be inserted into the list. N is an integer greater than zero). After that, the candidates in the NAMVP list will be inserted into the IBC merging candidate list with a relaxed threshold.

[0179] IBC with diversity reordering

[0180] In the reordering process, a candidate is considered redundant if the cost difference between the candidate and its predecessor is smaller than a lambda value, e.g., |D1-D2|<λ, where D1 and D2 are the costs obtained during the first ARMC ordering, and λ is the Lagrangian parameter in the RD criterion used at the encoder side.

[0181] In one embodiment, if the reordering process is applied to the IBC merge candidate list, the lambda value for the IBC merge candidate list can be different from the lambda value for the inter merge candidate list. For example, the lambda value for the IBC merge list is N times the lambda value for the inter merge candidate list. As another example, the lambda value for the IBC merge list is the lambda value for the inter merge candidate list plus an offset. As yet another example, the lambda value for the IBC merge list can be derived from the lambda value for the inter merge candidate list by a linear function. In another embodiment, the lambda value for the IBC merge list can be designed according to the current CU size, bit depth, or picture resolution.

[0182] Bi-prediction IBC AMVP mode

[0183] In the current IBC design, an IBC coded block only needs to signal one block vector (BV) to refer to a previously coded block in the current picture. To further improve the IBC mode, a bi-prediction IBC AMVP mode is proposed.

[0184] In one embodiment, the first BV is derived by applying the search algorithm of IBC, i.e., IBC hash search or IBC motion estimation. After that, the reference block of the first derived BV will be used to derive the search pattern of the second BV. For example, the search pattern of the second BV can be the original block minus the reference block of the first derived BV.

[0185] In another embodiment, the BVDs of the first BV and the second BV are both signaled in the bitstream for the first BV and the second BV, respectively.

[0186] In another embodiment, the block vector predictors (BVPs) of the first block vector (BV) and the second block vector (BV) can be different, and two BVP indices are signaled in the bitstream for the first BV and the second BV, respectively. In another embodiment, the BVPs of the first BV and the second BV are the same. In this case, only one BVP index is signaled in the bitstream, and it will be shared by the first BV and the second BV.

[0187] In another embodiment, a block vector difference (BVD) is signaled in the bitstream to indicate the first BV. In addition, a second BVD is signaled in the bitstream to indicate the difference between the first BVD and the second BVD. The final second BVD can be derived by adding the signaled BVD difference to the first BVD. In another embodiment, a BVD is signaled in the bitstream to indicate the first BV. In addition, an index is signaled in the bitstream to indicate the BVD difference between the first BVD and the second BVD. The final BVD difference between the first BVD and the second BVD can be derived by a pre-defined look-up table (LUT) and the signaled index. In another embodiment, a BVD is signaled in the bitstream to indicate the first BV. In addition, an index is signaled in the bitstream to indicate the BVD difference (BVDd) between the first BVD and the second BVD. The signaled index can be used to derive the direction and distance of the BVDd. In another embodiment, a BVD is signaled in the bitstream to indicate the first BV, and the second BVD will be inferred as zero.

[0188] In another embodiment, after the first BV is derived, a search pattern of the second BV can be derived. The second BV is searched based on the derived search pattern, and the search region is around the first BV. For example, the distance between the first BV and the second BV is limited to N integer luma samples. N can be any integer greater than zero.

[0189] In another embodiment, the search region of the second BV around the first BV is designed based on a coding unit (CU) size, a CU width, a CU height, or a current picture resolution. In another embodiment, the search region design of the second BV around the first BV corresponds to the motion resolution of the first BV. In another embodiment, the search region of the second BV can be around the best BVP candidate. The search region can be designed based on the CU size, the CU width, the CU height, the motion resolution of the first BV, or the current picture resolution.

[0190] In one embodiment, the first BV is referenced by an IBC merge candidate. After that, the reference block of the first derived BV will be used to derive the search pattern of the second BV. For example, the search pattern of the second BV can be the original block doubled minus the reference block of the first derived BV. For another example, a spatial candidate is referenced as the first BV. For another example, an HMVP candidate is referenced as the first BV.

[0191] In another embodiment, a merge candidate list is generated, and then a TM reordering technique is applied to reorder the merge candidate list. By doing so, the merge index of the promising candidate can be shorter.

[0192] In another embodiment, a merge index is signaled in the bitstream to indicate which merge candidate is used.

[0193] In one embodiment, after the first predictor and the second predictor of IBC are derived, the predictors are further refined using a motion refinement technique. For example, DMVR technique, BDOF technique can be applied.

[0194] In one embodiment, the mixing weights of the first predictor and the second predictor can be different. For example, the weight pair of the first predictor and the second predictor can be derived based on TM cost. For another example, the weight pair of the first predictor and the second predictor can be derived based on BV cost. The reference blocks far away from the current block will apply lighter weight, and the reference blocks close to the current block will apply higher weight. For another example, the weight pair of the first predictor and the second predictor can be derived based on reference block type. For example, the reference blocks derived from merge candidate will apply lower weight, while the blocks derived from IBC search process will apply higher weight. For another example, the weight pair can be signaled in the bitstream. For example, an index is signaled to indicate the value of the weight pair.

[0195] In one embodiment, a flag is signaled in the bitstream to indicate the switch of bi-predictive IBC AMVP mode. Only when the bi-predictive IBC AMVP indication flag is true, the syntax related to bi-predictive IBC AMVP mode is signaled. For example, the syntax corresponds to merge_idx, or BVP and BVD of the second BV.

[0196] In one embodiment, a high-level syntax flag is signaled in the bitstream to indicate the switch of bi-predictive IBC AMVP mode. For example, the syntax corresponds to sps_bi_ibc_amvp_enabled_flag, pps_bi_ibc_amvp_enabled_flag, or ph_bi_ibc_amvp_enabled_flag. The switch of bi-predictive IBC AMVP mode is consistent with the IBC fractional pixel enabling flag. They can be enabled under the same condition. For example, if the hash hit rate of the current picture is low enough, the bi-predictive IBC AMVP mode and the IBC fractional pixel mode will be enabled.

[0197] In another embodiment, only when the high-level syntax flag of bi-predictive IBC AMVP mode is true, the low-level (i.e., CU-level) syntax of bi-predictive IBC AMVP mode can be signaled.

[0198] In one embodiment, the inter direction index is used to indicate the switch of the bi-predictive IBC AMVP mode. For example, if the current block is enabled with the bi-predictive IBC AMVP mode, the inter direction index needs to be signaled and the inter direction index should be set to 2. In one embodiment, the inter direction index is used to indicate the switch of the bi-predictive IBC AMVP mode and the context table of the inter direction index for the IBC and non-IBC modes should be different.

[0199] In one embodiment, the maximum number of prediction bins for the bi-predictive IBC AMVP mode can be different from the conventional uni-predictive IBC AMVP mode. For example, the maximum number of prediction bins for the bi-predictive IBC AMVP mode is consistent with the non-IBC inter prediction mode.

[0200] In one embodiment, the BVD prediction for the bi-predictive IBC AMVP mode is consistent with the non-IBC inter prediction mode.

[0201] In one embodiment, the bi-predictive IBC AMVP mode cannot be enabled together with IBC-CIIP, IBC-GPM, IBC-LIC. In one embodiment, the bi-predictive IBC AMVP mode can be enabled together with IBC-CIIP, IBC-GPM, IBC-LIC.

[0202] In one embodiment, if a block is coded with the bi-predictive IBC-AMVP mode, the first BVD is signaled as motion from reference picture L0. The second BVD is signaled as motion from reference picture L1. In this case, the reference picture index does not need to be signaled.

[0203] In one embodiment, if a bi-predictive IBC-AMVP coded block is referenced as a merge candidate, both BVs of the bi-predictive IBC-AMVP coded block will be used to generate the final predictor. In one embodiment, if a bi-predictive IBC-AMVP coded block is referenced as a merge candidate, the inherited weighting index will be used to derive the weighting pair to generate the final predictor. In one embodiment, if a block is coded with the bi-predictive IBC-AMVP mode, all the candidates in the BVP candidate list should be bi-predictive motion. Otherwise, if a block is coded with the uni-predictive IBC-AMVP mode, all the candidates in the BVP candidate list should be uni-predictive motion. In one embodiment, if a block is coded with the bi-predictive IBC-AMVP mode, only the first BV is used to derive the propagated intra mode.

[0204] In one embodiment, in the motion pruning process, both the motion of the bi-predictive IBC-AMVP block will be used for comparison. For example, when one candidate is bi-predictive IBC-AMVP coded and the other candidate is uni-predictive IBC-AMVP coded, it is considered different. Another example is when both candidates are bi-predictive IBC-AMVP coded, if the first BV or the second BV of the two candidates are different, it is considered different. In one embodiment, in the motion pruning process, if both candidates are bi-predictive IBC-AMVP coded blocks, not only the motion, but also the weighting index will be used for comparison.

[0205] In one embodiment, if a block is coded by bi-predictive IBC-AMVP mode, only the first BV is used to derive the BS value in the deblocking filter stage. In one embodiment, if a block is coded by bi-predictive IBC-AMVP mode, only the second BV is used to derive the BS value in the deblocking filter stage.

[0206] In one embodiment, if a block is coded by bi-predictive IBC-AMVP mode, the average of the two BVs will be used to derive the BS value in the deblocking filter stage.

[0207] Bi-predictive IBC mode

[0208] In one embodiment, when performing bi-predictive IBC mode, the current slice type can be considered.

[0209] In one embodiment, when the current slice type is I slice, the bi-predictive IBC-AMVP mode will mix one IBC AMVP predictor and one IBC merge predictor. When the current slice type is non-I slice, the bi-predictive IBC-AMVP mode will mix one IBC-AMVP predictor and one inter merge predictor. In this case, the motion of the inter merge predictor does not need to point to a reference block in the current picture.

[0210] In one embodiment, when the current slice type is I slice, the bi-predictive IBC-AMVP mode will mix one IBC AMVP predictor and one IBC merge predictor. When the current slice type is non-I slice, the bi-predictive IBC-AMVP mode will mix one inter-AMVP predictor and one IBC merge predictor. In this case, the motion of the inter-AMVP predictor does not need to point to a reference block in the current picture. It can be a motion pointing to a reference block in any reference picture.

[0211] In one embodiment, the above techniques can be applied to IBC-CIIP mode. Depending on the current slice type, the IBC-CIIP mode can mix different types of predictors.

[0212] In one embodiment, when the current slice type is I slice, the IBC-CIIP mode will mix one IBC merge predictor and one intra predictor. When the current slice type is non-I slice, the IBC-CIIP mode will mix one inter merge predictor and one IBC merge predictor. In this case, the motion of the inter merge predictor does not need to point to a reference block in the current picture.

[0213] In one embodiment, when mixing one IBC predictor and one inter merge predictor, equal weight pair will be applied. In one embodiment, when mixing one IBC predictor and one inter merge predictor, the weight of the inter merge predictor will be greater than the weight of the IBC predictor. In one embodiment, when mixing one IBC predictor and one inter merge predictor, the weight of the inter merge predictor will be less than the weight of the IBC predictor. In one embodiment, when mixing one IBC predictor and one inter merge predictor, if the IBC predictor is derived from IBC AMVP mode, the weight of the IBC predictor will be greater than the weight of the inter merge predictor.

[0214] In one embodiment, when mixing one inter-AMVP predictor and one IBC merge predictor, the weight of the inter-AMVP predictor will be greater than the weight of the IBC merge predictor.

[0215] In one embodiment, not only slice type can be used to determine the predictors to be mixed, other criteria can also be used.

[0216] In one embodiment, the criteria can be a syntax used to indicate whether the current slice is inclined to screen content. For example, if the slice is non-I slice, it can be found after analyzing the slice that most of the parts of the slice are flat, the IBC merge predictor will be mixed with the IBC-AMVP predictor (similar to I slice behavior). In one embodiment, the criteria can be a syntax used to indicate whether fractional MVD is used in MMVD or IBC-MBVD. In one embodiment, the above criteria can also be used to restrict the switching of IBC with multiple hypothesis mode.

[0217] In one embodiment, if IBC with multiple hypothesis mode is not allowed for the slice / picture, the related syntax does not need to be further signaled.

[0218] In one embodiment, two IBC merge predictors can be mixed in IBC merge MHP mode. In this case, the mixing pair is predefined. In one embodiment, in IBC merge MHP mode, a CU level switching flag is signaled to indicate the switching of the mode. After that, if IBC merge MHP mode is enabled, an ibc_mhp_idx is signaled to further indicate which mixing pair is used.

[0219] To further improve the coding efficiency of ibc_mhp_idx, the mixed pairs in PredOList and PredlList will be reordered according to TM cost. In this case, the more promising candidates will be assigned smaller indices.

[0220] In one embodiment, the syntax for indicating the type of hybrid predictor does not need to be signaled in I slice. In non-I slice, the syntax for indicating the type of hybrid predictor will be signaled.

[0221] In one embodiment, if the use of IBC with multiple hypothesis predictors is not allowed in the current CU, the syntax for indicating the type of hybrid predictor does not need to be signaled.

[0222] In one embodiment, in I slice, the syntax for indicating the to-be-mixed predictor is used to indicate the IBC predictor in the IBC merge list. In non-I slice, the syntax for indicating the to-be-mixed predictor is used to indicate the inter predictor in the inter merge list.

[0223] In one embodiment, a syntax is signaled to indicate whether the IBC predictor can be combined with other non-IBC predictors.

[0224] In one embodiment, if a syntax is signaled to indicate the IBC predictor is used to be combined with other non-IBC predictors, the merge index does not need to be signaled. It will be inferred as zero.

[0225] In one embodiment, if a syntax is signaled to indicate the IBC predictor is used to be combined with other non-IBC predictors, the hyp_weight_idx for indicating the hybrid weight does not need to be signaled. It will be inferred as equal weight or a predefined weight pair will be used in the hybrid.

[0226] IBC-CIIP mode

[0227] In one embodiment, the behavior of inter CIIP mode, IBC-CIIP in AMVP and IBC-CIIP in merge mode can be kept consistent.

[0228] In one embodiment, the behavior can be intra list construction. In one embodiment, the behavior can be hybrid weight design. In one embodiment, the behavior can be related to position dependent prediction combination (PDPC). PDPC is an intra prediction mode that has been adopted by VVC. For VVC, PDPC is applied to planar, DC, horizontal, vertical intra modes, and certain angular intra modes. When PDPC is applied, the prediction sample at a certain position is calculated as a combination of the current sample and two neighboring reference samples, where one neighboring sample is from the above neighboring reference line and the other neighboring sample is from the left neighboring reference line. In this case, both inter CIIP mode and IBC-CIIP mode can support PDPC, or neither inter CIIP mode nor IBC-CIIP mode can support PDPC. In one embodiment, when the current block is inter CIIP mode or IBC-CIIP mode, the behavior corresponds to deblocking filtering. In one embodiment, the behavior can be size constraint. For example, when the CU region is too large or too small, inter CIIP and IBC-CIIP will be disabled. In one embodiment, the behavior corresponds to motion reference of other CUs. For example, if a block is inter CIIP mode or IBC-CIIP mode, its motion cannot be referenced by other blocks. Another example is that if a block is inter CIIP mode or IBC-CIIP mode, its intra prediction cannot be referenced by other blocks.

[0229] In one embodiment, IBC-CIIP can merge two IBC predictors. For example, mixing one IBC TM merge predictor and one regular merge predictor.

[0230] In one embodiment, multiple binary bits are signaled to indicate the predictors to be mixed in IBC-CIIP. For example, the first binary bit is used to indicate the type of predictor (i.e., intra predictor or IBC predictor), and the second binary bit is used to indicate the index of the predictor (i.e., 0 or 1).

[0231] In one embodiment, only one bin is signaled to indicate the predictor to be mixed in IBC-CIIP. For example, the bin is used to indicate the type of predictor (i.e., intra predictor or IBC predictor). In one embodiment, only one bin is signaled to indicate the predictor to be mixed in IBC-CIIP. For example, the bin is used to indicate the type of predictor (i.e., intra predictor or IBC predictor). The index of the predictor is derived by an implicit method (i.e., template matching related method). In one embodiment, multiple bins are signaled to indicate the predictor to be mixed in IBC-CIIP. For example, two bins are used and two independent context tables are used for the two bins. In one embodiment, multiple bins are signaled to indicate the predictor to be mixed in IBC-CIIP. For example, two bins are used. The first bin is coded by context coded bin and the other bin is coded by EP bin.

[0232] In one embodiment, the syntax used to indicate the predictor to be mixed in IBC-CIIP is coded by unary code.

[0233] IBC merge list and related syntax design

[0234] In current intra block copy (IBC) design, IBC includes IBC merge coding type and IBC advanced motion vector prediction (AMVP) coding type. In IBC merge mode, neighboring, historical, temporal motion vector prediction (TMVP) or non-neighboring spatial candidate will be inserted into IBC merge candidate list. Then, a merge index will be signaled in bitstream to indicate which merge candidate is used.

[0235] In IBC AMVP mode, a motion estimation process will be performed to determine the reference block vector (BV), the motion information of the target BV will be directly signaled in bitstream.

[0236] In one embodiment, it is proposed to add IBC merge regression model filtering mode. For example, the regression model filter can be linear model, non-linear model, gradient linear model using gradient value to derive model parameters or cross component linear model.

[0237] In one embodiment, a list will be generated for IBC merge regression model filtering mode. Each candidate in the list will be optimized by applying regression model filtering. For example, the filter coefficients are derived by minimizing the difference between the template samples of the current block and their corresponding reference samples based on the regression model.

[0238] Regarding the syntax design, in one embodiment, a flag will be signaled in the bitstream to indicate the on or off of IBC merge regression model filtering mode. If the flag is true (i.e. IBC merge regression model filtering mode is enabled), a merge index will be further signaled in the bitstream to indicate which IBC merge regression model filtering candidate is used.

[0239] In one embodiment, a list will be generated for IBC merge mode. After that, each candidate will be tested with or without regression model filtering. The best type will be selected.

[0240] Regarding the syntax design, in one embodiment, a merge index will be signaled in the bitstream to indicate which candidate in the IBC merge list is used. After that, a regression model filtering on or off flag will be further signaled to indicate whether to apply regression model filtering or not.

[0241] In one embodiment, a list will be generated for IBC merge mode. After that, each candidate will be inserted into the candidate list again by interleaving. For example, odd indexed candidates will perform regression model filtering, even indexed candidates will not perform regression model filtering.

[0242] In one embodiment, non-regression filtered candidates and regression filtered candidates can be mixed together in one list (e.g. by some cost interleaving or reordering). For example, template matching (TM) cost is used to reorder the candidates. Another example is block vector (BV) cost is used to reorder the candidates.

[0243] Regarding the syntax design, in one embodiment, a merge index will be signaled in the bitstream to indicate which candidate in the IBC merge list is used. After that, based on the signaled index, the decoder can know whether to perform regression model filtering or not.

[0244] In another embodiment, in the aforementioned embodiment, all candidates in the second round can also be inserted at the end of the IBC merge list.

[0245] In one embodiment, a list will be generated for IBC merge mode. After that, TM based reordering is applied to the generated list. Only the top N candidates in the reordered list can perform regression model filtering.

[0246] Regarding the syntax design, in one embodiment, a merge index will be signaled in the bitstream to indicate which candidate in the IBC merge list is used. After that, if the signaled merge index is less than N, a regression model filtering on or off flag will be further signaled to indicate whether to apply regression model filtering or not.

[0247] In one embodiment, a list will be generated for IBC merge mode. Then, TM based reordering will be applied to the generated list. If the TM cost of a candidate is greater than a threshold, regression model filtering will be performed.

[0248] Regarding the syntax design, in one embodiment, a merge index will be signaled in the bitstream to indicate which candidate in the IBC merge list is used. Then, if the TM cost of the target candidate is greater than a threshold, regression model filtering technique will be implicitly performed.

[0249] In one embodiment, the regression model filtering technique is mutually exclusive with IBC local illumination compensation (LIC). In one embodiment, the regression model filtering technique is mutually exclusive with reconstruction reordered intra block copy (RRIBC). In one embodiment, the parameters derived for the regression model filtering can be used to define the parameters of IBC-LIC.

[0250] In one embodiment, the neighboring samples used for regression model filter parameter derivation should be determined according to the RRIBC type. For example, if the RRIBC is horizontal flipping type, the samples on the top and right side of the reference block will be used. If the RRIBC is vertical flipping type, the samples on the bottom and left side of the reference block will be used.

[0251] In one embodiment, it is proposed to add IBC merge multi-model mode.

[0252] In one embodiment, a list will be generated for IBC merge multi-model mode, containing the motion in T predefined position references. T can be any integer greater than zero. For example, the predefined positions are designed according to the coding unit (CU) width and CU height. Then, TM based reordering is performed by comparing the top template of the reference block and the top template of the current block. The top N candidates with the lowest TM cost are reserved. Then, TM based reordering is performed again by comparing the left template of the reference block and the left template of the current block. The top M candidates with the lowest TM cost are reserved. Then, TM based reordering is performed again by comparing the top and left templates of the reference block and the top and left templates of the current block. The top K candidates with the lowest TM cost are reserved.

[0253] In one embodiment, the list will be reordered according to the TM cost. In one embodiment, the final block vector can be any candidate in the list. In one embodiment, the final block vector can be the average block vector derived from the top N candidates in the list. In one embodiment, the final predictor can be the average of the N predictors generated from the N candidates in the list. N can be any integer value greater than 0.

[0254] Regarding the syntax design, in one embodiment, a flag is signaled in the bitstream to indicate the switch of IBC merge multi-model mode. If the flag is true (i.e. IBC merge multi-model mode is enabled), a merge index is further signaled in the bitstream to indicate which IBC merge multi-model candidate should be used to generate the final predictor.

[0255] Regarding the syntax design, in one embodiment, a flag is signaled in the bitstream to indicate the switch of IBC merge multi-model mode. If the flag is true (i.e. IBC merge multi-model mode is enabled), a merge index is further signaled in the bitstream to indicate which IBC merge multi-model candidate should be used to generate the final predictor.

[0256] In one embodiment, in a single tree, for an IBC coded block, if the luma is coded by IBC, the chroma can only be coded by IBC. The above-mentioned regression model filtering method can be applied to refine the chroma predictor.

[0257] In one embodiment, the regression model can be derived using the neighboring samples of the luma block and the neighboring samples of the chroma block. Then, the regression model can be applied to further refine the chroma predictor after the chroma predictor is derived based on the scaled luma-based block vector (BV).

[0258] Regarding the syntax design, in one embodiment, a flag is signaled in the bitstream to indicate the switch of IBC chroma refinement. In another embodiment, if the coding unit (CU) size satisfies certain conditions, IBC chroma refinement will be applied always without signaling.

[0259] The above-mentioned refinement method can also be applied to screen content related tools (e.g. the mode using the current picture as the reference picture).

[0260] Direct reference mode for intra chroma prediction

[0261] In this invention, an intra chroma mode is proposed, which utilizes a block vector (BV) referenced chroma mode to generate the prediction. The BV referenced chroma mode mentioned here is the chroma mode of the reference block corresponding to the BV obtained in the intra luma prediction.

[0262] In one embodiment, if the current block is coded by the proposed intra chroma mode, the prediction of the current block can be generated by directly applying the BV referenced chroma mode to the L-shaped neighboring reference samples of the current block. For example, if the BV referenced chroma mode is the planar mode, the chroma prediction of the proposed chroma mode is generated by applying the planar mode to the L-shaped neighboring reference samples of the current block.

[0263] In another embodiment, if the current block is coded by the proposed intra chroma mode, the prediction of the current block can be the fusion of the prediction generated by the BV reference chroma mode and the prediction generated by another pre-defined intra chroma mode. The fusion weights can be a pre-defined weight set or obtained by minimizing the mean square error (MSE) on the neighboring template.

[0264] In another embodiment, if the current block is coded by the proposed intra chroma mode, the prediction of the current block can be the fusion of the prediction generated by the BV reference chroma mode and the prediction generated by another pre-defined intra chroma mode. The fusion weights can be a pre-defined weight set or obtained by minimizing the mean square error (MSE) on the neighboring template.

[0265] In one embodiment, if the luma component of the current block is coded by one or a set of BV related coding tools (e.g., Intra TMP mode or IBC mode), a flag can be signaled to indicate whether the proposed chroma mode is used to code the chroma component of the current block. If the flag is false, the method falls back to the original intra chroma coding.

[0266] In one embodiment, if the co-located luma block of the current chroma block is coded in the BV related coding tools, the proposed fusion will automatically apply some constraints. For example, if the current block size is smaller or larger than a pre-defined threshold, the proposed fusion will be applied. In this case, no extra flag is needed to signal the switch of the proposed fusion mode.

[0267] In one embodiment, when the proposed fusion mode is applied, the chroma predictor will be fused with the reference block pointed by the BV, the target fusion BV can depend on the luma BV.

[0268] In one embodiment, when the proposed fusion mode is applied, the chroma predictor will be fused with the reference block pointed by the BV, the target fusion BV can depend on the luma BV. The weights of the two predictors can be determined by some cost (e.g., template matching cost). For another embodiment, sample based weighting can be used. For the samples close to the boundary, a larger weight will be applied to the predictor generated by the intra mode. Otherwise, a larger weight will be applied to the predictor pointed by the luma BV.

[0269] In one embodiment, a high level syntax can be signaled in SPS, PPS (Picture Parameter Set), PH (Picture Header) or SH (Slice Header) to indicate whether the proposed chroma mode is allowed to be used in the current sequence, picture or slice.

[0270] In one embodiment, the aforementioned high-level syntax can only be signaled when both BV-related coding tools (e.g., IntraTMP mode or IBC mode) are turned on or at least one is turned on.

[0271] Chroma mode coding improvement

[0272] As a block with complex texture can be difficult to be coded by regular intra modes, IntraTMP mode can be more suitable than other regular intra luma modes if the neighboring region has similar texture as the current block. At this time, cross-component prediction mode can also be more suitable than other regular intra chroma modes.

[0273] In the present disclosure, a method is proposed that, when the coded intra luma mode is a BV-related coding tool (e.g., IntraTMP mode or IBC mode), the cross-component prediction mode is prioritized.

[0274] In one embodiment, there can be additional context models for coding the intra chroma LM flag, and the selection of the multiple context models can depend on whether the luma component of the current block is coded by one or a set of BV-related coding tools (e.g., IntraTMP mode or IBC mode).

[0275] In one embodiment, if the luma component of the current block is coded by one or a part of the BV-related coding tools, the intra chroma LM flag cannot be signaled, and the chroma mode can be inferred to be one of the cross-component prediction modes.

[0276] In one embodiment, the syntax order or the chroma mode candidate order can be adaptively changed according to whether the luma block is coded in BV-related coding tools or not.

[0277] According to JVET-AC0193, when the prediction and the reference IBC block overlap, the cost in TM-related tools that can interact with IBC (e.g., TM AMVP, TM MRG, IBC TM, IBC BVD, IBC MRG, etc.) is adjusted using a cost factor as follows, with a fixed IBC_OVERLAP_TM_COST_FACTOR = 2.5:

[0278] if(((xBv + nwidth) >= 1 && ((yBv + nHeight) >= 1))

[0279] {

[0280] uiCost = (unit64_t)(uiCost * IBC_OVERLAP_TM_COST_FACTOR);

[0281] }

[0282] In one approach, IBC_OVERLAP_TM_COST_FACTOR is not a fixed value, but can have multiple values. In one embodiment, the value of the cost factor can be derived based on the size of the overlap region. In one embodiment, there are multiple predefined cost factors, and the cost factor used for the current block is signaled to the decoder or implicitly defined in the decoder (at CU / PU / CTU / SH / PH / APS / SPS or other level). In one embodiment, a subset of the cost factors in the predefined set is used. In one embodiment, the cost factor can be adjusted based on the tool / prediction method. In this case, different cost factors are used for different tools. In one embodiment, the cost factor is adjusted based on the slice type (i.e. I / P / B). In one embodiment, the cost factor is adjusted based on the neighboring information (e.g. the coding / motion information of the neighboring CUs). In one embodiment, multiple adjustment factors are used in combination. For example, the factor is adjusted based on the slice type and the neighboring information.

[0283] In one approach, when the prediction block is outside the prediction region and needs to use padding to fill in the unavailable prediction samples, these unavailable samples can not be accurate enough. Therefore, when these unavailable samples are used for prediction, they need to have a smaller "value" or "weight" (i.e. meaning less important). In one embodiment, such a prediction block has one or more special adjustment factors to compensate for the part of the prediction that is not accurate. In one embodiment, the cost factor is predefined, and in another embodiment, the cost factor is signaled to the decoder at CU / PU / CTU / SH / PH / APS / SPS or other level.

[0284] In one approach, the cost factor is not only applied to adjust the cost in IBC related tools, but also to other intra and / or inter tools. In one embodiment, the cost factor is used to adjust the cost of prediction, in case these tools interact with TM. In other words, whenever a tool uses TM related cost to adjust the cost of prediction, the TM related cost will be modified by the factor to adjust the priority of such prediction. In one embodiment, if a CU is predicted using one of the inter prediction tools, and the reference block overlaps with the current prediction block, then additional weight factor(s) can be applied based on the overlap region.

[0285] The hybrid IBC prediction mode and the aligned bi-predictive IBC mode as described above can be implemented at the encoder side or the decoder side. For example, any of the proposed candidate derivation methods can be implemented in the intra / inter coding module in the decoder (e.g. in the reference picture buffer or in the prediction module) or in the encoder (e.g. in the reference picture buffer or in the prediction module). FIG. 1Bimplemented in the intra prediction 150 / MC 152 in the encoder or the intra / inter coding module is an encoder (e.g. FIG. 1A implemented in the intra prediction 110 / inter prediction 112 in the decoder. However, the decoder or the encoder can also use additional processing units to implement the required IBC processing. While the intra / inter processing is shown as separate processing units, they can correspond to executable software or firmware code stored on a medium, e.g. a hard disk or flash memory, for a CPU (Central Processing Unit) or programmable device (e.g. a DSP (Digital Signal Processor) or FPGA (Field Programmable Gate Array)).

[0286] FIG. 10 A flowchart of an exemplary video coding system using hybrid IBC prediction mode according to an embodiment of the present application is shown. The steps shown in the flowchart can be executed as program code on one or more processors (e.g. one or more CPUs) at the encoder side. The steps shown in the flowchart can also be implemented based on hardware, e.g. one or more electronic devices or processors arranged to perform the steps in the flowchart. According to the method, input data associated with a current block is received in step 1010, wherein the input data comprises pixel data to be encoded at the encoder side or data associated with the current block to be decoded at the decoder side. A first predictor or first hypothesis is derived in step 1020 by using a first prediction mode comprising an IBC (Intra Block Copy) prediction mode or an Intra TMP (Intra Template Matching Prediction) mode, wherein the IBC prediction mode derives the first predictor or first hypothesis by locating a target reference block in the same picture as the current picture according to a first block vector, or the Intra TMP mode derives the first predictor or first hypothesis according to a template matching cost between a reference template of the target reference block and a current template of the current block. A second predictor is derived in step 1030. A hybrid predictor is derived in step 1040 by blending the first predictor or first hypothesis with the second predictor. The current block is encoded or decoded in step 1050 using a predictor candidate set comprising the hybrid predictor.

[0287] FIG. 11A flowchart of an exemplary video coding system that uses IBC-CIIP prediction mode to maintain consistency between IBC AMVP mode, IBC merge mode and inter mode, according to an embodiment of the present application is shown. According to the method, input data associated with a current block is received in step 1110, where the input data comprises pixel data to be encoded at an encoder end or data associated with the current block to be decoded at a decoder end. A first predictor is derived in step 1120 by using an intra block copy (IBC) prediction mode or an inter prediction mode, where the IBC prediction mode corresponds to IBC advanced motion vector prediction (AMVP) mode or IBC merge mode, and the first predictor is derived by locating a target reference block in the same picture as the current picture according to a first block vector selected from an IBC merge list or by a search mode of IBC AMVP mode when the IBC prediction mode is used. A second predictor is derived in step 1130 according to an intra prediction mode. A mixed predictor is derived in step 1140 by mixing the first predictor and the second predictor. The current block is encoded or decoded in step 1150 using a predictor candidate set including the mixed predictor, where one or more processes and / or one or more conditions associated with deriving the second predictor, deriving the mixed predictor or both are consistent with the inter prediction mode, the IBC AMVP mode and the IBC merge mode.

[0288] The flowchart shown is intended to illustrate one example of video coding according to the present application. A skilled person can modify each step, rearrange the steps, split a step or combine steps to implement the present application without deviating from the spirit of the present application. In the present disclosure, specific syntax and semantics are used to illustrate examples of implementing the present application. A skilled person can implement the present application by replacing with equivalent syntax and semantics without deviating from the spirit of the present application.

[0289] The above description aims to enable a person of ordinary skill to implement the present application according to a specific application and its requirements. Various modifications to the described embodiments will be apparent to a skilled person, and the general principles defined herein can be applied to other embodiments. Thus, the present application is not intended to be limited to the particular embodiments shown and described, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details are shown in order to provide a thorough understanding of the present application. However, skilled persons will understand that the present application can be implemented without these specific details.

[0290] Embodiments of the present application as described above can be implemented in various hardware, software code, or a combination of both. For example, an embodiment of the present application can be one or more circuit integrated into a video compression chip, or program code integrated into a video compression software to perform the processing described herein. An embodiment of the present application can also be program code to be executed on a digital signal processor (DSP) to perform the processing described herein. The present application can also involve several functions to be performed by a computer processor, a digital signal processor, a microprocessor, or field programmable gate array (FPGA). These processors can be configured to perform particular tasks according to the software code or firmware code that is made to embody the techniques of the application. Software code or firmware code can be developed in different programming languages and different formats to be used in different platforms. Software code can also be compiled into different object code formats or converted into different interpretable code formats. Different code formats, styles, and languages of software codes, and other means of configuring code to perform the tasks in accordance with the application will not depart from the spirit and scope of the application.

[0291] The present application can be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The described examples are to be considered in all respects only as illustrative and not restrictive. The scope of the application is, therefore, indicated by the appended claims, rather than by the foregoing description. All changes that come within the meaning of the claims are intended to be embraced therein.

Claims

1. A method of video coding, the method comprising: receiving input data associated with a current block, wherein the input data comprises pixel data to be encoded at an encoder end or data associated with the current block to be decoded at a decoder end; deriving a first predictor or a first hypothesis by using a first prediction mode comprising an intra block copy (IBC) prediction mode or an intra template matching prediction (IntraTMP) mode, wherein the IBC prediction mode derives the first predictor or the first hypothesis by locating a target reference block in a same picture as the current picture according to a first block vector, or the IntraTMP mode derives the first predictor or the first hypothesis according to a template matching cost between a reference template of the target reference block and a current template of the current block; deriving a second predictor; deriving a hybrid predictor by blending the first predictor or the first hypothesis with the second predictor; and encoding or decoding the current block using a predictor candidate set comprising the hybrid predictor.

2. The method of claim 1, wherein, The first predictor or the first hypothesis is generated by an IBC advanced motion vector prediction (AMVP) mode or an IBC merge mode.

3. The method of claim 1, wherein, The first predictor or the first hypothesis is generated by weighting a first IBC predictor in an IBC AMVP mode and a second IBC predictor in an IBC merge mode.

4. The method of claim 1, wherein, A plurality of first predictors or a plurality of first hypotheses is generated.

5. The method of claim 4, wherein, A redundancy check is performed when generating the plurality of first predictors or the plurality of first hypotheses.

6. The method of claim 1, wherein, The first prediction mode comprises an IBC merge mode, an IBC AMVP mode, an IBC geometric partitioning (GPM) mode, an IBC combined intra-inter prediction (IBC-CIIP) mode, a reconstructed reorder intra block copy (RRIBC) mode, an IBC template matching inter (IBC-TM-inter) mode, an IBC template matching merge (IBC-TM-merge) mode, an IBC regular merge (IBC-regular-merge) mode, an IBC motion vector difference merge (IBC-MBVD-merge) mode, an IBC local illumination compensation (IBC-LIC) mode, an IBC subblock (IBC-subblock) mode, or a combination thereof.

7. The method of claim 1, wherein, The second predictor is derived by using a second prediction mode, and the second prediction mode comprises an IBC merge mode, an IBC AMVP mode, an IBC GPM mode, an IBC-CIIP mode, an RRIBC mode, an IBC template matching inter mode, an IBC template matching merge mode, an IBC regular merge mode, an IBC motion vector difference merge mode, an IBC-LIC mode, an IBC subblock mode, or a combination thereof.

8. The method of claim 1, wherein, The first prediction mode corresponds to IBC with bi-prediction.

9. The method of claim 8, wherein, A first block vector (BV) is derived by applying an IBC search algorithm, and a search mode of a second BV is located using the first BV.

10. The method of claim 8, wherein, A block vector difference (BVD) associated with the first BV of the bi-prediction is signaled or parsed from a bitstream including coded data of the current block, and a second BVD associated with the second BV of the bi-prediction is inferred as 0.

11. The method of claim 8, wherein, A block vector difference (BVD) associated with the first BV of the bi-prediction is signaled or parsed from a bitstream including coded data of the current block, and a second BVD associated with the second BV of the bi-prediction is inferred as 0.

12. The method of claim 8, wherein, A merge index is signaled or parsed from a bitstream including coded data of the current block to indicate a target merge candidate in an IBC merge list for the bi-prediction.

13. The method of claim 8, wherein, A flag is signaled or parsed from a bitstream including coded data of the current block to indicate whether to use IBC AMVP mode for the bi-prediction.

14. The method of claim 8, wherein, If the current block is coded in IBC AMVP mode, all candidates in an AMVP candidate list of the current block are bi-predictive motion.

15. The method of claim 8, wherein, If the current block is coded in IBC AMVP mode, a dominant BV of the current block is used to derive a propagated Intra mode.

16. The method of claim 8, wherein, When a current slice type of the current block is an I slice, the hybrid predictor is generated by one IBC AMVP predictor and one IBC merge predictor.

17. The method of claim 8, wherein, When the current slice type of the current block is a non-I slice, the hybrid predictor is generated by mixing one IBC AMVP predictor and one inter merge predictor, or by mixing one inter AMVP predictor and one IBC merge predictor.

18. An apparatus for video coding, the apparatus comprising one or more electronic devices or processors arranged to: receive input data associated with a current block, the input data comprising pixel data to be encoded at an encoder end or data related to the current block to be decoded at a decoder end; derive a first predictor or first hypothesis by using a first prediction mode comprising an Intra Block Copy (IBC) prediction mode or an Intra TMP (Intra Template Matching Prediction) mode, wherein the IBC prediction mode derives the first predictor or first hypothesis by locating a target reference block in a same picture as the current picture according to a first block vector, or the Intra TMP mode derives the first predictor or the first hypothesis according to a template matching cost between a reference template of the target reference block and a current template of the current block; derive a second predictor; derive a hybrid predictor by mixing the first predictor or first hypothesis and the second predictor; encode or decode the current block using a predictor candidate set comprising the hybrid predictor.

19. A method of video coding, the method comprising: receiving input data associated with a current block, the input data comprising pixel data to be encoded at an encoder end or data related to the current block to be decoded at a decoder end; deriving a first predictor using an intra block copy (IBC) prediction mode or an inter prediction mode, wherein the IBC prediction mode corresponds to an IBC advanced motion vector prediction (AMVP) mode or an IBC merge mode, and the first predictor is derived by locating a target reference block in the same picture as the current picture according to a first block vector selected from an IBC merge list or by a search mode of the IBC AMVP mode when the IBC prediction mode is used; deriving a second predictor according to an intra prediction mode; deriving a combined predictor by combining the first predictor and the second predictor; encoding or decoding the current block using a predictor candidate set including the combined predictor, wherein one or more processes and / or one or more conditions associated with deriving the second predictor, deriving the combined predictor, or both are consistent with the inter prediction mode, the IBC AMVP mode, and the IBC merge mode.

20. The method of claim 19, wherein the one or more processes and / or one or more conditions comprise intra list construction.

21. The method of claim 19, wherein the one or more processes and / or one or more conditions comprise combined weight design.

22. The method of claim 19, wherein the one or more processes and / or one or more conditions are related to position dependent prediction combination (PDPC).

23. The method of claim 19, wherein the one or more processes and / or one or more conditions comprise deblocking filtering.

24. The method of claim 19, wherein the one or more processes and / or one or more conditions comprise one or more constraints related to current block size.

25. The method of claim 19, wherein the one or more processes and / or one or more conditions comprise motion reference of other blocks.

26. An apparatus for video coding, the apparatus comprising one or more electronic devices or processors configured to: receive input data associated with a current block in a current picture, wherein the input data comprises pixel data to be encoded at an encoder end or data associated with the current block to be decoded at a decoder end; derive a first predictor value using an intra block copy (IBC) prediction mode or an inter prediction mode, wherein the IBC prediction mode corresponds to an IBC advanced motion vector prediction (AMVP) mode or an IBC merge mode, and wherein the first predictor value is derived by locating a target reference block in the same picture as the current picture according to a first block vector selected from an IBC merge list or by a search mode of the IBC AMVP mode; derive a second predictor value according to an intra prediction mode; derive a combined predictor value by combining the first predictor value and the second predictor value; and The current block is encoded or decoded by using a prediction candidate set including the hybrid prediction value, wherein one or more processes and / or one or more conditions associated with the deriving the second prediction value, the deriving the hybrid prediction value, or both, are consistent with the inter prediction mode, the IBC AMVP mode, and the IBC merge mode.

Images