Application of template matching in video coding

Abstract

A method for template matching (TM) in video coding is described. The proposed method includes using constrained top and left neighbors in template matching, enabling TM only at coding tree unit boundaries, using approximated reconstructed samples, a new processing pipeline for deriving decoder-side intra mode derivation (DIMD) in combination with template-based intra mode derivation (TIMD), and using filtered pixels from neighbors instead of using reconstructed pixels. Furthermore, methods are described for applying template matching in combination with intra, sub-partitioning modes, interpolation filtering in intra prediction, block partitioning, bi-prediction with coding unit level weights, and adaptive motion vector resolution.

Description

[Technical Field]

[0001] [Related Applications] This application claims the benefit of priority to Indian Provisional Patent Application No. 202241021946, filed April 12, 2022, which is incorporated herein by reference in its entirety.

[0002] [Technical field] FIELD OF THE INVENTION This disclosure relates generally to image and video coding. More particularly, embodiments of the present invention relate to the application of template matching in video coding. [Background technology]

[0003] In 2020, the MPEG group of the International Organization for Standardization (ISO), in collaboration with the International Telecommunication Union (ITU), released the first version of the Versatile Video Coding Standard (VVC), also known as H.266 (Ref. [7]). More recently, the same group has been working on developing a next-generation coding standard that will improve coding performance over existing video coding techniques. As part of this research, new coding techniques are also being considered.

[0004] As recognized by the inventors herein, improved techniques for applying template matching in image and video coding are desirable, and are described herein.

[0005] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise noted, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, problems identified with one or more approaches should not be assumed to have been recognized in any prior art under this section unless specifically indicated. [Brief explanation of the drawings]

[0006] Embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the accompanying figures, in which like reference symbols represent similar elements and in which:

[0007] [Figure 1] 1 illustrates an example of template matching in video coding.

[0008] [Figure 2] 1 illustrates an exemplary subdivision of a picture for processing coding units, according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Exemplary embodiments relating to applying template matching in video coding are described herein. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of various embodiments of the present invention. It will be apparent, however, that various embodiments of the present invention may be practiced without some of these specific details. In other instances, well-known structures and devices have not been described in exhaustive detail to avoid unnecessarily occluding, obscuring, or obscuring embodiments of the present invention.

[0010] <Summary> Example embodiments described herein relate to template matching (TM) in image and video coding. Proposed methods include the use of constrained top and left neighbors in template matching, enabling TM only at coding tree unit boundaries, using approximated reconstructed samples, a new processing pipeline for deriving decoder-side intra mode derivation (DIMD) in combination with template-based intra mode derivation (TIMD), and using filtered pixels from neighbors instead of using reconstructed pixels. Furthermore, example embodiments describe how to apply template matching in combination with intra modes, subpartitioning modes, interpolation filtering in intra prediction, block partitioning, bi-prediction with coding unit level weights, and adaptive motion vector resolution.

[0011] Template Matching in Video Coding Figure 1 shows an example of template matching in video coding (Ref. [1]). The term "template matching" refers to a decoder-side motion vector (MV) derivation method that refines the motion information of a current coding unit (CU) by finding the closest match between a template in the current picture (i.e., the current CU's upper and / or left neighboring blocks (105)) and a block in a reference picture (i.e., the same size as the template). As shown in Figure 1, in one embodiment, given an initial motion vector (110), a better MV is searched for around the initial motion vector of the current coding unit (CU) within a [-8, +8]-pel search range (125). The search step size is determined based on the advanced motion vector resolution (AMVR) mode, and TMs can be cascaded in the bilateral matching process in merge mode.

[0012] In advanced motion vector prediction (AMVP) mode, a motion vector predictor (MVP) candidate is determined based on template matching error, and the one that achieves the smallest difference between the current block template (105) and the reference block template (115) is selected. TM is then performed only on this specific MVP candidate for MV refinement. TM refines this MVP candidate using an iterative diamond search, starting with full-pel motion vector difference (MVD) accuracy (or 4-pel for 4-pel AMVR mode) within a [-8, +8] pel search range (125). The AMVP candidate is further refined using a cross search at full-pel MVD accuracy (or 4-pel for 4-pel AMVR mode), followed by 1 / 2-pel and 1 / 4-pel ones, successively depending on the AMVR mode. This search process ensures that the MVP candidate continues to maintain the same MVD accuracy as indicated by AMVR mode after the TM process.

[0013] In merge mode, a similar search method is applied to the merge candidates indicated by the merge index. TM can run all methods up to 1 / 8-pel MVD accuracy or skip those beyond 1 / 2-pel MVD accuracy, depending on whether an alternative interpolation filter (used when AMVR is in 1 / 2-pel mode) is used according to the merged motion information. Furthermore, when TM mode is enabled, template matching can operate as an independent process between block-based and sub-block-based bilateral matching (BM) methods or as an additional MV refinement process, depending on whether BM can be enabled according to its enablement condition check.

[0014] As recognized by the inventors, the current version of the inter-template matching tool knows the following: 1) Refined motion vectors (MVs) from neighboring blocks are needed to start the current block MV prediction; 2) We need the current template pixel, which is a reconstructed pixel from a neighboring block (e.g., the top and fourth row from the left), 3) VVC uses the current pixel in the reshaped domain for intra prediction, but if we use the current reconstructed pixel for template matching, we need inverse shaping to return to the original domain because the reference template pixel is in the original domain. 4) The boundary strength calculation process needs to be delayed in the HW pipeline design because refinement MV is used.

[0015] In addition to inter-template matching tools, the idea of ​​template matching is also widely utilized by other coding tools to aid decisions at the decoder side by finding the closest match between the template of the current region (i.e., the above and / or left neighboring blocks of the current CU) and a reference region. Examples include: 1) Intra Template Matching (Ref. [2]): A special intra prediction mode that copies the best predicted block from the reconstructed part of the current frame, and an L-shaped template is matched with the current template. 2) Template-based Intra Mode Derivation (TIMD) using most probable mode (MPM) (Ref. [3]). This is an intra mode derivation method using most probable mode (MPM) using neighboring templates. TIMD mode is used as an additional intra prediction method for CUs. 3) Adaptive re-ordering of merge candidates (ARMC) using TM (Ref. [4]). In this tool, after constructing a merge candidate list, the merge candidates are divided into several subgroups. The merge candidates in each subgroup are sorted in ascending order according to their cost values ​​based on template matching. 4) MVD code prediction using TM (Ref. [5]). Here, motion vector difference code prediction can be applied in regular inter mode when the motion vector difference contains non-zero components. The possible MVD code combinations are sorted according to their template matching cost, and the index corresponding to the true MVD code is derived and coded with the context model. 5) TM with merge mode with motion vector difference (MMVD) (Ref. [5]), which is a template matching based sorting for extended MMVD.

[0016] The embodiments presented herein aim to improve the template matching process in various aspects. 1) Proposals for quality improvement (QI), which aim to improve compression efficiency, preferably without additional hardware (HW) implementation issues compared to current coding tools. 2) Proposals for removing HW dependency / pipeline issues (HWPI) and for hardware / software (HW / SW) complexity reduction (CR). The aim is to resolve HW dependency / pipeline issues and harmonize similar tools by considering specific guidelines. It also provides HW / SW-friendly solutions, such as complexity reduction with minimal impact on compression efficiency. 3) Extending the TM concept to other coding tools.

[0017] Use of constrained top or left neighbors for TM to reduce immediate neighbor dependencies in decoding order Motivation: TM requires the immediately above and left neighboring reconstructed pixels for the template. This introduces a strong pipeline dependency in the decoding pipeline, since the immediately neighboring reconstructed pixels are required to derive the motion information of the current CU.

[0018] Proposal 1: For TM, do not allow neighbor samples from the previous CU as follows. If the left CU is the previous CU in decoding order, use the CU above for TM. For example, in Figure 2, CU3 uses the sample above (CU1) to calculate the neighbor cost, but does not use the left sample (CU2), which was the previous CU in decoding order. Similarly, CU9 can use all of the above neighboring samples from CU8, but only partial left neighboring samples from CU7, because the left neighboring samples from CU8 belong to the previous CU in decoding order. If the CU above is the previous CU in decoding order, then use the left CU to calculate the neighbor cost. For example, in Figure 2, CU6 can use the left neighboring sample from CU4 for TM, but cannot use the top neighboring sample from CU5, which is the previous CU in decoding order.

[0019] Proposal 2: For TM, do not allow neighbor samples from "X" (X>1) previous CUs. This is an extension of Proposal 1, where neighboring samples from multiple previous CUs in decoding order are forbidden for TM to provide more HW parallelism. This is proposed because TM is a relatively complex tool with many stages of search and refinement. If neither left nor top neighbors can be used due to this constraint, TM must be completely and implicitly disabled for such CUs. For example, when X=2, -CU9 can only use the upper sample in Figure 2. On the other hand, CU3 and CU6 in Figure 2 cannot use left or top samples, so TM is implicitly disabled for them.

[0020] Proposal 3: For TM, disallow neighbor samples from a specific region size of the previous CU. This is similar to Proposal 2, but the number of prohibited CUs can be variable rather than a constant value X.

[0021] TM valid only at CTU boundaries with restrictions on the use of left neighbors To enable virtual pipeline data unit (VPDU)-level parallelism in TM, we propose to enable TM only at coding tree unit (CTU) boundaries as follows. 1) Upper CTU neighboring samples are always available (except at frame boundaries). 2) The use of left CTU neighbor samples is permitted only if the following conditions are met: 2-2) The current CTU root node is split into a quad-tree (QT) (NxN) or a binary tree (BT) (2NxN). 2-3) The left CTU root node is divided into QT (NxN) or horizontal BT (2NxN). 2-4) This ensures that the TMs of the four NxN VPDUs in the CTU are pipelined as follows: - The VPDU0 boundary sample of the current CTU can use the VPDU1 of the left CTU for the left neighboring sample and the upper CTU sample. Currently, the VPDU2 boundary sample of a CTU can use the VPDU3 of the left CTU for the left neighboring sample, but cannot be used for the upper sample. -For VPDU1 boundary samples, only the upper CTU samples can be used for TM. -VPDU3 cannot use TM. 2-5) Note that CUs that are not part of the top or left CTU boundary cannot use TM.

[0022] TM using approximate reconstruction samples of top and left inter-predicted CUs The motivation is to use the approximate reconstructed samples of neighboring inter-CUs for TM, which are derived by adding the filtered (e.g., bilinearly interpolated) predicted samples and the inverse transform residuals based on a lookup table (LUT) of the main transform coefficients (top 4 or top 8).

[0023] TM introduces a dependency on using reconstructed neighboring samples, which causes significant hardware pipeline delays for inter-reconstruction. Currently, a CU must wait for the CUs above and to the left to complete reconstruction (e.g., make reconstructed samples available) before starting the TM process. This proposal aims to reduce the pipeline delay by replacing the use of reconstructed samples with approximate reconstructed samples, so that the current CU can start the TM process once the predicted and dequantized transform coefficients of the neighboring CUs are available. In brief, ReconSample_Actual=Pred+Res where Pred represents the predicted pixel, and Res=InvT(QuantCoeff), where InvT(QuantCoeff) represents the inverse transform of the quantization coefficients. ReconSample_Approximated=filtered(Pred)+LUT[QuantCoeff] The LUT-based operation on the inverse quantized transform coefficients is a fast approximation for estimating the residual without performing the actual inverse transform. The idea of ​​using filtering for prediction is similar to the motivation for adaptive loop filtering (ALF) to improve the accuracy of the approximation. However, to reduce complexity, the filtering applied here should not be too complex.

[0024] For luma mapping, chroma scaling (LMCS), when enabled in VVC, a mapping operation (LMCSFwdMap) is required because inter prediction is in the original domain, while the reconstruction and residual are in the reshaped domain. Filtering of the prediction can be used in either domain. ReconSample_Actual=LMCSFwdMap[Pred]+Res where Res=InvT(QuantCoeff). ReconSample_Approximated=filtered(Pred)+LUT[QuantCoeff] where: filtered(Pred) = filtered(LMCSFwdMap[Pred]) or LMCSFwdMap[filtered(Pred)].

[0025] The following restrictions / modifications can be further applied to reduce TM complexity: 1) TM uses four neighboring samples, while code prediction uses two neighboring samples. To reduce the overhead of inter-CU approximate reconstruction using -LUT, both TM and sign prediction can be harmonized to use only two neighboring samples. 2) Another option is to use only the left and top inter prediction samples for inter TM (without the approximate residual). -This assumes that inter-CU residuals are small (small number of small coded coefficients), so the prediction contains most of the information of the reconstructed CU. To apply this assumption, the following constraints may also be applied: Use inter-predicted samples of neighboring CUs that have a small number of coded coefficients (e.g., fewer than 3 coefficients) coded in the neighboring CU.

[0026] Limitations on the use of TM refined MVs for the merging and MV list construction process Motivation: Using TM-based MV or merge modes introduces strong pipeline dependencies into the decoding pipeline, since immediate neighboring reconstructed pixels are required to derive the merge / AMVP lists of the current CU. This causes almost all HW decoding operations to be serialized at the CU level.

[0027] Proposal: Limit the use of TM refinement MVs for the motion vector prediction process, such as merge list and AMVP list construction, as follows: 1. Use TM refinement motion only at CTU boundaries, and only from the top, top-left, and top-right CTUs. Prior TM motion information used for other spatial neighbors. 2. TM refined MVs can be used for other forward dependencies, such as boundary strength (BS) calculations and temporal MV storage. 3. Tools such as ARMC, TM using MMVD, and TM using MVD code should be completely disabled or enabled only at the TOP CTU boundary.

[0028] Harmonizing the intra prediction process across DIMD and TIMD Motivation: Decoder-side intra mode derivation (Ref. [6]) is a new tool in JVET's current enhanced compression model (ECM). The DIMD process uses a fusion of three intra modes, while TIMD uses a fusion of one or two intra modes. TIMD uses the DIMD mode to determine the best mode based on template cost. Therefore, the TIMD process is the worst case in terms of HW processing latency. We harmonize aspects of DIMD and TIMD to help improve compression efficiency or reduce HW complexity. In ECM, the following shorthand notation can be used to describe the computation engines required for DIMD and TIMD modes: C0: Deriving the MPM list Input: Neighborhood Intra Mode Output: Set Mode C1: Find the set of intra-modes from the gradient histogram and the magnitude of the gradient histogram. Input: Current neighboring reconstructed pixel Output: Top two intra modes based on TM cost C2: Calculate the TIMD cost for a given set of intra-modes and select the top two intra-modes. Input: current set of neighboring reconstructed pixels and intra mode Output: Top 2 intra modes based on TM cost C3: Fusion of three intra modes (one fixed to Planar) Inputs: 2 Intra and 1 Planar modes Output: Final intra prediction data C4: Fusion of two intra-modes Input: 2 Intra modes Output: Final intra prediction data Current methods of ECM DIMD mode: C1+C3 TIMD mode: C0+C1+C2+C4 Suggestion 1: DIMD mode: C1 (set of modes) + C2 + C4 TIMD mode: C0+C1(only two modes)+C2+C4 Suggestion 2: DIMD mode: C1 (set of N modes) + C2 + C3 TIMD mode: C0+C1(only two modes)+C2+C3 The term "N modes" refers to N intra-prediction modes, such as angular mode, DC mode, and planar mode.

[0029] Reduction of IntraTM on-chip memory Motivation: The on-chip memory requirements for Intra TM are very high compared to Intra block copy (IBC). In the current intra template matching method of ECM, when the CU size is large, top pixels from several top CTU rows are used.

[0030] Proposal: Limit the use of currently reconstructed pixels from the top CTU row. Allow the bottom four lines of reconstructed pixels from the top CTU because they are already used for TM or intra prediction. Limit the on-chip memory size to a × CTU size, where the factor a can be between 1 and 5.

[0031] Use filtered predictions as a substitute for neighboring reconstructed pixels. Motivation: TM requires the reconstructed pixels above and to the left for template construction. This introduces a strong pipeline dependency in the decoding pipeline, since the neighboring reconstructed pixels are required to derive the motion information of the current CU.

[0032] Proposal: In template construction, a neighboring prediction, even a filtered version, can be used instead of a reconstructed pixel. In one embodiment, the filter can be a Wiener filter, derived using statistical properties of the predicted and reconstructed pixels from the region in the reference frame pointed to by the unrefined MV. This process applies only if at least one of the neighbors is inter-coded. This proposal aims to find an appropriate replacement for using the neighboring reconstructed pixel. The neighboring prediction can be thought of as a noisy version of the neighboring reconstruction. Therefore, if we find a linear filter to apply to the prediction so that the difference between the filter output and the actual reconstructed pixel is minimized, we can find an optimal replacement, assuming we are constrained by the linear filter. For example, in FIG. 1, template 105 can be filtered because it is the hardware pipeline bottleneck. (In contrast, for template 115, all reconstructed samples are already available.) The TM needs to use the reconstructed samples of 105 (InterRecon occurs at a later stage in the pipeline). The proposal is to use a filtered version of the prediction of 105 to replace the reconstruction of 105. Therefore, a CU's TM can now start as soon as predicted samples that occur earlier in the pipeline become available to neighboring CUs. For example, ReconSample_Approximated=f(Pred) where f() is some kind of filtering operation, such as a Wiener filter, a nonlinear filter, or a neural network-based filter.

[0033] The filter coefficients shall be derived using reconstructed pixels from the reference region pointed to by the unrefined TM MV as the reference signal and a prediction from the same region as the noisy version of the reference signal.

[0034] Below are some variations on this proposal that address various complexities implied by the above paragraph. 1) Instead of deriving a filter for every TM call, one can signal a set of predefined filters via an adaptation parameter set (APS) and define a process for deriving filter indices based on prediction and reconstruction statistics or other properties. 2) Since using neighboring predictions means waiting for motion compensation (MC) of neighboring CUs, this involves waiting for the refined MV output by the neighboring CU's TM and similar MV refinement algorithms, and using the MV before the start of MV refinement to derive the neighboring predictions used by the current CU for template construction.

[0035] Harmony between ARMC and TM Motivation: Improve the coding efficiency of ARMC. The improved ARMC compensates for the loss of coding efficiency by removing TM refinement.

[0036] Proposal: Adaptive reordering of merge candidates with TM refinement (ARMC-R). Start with -ARMC-TM, which brings in the referenced data for each merge candidate. In the current ARMC, top and left reference template regions are used, with the size of the top reference template region being (BlkWidthx4) and the size of the left reference template region being (4xBlkHeight). In an embodiment, the reference template can be improved by adding pixels around it so that + / -1MV refinement can be performed. Thus, the size of the top reference template region will be ((BlkWidth+2)x(4+2)) and the left template region will be ((4+2)x(BlkHeight+2)). The "+2" indicates an additional line of pixels around the reference template region for + / -1MV refinement. For + / -2MV refinement, "+4" is needed. - 9-point (center-inclusive, square pattern) integer-pixel distance costs are evaluated. Sub-pixel costs based on error surfaces can also be derived from the 9-point costs, as used in VVC decoder-side motion vector refinement (DMVR). -Merge MV Instead of TM cost, select the minimum cost from the 9-point refinement of each merge candidate. - Sort the merge list based on minimum refinement cost. -Remove duplicate MVs. Method 1: (QI) Replace ARMC-TM with the proposed ARMC-R. Method 2: (CR) Remove TM refinement above Method 1, since Method 1 already covers refinement.

[0037] Simplifying TM-based MV refinement for integer-pixel refinement Motivation,MV refinement methods based on template matching are highly,sequential, involving interpolation and template cost calculation,using a diamond pattern followed by one final step of,cross.

[0038] Proposal: Use integer MV positions corresponding to motion vectors from the merge / AMVP list as the search starting point. This helps avoid the need for interpolation for integer pixel refinement.

[0039] The search range is limited to an optimal value so that the TM costs for all integer pixel locations around the center can be calculated in parallel. For a search range of + / - 2 pixels around the center, the TM costs need to be calculated for a total of 25 points, and for a search range of + / - 3 pixels around the center, the TM costs need to be calculated for 49 points.

[0040] Extending the TM concept to other tools Intra sub-partition mode (ISP) and TM: In VVC, an intra-predicted block can be subdivided horizontally or vertically into smaller blocks called sub-partitions. In each of them, prediction and transform coding operations are performed separately, but the intra-mode is shared by all sub-partitions. In one embodiment, we propose combining ISP and TM to allow each sub-partition to have a different intra-mode. The basic idea is to use TM to refine the shared intra-mode of each sub-partition using neighboring angular intra-prediction modes or the most probable mode (MPM) mode of this block partition.

[0041] Interpolation Filtering for Intra Prediction and TM: In VVC, interpolation filtering is applied in fractional gradient mode. For luma, the interpolation filter represents either a 4-tap DCT-based interpolation filter (DCTIF) or a 4-tap smoothing interpolation filter (SIF). The type of interpolation filter is not signaled in the bitstream, but is determined based on the size of the block and the intra-prediction mode index. In one embodiment, one approach is to use the TM to determine the IF to use without explicit signaling. It is also possible to add more candidate IFs, such as 8-tap DCTIFs or SIFs, to the pool and let the TM decide which IF to use for optimal coding efficiency.

[0042] Block Partitioning and TM: For a given CU, we can use TM to find the optimal integer MV. Then we can copy the block partition from the optimal MV as the estimated partition for the current block, in order to preserve bits for the partition.

[0043] Bi-prediction with CU level weights (BCW) and TM: In VVC, for BCW, a set of candidate weight values ​​can be selected for bidirectional inter prediction. The index of the selected weight value is signaled for AMVP mode and is inherited to merge mode if allowed. In one embodiment, TM can improve BCW in two aspects: 1) Using TM, avoiding signaling of weight index; and 2) Allowing more weights and using TM to select and signal a limited set.

[0044] Adaptive motion vector resolution with TM: Instead of explicitly signaling the motion vector resolution, we can estimate the resolution based on TM. Essentially, TM allows us to try different motion vector resolution (MVR) techniques, and the best MV-driven resolution is the current CU resolution. References Each of the references listed herein is incorporated herein by reference in its entirety. The term JVET refers to the Joint Video Experts Team of ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29. [1] JVET-U0100, “Compression efficiency methods beyond VVC,” Y.-J. Chang et al., teleconference, Jan. 2021. [2] JVET-V0130, “EE2: Intra Template Matching,” K. Naser et al., teleconference, April 2021. [3] JVET-V0098, “EE2-related: Template-based intra mode derivation using MPMs,” Y. Wang et al., teleconference, April 2021. [4] JVET-W0090, “EE2-3.1 / EE2-3.2: Adaptive Reordering of Merge Candidates with Template / Bilateral Matching,” N. Zhang et al., teleconference, July 2021. [5] JVET-Y0067, “EE2-3.9 and EE2-3.10: TM based reordering for MMVD and affine MMVD and MVD sign prediction,” M. Salehifar et al., teleconference, Jan. 2022. [6] JVET-O0449, “Non-CE3: Decoder-side Intra Mode Derivation (DIMD) with prediction fusion using Planar,” M. Abdoli et al., Gothenburg, July 2019. [7] “Versatile Video Coding,” Rec. ITU-T H.266, August 2020.

[0045] <Implementation of an exemplary computer system> Embodiments of the present invention may be implemented by a computer system, a system configured in electronic circuits and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA) or other configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or perform instructions related to applying template matching in image and video coding as described herein. The computer and / or IC may calculate any of various parameters or values ​​related to applying template matching in image and video coding as described herein. Image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0046] A specific implementation of the present invention includes a computer processor executing software instructions that cause the processor to perform the methods of the present invention. For example, one or more processors in a display, encoder, set-top box, transcoder, etc. may implement the methods related to applying template matching in image and video coding described above by executing software instructions in a program memory accessible to the processor. An embodiment of the present invention may be provided in the form of a program product. The program product may include any non-transitory tangible medium that carries a set of computer-readable signals containing instructions that, when executed by a data processor, cause the data processor to perform the methods of the present invention. A program product according to the present invention may be in any of a variety of non-transitory tangible forms. The program product may include physical media such as magnetic data storage media including floppy disks, hard disk drives, optical data storage media including CD-ROMs and DVDs, ROMs, electronic data storage media including flash RAM, etc. The computer-readable signals on the program product may be optically compressed or encrypted. Although components (e.g., software modules, processors, components, devices, circuits, etc.) have been referred to above, unless otherwise indicated, references to those components (including references to "means") should be interpreted to include equivalents of those components, any components that perform the functions of the described components (e.g., are functionally equivalent), and components that are not structurally equivalent to the disclosed structures that perform the functions in the illustrated exemplary embodiments of the present invention.

[0047] <Equivalents, Extensions, Alternatives and Miscellaneous> Exemplary embodiments relating to applying template matching in video coding are described herein. In the foregoing specification, embodiments of the present invention have been described with reference to numerous specific details that may vary from implementation to implementation. Accordingly, the sole and exclusive indication of what the invention is, and what Applicant intends to be the invention, is set forth in the claims as issued in particular form hereby, including any subsequent amendments. Any definitions expressly set forth herein for terms contained in such claims shall control the meaning of such terms as used in the claims. Accordingly, any limitation, element, feature, advantage, or attribute not expressly recited in the claims should not in any way limit the scope of the claims. The specification and drawings are, therefore, to be considered in an illustrative, rather than a restrictive, sense.

Claims

1. 1. A method of processing one or more pictures using template matching, the method comprising: receiving a picture having a current coding unit (CU) and a previous decoded coding unit; Applying template matching using a neighborhood pixel region of the current CU, the template matching comprising: Except for frame boundaries, upper CTU neighboring samples are always available. The use of left CTU neighbor samples is permitted only if the following conditions are met: - The current CTU root node is split into a quadtree (QT) (NxN) or a horizontal binary tree (BT) (2NxN), - The left CTU root node is split into QT (NxN) or horizontal BT (2NxN), The steps are constrained as follows: A method comprising:

2. 1. A method of processing one or more pictures using template matching, the method comprising: receiving a picture having a current coding unit (CU) and a previous decoded coding unit; Applying template matching using a neighboring pixel region of the current CU, wherein the samples of the neighboring CUs in the template matching are derived based on the use of one or more of filtered prediction samples or approximated residuals; A method comprising:

3. 1. A method of processing one or more pictures using template matching, the method comprising: receiving a picture having a current coding unit (CU) and a previous decoded coding unit; applying template matching using a neighboring pixel region of the current CU, wherein only inter-predicted samples of neighboring CUs are used in the template matching; A method comprising:

4. 3. The method of claim 2, wherein the filter for generating the filtered prediction samples is derived using statistical properties of predicted pixels and reconstructed pixels from a region in a reference frame indicated by the unrefined motion vector.

5. 1. A method for decoder-side intra mode derivation (DIMD) in combination with template-based intra mode derivation (TIMD), the method comprising: below: DIMD mode by combining C1 (which generates a set of N modes), C2, and C4, TIMD mode by calculating C0, C1 (selects only two modes), C2, and C4; or below: DIMD modes by calculating C1 (which generates a set of N modes), C2, and C3, TIMD mode by calculating C0, C1 (selects only two modes), C2, and C3; Calculating Including, C0, C1, C2, C3, and C4 include: C0: Derive an MPM list with a set of neighboring intra-modes at input and modes at output. C1: Find a set of intra modes from the gradient histogram and gradient histogram magnitude, with the current neighboring reconstructed pixel as C1 input and the top two intra modes based on TM cost as C1 output. C2: Compute the TIMD cost of a given set of intra modes, with the current neighboring reconstructed pixel and the set of intra modes as C2 input, and the top two intra modes based on the TM cost as C2 output, and select the top two intra modes; C3: A fusion of three intra modes (one fixed to planar) with two intra modes and a planar mode at the C3 input and the final intra prediction data at the C3 output. C4: Fusion of two intra modes with two intra modes at the C4 input and final intra prediction data at the C4 output; method.

6. 1. A method for adaptive reordering of merge candidates using template matching (ARMC), the method comprising: performing adaptive reordering of merge candidates (ARMC) by extending the reference template with two or four additional rows of pixels for motion vector refinement, and performing corresponding adaptive reordering of ARMC costs; Calculating a 9-point refined integer pixel distance cost; selecting the smallest cost among the 9-point refinement costs as a minimum refinement cost instead of the ARMC cost; reordering the merge candidates based on the minimum refinement cost; A method comprising:

7. 1. A method for applying template matching (TM) in video coding or decoding, the method comprising: combining an intra sub-partition mode (ISP) with a TM, where each sub-partition has its own intra mode, determined by applying the TM to refine the shared intra mode using either neighboring angular intra prediction modes or Most Probable Mode (MPM) modes; combining interpolation filtering and TM in intra prediction, where TM is applied to determine which interpolation filter to apply without explicitly signaling an intra prediction mode index; Combining block partitioning and TM, where for a given coding unit (CU), the best integer motion vector can be determined first using TM, and then copying the block partition from the best MV as the estimated partition for the current block; Combining CU-level weighted bi-prediction (BCW) and TM, where candidate weights are selected via template matching instead of being signaled; Combining adaptive motion vector resolution with TM, where instead of explicitly signaling the motion vector resolution, the resolution can be estimated using template matching; A method comprising one or more of the following:

8. A non-transitory computer-readable storage medium storing computer-executable instructions for executing the method of any one of claims 1 to 7 by one or more processors.

9. An apparatus including a processor and configured to perform the method of any one of claims 1 to 7.

Images