Application of template matching in video coding
By using constrained top and left neighbors, approximate reconstructed sample points and new processing pipelines in video encoding, combined with intra-mode and interpolation filtering, the hardware dependency and pipeline delay problems in template matching process are solved. , improves the compression efficiency and encoding performance of video encoding.
Patent Information
- Application Number
- CN202380033833.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-12
- Filing Date
- 2023-04-05
- Publication Date
- 2024-11-22
AI Technical Summary
In the existing video encoding technology, the template matching process has hardware dependence and pipeline delay problems, resulting in low compression efficiency and high complexity, and it is difficult to achieve optimization in hardware design.
By using constrained top and left neighbors in video encoding, template matching is enabled only at the boundary of the encoding tree unit, using an approximate reconstruction sample points and a new processing pipeline, combining intra-mode, sub-partition mode, and interpolation filtering and adaptive motion vector resolution, reducing dependency on immediate neighbors, and enabling template matching at CTU boundaries for parallelism.
Improves the compression efficiency of video encoding, reduces hardware and software complexity, reduces pipeline delay, and improves encoding performance without increasing hardware costs.
Smart Images

Figure CN119013985A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to Indian Provisional Patent Application No. 202241021946 filed on April 12, 2022, which is incorporated herein by reference in its entirety. Technical Field
[0003] This document relates generally to image and video coding. More specifically, embodiments of the invention relate to the application of template matching in video coding. Background Art
[0004] In 2020, the MPEG expert group within the International Organization for Standardization (ISO) and the International Telecommunication Union (ITU) jointly released the first version of the Versatile Video Coding Standard (VVC), also known as H.266 (reference [7]). Recently, the same expert group has been working on the development of a next-generation coding standard with improved coding performance compared to existing video coding techniques. As part of this investigation, new coding techniques are also being studied.
[0005] As appreciated by the inventors herein, improved techniques for applying template matching in image and video coding are desired and such techniques are described herein.
[0006] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any approach described in this section qualifies as prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise indicated, issues identified with respect to one or more approaches should not be assumed to be recognized in any prior art based on this section. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Embodiments of the invention are illustrated by way of example and not limitation in the accompanying drawings and in which like reference numerals refer to similar elements and in which:
[0008] Figure 1 An example of template matching in video coding is depicted; and
[0009] Figure 2 An example subdivision of a picture for processing coding units according to an embodiment of the present invention is depicted. DETAILED DESCRIPTION
[0010] This article describes example embodiments involving the application of template matching in video coding. In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of various embodiments of the present invention. However, it is apparent that various embodiments of the present invention can be practiced without these specific details. In other cases, in order to avoid unnecessarily obscuring, obscuring or confusing embodiments of the present invention, well-known structures and devices are not described in detail.
[0011] Overview
[0012] Example embodiments described herein relate to template matching (TM) in image and video coding. The proposed method includes: using constrained top and left neighbors in template matching, enabling TM only in coding tree unit boundaries, using approximate reconstructed samples, a new processing pipeline to derive decoder side intra mode derivation (DIMD) combined with template-based intra mode derivation (TIMD), and using filtered pixels from neighbors instead of reconstructed pixels. In addition, example embodiments describe how template matching can be applied in combination with intra modes, sub-partition modes, interpolation filtering in intra prediction, block partitioning, bidirectional prediction with coding unit level weights, and adaptive motion vector resolution.
[0013] Template Matching in Video Coding
[0014] Figure 1 An example of template matching (TM) in video coding is described (reference [1]). The term "template matching" refers to a decoder-side motion vector (MV) derivation method for refining the motion information of a current coding unit (CU) by finding the closest match between a template in the current picture (i.e., the top neighboring block and / or the left neighboring block (105) of the current CU) and a block in a reference picture (i.e., the same size as the template). Figure 1 As shown, in an embodiment, given an initial motion vector (110), a better MV is searched around the initial motion vector of the current coding unit (CU) within a [-8, +8] pixel search range (125). The search step size is determined based on the advanced motion vector resolution (AMVR) mode and can be cascaded with the bilateral matching process in merge mode TM.
[0015] In advanced motion vector prediction (AMVP) mode, motion vector predictor (MVP) candidates are determined based on template matching error to select one MVP candidate that achieves the minimum difference between the current block template (105) and the reference block template (115), and then TM is performed only on this specific MVP candidate for MV refinement. TM refines the MVP candidate by using an iterative diamond search starting with full-pixel motion vector difference (MVD) accuracy (or 4 pixels for 4-pixel AMVR mode) within a [-8, +8] pixel search range (125). The AMVP candidate can be further refined according to the AMVR mode by using a cross search with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode), followed by half-pixel and quarter-pixel accuracy in sequence. This search process ensures that the MVP candidate continues to maintain the same MV accuracy as indicated by the AMVR mode after the TM process.
[0016] In merge mode, a similar search method is applied to the merge candidates indicated by the merge index. Depending on whether an alternative interpolation filter is used based on the merged motion information (used when AMVR adopts half-pixel mode), TM can be performed all the way to 1 / 8 pixel MVD accuracy or skip those parts that exceed half-pixel MVD accuracy. In addition, when TM mode is enabled, template matching can work as an independent process between block-based and sub-block-based bilateral matching (BM) methods or as an additional MV refinement process, depending on whether BM can be enabled according to its enabling condition check.
[0017] As understood by the inventors, in the current version of the inter-frame template matching tool, the following observations can be made:
[0018] 1) A refined motion vector (MV) from a neighboring block is needed to start MV prediction for the current block
[0019] 2) The current template pixel (e.g., top left 4 rows) is needed, which is the reconstructed pixel from the neighboring block
[0020] 3) VVC uses the current pixel in the reshaped domain for intra prediction, but when using the current reconstructed pixel for template matching, inverse reshaping is required to return to the original domain because the reference template pixel is located in the original domain
[0021] 4) In HW pipeline design, the boundary strength calculation process needs to be delayed because of the use of refined MV
[0022] In addition to the inter-frame template matching tool, the idea of template matching is also widely used by other coding tools to help the decoder side make decisions by finding the closest match between the template in the current area (i.e., the top neighboring block and / or the left neighboring block of the current CU) and the reference area. Examples include:
[0023] 1) Intra-frame template matching (reference [2]): A special intra-frame prediction mode that copies the best prediction block from the reconstructed part of the current frame, whose L-shaped template matches the current template.
[0024] 2) Template based Intra Mode Derivation (TIMD) using most probable mode (MPM) (Reference [3]): This is an intra mode derivation method that uses the most probable mode (MPM) using neighboring templates. The TIMD mode is used as an additional intra prediction method for the CU.
[0025] 3) Adaptive re-ordering of merge candidates (ARMC) using TM (reference [4]): In this tool, after building a list of merge candidates, the merge candidates are divided into several subgroups. The merge candidates in each subgroup are reordered in ascending order according to the template matching-based cost value.
[0026] 4) MVD symbol prediction using TM (reference [5]): Here, if the motion vector difference contains non-zero components, motion vector difference symbol prediction can be applied in regular inter-frame mode. The possible MVD symbol combinations are sorted according to the template matching cost, and the index corresponding to the true MVD symbol is derived and encoded with the context model.
[0027] 5) TM with merge mode with motion vector difference (MMVD) (reference [5]): This is an extended MMVD reordering method based on template matching.
[0028] The embodiments proposed here aim to improve the template matching process from different aspects:
[0029] 1) Quality improvement (QI) proposals: aim to improve compression efficiency compared to current coding tools, preferably without any additional hardware (HW) implementation issues.
[0030] 2) Recommendations for HW dependency / pipeline issue elimination (HWPI) and hardware / software (HW / SW) complexity reduction (CR): Aims to address HW dependency / pipeline issues and coordinate similar tools by considering certain guidelines and HW / SW-friendly solutions (such as complexity reduction while minimizing the impact on compression efficiency)
[0031] 3) Extend the TM concept to other coding tools.
[0032] Use constrained top neighbors or left neighbors for TM to reduce dependencies on immediate neighbors in decoding order
[0033] Motivation: TM requires the immediate top and left neighboring reconstructed pixels for the template. This introduces a strong pipeline dependency in the decoding pipeline, since the reconstructed pixels of the immediate neighbors are needed to derive the motion information of the current CU.
[0034] Recommendation 1: Do not allow neighboring samples from the immediately previous CU to be TM, as follows:
[0035] ○ If the left CU is immediately adjacent to the previous CU in decoding order, use the top CU for TM. Figure 2 , CU3 will use the top samples (CU 1) to calculate the neighboring costs, but will not use the left samples (CU 2), which is its immediately previous CU in the decoder order. Similarly, CU 9 can use all top neighboring samples, but can only use some of the left neighboring samples from CU 7, because the left neighboring samples from CU 8 belong to the immediately previous CU in the decoding order.
[0036] ○ If the top CU is immediately adjacent to the previous CU in decoding order, the left CU is used for computing the neighboring cost. Figure 2 , CU 6 can use the left neighboring samples from CU 4 for TM, but cannot use the top neighboring samples from CU 5, which is the immediately previous CU in decoding order.
[0037] Suggestion 2: Do not allow neighboring samples from 'X' (X>1) previous CUs to be TM.
[0038] This is an extension of Proposal 1, where neighboring samples from multiple previous CUs in decoding order are disabled for TM to provide more HW parallelism. This is because TM is a relatively complex tool with many search and refinement stages. If the left neighbor or top neighbor cannot be used due to this constraint, then TM must be completely implicitly disabled for such CUs. For example, if X=2,
[0039] CU 9 can only be used Figure 2 The top sample point in
[0040] ○On the other hand, Figure 2 CU 3 and CU 6 in CAN not use left or top samples, so TM is implicitly disabled for them.
[0041] Suggestion 3: Do not allow neighboring samples of a certain region size from the previous CU to be TM. This is similar to Suggestion 2, but the number of prohibited CUs can be a variable instead of a constant value X.
[0042] TM is enabled only at CTU boundaries and constrains the use of the left neighbor.
[0043] This proposal allows TM to be enabled only at coding tree unit (CTU) boundaries to allow virtual pipeline data unit (VPDU) level parallelism for TM as described below:
[0044] - Top CTU neighboring samples are always available (except frame boundaries).
[0045] - The use of the left CTU neighboring samples is allowed only when the following conditions are met:
[0046] ○The current CTU root node is split into a quad-tree (QT) (N×N) or a horizontal binary tree (BT) (2N×N)
[0047] ○The left CTU root node is split into QT (N×N) or horizontal BT (2N×N)
[0048] ○ This ensures that the TM of the 4 N×N VPDUs in a CTU is pipelined as follows:
[0049] ■The VPDU0 boundary sample of the current CTU can use the VPDU1 of the left CTU as the left adjacent sample and the top CTU sample
[0050] ■The VPDU2 boundary sample of the current CTU can use the VPDU3 of the left CTU as the left adjacent sample, but cannot use the top sample
[0051] ■VPDU1 boundary samples can only use top CTU samples for TM
[0052] ■VPDU3 cannot use TM
[0053] ○ Note that CUs that are not part of the top or left CTU boundary cannot use TM.
[0054] TM using approximate reconstructed samples of top and left inter-prediction CUs
[0055] The motivation is to use the approximate reconstructed samples of the neighboring inter CUs for TM. The approximate reconstructed samples of the neighbors are derived by adding the filtered (e.g., bilinearly interpolated) prediction samples and the inverse transform residuals (top 4 or top 8) of the main transform coefficients based on the lookup table (LUT).
[0056] TM causes significant hardware pipeline delay for inter-frame reconstruction because it introduces a dependency on using neighboring samples for reconstruction. The current CU needs to wait for its top and left CUs to complete reconstruction (e.g., allowing reconstruction samples to be available) before starting the TM process. This proposal aims to reduce the pipeline delay by replacing the use of reconstruction samples with approximate reconstruction samples, so that the current CU can start the TM process as soon as the predicted and dequantized transform coefficients of the neighbors are available.
[0057] in short:
[0058] ReconSample_Actual=Pred+Res
[0059] Where Pred represents the predicted pixel, and Res = InvT(QuantCoeff), where InvT(QuantCoeff) represents the inverse transform of the quantization coefficient, and
[0060] ReconSample_Approximated=filtered(Pred)+LUT[QuantCoeff].
[0061] The LUT-based operation on the dequantized transform coefficients is a fast approximation for estimating the residual without performing the actual inverse transform. The idea of using filtering for prediction is similar to the motivation of adaptive loop filtering (ALF) to improve the accuracy of the approximation. However, for the sake of reducing complexity, the filtering to be applied here should not be too complex.
[0062] For the case of luma mapping and chroma scaling (LMCS), when enabled in VVC, inter prediction is in the original domain, while reconstruction and residual are in the shaped domain, so a mapping operation (LMCSFwdMap) is required. Prediction filtering can be used in either domain.
[0063] ReconSample_Actual=LMCSFwdMap[Pred]+Res,
[0064] Where Res = InvT(QuantCoeff).
[0065] ReconSample_Approximated=filtered(Pred)+LUT[QuantCoeff],
[0066] in,
[0067] filtered(Pred)=filtered(LMCSFwdMap[Pred]) or LMCSFwdMap[filtered(Pred)].
[0068] The following restrictions / modifications can be further applied to reduce TM complexity:
[0069] -TM uses 4 neighboring samples, while symbol prediction uses 2 neighboring samples.
[0070] ○ TM and symbol prediction can be coordinated to use only two neighboring samples, thereby reducing the overhead of reconstructing inter-frame CUs using LUT approximation.
[0071] - Another option is to use only the left and top inter-prediction samples for inter-TM (without using the approximation residual).
[0072] o This assumes that the inter CU has less residual (smaller number of small coded coefficients) and hence the prediction has most of the information for the reconstructed CU.
[0073] ○ To strengthen this assumption, constraints can also be applied, such as using inter-prediction samples from neighboring CUs that have fewer coded coefficients (e.g., less than 3 coefficients) coded in the neighboring CUs
[0074] Restricted use of MVs for TM refinement during merging and MV list construction
[0075] Motivation: Using TM-based MV or merge mode introduces strong pipeline dependencies in the decoding pipeline, since the reconstructed pixels of the immediate neighbors are needed to derive the merge list / AMVP list of the current CU, which makes almost all HW decoding operations serialized at the CU level.
[0076] Recommendation: Limit the use of TM-refined MVs for motion vector prediction processes, such as merge list and AMVP list construction, as shown below:
[0077] 1. Use TM refined motion only at CTU boundaries (only from top, top left and top right CTUs). Advance TM motion information for other spatial neighbors
[0078] 2.TM-refined MVs can be used for other forward dependencies, such as boundary strength (BS) calculation and temporal MV storage.
[0079] 3. Tools such as ARMC, TM with MMVD, TM with MVD symbols, etc. must be completely disabled or enabled only at the top CTU boundary.
[0080] Coordination of intra prediction process across DIMD and TIMD
[0081] Motivation: Decoder-side intra mode derivation (reference [6]) is a new tool in the current enhanced compression model (ECM) in JVET. The DIMD process uses a fusion of three intra modes, while TIMD uses a fusion of one or two intra modes. TIMD uses the DIMD mode to decide the best mode based on the template cost, so the TIMD process is the worst case in terms of HW processing latency. Coordinating various aspects of DIMD and TIMD can help improve compression efficiency or reduce HW complexity. In ECM, the following simplified notation can be used to describe the computational engines required for DIMD and TIMD modes:
[0082] C0: MPM list derivation
[0083] Input: Neighborhood Intra Mode
[0084] Output: A collection of patterns
[0085] C1: Gradient histogram, and find the intra-frame mode set from the magnitude of the gradient histogram
[0086] Input: Current neighboring reconstructed pixels
[0087] Output: First two intra modes based on TM cost
[0088] C2: Calculate the TIMD cost for a given set of intra modes and select the top 2 intra modes
[0089] Input: Current neighboring reconstructed pixels and intra mode set
[0090] Output: First two intra modes based on TM cost
[0091] C3: Fusion of three intra-frame modes (one fixed as planar mode)
[0092] Input: Two intra modes and one planar mode
[0093] Output: Final intra-frame prediction data
[0094] C4: Fusion of two intra-frame modes
[0095] Input: Two intra modes
[0096] Output: Final intra-frame prediction data
[0097] Current methods in ECM:
[0098] DIMD mode: C1+C3
[0099] TIMD mode: C0+C1+C2+C4
[0100] Recommendation 1
[0101] DIMD mode: C1 (a collection of N modes) + C2 + C4
[0102] TIMD mode: C0+C1 (only 2 modes)+C2+C4
[0103] Recommendation 2
[0104] DIMD mode: C1 (a collection of N modes) + C2 + C3
[0105] TIMD mode: C0+C1 (only 2 modes)+C2+C3
[0106] The term "N modes" refers to N intra prediction modes, such as angular mode, DC mode, planar mode, and the like.
[0107] Reducing on-chip memory for intra-TM
[0108] Motivation: Intra TM has very high on-chip memory requirements compared to Intra block copy (IBC). If the current CU size is large, the current intra template matching technique in ECM uses the top pixels from several top CTU rows.
[0109] Recommendation: Limit the use of current reconstructed pixels from the top CTU row. Allow 4 bottom lines of reconstructed pixels from the top CTU, as this is already used for TM or intra prediction. Limit on-chip memory size to a*CTU size, where scaler a can be 1 to 5.
[0110] Using filtered predictions as an alternative to neighboring reconstructed pixels
[0111] Motivation: TM requires top and left neighboring reconstructed pixels for template construction. This introduces strong pipeline dependencies in the decoding pipeline, since the reconstructed pixels of the neighbors are needed to deduce the motion information of the current CU.
[0112] Proposal: For template construction, instead of reconstructed pixels, predictions from neighbors (albeit filtered versions) may be used. In an embodiment, the filter may be a Wiener filter, which may be derived using the statistical properties of the predicted and reconstructed pixels from the region pointed to by the unrefined MV in the reference frame. This process is only applied if at least one neighbor is inter coded. The proposal aims to find a suitable substitute for using reconstructed pixels from neighbors. The predictions of the neighbors can be thought of as noisy versions of the neighbor reconstructions. Therefore, if one were to find a linear filter to apply to the prediction such that the difference between the output of the filter and the actual reconstructed pixels is minimized, then assuming the constraints of the linear filter, the best substitute would be found. For example, in Figure 1 In , template 105 can be filtered because it is the hardware pipeline bottleneck. (In contrast, in template 115, all reconstruction samples are already available). TM needs to use the reconstructed samples of 105 (InterRecon happens in a later pipeline stage). The suggestion is to use a filtered version of the prediction in 105 instead of the reconstruction of 105, so that TM of the current CU can start immediately after the neighboring CU has available prediction samples, which happens in an early pipeline stage. For example,
[0113] ReconSample_Approximated=f(Pred),
[0114] Here, f() is some filtering operation, such as a Wiener filter, a nonlinear filter, a neural network-based filter, etc.
[0115] The filter coefficients should be derived using the reconstructed pixels from the reference region pointed to by the unrefined TM MV as reference signal and the prediction from the same region as the noisy version of the reference signal.
[0116] Here are a few variations on this suggestion that address the various complexities hinted at in the paragraphs above:
[0117] 1) Instead of deriving one filter for each TM call, a predefined set of filters can be signaled via an adaptation parameter set (APS), and a process is defined to derive the filter index based on prediction and reconstruction statistics or some other properties.
[0118] 2) Since the use of neighbor prediction means waiting for motion compensation (MC) of the neighboring CU, which will require waiting for the TM of the neighboring CU and the refined MV output of the MV refinement algorithm, the MV is used to derive the neighbor prediction of the current CU for template construction before MV refinement starts.
[0119] Coordination of ARMC and TM
[0120] Motivation: Improve the coding efficiency of ARMC. The improved ARMC compensates for any coding efficiency loss by removing TM refinement.
[0121] Suggestion: Adaptive reordering of merge candidates with TM refinement (ARMC-R), in an embodiment, includes the following steps:
[0122] - Starting from ARMC-TM, it has brought reference data for each merge candidate. In current ARMC, top and left reference template areas are used, where the size of the top reference template area is (BlkWidth×4) and the size of the left reference template area is (4×BlkHeight).
[0123] - In an embodiment, the reference template may be improved by adding pixels around it so that a refinement of + / -1MV can be performed. Thus, the size of the top reference template area will be ((BlkWidth+2)x(4+2)) and the size of the left template area will be ((4+2)×(BlkHeight+2)). The "+2" represents an additional row of pixels around the reference template area for a refinement of + / -1MV. For a refinement of + / -2MV, a "+4" would be required.
[0124] -9-point (including center, square pattern) integer pixel distance costs will be evaluated. Error surface-based sub-pixel costs can also be derived from 9-point costs, as used in VVC's decoder-side, motion vector refinement (DMVR).
[0125] - Instead of merging MV TM costs, select the minimum cost from the 9-point refinement of each merge candidate
[0126] - Reorder the merge list based on minimum refinement cost.
[0127] -Remove duplicate MVs.
[0128] Method 1: (QI) Replace ARMC-TM with the proposed ARMC-R.
[0129] Method 2: (CR) Remove TM refinement based on method 1, because method 1 has already covered this refinement.
[0130] Simplification of TM-based MV refinement for integer pixel refinement
[0131] Motivation: Template matching based MV refinement methods are strongly sequential, involving interpolation and template cost computation using diamond patterns, followed by a cross as the final step.
[0132] Recommendation: Use the integer MV position corresponding to the motion vector from the merge / AMVP list as the starting point for the search. This helps avoid the need for interpolation of integer pixel refinement.
[0133] The search range will be restricted to an optimal value such that the TM costs for all integer pixel positions around the center can be calculated in parallel. For a search range of + / -2 pixels around the center, a total of 25 points of TM costs need to be calculated, and for a search range of + / -3 pixels around the center, a total of 49 points of TM costs need to be calculated.
[0134] Extending TM concepts to other tools
[0135] Intra sub-partition mode (ISP) plus TM: In VVC, intra prediction blocks can be subdivided horizontally or vertically into smaller blocks, called sub-partitions. On each sub-partition, prediction and transform coding operations are performed separately, but all sub-partitions share the intra mode. In an embodiment, it is recommended to combine ISP with TM to allow each sub-partition to have a different intra mode. The basic idea is to use TM to refine the shared intra mode of each sub-partition using the neighboring angle intra prediction mode or the most probable mode (MPM) mode of the block partition.
[0136] Intra-frame prediction plus interpolation filtering in TM: In VVC, interpolation filtering is applied in fractional slope mode. For luminance, the interpolation filter represents a 4-tap DCT-based interpolation filter (DCTIF) or a 4-tap smoothing interpolation filter (SIF). The type of interpolation filter is not signaled in the bitstream, but is determined based on the size of the block and the intra-frame prediction mode index. In an embodiment, one method is to use TM to decide which IF should be used without explicit signaling. It is also possible to add more candidate IFs to the pool, such as 8-tap DCTIF or SIF, and let TM decide which IF to use for optimal coding efficiency.
[0137] Block partition plus TM: For a given CU, TM can be used to find the best integer MV. Then, the block partition can be copied from the best MV as the inferred partition of the current block. This is to save bits used for partitioning.
[0138] BCW (bidirectional prediction with CU-level weights) plus TM: In VVC, for BCW, a candidate set of weight values can be selected for bidirectional inter-frame prediction. If allowed, the index of the selected weight value is signaled for AMVP mode and inherited for merge mode. In an embodiment, BCW can be improved by TM in two aspects: 1) Use TM to avoid signaling weight index; 2) Allow more weights and use TM to select a limited set for signaling.
[0139] Adaptive motion vector resolution with TM: Instead of explicitly signaling the motion vector resolution, the resolution can be inferred based on the TM. Basically for TM, different motion vector resolution (MVR) techniques can be tried, and the resolution with the best MV is the resolution of the current CU.
[0140] References
[0141] Each of the references listed herein is incorporated herein by reference in its entirety.The term JVET refers to the Joint Video Experts Group of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29.
[0142] [1] JVET-U0100, “Compression efficiency methods beyond VVC,” Y.-J. Chang et al., conference call, January 2021.
[0143] [2] JVET-V0130, “EE2: Intra Template Matching,” K. Naser et al., conference call, April 2021.
[0144] [3] JVET-V0098, “EE2-related: Template-based intra mode derivation using MPMs,” Y. Wang et al., conference call, April 2021.
[0145] [4] JVET-W0090, “EE2-3.1 / EE2-3.2: Adaptive Reordering of MergeCandidates with Template / Bilateral Matching,” N. Zhang et al., conference call, July 2021.
[0146] [5] JVET-Y0067, “EE2-3.9and EE2-3.10: TM based reordering for MMVD andaffine MMVD and MVD sign prediction”, M. Salehifar et al., conference call, January 2022.
[0147] [6] JVET-O0449, “Non-CE3: Decoder-side Intra Mode Derivation (DIMD) with prediction fusion using Planar”, M. Abdoli et al., Gothenburg, July 2019.
[0148] [7] “Versatile Video Coding”, Rec. ITU-T H.266, August 2020.
[0149] Example Computer System Implementation
[0150] Embodiments of the present invention may be implemented using a computer system, a system configured with electronic circuits and components, an integrated circuit (IC) device (such as a microcontroller, a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or implement instructions related to applying template matching in image and video encoding, such as those described herein. The computer and / or IC may calculate any of the various parameters or values related to applying template matching in image and video encoding described herein. The image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.
[0151] Some embodiments of the present invention include a computer processor that executes software instructions, which cause the processor to perform the method of the present invention. For example, one or more processors in a display, an encoder, a set-top box, a transcoder, etc. can implement the method related to applying template matching in image and video coding as described above by executing software instructions in a program memory accessible to the processor. An embodiment of the present invention can also be provided in the form of a program product. The program product may include any non-transient tangible medium that carries a set of computer-readable signals including instructions, which cause the data processor to perform the method of the present invention when executed by a data processor. The program product according to the present invention may take any form in various non-transient and tangible forms. The program product may include, for example, a physical medium, such as a magnetic data storage medium including a floppy disk, a hard disk drive, an optical data storage medium including a CD ROM, a DVD, an electronic data storage medium including a ROM, a flash RAM, etc. The computer-readable signal on the program product may be optionally compressed or encrypted.
[0152] Where components (e.g., software modules, processors, components, devices, circuits, etc.) are mentioned above, unless otherwise indicated, references to such components (including references to "means") should be interpreted to include any components that perform the function of the described components as equivalents (e.g., functionally equivalent) to such components, including components that are not structurally equivalent to the disclosed structures that perform the functions in the illustrated example embodiments of the invention.
[0153] Equivalents, extensions, alternatives and miscellaneous
[0154] Thus described are example embodiments related to the application of template matching in image and video coding. In the foregoing description, embodiments of the present invention have been described with reference to many specific details, which may vary from implementation to implementation. Therefore, the only and exclusive indication of the present invention and the applicant's inventive intent is the set of claims issued in a specific form according to the present application, wherein such claim issuance includes any subsequent corrections. Any definitions expressly set forth herein for terms contained in such claims should govern the meaning of such terms as used in the claims. Therefore, limitations, elements, properties, features, advantages or attributes not expressly cited in a claim should not limit the scope of such claim in any way. Therefore, this specification and drawings should be viewed in an illustrative rather than a restrictive sense.
Claims
1. A method for processing one or more images using template matching, the method comprising: Receiving a picture having a current coding unit (CU) and a previously decoded coding unit; as well as Template matching is applied using the neighboring pixel area of the current CU, wherein in the template matching, the neighboring CU is constrained as follows: If the left CU is immediately adjacent to the previous CU in decoding order, the top CU is used to calculate the neighbor cost; If the top CU is immediately adjacent to the previous CU in decoding order, then using the left CU to calculate the neighboring cost; or Neighboring samples from X (X>1) previous CUs are not allowed.
2. A method for processing one or more images using template matching, the method comprising: Receiving a picture having a current coding unit (CU) and a previously decoded coding unit; as well as Template matching is applied using a neighboring pixel region of the current CU, wherein template matching is constrained at a coding tree unit (CTU) as follows: Top CTU neighboring samples are always available (except frame boundaries); The use of the left CTU neighboring samples is allowed only when the following conditions are met: ○ The current CTU root node is split into a quadtree (QT) (N×N) or a horizontal binary tree (BT) (2N×N); as well as ○The left CTU root node is split into QT (N×N) or horizontal BT (2N×N).
3. A method for processing one or more images using template matching, the method comprising: Receiving a picture having a current coding unit (CU) and a previously decoded coding unit; as well as Applying template matching using the neighboring pixel area of the current CU, wherein in the template matching, samples of the neighboring CU are derived based on using one or more of the following: The prediction samples or approximate residuals after filtering.
4. A method for processing one or more images using template matching, the method comprising: Receiving a picture having a current coding unit (CU) and a previously decoded coding unit; as well as Template matching is applied using a neighboring pixel area of the current CU, wherein in the template matching, only inter-frame prediction samples of neighboring CUs are used.
5. The method of claim 3, wherein: Using the statistical properties of the predicted and reconstructed pixels, filters for generating filtered prediction samples are derived from the region pointed to by the unrefined motion vector in the reference frame.
6. A method for decoder-side intra mode derivation (DIMD) in combination with template-based intra mode derivation (TIMD), the method comprising computing: DIMD mode: by combining C1 (generating a set of N modes), C2, and C4; TIMD mode: by calculating C0, C1 (only 2 modes are selected), C2 and C4; Or calculate: DIMD mode: by calculating C1 (generating a set of N modes), C2 and C3; TIMD mode: By calculating C0, C1 (only 2 modes are selected), C2 and C3, in, C0, C1, C2, C3 and C4 include: C0: MPM list derivation, with input: neighboring intra patterns, and output: a set of patterns; C1: gradient histogram, and find the intra mode set from the magnitude of the gradient histogram, with C1 input: the current neighboring reconstructed pixels, and C1 output: the intra modes of the first two based on the TM cost; C2: Calculate the TIMD cost for a given set of intra modes and select the top two intra modes, with C2 input: current neighboring reconstructed pixels and the set of intra modes, and C2 output: top two intra modes based on TM cost; C3: Fusion of three intra modes (one fixed as planar mode), with C3 input: two intra modes and one planar mode, and C3 output: final intra prediction data; C4: Fusion of 2 intra modes, with C4 input: two intra modes, and C4 output: final intra prediction data.
7. A method for adaptively reordering merging candidates (ARMC) using template matching, the method comprising: performing adaptive reordering of merge candidates (ARMC) and corresponding ARMC costs for motion vector refinement by enlarging the reference template by two or four additional rows of pixels; Calculate the 9-point refinement integer pixel distance cost; In place of the ARMC cost, a minimum cost is selected among the 9-point refinement costs as a minimum refinement cost; as well as The merge candidates are reordered based on the minimum refinement cost.
8. A method of applying Template Matching (TM) in video encoding or decoding, the method comprising one or more of the following: The intra-frame sub-partitioning mode (ISP) is combined with TM, where Each sub-partition has its own intra mode, which is determined by applying TM to refine the shared intra mode using the neighboring angle intra prediction mode or the most probable mode (MPM) mode; Combining interpolation filtering in intra prediction with TM, wherein TM is applied to determine what interpolation filter to apply without explicitly signaling its identity index; Block partitioning is combined with TM, where for a given coding unit (CU), First, the TM can be used to determine the best integer motion vector; and Then copy the block partition from the best MV as the inferred partition of the current block; Combining BCW (Bidirectional Prediction with CU Level Weights) with TM, where weighted value candidates are selected via template matching instead of being signaled; and Adaptive motion vector resolution is combined with TM, where the motion vector resolution can be inferred using template matching instead of explicitly signaling the resolution.
9. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon, the instructions being for executing the method according to any one of claims 1 to 8 using one or more processors.
10. An apparatus comprising a processor and configured to perform any of the methods of claims 1 to 8.
Citation Information
Cited By
Intra-prediction method with side template matching and apparatus incorporating the same
WO2026148463A1
