Video encoding and decoding method and apparatus for MV refinement or candidate reordering using template matching

Abstract

The invention discloses a video encoding / decoding method and apparatus employing an adaptive cost function setting. The method first determines one or more templates for the current block. Then, based on size information associated with the current block, the one or more templates, or both, a target cost function or a target weighted coefficient set is selected from multiple cost functions or multiple sets of weighted coefficients. Next, based on the one or more templates, a target cost is determined using the selected target cost function or target weighted coefficient set. Finally, a target encoding / decoding tool belonging to the encoding / decoding tool group is applied to the current block, wherein the target encoding / decoding tool performs target processing based on information including the target cost.

Description

[0001] Cross-referencing This invention is a non-provisional application, U.S. Provisional Patent Application No. 63 / 621,615, filed January 17, 2024, and claims priority to the aforementioned provisional application. The entire contents of the aforementioned U.S. Provisional Patent Application are incorporated herein by reference. [Technical Field] This invention relates to video encoding and decoding systems. More specifically, it relates to setting a cost function for cost function evaluation of one or more templates within a video encoding and decoding system. [Background Technology] Versatile Video Coding (VVC) is the latest international video coding standard developed by the ITU-T Video Coding Experts Group (VCEG) and the Joint Video Experts Team (JVET) of the International Organization for Standardization / International Electrotechnical Commission (ISO / IEC) Moving Picture Experts Group (MPEG). This standard has been published as an ISO standard: ISO / IEC 23090-3:2021, Information technology—Coding representation of immersive media—Part 3: Versatile Video Coding, published in February 2021. VVC was developed based on its predecessor, High Efficiency Video Coding (HEVC), by adding more encoding and decoding tools to improve coding efficiency and enabling it to handle various types of video sources, including 3D video signals.

[0004] Figure 1AAn example adaptive inter-frame / intra-frame video coding system incorporating loop processing is demonstrated. For intra-frame prediction 110, the prediction data is derived from previously encoded video data in the current image. For inter-frame prediction 112, motion estimation (ME) is performed at the encoder, and motion compensation (MC) is performed based on the results of ME to provide prediction data derived from other images and motion data. Switch 114 selects either intra-frame prediction 110 or inter-frame prediction 112, and the selected prediction data is provided to adder 116 to form a prediction error, also known as a residual. The prediction error is then processed by transform (T) 118, followed by quantization (Q) 120. The transformed and quantized residual is then encoded by entropy encoder 122 to be included in the video bitstream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packaged with side information, such as the motion and coding modes associated with intra-frame and inter-frame prediction, and other information such as parameters associated with loop filters applied to the underlying image regions. Side information related to intra-frame prediction 110, inter-frame prediction 112, and loop filter 130, such as Figure 1A The reference image is provided to the entropy encoder 122. When using inter-frame prediction mode, the reference image must also be reconstructed at the encoding end. Therefore, the transformed and quantized residuals are processed by inverse quantization (IQ) 124 and inverse transformation (IT) 126 to recover the residuals. The residuals are then added back to the prediction data 136 at reconstruction (REC) 128 to reconstruct the video data. The reconstructed video data can be stored in the reference image buffer 134 and used for prediction of other frames.

[0005] like Figure 1A As shown, the input video data undergoes a series of processes in the encoding system. The reconstructed video data from REC 128 may be subject to various impairments due to these processes. Therefore, a loop filter 130 is typically applied to the reconstructed video data before it is stored in the reference image buffer 134 to improve video quality. For example, a deblocking filter (DF), sample adaptive offset (SAO), and adaptive loop filter (ALF) can be used. Loop filter information may need to be included in the bitstream so that the decoder can correctly recover the required information. Therefore, loop filter information is also provided to the entropy encoder 122 for inclusion in the bitstream. Figure 1A In this process, the loop filter 130 is applied to the reconstructed video before the reconstructed video data is stored in the reference image buffer 134. Figure 1A The system in this document is designed to demonstrate an example architecture of a typical video encoder, which may correspond to the High Efficiency Video Coding (HEVC) system, VP8, VP9, ​​H.264, or VVC.

[0006] like Figure 1B As shown, the decoder can use the same or partially the same functional modules as the encoder, except for transform 118 and quantization 120, because the decoder only needs inverse quantization 124 and inverse transform 126. The decoder uses entropy decoder 140 to decode the video bitstream into quantized transform coefficients and the required coding information (e.g., ILPF information, intra-frame prediction information, and inter-frame prediction information), instead of entropy encoder 122. Intra-frame prediction 150 at the decoder end does not require mode search; the decoder only needs to generate intra-frame predictions based on the intra-frame prediction information received from entropy decoder 140. Furthermore, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) based on the inter-frame prediction information received from entropy decoder 140, without performing motion estimation.

[0007] In VVC and emerging video coding, several new encoding and decoding tools have been released to improve encoding efficiency. This release focuses on techniques related to template matching.

[0008] Template Matching (TM) Template matching is a decoding-side motion vector (MV) derivation method that refines the motion information of the current CU by finding the closest match between the template of the current coding unit (CU) in the current image (i.e., the top and / or left neighboring blocks of the current CU) and a template of the same size in the reference image. For example... Figure 2 As shown, a better MV (i.e., a refined MV) is searched within a search range of [–8, +8] pixels around the initial motion of the current CU. The template matching method in JVET-J0021 has been modified as follows: the search step size is determined according to the AMVR mode, and the TM can be cascaded with the bilateral matching process in the merge mode.

[0009] In Advanced Motion Vector Prediction (AMVP) mode, motion vector prediction (MVP) candidates are determined based on template matching error. The candidate with the smallest difference between the current block template and the reference block template is selected, and then TM is performed only on that specific MVP candidate for MV refinement. TM refines the MVP candidate, starting with full-pixel MVD precision (or 4 pixels for 4-pixel AMVR mode) and employing an iterative 16-point diamond search within a search range of [–8, +8] pixels. AMVP candidates can be further refined through cross-search at full-pixel MVD precision (or 4 pixels for 4-pixel AMVR mode), followed by half-pixel and quarter-pixel searches sequentially according to the AMVR modes shown in Table 1. This search process ensures that the MVP candidate remains consistent with the MV precision indicated by the AMVR mode after TM processing. During the search process, if the difference between the previous minimum cost and the current minimum cost in an iteration is less than or equal to a threshold of the block area, the search process terminates.

[0010] Table 1. Search patterns for AMVR and merged modes with AMVR.

[0011] In merge mode, a similar search method is used for merge candidates indicated by the merge index. As shown in Table 1, TM can proceed up to 1 / 8 pixel MVD precision, or skip portions beyond half-pixel precision, depending on whether an alternative interpolation filter (used in AMVR in half-pixel mode) is used based on the merge motion information. Furthermore, when TM mode is enabled, template matching can be performed as a standalone process or as an additional MV refinement process between block-level and sub-block-level bilateral matching (BM) methods, depending on whether BM can be enabled according to its enable condition check.

[0012] When template matching (TM) is applied to bidirectional prediction blocks, an iterative process is employed. Specifically, the initial motion vectors of L0 and L1 are first refined, and the TM costs Cost0 and Cost1 for L0 and L1 are calculated respectively. When Cost0 is greater than Cost1, the refined motion vector of L1 is used. To derive further refined motion vectors of L0 () Then, use right Further refinement is performed. Similarly, when Cost0 is not greater than Cost1, the refined motion vector of L0 is used ( To derive further refined motion vectors of L1 ( ), and use right Further refinement is needed. Additionally, when the DMVR conditions are met, TM for bidirectional prediction is enabled.

[0013] TM-based sub-block motion refinement In JVET-AF0168 test3.4a, template matching is proposed for applying to sub-block-based motion tools, including affine and SbTMVP modes. More specifically, the control point motion vectors (CPMVs) of unidirectional predictive affine merging candidates and the motion offsets of SbTMVP candidates are refined using TM. For unidirectional predictive affine merging candidates, the same MV offset is assigned to all CPMVs, and the TM cost of that affine candidate is calculated accordingly. The optimal CPMV offset with the minimum TM cost can be used to refine the corresponding affine candidate. For SbTMVP candidates, the initial motion offset is refined using TM, and the refined motion offset is then used to derive the sub-block temporal domain.

[0014] Commonly used cost functions for template matching Commonly used cost functions for TM processing include the following: 1. SAD stands for Sum of Absolute Differences. 2. SSD stands for Sum of Square Differences. 3. SATD stands for Sum of Absolute Transformed Differences. The following is a brief overview of some codec tools that can be combined with TM for MV refinement, candidate reordering, or predictor generation.

[0015] Adaptive Merge Candidate Reordering (ARMC) In JVET-V0099 (Na Zhang et al., “AHG12: Adaptive Reordering of MergeCandidates with Template Matching”, ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG11 Joint Video Expert Group (JVET), 22nd Meeting, Remote Conference, April 20-28, 2021, Document: JVET-V0099), an adaptive reordering method for merge candidates based on template matching (ARMC) was proposed. This reordering method is applied to regular merge mode, template matching (TM) merge mode, and affine merge mode (excluding SbTMVP candidates). For TM merge mode, merge candidates are reordered before refinement.

[0016] After constructing the merge candidate list, the merge candidates are divided into several subgroups. The subgroup size is set to 5. Within each subgroup, the merge candidates are reordered in ascending order based on the cost value based on template matching. To simplify processing, the merge candidates in the last subgroup are not reordered unless there is only one subgroup.

[0017] Template matching cost is measured by the sum of absolute differences (SAD) between the current block template sample and its corresponding reference sample. The template includes a set of reconstructed samples from neighboring current blocks. The reference sample of the template is located using the same motion information of the current block.

[0018] Intra-block copy (IBC) mode Motion compensation is a key technique in hybrid video coding and decoding, used to uncover pixel correlations between adjacent images. It typically assumes that in a video sequence, a pattern corresponding to an object or background in one frame shifts, forming a corresponding object in a subsequent frame, or relating to other patterns within the current frame. By estimating this shift (e.g., using block matching), the pattern can be substantially reproduced without re-encoding it. Similarly, block matching and duplication have been attempted to allow the selection of reference blocks from the same image as the current block. However, applying this concept to camera-captured video has been found to be inefficient. This is partly because while the texture patterns in spatially neighboring regions may be similar to the currently coded block, they often exhibit some spatial gradients. In camera-captured video, it is difficult to find a completely matching block within the same image for a given block. Therefore, the improvement in coding performance is limited.

[0019] However, the spatial correlation between pixels within the same image differs for screen content. For typical videos containing text and graphics, recurring patterns often exist within the same image. Therefore, intra-block (image) compensation is considered highly effective. To leverage this characteristic, a new prediction mode for screen content coding has been introduced: Intra-Block Copy (IBC) mode, also known as Current Image Reference (CPR). In CPR mode, prediction units (PUs) are predicted from previously reconstructed blocks within the same image. Furthermore, a displacement vector (called the block vector or BV) is used to indicate the relative displacement between the current block position and the reference block position. The prediction error is then encoded through transform, quantization, and entropy coding.

[0020] Intra-frame template matching prediction Intra-Template Matching Prediction (IntraTMP) is a special intra-prediction mode that copies the best prediction block of an L-shaped template that matches the current template in the reconstructed portion of the current frame. For a predefined search range, the encoder searches for the template most similar to the current template in the reconstructed portion of the current frame and uses the corresponding block as the prediction block. The encoder then indicates the use of this mode and performs the same prediction operation at the decoder.

[0021] Joint Inter-Frame and Intra-Frame Prediction (CIIP) In VVC, when a coding unit (CU) is encoded in merged mode, if the CU contains at least 64 luma samples (i.e., CU width multiplied by CU height equals or greater than 64), and both the CU width and CU height are less than 128 luma samples, an additional flag is sent to indicate whether Joint Inter-Frame and Intra-Frame Prediction (CIIP) mode is applied to the current CU. As its name suggests, CIIP prediction combines inter-frame prediction signals with intra-frame prediction signals. The inter-frame prediction signal in CIIP mode... The intra-frame prediction signal is derived through the same inter-frame prediction processing as the regular merging mode. The intra-frame prediction signal is then derived using the planar mode in conventional intra-frame prediction processing. Subsequently, the intra-frame and inter-frame prediction signals are combined by a weighted average, where the weight value wt is calculated based on the coding modes of the top and left neighboring blocks of the current CU.

[0022] Merge pattern with MVD (MMVD) In addition to the merge mode (in which implicitly derived motion information is directly used to generate prediction samples for the current CU), VVC also introduces a merge mode with motion vector difference (MVD) (MMVD). The MMVD flag is sent immediately after the regular merge flag to specify whether the CU uses MMVD mode.

[0023] In Merge Mode with MVD (MMVD), after selecting a merge candidate, the MVD information is further refined via signaling. This further information includes a merge candidate flag, an index specifying the motion amplitude, and an index indicating the motion direction. In MMVD mode, one of the first two candidates in the merge list is selected as the MV basis. The MMVD candidate flag specifies via signaling which candidate to use between the first and second merge candidates.

[0024] The distance index specifies motion amplitude information and indicates a predefined offset of the starting points of the L0 and L1 reference blocks. The offset is added to the horizontal or vertical component of the starting MV.

[0025] Geometric Partitioning Mode (GPM) In VVC, Geometric Partitioning (GPM) supports inter-frame prediction, as described in JVET-W2002 (Adrian Browne et al., Description of the Versatile Video Coding and Test Model 14 (VTM 14) Algorithm, ITU-T / ISO / IEC Joint Video Exploration Group (JVET), 23rd Meeting, Remote Conference, July 7-16, 2021, Document: JVET-M2002). GPM signals via CU-level flags as a merging mode; other merging modes include regular merging, MMVD, CIIP, and sub-block merging. For each possible CU size, GPM supports a total of 64 partitions. ,in Excluding 8x64 and 64x8. GPM mode can be applied to skip or merge CUs whose size is within the above range and which have at least two regular merge modes.

[0026] Bi-Prediction with CU-level Weight (BCW) In HEVC, bidirectional predictive signals, By analyzing two prediction signals, and The two predicted signals are generated by averaging, with each signal derived from a different reference image and / or using different motion vectors. In VVC, the bidirectional prediction mode is extended to allow a weighted average of the two predicted signals.

[0027] (3) Five weights are allowed in the weighted average two-way forecast. For each bidirectional prediction CU, the weight w is determined in one of two ways: 1) for non-merged CUs, the weight index is signaled after the motion vector difference; 2) for merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is applied only to CUs with 256 or more luminance samples (i.e., CU width multiplied by CU height is greater than or equal to 256). For low-latency images, all five weights are used. For non-low-latency images, only three weights are used (w ∈ {3,4,5}). At the encoder, a fast search algorithm is employed to find the weight index without significantly increasing encoder complexity. These algorithms are summarized below.

[0028] This invention discloses a method and apparatus for adaptively selecting cost function settings associated with certain codec tools that use template matching for candidate reordering, MV refinement, or predictor generation. [Summary of the Invention] A video encoding / decoding method and apparatus using an adaptive cost function setting are disclosed. According to the method, input data associated with a current block is received, wherein the input data includes pixel data to be encoded at the encoding end, or encoded data associated with the current block to be decoded at the decoding end. One or more templates for the current block are determined. Based on size-related information associated with the current block, the one or more templates, or both, a target cost function or a target weighted coefficient set is selected from a plurality of cost functions or a plurality of weighted coefficient sets. Using the selected target cost function or target weighted coefficient set, a target cost is determined based on the one or more templates. A target encoding / decoding tool belonging to a set of encoding / decoding tools is applied to the current block, wherein the target processing procedure of the target encoding / decoding tool is performed based on information including the target cost.

[0030] In one embodiment, the size-related information includes width, height, block size, aspect ratio, or a combination thereof. In one embodiment, if the width is greater than twice the height, or the height is greater than twice the width, the top template and the left template use different sets of weighting factors.

[0031] In one embodiment, the first set of spatial weighting coefficients for the top template contains more elements than the second set of spatial weighting coefficients for the left template.

[0032] In one embodiment, one or more additional sets of distance-weighted coefficients and / or one or more additional sets of spatial-weighted coefficients are used. In one embodiment, the plurality of cost functions include sum of absolute differences (SAD), weighted SAD, sum of squared differences (SSD), weighted SSD, sum of absolute transformed differences (SATD), weighted SATD, or a combination thereof.

[0033] In one embodiment, the codec toolset includes Adaptive Reordering of Merge Candidates (ARMC), Template Matching (TM) for Affine Merging, Regular Merging, CIIP Merging, Bilateral Matching (BM) Merging, or a combination thereof. In one embodiment, the codec toolset includes motion vector differences (MVD) symbol prediction for Advanced Motion Vector Prediction (AMVP), merge mode with MVD (MMVD), affine MMVD, bi-prediction with CU-level weight (BCW) index derivation, combined inter and intra prediction (CIIP) with intra-block-copy (IBC), CIIP TM merging, IBC TM merging, TM merging mode, GPM, intra-template matching prediction (intraTMP), AMVP merged MV refinement, or a combination thereof.

[0034] In one embodiment, the target processing procedure of the target codec tool includes candidate reordering, MV refinement, predictor generation, or a combination thereof.

[0035] According to another method for the encoding end, multiple rate-distortion (RD) costs associated with cost function setting candidates are evaluated, each of which is evaluated against one of the cost function setting candidates for the target codec tool. The optimal cost function setting is selected from these candidates to obtain the minimum RD cost among the multiple RD costs. One or more syntaxes are transmitted in the bitstream to indicate this optimal cost function setting. In one embodiment, the RD costs are evaluated only against a subset of the cost function setting candidates.

[0036] According to another method used at the decoding end, one or more syntaxes in the bitstream are parsed to indicate the optimal cost function setting. One or more templates for the current block are determined. Using the optimal cost function setting on said one or more templates, the target processing procedure of the target codec tool is applied to the current block to generate processed data. The processed data is then provided.

[0037] In one embodiment, the optimal cost function setting includes a cost function, a set of weighted coefficients, a template size, or a combination thereof. In one embodiment, the one or more syntaxes include four syntaxes to indicate the optimal cost function, optimal template size, optimal distance weights, and optimal spatial weights. In one embodiment, the one or more syntaxes include two syntaxes to indicate the optimal distance weights and optimal spatial weights. [Attached Image Description] Figure 1A An adaptive inter-frame / intra-frame video codec system incorporating loop processing is illustrated as an example.

[0039] Figure 1B Explanation Figure 1A The decoder corresponding to the encoder.

[0040] Figure 2 An example of template matching is illustrated, where template matching is performed within a search region surrounding the initial motion vector (MV).

[0041] Figure 3 An example of an extended template is shown, in which both the top and left templates are extended.

[0042] Figure 4 Explained with Figure 3 A similar example of an extended template, but the extended template also includes the top-left neighboring sample.

[0043] Figures 5A-5C This illustrates a template matching example based on sub-blocks, where the center of a set of weights corresponds to the sub-block location, used to calculate the TM cost.

[0044] Figure 6 A flowchart illustrating an exemplary video encoding / decoding system employing an adaptive cost function setting according to an embodiment of the present invention is provided.

[0045] Figure 7 A flowchart illustrating an exemplary video decoding system for parsing one or more syntaxes to determine the optimal cost function settings, according to an embodiment of the present invention, is provided.

[0046] Figure 8 A flowchart illustrating an exemplary video coding system according to an embodiment of the present invention is provided, which determines the optimal cost function setting and sends one or more syntaxes to the optimal cost function setting.

Detailed Implementation Methods

[0048] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Those skilled in the art will recognize that the invention can be practiced without one or more specific details, or using other methods, components, etc. In other instances, well-known structures or operations have not been shown or described in detail to avoid obscuring certain aspects of the invention. Illustrative embodiments of the invention will be best understood by referring to the accompanying drawings, in which like parts are designated with the same numerals throughout the drawings. The following description is merely illustrative and only illustrates certain selected apparatus and method embodiments consistent with the claims of the invention.

[0049] To improve encoding and decoding efficiency, several embodiments of using template matching for MV refinement or candidate reordering related to certain encoding and decoding tools are disclosed.

[0050] Method 1: The table below lists codec tools that use template matching for MV refinement, candidate reordering, or other purposes.

[0051] Table 2. Encoding and decoding tools involving template matching in VTM This disclosure proposes a cost function calculation method for template matching costs that differs from existing VTM designs. The proposed embodiments are summarized in the table below, where each row represents an embodiment.

[0052] Table 3. Examples of implementations using different cost functions or template sizes than VTM. Table 3 illustrates some of our proposed embodiments. To summarize all our proposed embodiments, we propose a unified cost function for candidate reordering in a TM that is applicable to many codec tools, including: ARMC™ for affine merging, regular merging, CIIP™ merging, TM merging, and BM merging. BCW Index Export Based on TM Affine MMVD and MMVD MVD symbol prediction for AMVP, affine AMVP, and SMVD IBC CIIP and IBC regular merger In addition, we recommend a unified cost function for MV refinement in the TM, which is applicable to many codec tools, including: AMVP merges MV details CIIP™ Merge,™ Merge Mode, IBC™ Merge GPM (TM MV Candidate) Last but not least, we also recommend unifying the cost function used in TM for generating intra-template matching prediction (intraTMP) predictors.

[0053] The cost function can be Sum of Absolute Differences (SAD), Weighted SAD, Sum of Squared Differences (SSD), Weighted SSD, Sum of Absolute Transform Differences (SATD), or Weighted SATD. The template size can be 1, 2, or 4. The template region used can be top only, left only, or both top and left (i.e., L-shaped). The difference between L-shaped and "both top and left" is that L-shaped also uses the top-left neighbor pixel as the template, while "both top and left" does not.

[0054] If the cost function is weighted SAD, weighted SSD, or weighted SATD, the following weighting coefficients are used, which are the same as those used in the VTM implementation.

[0055] TM_DISATNCE_WEIGHT[2][4]= {{0,1,2,3},{1,2,3,3}} TM_SPATIAL_WEIGHT[2][4]= {{2,2,2,2},{0,1,1,2}} The current design uses the PU size to determine which set of weighting factors to use. If both the width and height of the PU are greater than 8, the second set of factors is used. Otherwise, the first set of factors is used.

[0056] For other codec tools, such as CCLM, CCCM, GLM, TIMD, DIMD, or LIC, we also recommend using TM for candidate reordering. The cost function can be SAD, weighted SAD, SSD, weighted SSD, SATD, or weighted SATD. Template size can include 1, 2, and 4.

[0057] Method 2: For all embodiments listed in Table 3, the cost function is weighted SAD, weighted SSD, or weighted SATD. In one embodiment, we propose using different cost functions or different weighting coefficients depending on the size of the CU / PU or the aspect ratio of the CU / PU. Furthermore, the number of weighting coefficients is not limited to the set described above.

[0058] If the width of the CU / PU is greater than or equal to twice the height of the CU / PU, according to one embodiment, the top template and the left template use different weighting coefficients. For example, the weighting coefficient of the top template is greater than that of the left template. Also, the set of spatial weighting coefficients for the top template contains more elements than the set of spatial weighting coefficients for the left template. That is, we divide the top template into more sub-regions horizontally. Here are some examples: tmWeightIdx = 1, if the width and height of CU / PU are both greater than or equal to 8.

[0059] tmWeightIdx = 0, otherwise.

[0060] Example 1: The distance weight of the top template = {{1,2,3,4}, {2,3,4,4}} Distance weights of the left template = {{0,1,2,3}, {1,2,3,3}} Spatial weight of top template = {{3,3,3,3}, {1,2,2,3}} Spatial weights of the left template = {{2,2,2,2}, {0,1,1,2}} Example 2: TM cost calculation is performed using only the top template.

[0061] Distance weight of top template = {{0,1,2,3}, {1,2,3,3}} The spatial weight of the top template is removed, meaning that each spatial location has an equal weight.

[0062] Example 3: Distance weights between the top and left templates = {{0,1,2,3}, {1,2,3,3}} Spatial weight of top template = {{2,2,2,2}, {0,0,0,1,1,2,2,2}} Spatial weights of the left template = {{2,2,2,2}, {0,1,1,2}} Example 4: Distance weights between the top and left templates = {{0,1,2,3}, {1,2,3,3}} Spatial weight of top template = {{2,2,2,2}, {0,1,1,1,1,1,1,2}} Spatial weights of the left template = {{2,2,2,2}, {0,1,1,2}} Example 5: Distance weights between the top and left templates = {{0,1,2,3}, {1,2,3,3}} Spatial weight of top template = {{2,2,2,2}, {0,0,1,1,1,2,2,2}} Spatial weights of the left template = {{2,2,2,2}, {0,1,1,2}} Similarly, if the height of the codec unit (CU) / prediction unit (PU) is greater than or equal to twice the width of the codec unit (CU) / prediction unit (PU), we use the following weighting factor to calculate the template matching (TM) cost: Example 1: Distance weight of top template = {{0,1,2,3}, {1,2,3,3}} Distance weights of the left template = {{1,2,3,4}, {2,3,4,4}} Spatial weight of top template = {{2,2,2,2}, {0,1,1,2}} Spatial weights of the left template = {{3,3,3,3}, {1,2,2,3}} Example 2: TM cost calculation is performed using only the template on the left.

[0063] Distance weights of the left template = {{0,1,2,3}, {1,2,3,3}} The spatial weights of the top template have been removed. That is, each spatial location is given an equal weight.

[0064] Example 3: Distance weights between the top and left templates = {{0,1,2,3}, {1,2,3,3}} Spatial weight of top template = {{2,2,2,2}, {0,1,1,2}} Spatial weights of the left template = {{2,2,2,2}, {0,0,0,1,1,2,2,2}} Example 4: Distance weights between the top and left templates = {{0,1,2,3}, {1,2,3,3}} Spatial weight of top template = {{2,2,2,2}, {0,1,1,2}} Spatial weights of the left template = {{2,2,2,2}, {0,1,1,1,1,1,1,2}} Example 5: Distance weights between the top and left templates = {{0,1,2,3}, {1,2,3,3}} Spatial weight of top template = {{2,2,2,2}, {0,1,1,2}} Spatial weights of the left template = {{2,2,2,2}, {0,0,1,1,1,2,2,2}} Method 3: This method is a variant of Method 2. We recommend adding more weighted coefficient sets for TM_DISATNCE_WEIGHT and TM_SPATIAL_WEIGHT to handle different codec unit (CU) / prediction unit (PU) sizes.

[0065] Example 1: When the width and height of the codec unit (CU) / prediction unit (PU) are both greater than or equal to 16, tmWeightIdx equals 2.

[0066] When the width and height of the codec unit (CU) / prediction unit (PU) are both greater than or equal to 8 but less than 16, tmWeightIdx equals 1.

[0067] In other cases, tmWeightIdx equals 0.

[0068] TM_DISATNCE_WEIGHT[3][4]= {{0,1,2,3},{1,2,2,3},{1,2,3,3}} TM_SPATIAL_WEIGHT[3][4]= {{2,2,2,2},{0,1,1,2},{0,0,1,1,1,2,2,2}} Example 2: When the width and height of the codec unit (CU) / prediction unit (PU) are both greater than or equal to 16, tmWeightIdx equals 2.

[0069] When the width and height of the codec unit (CU) / prediction unit (PU) are both greater than or equal to 8 but less than 16, tmWeightIdx equals 1.

[0070] In other cases, tmWeightIdx equals 0.

[0071] TM_DISATNCE_WEIGHT[3][4]= {{0,1,2,3},{1,2,3,3},{1,2,3,3}} TM_SPATIAL_WEIGHT[3][4]= {{2,2,2,2},{0,1,1,2},{0,0,1,1,1,2,2,2}} Example 3: When the width and height of the codec unit (CU) / prediction unit (PU) are both greater than or equal to 16, tmWeightIdx equals 2.

[0072] When the width and height of the coding unit (CU) / prediction unit (PU) are both greater than or equal to 8 but less than 16, tmWeightIdx equals 1.

[0073] In other cases, tmWeightIdx equals 0.

[0074] TM_DISATNCE_WEIGHT[3][4]= {{0,1,2,3},{1,2,3,3},{1,2,3,3}} TM_SPATIAL_WEIGHT[3][4]= {{2,2,2,2},{0,1,1,2},{0,0,1,1,1,1,2,2}} Example 4: When the width and height of the codec unit (CU) / prediction unit (PU) are both greater than or equal to 16, tmWeightIdx equals 2.

[0075] When the width and height of the codec unit (CU) / prediction unit (PU) are both greater than or equal to 8 but less than 16, tmWeightIdx equals 1.

[0076] In other cases, tmWeightIdx equals 0.

[0077] TM_DISATNCE_WEIGHT[3][4]= {{0,1,2,3},{1,2,2,3},{1,2,3,3}} TM_SPATIAL_WEIGHT[3][4]= {{2,2,2,2},{0,1,1,2},{0,0,1,1,1,1,2,2}} Example 5: When the width and height of the codec unit (CU) / prediction unit (PU) are both greater than or equal to 16, tmWeightIdx equals 2.

[0078] When the width and height of the codec unit (CU) / prediction unit (PU) are both greater than or equal to 8 but less than 16, tmWeightIdx equals 1.

[0079] In other cases, tmWeightIdx equals 0.

[0080] TM_DISATNCE_WEIGHT[3][4]= {{0,1,2,3},{1,2,3,3},{1,2,3,3}} TM_SPATIAL_WEIGHT[3][4]= {{2,2,2,2},{0,1,1,2},{0,0,0,1,1,2,2,2}} Example 6: When the width and height of the codec unit (CU) / prediction unit (PU) are both greater than or equal to 16, tmWeightIdx equals 2.

[0081] When the width and height of the codec unit (CU) / prediction unit (PU) are both greater than or equal to 8 but less than 16, tmWeightIdx equals 1.

[0082] In other cases, tmWeightIdx equals 0.

[0083] TM_DISATNCE_WEIGHT[3][4]= {{0,1,2,3},{1,2,2,3},{1,2,3,3}} TM_SPATIAL_WEIGHT[3][4]= {{2,2,2,2},{0,1,1,2},{0,0,0,1,1,2,2,2}} Example 7: We propose using different tmWeightIdx for the top and left templates.

[0084] When the width of CU / PU is greater than or equal to 16, the tmWeightIdx of the top template is equal to 2.

[0085] When the width of CU / PU is greater than or equal to 8 but less than 16, the tmWeightIdx of the top template is equal to 1.

[0086] In other cases, tmWeightIdx equals 0.

[0087] When the height of CU / PU is greater than or equal to 16, the tmWeightIdx of the left template is equal to 2.

[0088] When the width of CU / PU is greater than or equal to 8 but less than 16, the tmWeightIdx of the left template is equal to 1.

[0089] In other cases, tmWeightIdx equals 0.

[0090] TM_DISATNCE_WEIGHT[3][4]= {{0,1,2,3},{1,2,2,3},{1,2,3,3}} TM_SPATIAL_WEIGHT[3][4]= {{2,2,2,2},{0,1,1,2},{0,0,1,1,1,2,2,2}} Example 8: We propose using different tmWeightIdx for the top and left templates.

[0091] When the width of CU / PU is greater than or equal to 16, the tmWeightIdx of the top template is equal to 2.

[0092] When the width of CU / PU is greater than or equal to 8 but less than 16, the tmWeightIdx of the top template is equal to 1.

[0093] In other cases, tmWeightIdx equals 0.

[0094] When the height of CU / PU is greater than or equal to 16, the tmWeightIdx of the left template is equal to 2.

[0095] When the width of CU / PU is greater than or equal to 8 but less than 16, the tmWeightIdx of the left template is equal to 1.

[0096] In other cases, tmWeightIdx equals 0.

[0097] TM_DISATNCE_WEIGHT[3][4]= {{0,1,2,3},{1,2,3,3},{1,2,3,3}} TM_SPATIAL_WEIGHT[3][4]= {{2,2,2,2},{0,1,1,2},{0,0,1,1,1,2,2,2}} Example 9: We propose using different tmWeightIdx for the top and left templates.

[0098] When the width of CU / PU is greater than or equal to 16, the tmWeightIdx of the top template is equal to 2.

[0099] When the width of CU / PU is greater than or equal to 8 but less than 16, the tmWeightIdx of the top template is equal to 1.

[0100] In other cases, tmWeightIdx equals 0.

[0101] When the height of CU / PU is greater than or equal to 16, the tmWeightIdx of the left template is equal to 2.

[0102] When the width of CU / PU is greater than or equal to 8 but less than 16, the tmWeightIdx of the left template is equal to 1.

[0103] In other cases, tmWeightIdx equals 0.

[0104] TM_DISATNCE_WEIGHT[3][4]= {{0,1,2,3},{1,2,2,3},{1,2,3,3}} TM_SPATIAL_WEIGHT[3][4]= {{2,2,2,2},{0,1,1,2},{0,0,0,1,1,2,2,2}} Example 10: We propose using different tmWeightIdx for the top and left templates.

[0105] When the width of CU / PU is greater than or equal to 16, the tmWeightIdx of the top template is equal to 2.

[0106] When the width of CU / PU is greater than or equal to 8 but less than 16, the tmWeightIdx of the top template is equal to 1.

[0107] In other cases, tmWeightIdx equals 0.

[0108] When the height of CU / PU is greater than or equal to 16, the tmWeightIdx of the left template is equal to 2.

[0109] When the width of CU / PU is greater than or equal to 8 but less than 16, the tmWeightIdx of the left template is equal to 1.

[0110] In other cases, tmWeightIdx equals 0.

[0111] TM_DISATNCE_WEIGHT[3][4]= {{0,1,2,3},{1,2,3,3},{1,2,3,3}} TM_SPATIAL_WEIGHT[3][4]= {{2,2,2,2},{0,1,1,2},{0,0,0,1,1,2,2,2}} Example 11: We propose using different tmWeightIdx for the top and left templates.

[0112] When the width of CU / PU is greater than or equal to 16, the tmWeightIdx of the top template is equal to 2.

[0113] When the width of CU / PU is greater than or equal to 8 but less than 16, the tmWeightIdx of the top template is equal to 1.

[0114] In other cases, tmWeightIdx equals 0.

[0115] When the height of CU / PU is greater than or equal to 16, the tmWeightIdx of the left template is equal to 2.

[0116] When the width of CU / PU is greater than or equal to 8 but less than 16, the tmWeightIdx of the left template is equal to 1.

[0117] In other cases, tmWeightIdx equals 0.

[0118] TM_DISATNCE_WEIGHT[3][4]= {{0,1,2,3},{1,2,2,3},{1,2,3,3}} TM_SPATIAL_WEIGHT[3][4]= {{2,2,2,2},{0,1,1,2},{0,0,1,1,1,1,2,2}} Example 12: We propose using different tmWeightIdx for the top and left templates.

[0119] When the width of CU / PU is greater than or equal to 16, the tmWeightIdx of the top template is equal to 2.

[0120] When the width of CU / PU is greater than or equal to 8 but less than 16, the tmWeightIdx of the top template is equal to 1.

[0121] In other cases, tmWeightIdx equals 0.

[0122] When the height of CU / PU is greater than or equal to 16, the tmWeightIdx of the left template is equal to 2.

[0123] When the width of CU / PU is greater than or equal to 8 but less than 16, the tmWeightIdx of the left template is equal to 1.

[0124] In other cases, tmWeightIdx equals 0.

[0125] TM_DISATNCE_WEIGHT[3][4]= {{0,1,2,3},{1,2,3,3},{1,2,3,3}} TM_SPATIAL_WEIGHT[3][4]= {{2,2,2,2},{0,1,1,2},{0,0,1,1,1,1,2,2}} Method 4: We propose setting the rate-distortion (RD) cost for each cost function, including the cost function, the set of weighted coefficients, and / or the template size, and sending the optimal settings to the decoder. This method can be applied to one or more codecs listed in Table 2. The cost function can be weighted SAD, weighted SSD, or weighted SATD. The template size can be 1, 2, or 4. The distance weights can be {2,2,2,2}, {0,1,2,3}, {1,2,2,3}, or {1,2,3,3}. The spatial weights can be {2,2,2,2}, {0,1,1,2}, or {0,1,2,2}. Therefore, there are a total of 3 x 3 x 4 x 3 = 108 cost function settings. We propose sending four flags to the decoder according to one embodiment. One flag indicates the optimal cost function. One flag indicates the optimal template size. One flag indicates the optimal distance weights. One flag indicates the optimal spatial weights.

[0126] This search can be performed at the sequence level, image level, slice level, CTU level, or CU level. For example, we can perform this search at the CTU level for MV refinement of the TM merging mode. Four flags are sent to the decoder. One flag indicates the optimal cost function. One flag indicates the optimal template size. One flag indicates the optimal distance weights. One flag indicates the optimal spatial weights.

[0127] In addition to searching all elements (or candidates) of the cost function setting, another embodiment searches only a subset of the cost function setting. For example, with a fixed template size of 4, we search for the optimal cost function setting: (cost function, distance weight, spatial weight). After the search is complete, three flags are sent to the decoder according to one embodiment: one to indicate the cost function, one to indicate the distance weight, and one to indicate the spatial weight.

[0128] In another example, with a fixed template size of 4 and a fixed cost function of weighted SAD, we search for the optimal cost function setting: (distance weights, spatial weights). After the search is complete, two flags are sent to the decoder: one to indicate the distance weights and one to indicate the spatial weights.

[0129] Taking MMVD candidate reordering as an example, we can fix the distance weight and spatial weight, and search for the optimal cost function setting: (cost function, template size).

[0130] The cost function can be weighted SAD, weighted SSD, weighted SATD, SAD, SSD, or SATD. The template size can be 1, 2, or 4. We search for the optimal cost function setting among these 6 x 3 = 18 candidates. According to one embodiment, two flags are sent to the decoder: one indicating the cost function and the other indicating the template size.

[0131] Method 5: The proposed method is a variant of Method 4. Instead of thoroughly searching for the optimal cost function setting among the 54 candidate settings, we propose a hierarchical search for the suboptimal cost function setting.

[0132] Step 1: The template size is fixed to one element from the set {1,2,4}. The cost function can be weighted SAD, weighted SSD, or weighted SATD. The distance weights can be {2,2,2,2}, {0,1,2,3}, {1,2,2,3}, or {1,2,3,3}. The spatial weights can be {2,2,2,2}, {0,1,1,2}, or {0,1,2,2}. Therefore, there are 3 x 4 x 3 = 36 candidate options. We search for the optimal cost function setting among these 36 candidate options.

[0133] Step 2: Inherit the search results from Step 1 and search for the optimal template size. The template size can be 1, 2, or 4. We search for the optimal cost function setting among these three candidates.

[0134] The search can be performed at the chip, CTU, or CU level.

[0135] Method 6: The proposed method is a variant of Method 5. We propose a different approach to hierarchical search.

[0136] Step 1: The template size is fixed to an element in the set {1,2,4}. The cost criterion is fixed to a function in the set {Weighted SAD, Weighted SSD, Weighted SATD}. Distance weights can be {2,2,2,2}, {0,1,2,3}, {1,2,2,3}, or {1,2,3,3}. Spatial weights can be {2,2,2,2}, {0,1,1,2}, and {2,1,1,0}. Therefore, there are 4 x 3 = 12 candidate options. We search for the optimal cost function setting among these 12 candidate options.

[0137] Step 2: Continuing from the search results in Step 1, search for the optimal template size and the optimal cost criterion. The template size can be 1, 2, or 4. The cost criterion can be weighted SAD, weighted SSD, or weighted SATD. We search for the optimal cost function setting among these 9 candidates.

[0138] Method 7: The proposed method is a variant of Method 6. We propose fixing the distance and spatial weights in step 1 and searching for the optimal settings (template size, cost criterion). In step 2, inheriting the search results from step 1, we search for the optimal settings (set of distance weights, set of spatial weights).

[0139] Method 8: This method can be applied to each of the codec tools listed in Table 2. We propose selecting different cost function settings, including the cost function, weighting coefficients, and template size, based on the template region used (e.g., top only, left only, or top + left).

[0140] If both the top and left templates are used, then both distance weights and spatial weights are used, where tmWeightIdx = pu.width>= 8 and pu.height>= 8 ? 1 : 0.

[0141] An example of distance weighting is as follows: Example 1: TM_DISATNCE_WEIGHT[2][4] = {{0,1,2,3},{1,2,3,3}} Example 2: TM_DISATNCE_WEIGHT[2][4] = {{0,1,2,3},{1,2,2,3}} Example 3: TM_DISATNCE_WEIGHT[2][4] = {{0,1,2,3},{1,1,2,3}} Example 4: TM_DISATNCE_WEIGHT[2][4] = {{0,1,2,3},{1,1,1,2}}. An example of spatial weighting is as follows: Example 1: TM_SPATIAL_WEIGHT[2][4] = {{2,2,2,2},{0,1,1,2}} Example 2: TM_SPATIAL_WEIGHT[2][4] = {{2,2,2,2},{0,1,2,2}}. If only the top template is used, then only the distance weight is used for template matching (TM) cost calculation, and the template size is set to 2 or 4.

[0142] The distance weighting has the following implementation examples.

[0143] tmWeightIdx = pu.width>= 8 ? 1 : 0 Example 1: TM_DISATNCE_WEIGHT[2][4] = {{0,1,2,3},{1,2,3,3}} Example 2: TM_DISATNCE_WEIGHT[2][4] = {{0,1,2,3},{1,2,2,3}} Example 3: TM_DISATNCE_WEIGHT[2][4] = {{0,1,2,3},{1,1,2,3}} Example 4: TM_DISATNCE_WEIGHT[2][4] = {{0,1,2,3},{1,1,1,2}} Similarly, if only the left template is used, only the distance weight is used for TM cost calculation, and the template size is set to 2 or 4.

[0144] The distance weighting has the following implementation examples.

[0145] tmWeightIdx = pu.width>= 8 ? 1 : 0 Example 1: TM_DISATNCE_WEIGHT[2][4] = {{0,1,2,3},{1,2,3,3}} Example 2: TM_DISATNCE_WEIGHT[2][4] = {{0,1,2,3},{1,2,2,3}} Example 3: TM_DISATNCE_WEIGHT[2][4] = {{0,1,2,3},{1,1,2,3}} Example 4: TM_DISATNCE_WEIGHT[2][4] = {{0,1,2,3},{1,1,1,2}} Method 9: We propose an extended template region for template matching. An example of an extended template is shown below. Figure 3 As shown, the extended top template 320 and extended left template of the current coding unit (CU) 310 are displayed.

[0146] The proposed method requires the use of the lower left and upper right neighboring pixels.

[0147] Example 1: The width of the top template is twice the width of the CU / PU. Similarly, the height of the left template is twice the height of the CU / PU. The weighting factors are the same as those defined in the VTM software.

[0148] TM_DISATNCE_WEIGHT[2][4]= {{0,1,2,3},{1,2,3,3}} TM_SPATIAL_WEIGHT[2][4]= {{2,2,2,2},{0,1,1,2}} Example 2: As Figure 4 As shown, an L-shaped template is used. Therefore, the template and... Figure 3 Similar to the one in the image, but includes an additional top-left neighboring pixel 410.

[0149] The width of the top template is equal to twice the width of the CU / PU plus the template size. The height of the left template is equal to twice the height of the CU / PU. The weighting factors are the same as those defined in the VTM software.

[0150] TM_DISATNCE_WEIGHT[2][4]= {{0,1,2,3},{1,2,3,3}} TM_SPATIAL_WEIGHT[2][4]= {{2,2,2,2},{0,1,1,2}} Method 10: Sub-block-based template matching This method can be applied to all codec tools listed in Table 3, as well as CCLM, CCCM, GLM, TIMD, DIMD, and LIC. It can be used to refine the MV of each sub-block or to perform candidate reordering for each sub-block.

[0151] For each sub-block, the TM cost can be calculated using a set of TM weights centered at the sub-block's location. An example is shown below. Figures 5A-5C As shown. Figure 5A In the case where a sub-block-based TM is not used, the centroid is the entire current block. Figure 5B For the top-left sub-block, Figure 5C The case corresponding to the top right sub-block.

[0152] The weighting coefficients are indexed by (x,y) coordinates.

[0153] We defined four sets of weighting coefficients for the top-left, top-right, bottom-left, and bottom-right sub-blocks. The idea behind designing the weighting coefficients for each sub-block is to make the center of mass of the template as close as possible to the centroid of the sub-block.

[0154] Example: The weight matrix of the top template of the top left sub-block is [x][y] = [[2,2,1,0], [2,2,1,1], [2,3,2,1],[3,3,2,1]]. The weight matrix [x][y] of the template on the left side of the top-left sub-block is: [[2,2,2,3], [2,2,3,3], [1,1,2,2],[0,1,1,1]] The weight matrix of the top template of the top right sub-block is [x][y] = [[0,1,2,2], [1,1,2,2], [1,2,3,2],[1,2,3,3]]. The weight matrix [x][y] of the template on the left side of the top right sub-block is: [[2,2,2,3], [2,2,3,3], [1,1,2,2],[0,1,1,1]] The weight matrix of the top template of the bottom left sub-block is [x][y] = [[2,2,1,0], [2,2,1,1], [2,3,2,1],[3,3,2,1]]. The weight matrix [x][y] of the template on the left side of the bottom left sub-block is: [[0,1,1,1], [1,1,2,2], [2,2,3,3],[2,2,2,3]] The weight matrix of the top template of the bottom right sub-block is [x][y] = [[0,1,2,2], [1,1,2,2], [1,2,3,2],[1,2,3,3]]. The weight matrix [x][y] of the template on the left side of the bottom right sub-block is: [[0,1,1,1], [1,1,2,2], [2,2,3,3],[2,2,2,3]] As mentioned above, the method for setting the adaptive cost function related to template matching can be implemented at either the encoder or decoder. For example, any of the proposed methods can be implemented in the intra / inter-frame coding module of the decoder (e.g., Figure 1B Intra Pred. 150 / MC 152) or intra / inter-frame coding modules in the encoder (e.g. Figure 1A The proposed method can also be implemented as a circuit coupled to the intra / inter coding module of the decoder or encoder. However, the decoder or encoder can also use additional processing units to implement the required cross-component prediction processing. Figure 1A Units 110 / 112 and Figure 1B Units 150 / 152 in the diagram are shown as independent processing units, but they may correspond to executable software or firmware code stored on a medium, such as a hard disk or flash memory, for a central processing unit (CPU) or a programmable device (e.g., a digital signal processor (DSP) or a field-programmable gate array (FPGA)).

[0155] Figure 6A flowchart illustrating a video codec system using an adaptive cost function setting according to an embodiment of the present invention is presented. The steps shown in the flowchart can be executed at the encoding end as program code on one or more processors (e.g., one or more CPUs). The steps shown in the flowchart can also be implemented in hardware, such as one or more electronic devices or processors configured to execute the steps in the flowchart. According to the method, in step 610, input data associated with the current block is received, wherein the input data includes pixel data to be encoded at the encoding end, or encoded data associated with the current block to be decoded at the decoding end. In step 620, one or more templates for the current block are determined. In step 630, a target cost function or a target weighted coefficient set is selected from a plurality of cost functions or a plurality of weighted coefficient sets based on size-related information associated with the current block, the one or more templates, or both. In step 640, a target cost is derived based on the one or more templates using the selected target cost function or target weighted coefficient set. In step 650, a target codec tool belonging to a codec tool group is applied to the current block, wherein the target processing procedure of the target codec tool is performed based on information including the target cost.

[0156] Figure 7 A flowchart illustrating a video decoding system for parsing one or more syntaxes to determine an optimal cost function setting, according to an embodiment of the present invention, is shown. According to the method, in step 710, input data associated with the current block is received, wherein the input data includes encoded data associated with the current block to be decoded. In step 720, one or more syntaxes in the bitstream are parsed to indicate an optimal cost function setting. In step 730, one or more templates for the current block are determined. In step 740, using the optimal cost function setting on the one or more templates, a target processing procedure of a target codec tool is applied to the current block to generate processed data. The processed data is then provided in step 750.

[0157] Figure 8A flowchart illustrating a video coding system, according to an embodiment of the present invention, determines an optimal cost function setting and transmits one or more syntaxes for the optimal cost function setting. In step 810, input data associated with the current block is received, wherein the input data includes pixel data to be encoded. In step 820, one or more templates for the current block are determined. In step 830, multiple rate-distortion (RD) costs associated with cost function setting candidates are evaluated, wherein each of the multiple RD costs is evaluated against a target codec tool using one of the cost function setting candidates. In step 840, an optimal cost function setting is selected from the cost function setting candidates to obtain the minimum RD cost among the multiple RD costs. In step 850, one or more syntaxes are transmitted in the bitstream to indicate the optimal cost function setting.

[0158] The flowchart shown is intended to illustrate an example of video encoding and decoding according to the present invention. Those skilled in the art can modify each step, rearrange the steps, split the steps, or combine the steps to practice the invention without departing from its spirit. Specific syntax and semantics are used in this specification to illustrate examples of implementing embodiments of the invention. Those skilled in the art can practice the invention by replacing the syntax and semantics with equivalent syntax and semantics without departing from its spirit.

[0159] The foregoing description is intended to enable those skilled in the art to practice the invention in the context of specific applications and their requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments. Therefore, the invention is not intended to be limited to the specific embodiments shown and described, but should be given the broadest scope consistent with the principles and novel features disclosed herein. In the foregoing detailed description, various specific details have been shown to provide a full understanding of the invention. However, those skilled in the art will understand that the invention can be practiced.

[0160] As described above, embodiments of the present invention can be implemented in various hardware, software code, or combinations thereof. For example, one embodiment of the invention may be one or more circuits integrated into a video compression chip, or program code integrated into video compression software, to perform the processes described herein. Another embodiment of the invention may be program code that executes on a digital signal processor (DSP) to perform the processes described herein. The invention may also relate to multiple functions performed by a computer processor, digital signal processor, microprocessor, or field-programmable gate array (FPGA). These processors can be configured to execute machine-readable software code or firmware code to perform the specific methods embodied in the invention. The software code or firmware code can be developed in different programming languages ​​and different formats or styles. The software code can also be compiled for different target platforms. However, different software code formats, styles, and languages, as well as other configuration codes, to perform tasks consistent with the invention, do not depart from the spirit and scope of the invention.

[0161] This invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The examples described are for illustrative purposes only and not for limitation. Therefore, the scope of this invention is defined by the appended claims rather than the foregoing description. All variations within the meaning and equivalence of the claims should be included within their scope.

Claims

1. A video encoding / decoding method, the method comprising: Receive input data related to the current block, wherein the input data includes pixel data that will be encoded at the encoding end, or encoded data related to the current block that will be decoded at the decoding end; Determine one or more templates for the current block; Based on size-related information associated with the current block, one or more templates, or both, select a target cost function or a target weighted coefficient set from multiple cost functions or multiple weighted coefficient sets; Derive the target cost based on the one or more templates using the selected target cost function or the target weighting coefficient set; The target codec tool belonging to the codec tool group is applied to the current block, wherein the target processing procedure of the target codec tool is executed based on information including the target cost.

2. The video encoding and decoding method as described in claim 1, characterized in that, The size-related information includes width, height, block size, aspect ratio, or a combination thereof.

3. The video encoding and decoding method as described in claim 2, characterized in that, If the width is greater than twice the height, or the height is greater than twice the width, then the top template and the left template use different sets of weighting coefficients.

4. The video encoding and decoding method as described in claim 1, characterized in that, The first set of spatial weighting coefficients used for the top template contains more elements than the second set of spatial weighting coefficients used for the left template.

5. The video encoding and decoding method as described in claim 1, characterized in that, Use one or more additional sets of distance-weighted coefficients and / or one or more additional sets of spatial-weighted coefficients.

6. The video encoding and decoding method as described in claim 1, characterized in that, The plurality of cost functions include Sum of Absolute Differences (SAD), Weighted SAD, Sum of Squared Differences (SSD), Weighted SSD, Sum of Absolute Transformed Differences (SATD), Weighted SATD, or combinations thereof.

7. The video encoding and decoding method as described in claim 1, characterized in that, The encoding / decoding toolset includes Adaptive Merge Candidate Reordering (ARMC) Template Matching (TM) for affine merging, regular merging, CIIP merging, bilateral matching (BM) merging, or combinations thereof.

8. The video encoding and decoding method as described in claim 1, characterized in that, The codec toolset includes motion vector difference (MVD) symbol prediction for advanced motion vector prediction (AMVP), merge mode with MVD (MMVD), affine MMVD, bidirectional prediction (BCW) index derivation based on TM block weights, joint inter-frame and intra-frame prediction (CIIP) with intra-block copy (IBC), CIIP TM merging, IBC TM merging, TM merging mode, GPM, intra-frame template matching prediction (intraTMP), AMVP merge MV refinement, or combinations thereof.

9. The video encoding and decoding method as described in claim 1, characterized in that, The target processing procedure of the target encoding / decoding tool includes candidate reordering, MV refinement, predictor generation, or a combination thereof.

10. A video encoding / decoding apparatus, the apparatus comprising one or more electronic devices or processors configured to: Receive input data related to the current block, wherein the input data includes pixel data that will be encoded at the encoding end, or encoded data related to the current block that will be decoded at the decoding end; Determine one or more templates for the current block; Based on size-related information associated with the current block, one or more templates, or both, select a target cost function or a target weighted coefficient set from multiple cost functions or multiple weighted coefficient sets; Derive the target cost based on the one or more templates using the selected target cost function or the target weighting coefficient set; The target codec tool belonging to the codec tool group is applied to the current block, wherein the target processing procedure of the target codec tool is executed based on information including the target cost.

11. A video decoding method, the method comprising: Receive input data related to the current block, wherein the input data includes encoded data related to the current block that will be decoded; Parse one or more syntaxes in the bitstream to indicate the optimal cost function settings; Determine one or more templates for the current block; Using the optimal cost function set on the one or more templates, the target processing procedure of the target encoding / decoding tool is applied to the current block to generate processed data; Provide the processed data.

12. The video decoding method as described in claim 11, characterized in that, The optimal cost function settings include the cost function, the set of weighting coefficients, the template size, or a combination thereof.

13. The video decoding method as described in claim 11, characterized in that, The one or more syntaxes include four syntaxes to indicate the optimal cost function, optimal template size, optimal distance weight, and optimal spatial weight.

14. The video decoding method as described in claim 11, characterized in that, The one or more syntaxes include two syntaxes to indicate the optimal distance weights and the optimal spatial weights.

15. The video decoding method as described in claim 11, characterized in that, The target codec tool belongs to a codec tool group, which includes adaptive merge candidate reordering (ARMC) template matching (TM) for affine merge, regular merge, CIIP merge, bilateral matching (BM) merge, or combinations thereof.

16. The video decoding method of claim 11, wherein the target codec tool belongs to a group of codec tools including motion vector difference (MVD) symbol prediction for advanced motion vector prediction (AMVP), merge mode with MVD (MMVD), affine MMVD, TM-based BCW (block-weighted bidirectional prediction) index derivation, intra-block copy (IBC) joint inter-frame and intra-frame prediction (CIIP), CIIP TM merging, IBC TM merging, TM merging mode, GPM, intra-frame template matching prediction (intraTMP), AMVP merge MV refinement, or combinations thereof.

17. A video encoding method, the method comprising: Receive input data related to the current block, wherein the input data includes pixel data to be encoded; Determine one or more templates for the current block; Evaluate multiple rate-distortion (RD) costs associated with cost function setting candidates, wherein each of the multiple RD costs is evaluated for a target codec tool using one of the cost function setting candidates; Select the optimal cost function setting from the cost function setting candidates to obtain the minimum RD cost among the multiple RD costs; Send one or more syntaxes in the bitstream to indicate the optimal cost function settings.

18. The video encoding method as described in claim 17, wherein, The multiple RD costs are evaluated only for a subset of the candidates defined by the cost function.

Images