Geometric partition mode in video coding

Abstract

Methods related to geometric partition modes (GPM) in video coding are described. The proposed method comprises applying adaptive ordering (ARMC-TM) on merge candidates using template matching to obtain a GPM inter candidate list, applying merge motion vector differences under GPM, enabling GPM on full intra coding units (CUs), using inter template costs and intra template costs in an intra prediction mode of GPM, generating templates in template matching using GPM partitions, and generating a template in the template matching using GPM partitions. And deriving intercept of the top edge and the left edge using adjacent reconstructed sample points and edge criteria to generate partition candidates.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Indian Provisional Patent Application No. 202241021947 filed on April 12, 2022 and Indian Provisional Patent Application No. 202241038441 filed on July 4, 2022, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] This document relates generally to image and video coding. More particularly, embodiments of the invention relate to the application of Geometric Partitioning Mode (GPM) in video coding. Background Art

[0004] In 2020, the MPEG expert group within the International Organization for Standardization (ISO) and the International Telecommunication Union (ITU) jointly released the first version of the generic video coding standard (VVC), also known as H.266 (reference [1]). Recently, the MPEG expert group has been working on the development of the next generation coding standard with improved coding performance compared to existing video coding techniques. As part of this investigation, new coding techniques are also being studied.

[0005] As the inventors herein appreciate, what is needed are improved techniques for applying geometric partitioning modes (GPMs) in image and video coding, and such techniques are described herein.

[0006] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any approach described in this section qualifies as prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise indicated, issues identified with respect to one or more approaches should not be assumed to qualify as prior art based on this section. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Embodiments of the invention are illustrated by way of example and not limitation in the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0008] Figure 1 depicts an example of Geometric Partition Pattern (GPM) partitions in video coding that are grouped at specific angles but with varying offsets;

[0009] FIG. 2A to FIG. 2D depicts an example process for applying adaptive reordering (ARMC) to merge candidates in GPM processing according to an embodiment of the present invention;

[0010] Figure 3depicts an example of coding units segmented using a GPM and associated templates according to an embodiment of the present invention;

[0011] Figure 4 Depicts an example process flow for reordering an intra prediction mode (IPM) list and a GPM inter candidate list according to an embodiment of the present invention;

[0012] Figure 5 depicts examples of coding unit and partition related parameters according to an embodiment of the present invention;

[0013] Figure 6 depicts examples of intra candidates for geometric partitions generated according to an embodiment of the present invention; and

[0014] Figure 7 An example process for generating intra candidates for geometric partitions according to an embodiment of the present invention is depicted. DETAILED DESCRIPTION

[0015] Example embodiments related to applying geometric partitioning patterns in video coding are described herein. In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of various embodiments of the present invention. However, it is apparent that various embodiments of the present invention may be practiced without these specific details. In other cases, well-known structures and devices are not described in detail to avoid unnecessarily obscuring, obscuring or confusing embodiments of the present invention.

[0016] Overview

[0017] Example embodiments described herein relate to the application of geometric partitioning mode in image and video coding. Example embodiments include: applying adaptive ranking of merge candidates using template matching (ARMC-TM) to obtain a GPM inter-frame candidate list, applying merged motion vector difference under GPM, enabling GPM for all intra-frame coding units (CUs), using inter-frame template cost and intra-frame template cost in intra-frame prediction mode of GPM, using GPM partitioning to generate templates in template matching, and using neighboring reconstruction samples and edge criteria to obtain intercepts of top and left edges to generate partition candidates.

[0018] Geometric partitioning schemes in video coding

[0019] Figure 1 Describes an example of Geometric Partition Mode (GPM) in video coding (reference [1]). Geometric Partition Mode is an inter-frame prediction tool in which coding units (CUs) can be partitioned by angles. The line parameterized by the distance ρ is bisected. GPM provides the ability to predict arbitrary object boundaries without resorting to smaller CUs.

[0020] In VVC (reference [1]), the CU level flag is used as a merge mode to signal the geometric partition mode, where other merge modes include: normal merge mode, merge using motion vector difference (MMVD) mode, combined intra and inter prediction (CIIP) mode and sub-block merge mode. m ×2 n For each possible CU of (where m, n∈{3…6}, excluding 8×64 and 64×8), the geometric partitioning mode supports a total of 64 partitions, where W represents the width and H represents the height in pixels.

[0021] When this mode is used, the CU is split into two parts by a geometrically positioned straight line. The position of the split line is mathematically derived from the angle and offset parameters of the specific partition. Each part of the geometric partition in the CU is inter-predicted using its own motion. Only a single prediction is allowed for each partition, that is, each part has one motion vector and one reference index. A single prediction motion constraint is applied to ensure that only two motion compensated predictions are required for each CU, as in traditional bidirectional prediction.

[0022] If geometric partition mode is used for the current CU, a geometric partition index and two merge indexes (one for each partition) indicating the partition mode (angle and offset) of the geometric partition are further signaled. The number of maximum GPM candidate sizes is explicitly signaled in the sequence parameter set (SPS), and this number specifies the syntax binarization used for the GPM merge index. After predicting each part of the geometric partition, a blending process with adaptive weights is used to adjust the sample values ​​along the edges of the geometric partitions. This is the prediction signal for the entire CU. As in other prediction modes, the transform and quantization process will be applied to the entire CU. Finally, the motion field of the CU predicted using the geometric partition mode is stored, and some modifications are made to handle motion vector (MV) blending at the partition boundaries.

[0023] The latest developments in JVET regarding the Enhanced Compression Model (ECM version 4.0) include three GPM improvements:

[0024] 1) GPM with merged motion vector differences (GPM-MMVD) (reference [2]), where GPM in VVC is extended by applying motion vector refinement on the basis of the existing GPM unidirectional MV. First, a flag for GPMCU is signaled to specify whether this mode is used. If this mode is used, each geometric partition of the GPM CU can further decide whether to signal motion vector differences (MVD). If MVD is signaled for a geometric partition, the motion of the partition is further refined after the GPM merge candidate is selected.

[0025] 2) GPM with template matching (GPM-TM) (reference [3]), where when GPM mode is enabled for a CU, a CU-level flag is signaled to indicate whether template matching (TM) is applied to two geometric partitions. The motion information of each geometric partition is refined using TM. When TM is selected, the template is constructed using the left neighboring samples, the top neighboring samples, or the top left neighboring samples depending on the partition angle. The motion is then refined by minimizing the difference between the current template and the template in the reference picture using the same search pattern with the merge mode of the half-pixel interpolation filter disabled.

[0026] 3) GPM with inter prediction and intra prediction (reference [4]), where in GPM with inter prediction and intra prediction, the final prediction samples are generated by weighting the inter prediction samples and intra prediction samples of each GPM separation area. The inter prediction samples are obtained by the same scheme as GPM in ECM, while the intra prediction samples are obtained by the intra prediction mode (IPM) candidate list and the index transmitted by the encoder. The IPM list is filled with parallel modes, followed by candidates from template-based intra mode derivation (TIMD), decoder-side intra mode derivation (DIMD) and neighboring blocks. The size of the list is 3.

[0027] When analyzing the complexity of existing GPM-related technologies, the following observations can be made:

[0028] - The memory required to store all necessary look-up tables (LUTs) for calculating blending weights and motion vector (MV) storage type is large.

[0029] • GPM and GPM-TM utilizing inter-prediction and intra-prediction introduce spatial dependencies during the prediction calculation stage in the decoding pipeline.

[0030] The complexity of GPM-TM and TIMD is very high when used by GPM with inter-frame prediction and intra-frame prediction. This will introduce significant latency in multiple stages of the decoding pipeline, including motion vector predictor (MVPred) calculation and boundary strength (BS) calculation. One obvious result is that the motion vector predictor (MVPred) and most probable mode (MPM) calculation for subsequent virtual pipeline data units (VPDUs) will be blocked until TM and / or TIMD complete their processing.

[0031] When the Luma Mapping, Chroma Scaling (LMCS) tool in VVC is enabled, GPM-TM needs to reconstruct the spatial neighbors to convert back from the reshaped domain.

[0032] The embodiments presented here aim to improve the GPM process in different aspects:

[0033] 1) Quality Improvement (QI) proposals: aim to improve compression efficiency compared to current coding tools, preferably without any additional hardware (HW) implementation issues.

[0034] 2) Recommendations for HW dependency / pipelining elimination (HWPI) and hardware / software (HW / SW) complexity reduction (CR): Aims to address HW dependency / pipelining issues and harmonize similar tools by considering certain guidelines and HW / SW-friendly solutions such as complexity reduction while minimizing the impact on compression efficiency.

[0035] Adaptively reorder merge candidates (ARMC) to obtain GPM inter-frame candidate list

[0036] Motivation: In ECM 4.0, adaptive re-ranking of merge candidates using template matching (TM) (ARMC-TM) is enabled (reference [6]). After building the merge candidate list, the merge candidates are divided into several sub-groups. The merge candidates in each sub-group are re-ranked in ascending order according to the cost value based on template matching. ARMC-TM is not applied to the merge list construction used in the inter candidate list derivation of GPM. Using ARMC-TM in the inter candidate derivation will improve the efficiency of GPM because the merge index signaling cost is reduced.

[0037] proposal:

[0038] 1) Before constructing the GPM inter candidate list, ARMC-TM is used for the inter merge candidate list. The GPM list construction process based on merge index parity will remain unchanged and will be used to reorder the merge candidate list. Figure 2A An example process according to an embodiment is depicted in FIG. Figure 2A In the following discussion, part 0 and part 1 represent two partitions in a CU using the GPM mode, where each partition has its own merge candidate list. Figure 2AAs depicted, ARMC-TM (210) is performed after the merging of the candidate lists (205). Next, the output of the ARMC-TM is followed by a parity-based sorting (215). Parity-based reordering ensures that L0 and L1 references receive equal priority with respect to their position in the list. Without ARMC-TM, the list would be sorted based on spatial and temporal proximity to neighboring CUs, and thus parity-based reordering would select the best candidate based on proximity. Applying parity-based reordering after ARMC-TM ensures that L0 and L1 candidates are prioritized based on their template cost rather than their spatial or temporal proximity. Next, in step 220, the inter candidate list is constructed, and finally, in step 225, part 0 and part 1 lists are generated.

[0039] 2) Use the same mechanism as (1), but create separate lists for the two partitions and use partition-dependent templates when reordering with ARMC-TM. Figure 2B An example process according to an embodiment is depicted in FIG. Figure 2A In contrast, after the merge candidate list step (205), two separate templates (230-0 and 230-1) are created, and then in each path, the process is as follows Figure 2A Continue as shown.

[0040] 3) Before building the GPM inter-frame candidate list, ARMC-TM is used to generate the inter-frame merge candidate list. The GPM inter-frame candidate list is set to be the same as the inter-frame merge candidate list. The GPM list construction process based on merge index parity will not be used to reorder the merge candidate list. As mentioned above, parity-based reordering ensures that L0 and L1 references receive equal priority with respect to their position in the list. Without ARMC-TM, the list will be sorted based on the spatial and temporal proximity of neighboring CUs, and therefore parity-based reordering will select the best candidate based on proximity. However, the use of ARMC-TM ensures that the lists have been prioritized based on their template costs rather than their spatial or temporal proximity, and therefore parity-based reordering may be unnecessary. Figure 2C An example process according to an embodiment is depicted in Figure 2A In contrast, parity-based sorting (215) has been eliminated.

[0041] 4) Use the same mechanism as (3), but create separate lists for the two partitions and use partition-dependent templates when reordering using ARMC-TM. Figure 2D An example process according to an embodiment is depicted in Figure 2BSimilar, but again eliminating the parity-based sorting (215).

[0042] 5) Use ARMC-TM-like algorithm to reorder the GPM inter-frame candidate list. The GPM inter-frame candidate list will be obtained from the inter-frame merge candidate list. The GPM list construction process based on merge index parity will be replaced by the ARMC-TM-like mechanism.

[0043] MMVD-TM for GPM-MMVD

[0044] Motivation: Reference [5] introduces TM reordering for Merged Motion Vector Difference (MMVD) refinement positions. Using the same approach in GPM will improve quality, since the MMVD index signaling cost is reduced.

[0045] proposal:

[0046] 1) Template Matching (TM) is used to reorder the MMVD index used in GPM-MMVD.

[0047] 2) Use partition-based templates to reorder the MMVD index for each partition.

[0048] Enable GPM for all intra CUs

[0049] Motivation: Reference [4] prohibits encoding both partitions of a GPM CU as intra in order to reduce the signaling overhead of intra prediction mode (IPM) and avoid increasing the size of the intra prediction circuitry on the hardware decoder. As the inventors appreciate, the latter concern is incorrect because the TIMD / DIMD intra prediction circuitry can be reused.

[0050] proposal:

[0051] 1) Mark the GPM CU where both partitions are coded as intra CUs. For such CUs, the intra mode for each partition obtained from the IPM list is signaled via the corresponding IPM index instead of GPM_MAX_NUM_UNI_CANDS plus the IPM index. Another advantage of this proposal is that GPM can also be enabled in I slices.

[0052] a. In an example embodiment, two sets of syntax elements may be signaled in the bitstream

[0053] i.sps_allintra_gpm_enabled_flag - To be used in SPS signaling

[0054] ii. Coding Unit (CU) level syntax elements to be signaled as part of the CU.

[0055] Table 1 depicts an example of such signaling. New syntax compared to existing ECM syntax is indicated by italic font. This implementation assumes that the signaling of MIP (matrix-based intra prediction), ISP (intra sub-block partitioning) and MRL (multiple reference line intra prediction) within a GPM frame are mutually exclusive. In other embodiments, these modes can be enabled simultaneously, but at the cost of increased complexity.

[0056] Table 1: Example syntax of GPM in intra prediction mode (IPM)

[0057]

[0058] Example grammatical semantics for the proposed grammar include:

[0059] intra_gpm_flag specifies whether geometric partitioning mode is applied to the current coding unit. Array index x0, y0 specifies the position (x0, y0) of the upper left corner luma sample of the considered coding block relative to the upper left corner luma sample of the picture.

[0060] gpm_part_type_idx[x0][y0] specifies the partition shape of the geometric partition mode. The array index x0, y0 specifies the position (x0, y0) of the upper left corner luma sample of the considered coding block relative to the upper left corner luma sample of the picture.

[0061] intra_luma_ipm_idx[x0][y0][0] specifies the first intra prediction mode index for intra prediction based on geometric partitioning, where x0, y0 specify the position (x0, y0) of the upper left luma sample of the considered coding block relative to the upper left luma sample of the picture.

[0062] intra_luma_ipm_idx[x0][y0][1] specifies the second intra prediction mode index for intra prediction based on geometric partitioning, where x0, y0 specify the position (x0, y0) of the upper left luma sample of the considered coding block relative to the upper left luma sample of the picture.

[0063] b. Based on a), to further reduce the overhead of signaling the GPM intra mode, only gpm_part_type_idx can be signaled instead of the IPM mode. Instead, template matching can be used to help decide the IPM mode. Basically, the GPM partition line is extended to its top template and left template, and the extension line divides the left template or the top template into two partitions, such as Figure 3As shown. Then, for each partition in the template, DIMD or TIMD can be used to decide the best IPM mode, and then the IPM is assigned to the associated GPM partition of the current CU. In another embodiment, the resulting IPM mode can be used as the most probable mode (MPM) and the IPM is refined based on the MPM, and the remaining modes are signaled only based on the MPM.

[0064] c. Based on a), gpm_part_type_idx can be inferred. Method 1: For the top template or the left template, DIMD can be used to decide which is the best partition. Then, for each partition type index (idx), using DIMD in each partition, the best IPM and cost can be determined. Next, the costs of these two partitions are added together. The one with the smallest cost determines the type of GPM partition. Method 2: Same as Method 1, but TIMD is applied. In another embodiment, Method 1 or Method 2 can be used as a partition prediction, and then refinement can be done around the prediction and only the difference is signaled.

[0065] It should be noted that the method in c) can also be applied to the inter-frame correlation GPM mode.

[0066] Use the gradient of neighboring reconstructed sample points to obtain the geometric partition type within the entire frame

[0067] Motivation: For GPM partition mode where both partitions are coded as intra, for each partition, an index needs to be signaled to specify the partition type (from Table 36 in [1], a total of 64 partition types are allowed in VVC) and the intra mode. To reduce the signaling cost, one of the approaches used is template-based partition type and intra mode selection, where only the index corresponding to a list sorted based on the template cost is signaled. Template matching uses neighboring reconstructed samples on the decoder side to calculate the cost for each combination of partition type and intra mode. This increases the complexity of the decoder as it needs to calculate the template costs for 64 partition types multiplied by the number of intra modes for each partition. In [8], templates are used to generate shorter candidate lists. For example, in tests, only 26 of the 64 partition modes were used, and only MPM was used out of the 67 possible intra prediction modes.

[0068] Proposal: The gradients on the horizontal and vertical axes (denoted as G(x) and G(y)) are calculated using the neighboring reconstructed pixels from above the top boundary of the current CU to the left of the left boundary. The gradients can be calculated using an appropriate set of filter coefficients (e.g., a 3×3 edge detection filter such as a Sobel filter, etc.). The calculated gradients are then used to obtain the best partition type that is a subset of the maximum allowed partition type. For example, the points on the edge where the maximum values ​​of G(x) and G(y) occur can be calculated. This method reduces the number of partition types to be evaluated to calculate the template cost on the decoder side and helps reduce decoding complexity.

[0069] consider Figure 5 , depicting the current CU and the neighboring reconstructed samples around the top and left side of the CU. The intercept (507-a or 507-b) is a point on the edge (top or left) of the CU at which the gradient is used to detect the energy offset. In an embodiment, the partition type passing through the intercept is more likely to be selected as the full-frame intra-GPM partition type. Given the intercept, the arctan (G(y) / G(x)) of the vertical gradient value and the horizontal gradient value at the intercept is used to obtain the corresponding angle (502-a or 502-b). The partition type using the obtained angle is also more likely to be selected.

[0070] In an embodiment, without limitation, the following Sobel filter may be applied:

[0071]

[0072] The gradients along the x-axis and y-axis are calculated by element-wise multiplication of the Sobel filter coefficients with the reconstructed neighboring samples (NbrRecon(x,y)), where, for example, (x,y) represents the center of the filter kernel (see, for example, Figure 5 3×3 filter in ):

[0073] G(x)=F x *NbrRecon(x,y)

[0074] G(y)=F y *NbrRecon(x,y)

[0075] To calculate the maximum gradient (or detect edges), you can apply the L1 or L2 norm calculation, for example:

[0076] L1-norm variant of edge detection metric: |G(x)|+|G(y)|

[0077] L2-norm variant of edge detection metric:

[0078] Based on the angle and intercept position obtained for each edge intercept, next, a set of reference partition patterns are obtained, which have intercepts closer to the intercepts determined by the edge detection filter and are characterized by reference angles and / or position information. Then, for each reference partition pattern, a set of candidate partition patterns can be obtained as described below.

[0079] Figure 6 Depicted is an example of three sets of evaluation patterns used to determine candidate partitions. The solid line indicates the reference partition pattern, and the dashed line indicates additional patterns to be evaluated around the reference partition pattern. The unique elements in the union of all three sets are used as the list of partitions to be evaluated. Table 2 uses Figure 6 All three alternative assessment models are described using the example situation depicted in .

[0080] Table 2. Description of alternative zoning schemes that could be evaluated

[0081]

[0082] In an embodiment, as shown in Table 3, up to 29 possible partitioning patterns can be generated; however, in practice, the actual number of unique available partitions is much smaller. The system may also decide to remove certain options in order to reduce computational cost at the expense of reduced coding efficiency. Experiments with one million CUs yielded a median of 14 partitions.

[0083] Table 3. Examples of possible partitioning patterns to be evaluated using the criteria of Table 2

[0084]

[0085] It should be noted that given the mode M2, {(α L -1,x L ),(α L +1,x L )} may be a different set of modes from {(α–1,ρ),(α+1,ρ)}, since the former uses the intercept to characterize the mode, while the latter uses the distance index distanceIdx. The same applies to mode M3. The operation “mirror” represents the mirror operation under the 6.3 standard described in Table 2.

[0086] For a given CU size and any of the 64 partitions, the position parameters of the top and left intercepts can be obtained using the equations in Section 8.5.7.2 of reference [1]. For the intercepts generated by the edge detector, their distances can be calculated from the 64 available intercepts, and the intercept with the smallest distance (e.g., L1 or L2 distance) is selected. For example, if S describes the intercept of the partition in reference [1] and F describes the intercept generated by the edge detector, the mode that minimizes its L1 distance |S.left–F.left|+|S.top–F.top| can be selected. It should be noted that not all 64 modes have both left and top intercepts. In the case where only one of the two intercepts is available, the equations in Section 8.5.7.2 can be used to obtain the intercepts of all modes calculated based on the angle output generated by the edge detector, and then the mode with the intercept closest to one of the edge detector outputs is selected.

[0087] Figure 7 An example embodiment of the entire process is summarized in .

[0088] Step 705: Use the edge detection filter to calculate the horizontal gradient (G(x)) and the vertical gradient (G(y)) using the reconstructed pixels above the top edge and to the left of the left edge of the current CU.

[0089] Step 710: Find the top intercept point (x) based on the edge criterion (e.g., the L1 or L2 norm described previously) and the corresponding G(x) and G(y) values. T ) and the left intercept point (x L )

[0090] Step 715: For each intercept, calculate the angle atan(G(y) / G(x)) for the x and y values ​​of the filter at the intercept. For example, the left intercept is α L , the top intercept is α T

[0091] Step 720: Use the obtained intercepts and their angles to obtain a set of reference GPM partition patterns (e.g., patterns M1, M2, and M3 depicted in Table 3)

[0092] Step 725: Use the mirror angles (e.g., M4 and M5), the intercepts and angles closest to the set of reference GPM partition patterns to obtain additional sets of candidate GPM partition patterns (e.g., see the examples in Table 3)

[0093] Table 4 provides an example of a partition list obtained using the techniques described in Tables 2 and 3. In Table 4, NA means "not available", that is, no additional angle index and distance values ​​are allowed. The partition numbers correspond to those in Table 36 of the VVC specification (reference [1]). For example, M1=23 corresponds to angleIdx=11 and distanceIdx=3.

[0094] Table 4. Example of getting a list of partitions

[0095] Left boundary edge detector output <![CDATA[(α L ,x L )=(7,14)]]> Top boundary edge detector output <![CDATA[(α T ,x T )=(18,25)]]> Reference mode from joint intercept M1＝23 <![CDATA[Reference pattern from (α L , x L )]]> M2=NA <![CDATA[Reference pattern from (α T , x T )]]> M3＝38 M1=>{(α-1,ρ),(α+1,ρ),(α,ρ-1),(α,ρ+1)} {NA,27,22,NA} <![CDATA[M2=>{(a L -1.x L ),(a L +1, x L )}]]> {NA,18} M2=>{(α,ρ-1),(α,ρ+1)} {19,NA} M2=>Mirror M4＝50 M4=>{(α,ρ-1),(α,ρ+1)} {NA,51} <![CDATA[M3=>{(a T -1.x T ),(a T +1, x T )}]]> {NA,41} M3=>{(α,ρ-1),(α,ρ+1)} {2,39} M3=>Mirror M5＝5 M5=>{(α,ρ-1),(α,ρ+1)} {4,NA} Final partition list {2,4，5，18,19,22,23,27,38,39,41,50,51}

[0096] It should be noted that the final partition list in Table 4 is sorted to eliminate duplicate entries. This list is combined with the MPM or IPM list (see the embodiment presented next) and will be sorted based on the template cost.

[0097] In an embodiment, in order to further improve computational efficiency, an IPM list is used instead of an MPM list. More specifically, the proposed method is to use an IPM list (see reference [4]) where each partition has at most GEO_MAX_NUM_INTRA_CANDS (or any subset) modes. In the ECM software, GEO_MAX_NUM_INTRA_CANDS is a constant with a value of 3. The IPM list adds intra-frame modes parallel to the GPM partition type, DIMD mode, TIMD mode, adjacent mode, intra-frame modes perpendicular to the GPM partition type, and planar modes based on availability. In another embodiment, it can be mandatory that planar mode and DC mode are always in the IPM list.

[0098] IPM list and GPM inter-frame candidate list reordering

[0099] Motivation: When one of the partitions in a GPM is intra-coded, the corresponding mode is signaled using the value GPM_MAX_NUM_UNI_CANDS and the IPM index (reference [4]). Therefore, intra partitions will incur a significant cost for signaling their modes. This proposal defines a mechanism for minimizing this cost.

[0100] proposal: Figure 4 An example process of an embodiment with list reordering is depicted.

[0101] 1) Generate GPM inter-frame candidate list (405) and GPM intra-frame mode list (410)

[0102] 2) Calculate the inter-template cost and the intra-template cost (415, 420), and based on these costs assign the IPM index to the lower GPM merge index and the inter candidate index to the higher index, and vice versa.

[0103] 3) Finally, the calculated template costs are used to jointly reorder both the GPM inter candidate list and the IPM list (425). It should be noted that the two lists can also be staggered based on the cost.

[0104] In an embodiment, the common cost metric for intra-frame templates and inter-frame templates can be based on spatial domain or frequency domain metrics, such as sum of absolute differences (SAD), sum of absolute transform differences (SATD), variance, mean minus sum of absolute differences (MAD), etc.

[0105] Partition-related template size

[0106] Motivation: In ECM 4.0, in template matching, the template used is always a W×4 block from the top or a 4×H block from the left, or both, where W and H represent the width and height of the CU, respectively. The TM complexity can be reduced if the template size is also defined as a function of the partition type. For example, in Figure 3 , each partition in the CU can construct a template for the partition using only the pixels that are immediately adjacent to the partition from the top and from the left. The gray-shaded partition 0 can construct a template for the partition using only the adjacent gray pixels (305). Similarly, the white partition 1 can construct a template for the partition using only the adjacent white pixels (310).

[0107] proposal:

[0108] 1) For partitions adjacent to the left or top boundary, only the neighboring reconstructed pixels along the common boundary are used to build the template. For partitions that do not share any pixels with the top or left CU, the entire template is reused.

[0109] References

[0110] Each of the references listed herein is incorporated by reference in its entirety. The term JVET refers to the Joint Video Experts Group of ITU-TSG 16 WP 3 and ISO / IEC JTC 1 / SC 29.

[0111] [1] “Versatile Video Coding,” Rec. ITU-T H.266, August 2020.

[0112] [2] JVET-W0088, “EE2-3.3: GPM with MMVD (JVET-V0103 and JVET-V0125),” Z. Deng et al., conference call, July 2021.

[0113] [3] JVET-W0065, “EE2: Results of Test 3.4 and Test 3.5,” R.-L. Liao et al., conference call, July 2021.

[0114] [4] JVET-Y0065, “EE2-3.1: GPM with inter and intra prediction (JVET-X0166),” Y. Kidani et al., conference call, January 2022.

[0115] [5] JVET-Y0067, “EE2-3.9and EE2-3.10: TM based reordering for MMVD andaffine MMVD and MVD sign prediction,” M. Salehifar et al., conference call, January 2022.

[0116] [6] JVET-W0090: “EE2-3.1 / EE2-3.2: Adaptive Reordering of MergeCandidates with Template / Bilateral Matching,” N. Zhang et al., conference call, July 2021.

[0117] [7] JVET-W0097, “EE2-related: Combination of EE2-3.3, EE2-3.4and EE2-3.5,” Xiaoyu Xiu et al., conference call, July 2021.

[0118] [8] JVET-Z0124, “Non-EE2: Spatial GPM,” Fan Wang et al., conference call, April 2022.

[0119] Example Computer System Implementation

[0120] Embodiments of the present invention may be implemented using a computer system, a system configured with electronic circuits and components, an integrated circuit (IC) device (such as a microcontroller, a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may perform, control, or execute instructions related to applying geometric partitioning patterns in image and video encoding, such as those described herein. The computer and / or IC may calculate any of the various parameters or values ​​described herein related to applying geometric partitioning patterns in image and video encoding. The image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0121] Certain embodiments of the present invention include a computer processor that executes software instructions, which enable the processor to perform the method of the present invention. For example, one or more processors in a display, an encoder, a set-top box, a transcoder, etc. can implement the method related to applying a geometric partitioning pattern in image and video coding as described above by executing software instructions in a program memory accessible to the processor. An embodiment of the present invention can also be provided in the form of a program product. The program product may include any non-transient tangible medium that carries a set of computer-readable signals including instructions, which enable the data processor to perform the method of the present invention when executed by a data processor. According to the program product of the present invention, any of various non-transient and tangible forms can be used. The program product may include, for example, a physical medium, such as a magnetic data storage medium including a floppy disk, a hard disk drive, an optical data storage medium including a CD ROM, a DVD, an electronic data storage medium including a ROM, a flash RAM, etc. The computer-readable signal on the program product may be optionally compressed or encrypted.

[0122] Where components (e.g., software modules, processors, components, devices, circuits, etc.) are mentioned above, unless otherwise indicated, references to the components (including references to "means") should be interpreted to include any components that perform the functions of the described components as equivalents of the components (e.g., functionally equivalent), including components that are not structurally equivalent to the disclosed structures that perform the functions in the illustrated example embodiments of the invention.

[0123] Equivalents, extensions, alternatives and miscellaneous

[0124] Thus described are example embodiments related to the application of geometric partitioning patterns in image and video coding. In the foregoing description, embodiments of the present invention have been described with reference to many specific details, which may vary from implementation to implementation. Therefore, the only and exclusive indication of the present invention and the applicant's inventive intent is the set of claims issued in a specific form according to the present application, wherein such claim issuance includes any subsequent corrections. Any definitions expressly set forth herein for terms contained in such claims should govern the meaning of such terms as used in the claims. Therefore, limitations, elements, properties, features, advantages or attributes not expressly cited in a claim should not limit the scope of such claim in any way. Therefore, this specification and drawings should be viewed in an illustrative rather than a restrictive sense.

Claims

1. A method for obtaining a geometric partition mode (GPM) inter-frame candidate list, the method comprising: Build a list of merge candidates; applying adaptive re-ranking to the merge candidates using template matching (ARMC-TM) to generate an ARMC-TM output; applying a parity-based ordering to the ARMC-TM output to generate a parity-based reordering; generating a GPM inter-frame candidate list based on the parity-based reordering; as well as A partition 0 list and a partition 1 list are generated based on the GPM inter candidate list.

2. A method for obtaining a geometric partition mode (GPM) inter-frame candidate list, the method comprising: Build a list of merge candidates; Generate partition 0 template and partition 1 template; For each of the Partition 0 template or the Partition 1 template: applying adaptive re-ranking to the merge candidates using template matching (ARMC-TM) to generate an ARMC-TM output; applying a parity-based ordering to the ARMC-TM output to generate a parity-based reordering; generating a GPM inter-frame candidate list based on the parity-based reordering; as well as A partition list is generated based on the GPM inter-frame candidate list.

3. A method for obtaining a geometric partition mode (GPM) inter-frame candidate list, the method comprising: Build a list of merge candidates; applying adaptive re-ranking to the merge candidates using template matching (ARMC-TM) to generate an ARMC-TM output; generating a GPM inter-frame candidate list based on the ARMC-TM output; as well as A partition 0 list and a partition 1 list are generated based on the GPM inter candidate list.

4. A method for obtaining a geometric partition mode (GPM) inter-frame candidate list, the method comprising: Build a list of merge candidates; Generate partition 0 template and partition 1 template; For each of the Partition 0 template or the Partition 1 template: applying adaptive re-ranking to the merge candidates using template matching (ARMC-TM) to generate an ARMC-TM output; generating a GPM inter-frame candidate list based on the ARMC-TM output; as well as A partition list is generated based on the GPM inter-frame candidate list.

5. A method for signaling an intra prediction mode (IPM) in a geometric partition mode when both partitions are marked as intra, the method comprising: Signaling a GPM enable flag within the full frame; And if the signaling full intra GPM enable flag is set to enabled, then the intra horizontal syntax elements of each partition are signaled.

6. The method of claim 5, wherein: gpm_part_type_idx is signaled, but no signaling related to the IPM mode is performed.

7. The method of claim 6, wherein: The IPM mode is determined based on template matching.

8. A method for combining a GPM list and an IPM list, the method comprising: Generate GPM inter-frame candidate list (405) and GPM intra-frame mode list (410) Calculating an inter-frame template cost based on the generated GPM inter-frame candidate list; Calculating an intra-template cost based on the generated GPM intra-mode list; Assigning an IPM index to a lower GPM merge index based on the calculated inter-template cost and intra-template cost, and assigning an inter-candidate index to a higher index; Applying the calculated inter-frame template cost and the intra-frame template cost to Intra template costs (415, 420), and based on these costs assigning IPM indices to lower GPM merge indices and inter candidate indices to higher indices, or vice versa; and jointly reordering both the GPM inter candidate list and the IPM list (425).

9. A template matching method for a coding unit (CU) partitioned using a GPM pattern, the method comprising: Given two partitions in the current CU, When partition 0 and partition 1 share any pixels with the top CU of the current CU or the left CU of the current CU, then in template matching: Partition 0 can use a template that uses only the left pixel or the top pixel immediately adjacent to the partition 0 and does not use any pixel immediately adjacent to the partition 1; and Partition 1 can use a template that uses only the immediately left pixel or the top pixel of the partition 1 without using any pixel immediately adjacent to the partition 0.

10. A method for obtaining a geometric partition mode (GPM) intra candidate list, the method comprising: Access a coding unit (CU) with a top edge and a left edge; Applying a filter to neighboring pixels above a top edge of the CU along the x-axis and to neighboring pixels to the left of a left edge of the CU along the y-axis to generate a G(x) gradient and a G(y) gradient for each (x, y) position of a center of the filter; Determine position information of a top intercept on a top edge of the CU and a left intercept on a left edge of the CU according to an edge detection criterion and the G(x) gradient and the G(y) gradient; determining angle information of the top intercept and the left intercept based on the G(x) gradient and the G(y) gradient corresponding to the top intercept and the left intercept; as well as The GPM candidate list is determined based on the position information and the angle information of the top intercept and the left intercept.

11. The method of claim 10, wherein: Calculating the gradients G(x) and G(y) includes calculating G(x)=F x *NbrRecon(x,y), G(y)=F y *NbrRecon(x,y), Among them, F x and F y represents the edge filter kernels in the horizontal and vertical directions, and NbrRecon(x, y) represents a set of neighboring reconstructed pixels above the top edge or to the left of the left edge of the CU.

12. The method of claim 10, wherein: The edge detection criteria include maximizing the L1 norm: |G(x)|+|G(y)| or the L2 norm 13. The method of claim 10, wherein: Determining the angle information at the intercept at the filter location (x, y) includes calculating atan(G(y) / G(x)).

14. The method of claim 10, wherein: Determining the GPM candidate list based on the position information and the angle information of the top intercept and the left intercept includes: Calculating a reference mode partition closest to the top intercept and the left intercept; and determining the GPM candidate list by one or more of the following: characterizing the reference mode partitions using pairs (α, ρ), where α is an angle index and ρ is a range offset, and adding partitions corresponding to pairs (α–1, ρ), (α+1, ρ), (α, ρ−1), and (α, ρ+1) to a GPM candidate list; Use the pair (α L , x L ) and (α T , x T ) to characterize the reference mode partition, wherein α represents an angle index, and x represents the intercept positions of the left intercept and the top intercept, and will be compared with (α L -1,x L )、(α L +1,x L )、(α T -1,x T ) and (α T +1,x T ) The corresponding partition is added to the GPM candidate list; The reference pattern is partitioned by the angle to characterize the reference mode partition, and the angle is The mirror partition of the specified degree is added to the GPM candidate list.

15. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon for executing the method of any one of claims 1 to 14 using one or more processors.

16. An apparatus comprising a processor and configured to perform any of the methods of claims 1 to 14.

Images