Method and apparatus for improving intra mode derivation and prediction using gradients and templates

Abstract

Methods and apparatus for video coding and decoding. When the current intra angle prediction mode of the current block is not in the MPM list, a mode syntax related to the current intra prediction mode of the current block is transmitted or parsed according to first information derived based on DIMD or TIMD. A final intra predictor is generated based on second information including the first information and the mode syntax. And encoding or decoding the current block using a final mode derived based on the information including the syntax.

Description

[0001] Cross-references

[0002] The present invention is a non-provisional application of and claims priority to U.S. Provisional Patent Application No. 63 / 331,351, filed on April 15, 2022. The entire contents of the U.S. Provisional Patent Application are incorporated herein by reference. Technical Field

[0003] The present invention relates to intra-frame prediction in a video coding system, and in particular to a bit saving technique for intra-frame prediction modes using DIMD (Decoder Side Intra Mode Derivation) or TIMD (Template-based Intra Mode Derivation). Background Art

[0004] Versatile Video Coding (VVC) is the latest international video coding standard jointly developed by the Joint Video Experts Group (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG). The standard has been published as an ISO standard: ISO / IEC 23090-3:2021, Information technology - Coded representation of immersive media - Part 3: Versatile video coding, released in February 2021. VVC is an improvement on its predecessor HEVC (High Efficiency Video Coding) by adding more codec tools to improve codec efficiency, and can also handle various types of video sources, including 3D video signals.

[0005] Figure 1AAn exemplary adaptive inter / intra video coding system including a loop process is described. For intra prediction, prediction data is derived from previously encoded video data in the current picture. For inter prediction 112, motion estimation (ME) is performed on the encoder side and motion compensation (MC) is performed based on the results of ME to provide prediction data derived from other pictures and motion data. Switch 114 selects intra prediction 110 or inter prediction 112 and the selected prediction data is provided to adder 116 to form a prediction error, also known as a residual. The prediction error is then processed by transform (T) 118 and subsequent quantization (Q) 120. The transformed and quantized residual is then encoded by entropy encoder 122 to be included in a video bitstream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packaged with auxiliary information (such as motion and coding modes associated with intra prediction and inter prediction) and other information (such as parameters associated with loop filters applied to underlying image areas). Auxiliary information associated with intra prediction 110, inter prediction 112, and in-loop filter 130 is provided to entropy encoder 122, such as Figure 1A When inter-prediction mode is used, one or more reference pictures must also be reconstructed at the encoder end. Therefore, the transformed and quantized residual is inverse quantized (IQ) 124 and inverse transformed (IT)

[0006] The residual is then added back to the prediction material 136 at reconstruction (REC) 128 to reconstruct the video material. The reconstructed video material may be stored in a reference picture buffer 134 and used to predict other frames.

[0007] like Figure 1A As shown, the input video data undergoes a series of processes in the coding system. Due to a series of processes, the reconstructed video data from REC128 may be subject to various impairments. Therefore, loop filter 130 is often applied to reconstructed video data before the reconstructed video data is stored in reference picture buffer 134 to improve video quality. For example, deblocking filter (DF), sample adaptive offset (SAO) and adaptive loop filter (ALF) can be used. It may be necessary to merge loop filter information into the bitstream so that the decoder can correctly restore the required information. Therefore, loop filter information is also provided to entropy encoder 122 to be merged into the bitstream. Figure 1A In , a loop filter 130 is applied to the reconstructed video before the reconstructed samples are stored in a reference picture buffer 134 . Figure 1A The system in is intended to illustrate an exemplary structure of a typical video encoder. It may correspond to the High Efficiency Video Coding (HEVC) system, VP8, VP9, ​​H.264, or VVC.

[0008] like Figure 1BAs shown, the decoder can use similar or identical functional blocks as the encoder, except for transform 118 and quantization 120, because the decoder only needs inverse quantization 124 and inverse transform 126. Instead of entropy encoder 122, the decoder uses entropy decoder 140 to decode the video bitstream into quantized transform coefficients and required encoding information (such as ILPF information, intra-frame prediction information, and inter-frame prediction information). Intra-frame prediction 150 on the decoder side does not need to perform mode search. Instead, the decoder only needs to generate intra-frame prediction based on the intra-frame prediction information received from entropy decoder 140. In addition, for inter-frame prediction, the decoder only needs to perform motion compensation (MC152) based on the inter-frame prediction information received from entropy decoder 140 without motion estimation.

[0009] According to VVC, similar to HEVC, the input picture is divided into non-overlapping square block areas called CTUs (Coding Tree Units). Each CTU can be divided into one or more coding units (CUs) of smaller size. The generated CU partitions can be square or rectangular. In addition, VVC divides CTUs into prediction units (PUs) as units for applying prediction processes such as inter-frame prediction, intra-frame prediction, etc.

[0010] The VVC standard incorporates various new coding tools to further improve the coding efficiency over the HEVC standard. Among the various new coding tools, some coding tools related to the present invention are summarized as follows.

[0011] Use tree structure to divide CTU

[0012] In HEVC, CTUs are divided into CUs using a quadtree (QT) structure represented as a coding tree to accommodate various local characteristics. The decision to encode a picture area using inter-picture (temporal) or intra-picture (spatial) prediction is made at the leaf-CU level. Each leaf-CU can be further partitioned into one, two, or four PUs based on the PU partition type. Within a PU, the same prediction process is applied, and relevant information is transmitted to the decoder on a PU basis. After obtaining the residual block by applying the prediction process based on the PU partition type, the leaf-CU can be divided into transform units (TUs) according to another quadtree structure similar to the coding tree of the CU. One of the key features of the HEVC structure is that it has multiple partition concepts, including CU, PU, ​​and TU.

[0013] In VVC, a quadtree with nested multi-type trees using binary and ternary partitioning structures replaces the concept of multiple partitioning unit types, that is, it removes the separation of CU, PU and TU concepts except for CUs that are too large for the maximum transform length, and supports more flexible CU partition shapes. In the coding tree structure, CU can be square or rectangular. The coding tree unit (CTU) is first partitioned according to a quaternary tree (also called quadtree) structure. Then the quadtree leaf nodes can be further divided into multi-type tree structures. For example Figure 2 As shown, there are four types of splits in the multi-type tree structure, vertical binary split (SPLIT_BT_VER 210), horizontal binary split (SPLIT_BT_HOR 220), vertical ternary split (SPLIT_TT_VER 230), and horizontal ternary split (SPLIT_TT_HOR 240). The multi-type tree leaf node is called a coding unit (CU), and this segmentation will be used for prediction and transform processing without further division unless the CU is too large for the maximum transform length. This means that in most cases, CUs, PUs, and TUs have the same block size in a quadtree with a nested multi-type tree coding block structure. An exception occurs when the maximum supported transform length is less than the width or height of the CU color component.

[0014] Figure 3 The signaling mechanism for partition split information in a quadtree with a nested multi-type tree coding tree structure is shown. The coding tree unit (CTU) is regarded as the root of the quadtree and is first divided by the quadtree structure. Each quadtree leaf node (when large enough to allow it) is then further divided by the multi-type tree structure. In the multi-type tree structure, the first flag (mtt_split_cu_flag) is sent to indicate whether the node is further divided; when a node is further divided, the second flag (mtt_split_cu_vertical_flag) is sent to indicate the split direction, and then the third flag (mtt_split_cu_binary_flag) is sent to indicate whether the split is a binary split or a ternary split. Based on the values ​​of mtt_split_cu_vertical_flag and mtt_split_cu_binary_flag, the multi-type tree split mode (MttSplitMode) of a CU is derived, as shown in Table 1.

[0015] Table 1 – Deriving MttSplitMode based on multi-type tree syntax elements

[0016] MttSplitMode mtt_split_cu_vertical_flag mtt_split_cu_binary_flag SPLIT_TT_HOR 0 0 SPLIT_BT_HOR 0 1 SPLIT_TT_VER 1 0 SPLIT_BT_VER 1 1

[0017] Figure 4The CTU is shown to be partitioned into multiple CUs with a quadtree and nested multi-type tree coding block structure, where the bold block edges represent quadtree partitions and the remaining edges represent multi-type tree partitions. The quadtree with nested multi-type tree partitions provides a content-adaptive coding tree structure consisting of CUs. The size of the CU can be as large as the CTU or as small as 4×4 in units of luma samples. For the 4:2:0 chroma format, the maximum chroma CB size is 64×64 and the minimum chroma CB size consists of 16 chroma samples.

[0018] In VVC, the maximum supported luma transform size is 64 × 64, and the maximum supported chroma transform size is 32 × 32. When the width or height of the CB is larger than the maximum transform width or height, the CB is automatically split horizontally and / or vertically to meet the transform size limit in that direction.

[0019] The following parameters are defined and specified by the SPS syntax elements for quadtree with nested multi-type tree coding tree scheme.

[0020] –CTU size: the root node size of the quadtree

[0021] –MinQTSize: Minimum allowed quadtree leaf node size

[0022] –MaxBtSize: Maximum allowed binary tree root node size

[0023] –MaxTtSize: Maximum allowed ternary tree root node size

[0024] –MaxMttDepth: The maximum allowed depth of a multi-type tree split from a quad leaf node

[0025] –MinBtSize: The minimum allowed binary tree leaf node size

[0026] –MinTtSize: The minimum allowed ternary tree leaf node size

[0027] In one example of a quadtree with a nested multi-type tree coding tree structure, the CTU size is set to 128×128 luma samples and two corresponding 64×64 blocks of 4:2:0 chroma samples, MinQTSize is set to 16×16, MaxBtSize is set to 128×128, MaxTtSize is set to 64×64, MinBtSize and MinTtSize (width and height) are set to 4×4, and MaxMttDepth is set to 4. Tree partitioning is first applied to the CTU to generate quadtree leaf nodes. The size of the quadtree leaf node can range from 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If the leaf QT node is 128×128, the binary tree will not be further split because the size exceeds MaxBtSize and MaxTtSize (i.e., 64×64). Otherwise, the quadtree leaf node may be further split by the multi-type tree. Therefore, the quadtree leaf node is also the root node of the multitree, and its multitree depth (mttDepth) is 0. When the multitree depth reaches MaxMttDepth (i.e. 4), it is considered that no further segmentation is required. When the width of the multi-type tree node is equal to MinBtSize and less than or equal to 2*MinTtSize, no further horizontal segmentation is considered. Similarly, when the height of the multi-type tree node is equal to MinBtSize and less than or equal to 2*MinTtSize, no further vertical segmentation is considered.

[0028] In VVC, the coding tree scheme supports the ability to have separate block tree structures for luma and chroma. For P and B slices, the luma and chroma CTBs in one CTU must share the same coding tree structure. However, for I slices, luma and chroma can have separate block tree structures. When separate block tree modes are applied, the luma CTB is partitioned into CUs by one coding tree structure, and the chroma CTB is partitioned into chroma CUs by another coding tree structure. This means that a CU in an I slice may consist of coding blocks for the luma component or coding blocks for two chroma components, while a CU in a P or B slice always consists of coding blocks for all three color components unless the video is monochrome.

[0029] Virtual Pipeline Data Units (VPDU)

[0030] A Virtual Pipeline Data Unit (VPDU) is defined as a non-overlapping unit in a picture. In a hardware decoder, consecutive VPDUs are processed simultaneously by multiple pipeline stages. In most pipeline stages, the VPDU size is roughly proportional to the buffer size, so it is important to keep the VPDU size small. In most hardware decoders, the VPDU size can be set to the maximum transform block (TB) size. However, in VVC, ternary tree (TT) and binary tree (BT) partitioning may cause the VPDU size to increase.

[0031] To keep the VPDU size to 64x64 luma samples, the following canonical partitioning restrictions (with syntax signaling modifications) are applied in VTM, as Figure 5 As shown:

[0032] – For CUs with width or height or both width and height equal to 128, TT split is not allowed (e.g. Figure 5 ” in the figure).

[0033] – For 128xNCU with N ≤ 64 (i.e. width equals 128 and height less than 128), horizontal BT is not allowed.

[0034] For Nx128 CUs with N≤64 (i.e., height equal to 128 and width less than 128), vertical BT is not allowed. Figure 5 In the example, the luma block size is 128x128. The dashed line indicates a block size of 64x64. According to the above constraints, examples where partitioning is not allowed are indicated by "X", such as Figure 5 As shown in various examples (510-580).

[0035] Intra-mode coding with 67 intra-prediction modes

[0036] To capture arbitrary edge directions present in natural videos, the number of directional intra modes in VVC is expanded from 33 used in HEVC to 65. New directional modes not present in HEVC are introduced in Figure 6 Depicted as dashed arrows in , planar and DC modes remain unchanged. These more dense directional intra prediction modes apply to all block sizes and luma and chroma intra prediction.

[0037] In VVC, for non-square blocks, several traditional angular intra prediction modes are adaptively replaced with wide-angle intra prediction modes.

[0038] In HEVC, each intra-coded block has a square shape and the length of each of its sides is a power of 2. Therefore, no division operation is required to generate the intra predictor using the DC mode. In VVC, blocks can have a rectangular shape, which in general requires the use of a division operation for each block. To avoid the division operation for DC prediction, only the longer sides are used to calculate the average for non-square blocks.

[0039] In order to keep the complexity of the most probable mode (MPM) list generation low, an intra mode coding method with 6 MPMs is used by considering two available adjacent intra modes. The following three aspects are considered to construct the MPM list:

[0040] – Default intra mode

[0041] – Adjacent Intra Mode

[0042] – Intra mode for export.

[0043] A unified 6-MPM list is used for intra blocks, regardless of whether the MRL and ISP coding tools are applied. The MPM list is constructed based on the intra modes of the left and upper neighboring blocks. Assuming that the mode on the left is denoted as Left and the mode of the upper block is denoted as Above, the unified MPM list is constructed as follows:

[0044] – When a neighboring block is not available, its intra mode is set to planar by default.

[0045] – If both Left and Above modes are non-angle modes:

[0046] –MPM list → {plane,DC,V,H,V-4,V+4}

[0047] – If one of the Left and Above modes is an angle mode and the other is a non-angle mode:

[0048] – Set mode Max to the larger mode between Left and Above

[0049] – MPM list → {plane, Max, DC, Max-1, Max+1, Max-2}

[0050] – If Left and Above are both angled and they are different:

[0051] – Set mode Max to the larger mode between Left and Above

[0052] – If the difference between the modes Left and Above is in the range 2 to 62, inclusive

[0053] MPM list → {Plane,Left,Above,DC,Max-1,Max+1}

[0054] -otherwise

[0055] MPM list → {Plane,Left,Above,DC,Max-2,Max+2}

[0056] – If Left and Above are both angled and they are the same:

[0057] –MPM list → {plane,Left,Left-1,Left+1,DC,Left-2}

[0058] In addition, the first bin of the MPM index codeword is CABAC context coded. A total of three contexts are used, corresponding to whether the current intra block is MRL enabled, ISP enabled, or a normal intra block.

[0059] In the 6MPM list generation process, pruning is used to remove repeated patterns so that only unique patterns can be included in the MPM list. For entropy coding of the 61 non-MPM patterns, truncated binary code (TBC) is used.

[0060] Wide-angle intra prediction for non-square blocks

[0061] Conventional angular intra prediction directions are defined as clockwise from 45 degrees to -135 degrees. In VVC, several traditional angular intra prediction modes are adaptively replaced with wide-angle intra prediction modes for non-square blocks. The replaced modes are signaled using the original mode index, which is remapped to the index of the wide-angle mode after parsing. The total number of intra prediction modes remains unchanged, i.e. 67, and the intra mode encoding method remains unchanged.

[0062] To support these predictions, Fig. 7A and Figure 7B As shown, a top reference with a length of 2W+1 and a left reference with a length of 2H+1 are defined.

[0063] The number of alternative modes in wide-angle direction mode depends on the aspect ratio of the block. The alternative intra prediction modes are shown in Table 2.

[0064] Table 2 – Intra prediction modes replaced by Wide mode

[0065]

[0066]

[0067] In VVC, 4:2:2, 4:4:4, and 4:2:0 chroma formats are supported. The chroma derived mode (DM) derivation table for the 4:2:2 chroma format was originally ported from HEVC, expanding the number of entries from 35 to 67 to keep pace with the expansion of intra prediction modes. Since the HEVC specification does not support prediction angles below -135° and above 45°, the luma intra prediction modes are mapped from 2 to 5 to 2. Therefore, the chroma DM derivation table for the 4:2:2 chroma format is updated by replacing some values ​​of the mapping table entries to more accurately convert the prediction angles of the chroma blocks.

[0068] Decoder-side Intra Mode Derivation (DIMD)

[0069] When DIMD is applied, two intra modes are derived from the reconstructed neighboring samples and these two predictions are combined with the planar mode prediction with weights derived from the gradients. DIMD mode is used as an alternative prediction mode and is always checked in high complexity RDO (Rate-Distortion Optimization) mode.

[0070] To implicitly derive the intra prediction mode of a block, texture gradient analysis is performed on both the encoder and decoder side. This process starts with an empty gradient histogram (HoG) with 65 entries, corresponding to 65 angular modes. The magnitudes of these entries are determined during texture gradient analysis.

[0071] In the first step, DIMD selects a template of T=3 columns and rows from the left and top of the current block, respectively. This area is used as a reference for gradient-based intra prediction mode derivation.

[0072] In the second step, horizontal and vertical Sobel filters are applied to all 3×3 window locations, centered on the pixels in the template midline. At each window location, the Sobel filter computes the pure horizontal and vertical intensities G x and G y . The texture angle of the window is then calculated as:

[0073] angle=arctan(G x / G y ), (1)

[0074] can be converted to one of 65 angular intra prediction modes. Once the intra prediction mode index of the current window is derived as idx, the magnitude of its entry in HoG[idx] is updated by adding:

[0075] ampl=|G x |+|Gy | (2)

[0076] Fig. 8A -C shows an example of the HoG computed after applying the above operations for all pixel positions in the template. Fig. 8A An example of a template 820 selected for a current block 810 is illustrated. The template 820 includes T rows above the current block and T columns to the left of the current block. For intra prediction of the current block, an area 830 above and to the left of the current block corresponds to a reconstructed area, while an area 840 below and to the right of the block corresponds to an unavailable area. Figure 8B An example of T=3 is illustrated and the HoG is calculated for the pixel 860 in the middle row and the pixel 862 in the middle column. For example, for pixel 852, a 3x3 window 850 is used. Figure 8C An example of the amplitude (Ampl) calculated based on equation (2) for the angular intra prediction mode as determined from equation (1) is illustrated.

[0077] Once the HoG is calculated, the indices with the two highest histogram bins are selected as the two implicitly derived intra prediction modes for the block and further cooperated with the plane mode set as the prediction for the DIMD mode. Prediction fusion is applied as a weighted average of the above three predictors. For this purpose, the weight of the plane is fixed to 21 / 64 (~1 / 3). The remaining 43 / 64 (~2 / 3) weights are then shared between the two HoG IPMs, proportional to the amplitude of their HoG bins. Fig. 9 An example of the mixing process is shown. Fig. 9 As shown, two intra modes (M1 912 and M2 914) are selected according to the index of the two highest bars with the histogram bar 1110. Three predictors (Pred1 940, Pred2 942 and Pred3 944) are used to form a hybrid prediction. The three predictors correspond to applying M1, M2 and planar intra modes (920, 922 and 924, respectively) to the reference pixel 930 to form the corresponding predictor. The three predictors are weighted by the corresponding weighting factors (ω1, ω2 and ω3) 950. The weighted predictors are summed using an adder 952 to generate a hybrid predictor 960.

[0078] In addition, two implicitly derived intra modes are included in the MPM list so that the DIMD process is performed before constructing the MPM list. The primary derived intra mode of a DIMD block is stored with the block and used for MPM list construction for neighboring blocks.

[0079] Template-based Intra Mode Derivation (TIMD)

[0080] Template-based intra mode derivation (TIMD) mode implicitly derives the intra prediction mode of a CU using neighboring templates at the encoder and decoder, instead of signaling the intra prediction mode to the decoder. Fig.10 As shown, the reference samples (1020 and 1022) of the template of each candidate mode are used to generate the prediction samples (1012 and 1014) of the template of the current block 1010. The cost is calculated as the SATD (sum of absolute transform differences) between the prediction samples and the reconstructed samples of the template. The intra prediction mode with the smallest cost is selected as the DIMD mode and used for intra prediction of the CU. The candidate mode can be 67 intra prediction modes as in VVC or extended to 131 intra prediction modes. Generally, MPM can provide clues to indicate the directional information of the CU. Therefore, in order to reduce the intra mode search space and utilize the characteristics of the CU, the intra prediction mode can be implicitly derived from the MPM list.

[0081] For each intra prediction mode in the MPM, the SATD between the predicted and reconstructed samples of the template is calculated. The first two intra prediction modes with the smallest SATD are selected as TIMD modes. These two TIMD modes are fused with weights after applying the PDPC process, and this weighted intra prediction is used to encode the current CU. Positiondependent intra prediction combination (PDPC) is included in the derivation of TIMD modes.

[0082] The costs of the two selection modes are compared to a threshold, and in the test a cost factor of 2 is applied as follows:

[0083] costMode2<2*costMode1.

[0084] Among them, costMode2 is the cost of mode 2, and costMode1 is the cost of mode 1.

[0085] If this condition is true, fusion is applied, otherwise only mode 1 is used. The weight of the mode is calculated based on its SATD cost as follows:

[0086] weight1=costMode2 / (costMode1+costMode2)

[0087] weight2=1-weight1.

[0088] In the present disclosure, methods and apparatus for improving intra prediction modes to save bits are disclosed. Summary of the invention

[0089] A method and device for video encoding and decoding are disclosed. According to the method, pixel data associated with a current block is received on the encoder side or coded data associated with a current block to be decoded is received on the decoder side. When the current intra-frame angle prediction mode of the current block is not in a possible mode set, the mode syntax of the current intra-frame prediction mode for the current block is signaled or parsed according to first information based on DIMD (decoder-side intra-frame mode derivation) or TIMD (template-based intra-frame mode derivation), wherein the possible mode set includes candidate modes in a most probable mode (Most Probable Modes, MPM) list. A final intra-frame predictor is generated based on second information including the first information and the mode syntax.

[0090] In one embodiment, all intra angular prediction modes are divided into a plurality of sets, and the first information corresponds to a target set determined for the current block based on DIMD or TIMD. In another embodiment, the mode syntax is related to indicating the current intra angular prediction mode within the target set.

[0091] In one embodiment, the set of possible modes includes candidate modes in the MPM list, precise intra prediction modes derived using implicit coding tools other than DIMD and TIMD, or a combination thereof.

[0092] According to another method of the present invention, an initial MPM (most probable mode) list is determined for a current block. One or more DIMD candidate modes are generated using a template of the current block. One or more adjacent intra prediction modes associated with one or more adjacent blocks of the current block are determined. A final MPM list is generated by adding one or more additional candidate modes to the initial MPM list, wherein the one or more additional candidate modes include the one or more DIMD candidate modes. The current block is encoded or decoded using information containing the final MPM list.

[0093] In one embodiment, the one or more additional candidate modes include the one or more DIMD candidate modes, the one or more adjacent intra-frame prediction modes, one or more derived modes of the one or more DIMD candidate modes, one or more derived modes of the one or more adjacent intra-frame prediction modes, or a combination thereof. In one embodiment, the one or more derived modes of the one or more DIMD candidate modes include a mode number corresponding to (one DIMD candidate mode + k), where k is a non-zero integer. In one embodiment, the one or more derived modes of the one or more adjacent intra-frame prediction modes include a mode number corresponding to (one adjacent intra-frame prediction mode + k), where k is a non-zero integer.

[0094] In one embodiment, the one or more adjacent intra prediction modes include an upper adjacent intra prediction mode of an upper adjacent block, a top adjacent intra prediction mode of a top adjacent block, or both. In another embodiment, the one or more derived modes of the one or more adjacent intra prediction modes are included in the final MPM list after the one or more derived modes of the one or more adjacent intra prediction modes or after the one or more adjacent intra predictions. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1A An exemplary adaptive inter / intra video coding system incorporating loop processing is shown.

[0096] Figure 1B Show Figure 1A The corresponding decoder of the encoder in .

[0097] Figure 2 Examples of multi-type tree structures corresponding to vertical binary segmentation (SPLIT_BT_VER), horizontal binary segmentation (SPLIT_BT_HOR), vertical ternary segmentation (SPLIT_TT_VER), and horizontal ternary segmentation (SPLIT_TT_HOR) are shown.

[0098] Figure 3 An example of a signaling mechanism for partition split information in a quadtree with a nested multi-type tree coding tree structure is shown.

[0099] Figure 4 An example of a CTU being partitioned into multiple CUs with a quadtree and nested multi-type tree coding block structure is shown, where bold block edges represent quadtree partitions and the remaining edges represent multi-type tree partitions.

[0100] Figure 5 Some examples where TT splitting is prohibited when the width or height of the luma coding block is greater than 64 are marked.

[0101] Figure 6 The intra-frame prediction mode adopted by the VVC video coding standard is marked.

[0102] Fig. 7A -B shows blocks that are wider than they are tall ( Fig. 7A ) and blocks that are taller than they are wide ( Figure 7B ) is an example of wide-angle intra prediction.

[0103] Fig. 8A An example of a template selected for a current block is illustrated, where the template includes T rows above the current block and T columns to the left of the current block.

[0104] Figure 8BAn example of T=3 is shown, and the HoG (Histogram of Gradients) is calculated for the pixel in the middle row and the pixel in the middle column.

[0105] Figure 8C An example of the amplitude (Ampl) of the angular intra prediction mode is shown.

[0106] Fig. 9 An example of a blending process is illustrated, where two intra modes (M1 and M2) and a planar mode are selected based on the index with the two highest bins of the histogram bins.

[0107] Fig.10 An example of a Template-based Intra Mode Derivation (TIMD) mode is illustrated, where TIMD implicitly derives the intra prediction mode of a CU using neighboring templates at the encoder and decoder.

[0108] Fig.11 A flow chart of an exemplary video coding system is illustrated, in which the signaling of the current intra-frame angular prediction mode depends on the intra-frame angular mode derived by DIMD / TIMD according to one embodiment of the present invention.

[0109] Fig.12 A flow chart of an exemplary video codec system including a DIMD derivation mode in an MPM list according to one embodiment of the present invention is illustrated. DETAILED DESCRIPTION

[0110] It will be readily appreciated that the components of the present invention as generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of the embodiments of the systems and methods of the present invention as shown in the figures is not intended to limit the scope of the claimed invention, but is merely representative of selected embodiments of the present invention. References throughout this specification to "one embodiment," "an embodiment," or similar language mean that a particular feature, structure, or characteristic described in conjunction with that embodiment may be included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment" or "in an embodiment" appearing throughout this specification do not necessarily all refer to the same embodiment.

[0111] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. However, those skilled in the relevant art will recognize that the present invention may be practiced without one or more specific details, or using other methods, components, etc. In other cases, well-known structures or detailed descriptions of operations are not shown or are not shown to avoid obscuring aspects of the present invention. The illustrated embodiments of the present invention will be best understood with reference to the accompanying drawings, in which like parts are represented by like numbers throughout. The following description is intended to be exemplary only and simply illustrates certain selected embodiments of the apparatus and methods consistent with the present invention as claimed herein.

[0112] The following methods are proposed to improve intra mode derivation and prediction accuracy or coding performance:

[0113] Remaining mode signaling via TIMD / DIMD

[0114] As mentioned above, in HEVC and VVC, the MPM list is used to improve the coding efficiency of the current intra-frame prediction mode. When the current intra-frame prediction mode is in the MPM list, the current intra-frame prediction mode can be efficiently encoded because the MPM list contains only a small number of candidates (for example, 6). If the current intra-frame prediction mode is not in the MPM list, the current intra-frame prediction mode is called the remaining mode. In this case, the encoder needs to signal which of the remaining modes is the current intra-frame prediction mode. Since there are many candidates for the remaining modes, it is desirable to improve the coding efficiency when the current intra-frame prediction mode is the remaining mode. Therefore, a method using DIMD / TIMD is disclosed to improve signaling when the current intra-frame prediction mode is a remaining mode.

[0115] According to the method, if the current intra-frame angle prediction mode is a residual mode, the signaling of the current intra-frame angle prediction mode depends on the derived intra-frame angle mode derived by using DIMD / TIMD. In HEVC and VVC, when the current intra-frame prediction mode is not in the MPM list, it is called a residual mode. In the present invention, when the current intra-frame prediction mode is not in the MPM (Most Probable Modes) list, or is accurately predicted by other implicit intra-frame coding tools, it is called a residual mode. The present invention has expanded the possible mode set to include the MPM list and the precise intra prediction mode (precise intra prediction mode) derived using implicit coding tools other than DIMD and TIMD. The key idea of ​​using information related to the derived intra-frame angle mode derived by DIMD / TIMD is to narrow or reduce the number of candidates for the residual mode to be signaled. For example, all angle modes are first divided into multiple mode sets, and then the most likely mode set to which the current intra-frame prediction mode belongs is derived using DIMD / TIMD. When the most probable mode set is determined, some additional coded bits (corresponding to one or more syntaxes, such as mode syntax) are signaled to indicate the actual current intra-frame angle prediction mode within the most probable mode set. It is expected that the number of candidates in the most probable mode set is much smaller than the number of remaining modes. Therefore, the coding efficiency is improved according to the present invention.

[0116] Include DIMD export mode in MPM list

[0117] When exporting the MPM list, one or more DIMD export modes, one or more export modes of the DIMD export mode, or both may be included in the MPM list. In one embodiment, the MPM list includes: a plane mode, one or more upper adjacent intra modes (i.e., the intra mode of the upper adjacent block), one or more left adjacent intra modes (i.e., the intra mode of the left adjacent block), one or more DIMD export modes, one or more export modes of adjacent modes (e.g., adjacent mode + k, or adjacent mode - k), one or more export modes of DIMD export modes (e.g., DIMD export mode + k, or DIMD export mode - k), one or more default modes, or any combination thereof. In the above description, (adjacent mode + k) corresponds to an intra mode whose mode number is equal to ((mode number of adjacent mode) + k), and k is a positive integer. For VVC, the mode number is from 0 to 66, and the mode numbers of other coding standards may be different.

[0118] In one embodiment, the derived modes of the DIMD derived modes are included in the MPM list after the derived modes that include the adjacent modes, or after the derived modes that include the upper adjacent intra mode or the left adjacent intra mode. In another embodiment, only the derived modes with the highest amplitude among the DIMD derived modes are included in the MPM list. In another embodiment, only the derived modes with the DIMD derived modes with the i-th highest amplitude are included in the MPM list.

[0119] Indicates DIMD and TIMD mode on / off

[0120] Before indicating the on / off control syntax of DIMD and TIMD, a first syntax may be signaled to indicate whether one of DIMD or TIMD is allowed / enabled for the current block. For example, if the first syntax is false, it is inferred that DIMD or TIMD is not allowed / enabled for the current block. In this case, the on / off control syntax of DIMD and TIMD is not signaled. For another example, if the first syntax is true and the DIMD on / off control syntax is false, TIMD is implicitly inferred to be allowed / enabled for the current block. For another example, if the first syntax is true and the DIMD on / off control syntax is true, TIMD is implicitly inferred to be not allowed / enabled for the current block.

[0121] Any of the aforementioned remaining modes signaled by TIMD / DIMD and including the DIMD derivation mode in the MPM list method can be implemented in the encoder and / or decoder. For example, any of the proposed methods can be implemented in the intra prediction module of the encoder (e.g., Figure 1A Intra-frame prediction 110 in the decoder) and / or an intra-frame prediction module of the decoder (e.g., Figure 1B Intra-frame prediction 150 in the encoder). However, the encoder or decoder may also use additional processing units to implement the required processing. Alternatively, any of the proposed methods may be implemented as a circuit coupled to the inter-frame / intra-frame / prediction module of the encoder and / or the inter-frame / intra-frame / prediction module of the decoder to provide the information required by the inter-frame / intra-frame / prediction module. In addition, the entropy encoder 122 in the encoder or the entropy decoder 140 in the decoder may be used to implement the signaling associated with the proposed method.

[0122] Fig.11A flow chart of an exemplary video coding system is illustrated, in which the signaling of the current intra-frame angle prediction mode depends on the derived intra-frame angle mode derived by DIMD / TIMD as described in one embodiment of the present invention. The steps shown in the flow chart can be implemented as program codes that can be executed on one or more processors (e.g., one or more CPUs) on the encoder side. The steps shown in the flow chart can also be implemented based on hardware, such as one or more electronic devices or processors arranged to perform the steps in the flow chart. According to the method, in step 1110, pixel data associated with the current block is received on the encoder side or encoded data associated with the current block to be decoded on the decoder side. In step 1120, it is checked whether the current intra-frame angle prediction mode of the current block exists in the most likely mode set. If the current intra-frame angle prediction mode of the current block is not in the possible mode set (i.e., the "no" branch from step 1120), steps 1130 to 1150 are performed. Otherwise, (i.e., the "yes" branch from step 1120), steps 1130 to 1150 are skipped. In step 1130, according to the first information derived based on DIMD (decoder-side intra-frame mode derivation) or TIMD (template-based intra-frame mode derivation), the mode syntax related to the current intra-frame prediction mode of the current block is signaled or parsed, wherein the possible mode set includes the candidate mode in the MPM (MostProbable Modes) list, the precise intra-frame prediction mode derived using an implicit coding tool other than DIMD and TIMD, or a combination thereof. In step 1140, a final intra-frame predictor is generated based on the second information including the first information and the mode syntax. In step 1150, the current block is encoded or decoded using the final mode derived based on the information including the mode syntax.

[0123] Fig.12A flowchart of an exemplary video codec system including a DIMD derivation mode in an MPM list according to an embodiment of the present invention is illustrated. According to the method, in step 1210, pixel data associated with a current block is received at the encoder side or coded data associated with a current block to be decoded at the decoder side. In step 1220, an initial MPM (most probable mode) list of the current block is determined. In step 1230, one or more DIMD (decoder-side intra-frame mode derivation) candidate modes are generated using a template of the current block. In step 1240, one or more adjacent intra-frame prediction modes associated with one or more adjacent blocks of the current block are determined. In step 1250, a final MPM list is generated by adding one or more additional candidate modes to the initial MPM list, wherein the one or more additional candidate modes include the one or more DIMD candidate modes. Or the one or more additional candidate modes include the one or more DIMD candidate modes, the one or more adjacent intra-frame prediction modes, the one or more derived modes of the one or more DIMD candidate modes, the one or more derived modes of the one or more adjacent intra-frame prediction modes, or a combination. In step 1260, the current block is encoded or decoded using information including the final MPM list.

[0124] The flowchart shown is intended to illustrate an example of video encoding according to the present invention. Without departing from the spirit of the present invention, those skilled in the art may modify each step, rearrange the steps, split the steps, or combine the steps to implement the present invention. In the present disclosure, specific syntax and semantics have been used to illustrate examples to implement embodiments of the present invention. Without departing from the spirit of the present invention, those skilled in the art may implement the present invention by replacing syntax and semantics with equivalent syntax and semantics.

[0125] The above description is provided to enable those of ordinary skill in the art to practice the present invention provided in the context of a specific application and its requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the specific embodiments shown and described, but to conform to the widest scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details are illustrated to provide a thorough understanding of the present invention. However, those skilled in the art will appreciate that the present invention may be implemented.

[0126] The embodiments of the present invention as described above can be implemented in various hardware, software codes or a combination of the two. For example, one embodiment of the present invention can be one or more circuits integrated into a video compression chip or a program code integrated into the video compression software to perform the processing described herein. The embodiments of the present invention can also be a program code to be executed on a digital signal processor (DSP) to perform the processing described herein. The present invention can also be related to many functions performed by a computer processor, a digital signal processor, a microprocessor or a field programmable gate array (FPGA). These processors can be configured to perform specific tasks according to the present invention by executing machine-readable software codes or firmware codes that define the specific methods embodied by the present invention. The software code or firmware code can be developed in different programming languages ​​and different formats or styles. The software code can also be compiled for different target platforms. However, the different code formats, styles and languages ​​of the software code and other ways of configuring the code to perform tasks according to the present invention will not depart from the spirit and scope of the present invention.

[0127] The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples should be considered in all respects as illustrative and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than by the foregoing description. All changes that fall within the meaning and scope of equivalents of the claims should be included within their scope.

Claims

1. A video encoding and decoding method, the method comprising: Receiving pixel data associated with a current block at the encoder side or receiving coded data associated with a current block to be decoded at the decoder side; as well as When the current intra-frame angular prediction mode of the current block is not in the possible mode set: Signaling or parsing mode syntax associated with a current intra prediction mode of the current block based on first information derived from a decoder-side intra mode derivation (DIMD) or a template-based intra mode derivation (TIMD), wherein the set of possible modes includes modes in a candidate most probable mode (MPM) list; generating a final intra-frame predictor based on second information including the first information and the mode syntax; as well as The current block is encoded or decoded using a final mode derived based on information including the mode syntax.

2. The method according to claim 1, characterized in that All intra-frame angular prediction modes are divided into a plurality of sets, and the first information corresponds to a target set determined for the current block based on DIMD or TIMD.

3. The method according to claim 2, characterized in that The mode syntax is related to indicating the current intra-frame angular prediction mode within the target set.

4. The method according to claim 1, characterized in that: The possible mode set includes candidate modes in MPM, precise intra prediction modes derived using implicit coding tools other than DIMD and TIMD, or a combination thereof.

5. A device for video encoding and decoding, the device comprising one or more electronic devices or processors, for: Receiving pixel data associated with a current block at the encoder side or receiving coded data associated with a current block to be decoded at the decoder side; When the current intra-frame angular prediction mode of the current block is not in the possible mode set: Signaling or parsing mode syntax associated with the current intra prediction mode of the current block based on first information derived by decoder-side intra mode derivation (DIMD) or template-based intra mode derivation (TIMD), wherein the possible mode set includes modes in a candidate most probable mode (MPM) list; generating a final intra-frame predictor based on second information including the first information and the mode syntax; as well as The current block is encoded or decoded using a final mode derived based on information including the mode syntax.

6. A video encoding and decoding method, the method comprising: Receiving pixel data associated with a current block at the encoder side or receiving coded data associated with a current block to be decoded at the decoder side; determining an initial most probable mode (MPM) list for the current block; generating one or more decoder-side intra mode derivation (DIMD) candidate modes using a template of the current block; generating a final MPM list by adding one or more additional candidate patterns to the initial MPM list, wherein the one or more additional candidate patterns include the one or more DIMD candidate patterns; as well as The current block is encoded or decoded using the information including the final MPM list.

7. The method according to claim 6, characterized in that Include, determining one or more neighboring intra prediction modes associated with one or more neighboring blocks of the current block; The one or more additional candidate modes include the one or more DIMD candidate modes, the one or more adjacent intra-frame prediction modes, one or more derived modes of the one or more DIMD candidate modes, one or more derived modes of the one or more adjacent intra-frame prediction modes, or a combination thereof.

8. The method according to claim 7, characterized in that The one or more derived patterns of the one or more DIMD candidate patterns include a pattern corresponding to a pattern number of (one DIMD candidate pattern+k), where k is a non-zero integer.

9. The method according to claim 7, characterized in that: The one or more derived modes of the one or more neighboring intra prediction modes include a mode whose mode number corresponds to (one neighboring intra prediction mode+k), where k is a non-zero integer.

10. The method according to claim 7, characterized in that The one or more adjacent intra prediction modes include an upper adjacent intra prediction mode of an upper adjacent block, a left adjacent intra prediction mode of a left adjacent block, or both.

11. The method according to claim 7, characterized in that The one or more derived modes of the one or more DIMD candidate modes are included in a final MPM list after the one or more derived modes of the one or more neighboring intra prediction modes or after the one or more neighboring intra prediction modes.

12. A device for video encoding and decoding, the device comprising one or more electronic devices or processors, arranged to: Receiving pixel data associated with a current block at the encoder side or receiving coded data associated with a current block to be decoded at the decoder side; determining an initial most probable mode (MPM) list for the current block; generating one or more decoder-side intra mode derivation (DIMD) candidate modes using a template of the current block; determining one or more neighboring intra prediction modes associated with one or more neighboring blocks of the current block; generating a final MPM list by adding one or more additional candidate modes to the initial MPM list, wherein the one or more additional candidate modes include the one or more DIMD candidate modes, the one or more neighboring intra prediction modes, one or more derived modes of the one or more DIMD candidate modes, one or more derived modes of the one or more neighboring intra prediction modes, or a combination thereof; as well as The current block is encoded or decoded using information including the final MPM list.

Images