Method and apparatus for selecting transform according to intra prediction mode in video coding system

Abstract

A method and apparatus for improved transform processing for video coding. According to the method, input data related to a current block is received, where the input data includes residual data processed at an encoder side or encoded transform data related to the current block for decoding at a decoder side. A transformation process is applied to the input data, wherein the transformation process determines a target transformation of the current block. A target transform is selected from a set of transform candidates comprising at least one additional transform candidate with respect to the original set of transform candidates, or the target transform depends on an intra prediction mode associated with the current block, the reference region, or the one or more neighboring blocks. A transformed output is provided, wherein the transformed output includes the transform coefficients of the current block at the encoder side or the residual data recovered at the decoder side.

Description

[0001] RELATED APPLICATIONS This application is a non-provisional application claiming priority to U.S. Provisional Patent Application No. 63 / 486,675, filed February 24, 2023; U.S. Provisional Patent Application No. 63 / 486,676, filed February 24, 2023; and U.S. Provisional Patent Application No. 63 / 488,206, filed March 3, 2023. The contents of the above U.S. Provisional Patent Applications are incorporated by reference into the present disclosure. TECHNICAL FIELD

[0002] The present disclosure relates to video coding systems. In particular, the present disclosure relates to schemes for improving transform processing by including additional transform candidates and / or applying transform processing according to an intra prediction mode associated with a current block, a reference region, or one or more neighboring blocks. BACKGROUND

[0003] Versatile Video Coding (VVC) is the latest international video coding standard developed by the Joint Video Experts Team (JVET) of ITU-T Video Coding Experts Group (VCEG) and ISO / IEC Moving Picture Experts Group (MPEG). The standard was published as an ISO standard in February 2021: ISO / IEC 23090-3:2021, Information technology — Coding of immersive media — Part 3: Versatile Video Coding. VVC is based on its predecessor, High Efficiency Video Coding (HEVC), by adding more coding tools to improve coding efficiency and handle various types of video sources including 3-dimensional (3D) video signals.

[0004] FIG. 1AAn example adaptive inter / intra video coding system incorporating in-loop processing is shown. For intra prediction, prediction data is derived based on previously coded video data in the current picture. For inter prediction 112, Motion Estimation (ME) is performed at the encoder side and Motion Compensation (MC) is performed based on the results of ME to provide prediction data derived from other pictures and motion data. A switch 114 selects either intra prediction 110 or inter prediction 112, and the selected prediction data is provided to an adder 116 to form prediction error, also known as residual. The prediction error is then processed by Transform (T) 118 followed by Quantization (Q) 120. The transformed and quantized residual is then coded by an entropy coder 122 to be included in a video bitstream corresponding to compressed video data. The bitstream associated with the transform coefficients is then packed with side information such as motion and coding mode associated with intra and inter prediction, and other information such as parameters associated with in-loop filters applied to regions of the underlying picture, etc. As FIG. 1A shown, side information associated with intra prediction 110, inter prediction 112, and in-loop filters 130 is provided to the entropy coder 122. When inter prediction mode is used, one or more reference pictures must also be reconstructed at the encoder side. Thus, the transformed and quantized residual is processed by Inverse Quantization (IQ) 124 and Inverse Transformation (IT) 126 to recover the residual. The residual is then added back to the prediction data 136 at reconstruction (REC) 128 to reconstruct the video data. The reconstructed video data can be stored in a reference picture buffer 134 and used for prediction of other frames.

[0005] As FIG. 1A shown, the input video data undergoes a series of processing in the encoding system. Due to the series of processing, the reconstructed video data from REC 128 can suffer from various impairments. Thus, in-loop filters 130 are typically applied to the reconstructed video data to improve the video quality before the reconstructed video data is stored in the reference picture buffer 134. For example, a deblocking filter (DF), Sample Adaptive Offset (SAO), and Adaptive Loop Filter (ALF) can be used. The in-loop filter information can need to be incorporated into the bitstream so that the decoder can properly recover the needed information. Thus, the in-loop filter information is also provided to the entropy coder 122 to be incorporated into the bitstream. InFIG. 1A In the process, before the reconstructed samples are stored in the reference image buffer 134, the loop filter 130 is applied to the reconstructed video. FIG. 1A The system described herein is intended to illustrate an example architecture of a typical video encoder. It may correspond to a High Efficiency Video Coding (HEVC) system, VP8, VP9, ​​H.264, or VVC.

[0006] like FIG. 1B The decoder shown can use function blocks similar to or partially identical to the encoder, except for transform 118 and quantization 120, since the decoder only needs inverse quantization 124 and inverse transform 126. The decoder uses entropy decoder 140 instead of entropy encoder 122 to decode the video bitstream into quantized transform coefficients and the required encoding / decoding information (e.g., ILPF information, intra-frame prediction information, and inter-frame prediction information). Intra-frame prediction 150 at the decoder end does not require mode search. Instead, the decoder only needs to generate intra-frame predictions based on the intra-frame prediction information received from entropy decoder 140. Furthermore, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) based on the intra-frame prediction information received from entropy decoder 140, without motion estimation.

[0007] According to VVC, the input image is divided into non-overlapping square block regions called Coding Tree Units (CTUs), similar to HEVC. Each CTU can be further divided into one or more smaller coding units (CUs). The resulting CU partitions can be squares or rectangles. Furthermore, VVC divides the CTUs into prediction units (PUs) as a single unit to apply prediction processing, such as inter-frame prediction and intra-frame prediction.

[0008] Use a tree structure to partition the CTU. In HEVC, CTU is divided into CUs by using a quaternary-tree (QT) structure denoted as coding tree, to adapt to various local characteristics. The decision of coding a picture region using inter-picture (temporal) or intra-picture (spatial) prediction is made at leaf CU level. Each leaf CU can be further partitioned into one, two or four PUs according to the PU partition type. Within one PU, the same prediction process is applied, and the related information is transmitted to the decoder on a PU basis. After obtaining the residual block by applying the prediction process based on the PU partition type, a leaf CU can be divided into transform units (TUs) according to another quaternary-tree structure similar to the coding tree of CU. One of the key features of the HEVC structure is that it has multiple partition concepts, including CU, PU and TU.

[0009] In VVC, the concept of multiple partition unit types is replaced by a quaternary-tree with nested multi-type tree (MTT) having binary and ternary splitting partition structure, i.e. it removes the independence of CU, PU and TU concepts, except for the case where the size of CU is too large for the maximum transform length, and supports more flexible CU partition shapes. In the coding tree structure, a CU can be square or rectangular. A coding tree unit (CTU) is first partitioned in a quaternary-tree (also referred to as quad-tree) structure. Then the quaternary-tree leaf nodes can be further partitioned into a multi-type tree structure. As shown in FIG. 2 there are four types of splitting in the multi-type tree structure, vertical binary splitting (SPLIT_BT_VER 210), horizontal binary splitting (SPLIT_BT_HOR 220), vertical ternary splitting (SPLIT_TT_VER 230), and horizontal ternary splitting (SPLIT_TT_HOR 240). The multi-type tree leaf nodes are referred to as coding units (CUs), except for the case where the CU is too large for the maximum transform length, this segment will be used for prediction and transform processing without further division. This means that, in most cases, CU, PU and TU have the same block size in the quaternary-tree with nested multi-type tree coding block structure. An exception occurs when the supported maximum transform length is smaller than the width or height of the CU color component.

[0010] FIG. 3A signaling mechanism for quad-tree in partition split information with nested multi-type tree coding tree structure is shown. A coding tree unit (CTU) is considered as the root of the quad-tree, and is first partitioned by the quad-tree structure. Then, each quad-tree leaf node (when large enough) is further partitioned by the multi-type tree structure. In quad-tree with nested multi-type tree coding tree structure, for each CU node, a first flag (split_cu_flag) is signaled to indicate whether the node is further partitioned or not. If the current CU node is a quad-tree CU node, a second flag (split_qt_flag) is signaled to indicate whether QT partition or MTT partition mode is used. When the node is partitioned using MTT partition mode, a third flag (mtt_split_cu_vertical_flag) is signaled to indicate the split direction, and then a fourth flag (mtt_split_cu_binary_flag) is signaled to indicate whether the split is binary split or ternary split. According to the values of mtt_split_cu_vertical_flag and mtt_split_cu_binary_flag, the multi-type tree split mode (MttSplitMode) of the CU is derived, as shown in Table 1.

[0011]

[0012] FIG. 4 A CTU is shown to be partitioned into CUs with quad-tree and nested multi-type tree coding block structure, where the bold block edges represent quad-tree partition and the remaining edges represent multi-type tree partition. Quad-tree with nested multi-type tree partition provides a content-adaptive coding tree structure consisting of CUs. The size of a CU can be as large as the CTU or as small as 4x4 in luma samples. For 4:2:0 chroma format, the maximum chroma CB size is 64x64 and the smallest size chroma CB consists of 16 chroma samples.

[0013] In VVC, the maximum luma transform size supported is 64x64 and the maximum chroma transform size supported is 32x32. When the width or height of a CB is larger than the maximum transform width or height, the CB is automatically split in the horizontal and / or vertical direction to meet the transform size limit in that direction.

[0014] The following parameters can be defined for quad-tree definition with nested multi-type tree coding tree scheme. These parameters are specified by SPS syntax elements, and can be further refined by picture header syntax elements.

[0015] - CTU size: the root node size of the quad-tree - MinQTSize : the minimum quad-tree leaf node size allowed – MaxBtSize : Maximum allowed size of the root node of a binary tree – MaxTtSize : Maximum allowed size of the root node of a ternary tree – MaxMttDepth The maximum allowed level depth of a multi-type tree segmented from a quadtree leaf. – MinCbSize The smallest allowed decoding block size In one example of a quadtree with a nested multi-type tree codec tree structure, the CTU size is set to 128×128 luma samples and two corresponding 64×64 blocks of 4:2:0 chroma samples. MinQTSize It was set to 16×16. MaxBtSize It was set to 128×128. MaxTtSize It was set to 64×64. MinCbsize (Width and height) are set to 4×4. MaxMttDepth It is set to 4. Quadtree partitioning is first applied to CTU to generate quadtree leaf nodes. The size of quadtree leaf nodes can be from 16×16 (i.e., ... MinQTSize The size is 128×128 (i.e., CTU size). If the leaf QT node is 128×128, it exceeds the limit. MaxBtSize and MaxTtSize (i.e., 128×128), the binary tree will not be further divided. Otherwise, the leaf nodes of the quadtree might be further divided by the multi-type tree. Therefore, the leaf nodes of the quadtree are also the root nodes of the multi-type tree, and its multi-type tree depth (mttDepth) is 0. When the depth of the multi-type tree reaches... MaxMttDepth (i.e., 4) , further splitting is not considered. When the width of a multi-type tree node is equal to MinCbsize When the height of a multi-type tree node is equal to [a certain value], further vertical partitioning is no longer considered. Similarly, when the height of a multi-type tree node is equal to [a certain value], [further vertical partitioning is not considered]. MinCbsize At that time, further horizontal segmentation is not considered.

[0016] In VVC, the codec tree scheme supports the ability for luma and chroma to have separate block tree structures. For P and B films, the luma and chroma CTBs within a CTU must share the same codec tree structure. However, for I films, luma and chroma can have separate block tree structures. When the separate block tree mode is applied, the luma CTB is divided into CUs by one codec tree structure, and the chroma CTB is divided into chroma CUs by another codec tree structure. This means that a CU in an I film may consist of codec blocks for the luma component or codec blocks for both chroma components, while a CU in a P or B film always consists of codec blocks for all three color components, unless the video is monochrome.

[0017] Virtual Pipeline Data Unit (VPDU) A Virtual Pipeline Data Unit (VPDU) is defined as a non-overlapping unit in a picture. In a hardware decoder, consecutive VPDU are processed by multiple pipeline stages simultaneously. In most pipeline stages, the VPDU size is roughly proportional to the buffer size, so it is important to keep the VPDU size small. In most hardware decoders, the VPDU size can be set to the maximum transform block (TB) size. However, in VVC, ternary tree (TT) and binary tree (BT) partitioning can cause the VPDU size to increase.

[0018] To keep the VPDU size to 64x64 luma samples, the following specification partitioning restrictions (with syntax signaling modification) are applied to VTM, as shown in FIG. 5 For a CU with width or height, or both width and height equal to 128, TT partitioning (as indicated by “X” in FIG. 5 is not allowed.

[0019] For a 128xN CU with N≤64 (i.e., width equal to 128 and height less than 128), horizontal BT is not allowed.

[0020] For an Nx128 CU with N≤64 (i.e., height equal to 128 and width less than 128), vertical BT is not allowed.

[0021] In FIG. 5 , the luma block size is 128x128. The dashed line indicates the block size of 64x64. According to the above restrictions, the disallowed partitioning examples are indicated using “X”, as shown in various examples (510-580) in FIG. 5

[0022] Intra mode coding with 67 intra prediction modes To capture arbitrary edge directions that occur in natural videos, the number of directional intra modes in VVC is extended from 33 used in HEVC to 65. The new directional modes not present in HEVC are indicated with dashed arrows in FIG. 6 . The planar and DC modes remain unchanged. These denser directional intra prediction modes are applied to all block sizes and for both luma and chroma intra prediction.

[0023] In VVC, for non-square blocks, several traditional angular intra prediction modes are replaced adaptively with wide-angle intra prediction modes.

[0024] ​​In HEVC, each intra-coded block has a square shape and each side of it has a length of power of 2. Therefore, no division operation is needed to generate an intra predictor using the DC mode. In VVC, a block can have a rectangular shape, which in general case requires a division operation for each block. To avoid the division operation for DC prediction, only the longer side is used to calculate the average for non-square blocks.

[0025] To keep the complexity of the most probable mode (MPM) list generation low, the intra mode coding method with 6 MPMs uses by considering two available neighboring intra modes. The MPM list is constructed considering the following three aspects: default intra mode neighboring intra modes derived intra mode Regardless of whether the MRL and ISP coding tools are applied or not, a uniform 6-MPM list is used for intra blocks. The MPM list is constructed based on the intra modes of the left and above neighboring blocks. Assuming the mode of the left is labeled as Left and the mode of the above block is labeled as Above, the uniform MPM list is constructed as follows: When a neighboring block is not available, its intra mode is set to planar by default.

[0026] If both Left and Above modes are non-angular modes: MPM list -> {planar, DC, V, H, V - 4, V + 4} If one of the Left and Above modes is an angular mode and the other is a non-angular mode: Set mode Max to the larger one of Left and Above MPM list -> {planar, Max, Max - 1, Max + 1, Max - 2, Max + 2} If both Left and Above are angular and they are different: Set mode Max to the larger one of Left and Above If Max - Min is equal to 1: MPM list -> {planar, Left, Above, Min - 1, Max + 1, Min - 2} Otherwise, if Max - Min is greater than or equal to 62: MPM list -> {planar, Left, Above, Min + 1, Max - 1, Min + 2} Otherwise, if Max - Min is equal to 2: MPM list -> {Planar, Left, Above, Min+1, Min-1, Max+1} Otherwise: MPM list -> {Planar, Left, Above, Min-1, Min+1, Max-1} If both Left and Above are angular and they are the same: MPM list -> {Planar, Left, Left - 1, Left + 1, Left - 2, Left+2} In addition, the first bin of the MPM index codeword is CABAC context coded. In total, three contexts are used, corresponding to whether the current intra block is MRL enabled, ISP enabled, or normal intra block.

[0027] In the 6 MPM list generation process, pruning is used to remove duplicate modes so that only unique modes can be included in the MPM list. For the entropy coding of the 61 non-MPM modes, Truncated Binary Code (TBC) is used.

[0028] As described in JVET-D0114 (Seregin et al., “Block shape dependent intra mode coding”, ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 Joint Video Exploration Team (JVET), 4th Meeting: Chengdu, CN, 15th-21st Oct. 2016, document JVET-D0114), an auxiliary MPM list is introduced. The existing primary MPM (PMPM) list contains 6 entries, and the auxiliary MPM (SMPM) list contains 16 entries. A general MPM list containing 22 entries is first constructed, and then the first 6 entries in the general MPM list are included into the PMPM list, and the remaining entries form the SMPM list. The first entry in the general MPM list is the planar mode. The remaining entries are composed of the intra modes of the left (L), above (A), bottom-left (BL), above-right (AR), and above-left (AL) neighboring blocks, the directional modes with an offset of 2 from the first two available directional modes of the neighboring blocks, and the default mode, as shown below.

[0029] If the CU block is in vertical direction, the order of the neighboring blocks is A, L, BL, AR, AL; otherwise, it is L, A, BL, AR, AL. FIG. 7 The positions of the neighboring blocks (L, A, BL, AR, AL) used to derive the general MPM list for the current block 710 are shown.

[0030] First the PMPM flag is parsed, if equal to 1, the PMPM index is parsed to determine which entry in the PMPM list is selected, otherwise the SPMPM flag is parsed to determine whether the SMPM index is parsed or the rest of the modes.

[0031] Wide-Angle Intra Prediction for Non-Square Blocks The regular angular intra prediction directions are defined as clockwise directions from 45 degrees to -135 degrees. In VVC, several of the traditional angular intra prediction modes are replaced by the wide-angle intra prediction modes for non-square blocks. The replaced modes are signaled using the original mode index, which is remapped to the index of the wide-angle modes after parsing. The total number of intra prediction modes is unchanged, i.e., 67, and the intra mode coding scheme is unchanged.

[0032] To support these prediction directions, the top reference of length 2W+1 and the left reference of length 2H+1 are defined as shown in FIG. 7 A and FIG. 7 B, respectively. FIG. 7 A and FIG. 7 B, Dia. mode is the diagonal mode, i.e., mode 34.

[0033] The number of replaced modes in the wide-angle direction modes depends on the aspect ratio of the block. The replaced intra prediction modes are shown in Table 2.

[0034]

[0035] In VVC, 4:2:2 and 4:4:4 and 4:2:0 chroma formats are supported. The chroma derived mode (DM) derivation table for 4:2:2 chroma format is originally ported from HEVC, with the number of entries extended from 35 to 67 to keep consistent with the extension of the intra prediction modes. Since the HEVC specification does not support prediction angles below -135 degrees and above 45 degrees, the luma intra prediction modes from 2 to 5 are remapped to 2. Therefore, the chroma DM derivation table for 4:2:2 chroma format is updated by replacing some values of the mapping table entries to convert the prediction angles of the chroma blocks more accurately.

[0036] Decoder Side Intra Mode Derivation (DIMD) When applying DIMD, two intra modes are derived from the reconstructed neighborhood samples and combined with a planar mode predictor with weights derived from the gradient. The DIMD mode is used as an optional prediction mode and is always checked in the high complexity RDO mode.

[0037] To implicitly derive the intra prediction mode of a block, a texture gradient analysis needs to be performed at the encoder and decoder side. The process starts with an empty histogram of gradients (HoG) containing 65 entries, each corresponding to one of the 65 angular modes. The amplitudes of these entries are determined during the texture gradient analysis process.

[0038] As a first step, DIMD selects a template of T=3 columns and T=3 rows from the left and above of the current block, respectively. This region will be used as a reference for the gradient-based intra prediction mode derivation.

[0039] As a second step, Sobel filters for horizontal and vertical directions are applied to all 3x3 window positions, centered on the line pixel in the template. At each window position, the Sobel filters compute the intensity in pure horizontal and vertical direction, respectively, i.e. G x and G y The texture angle of the window is then computed as follows: (1) which can be converted into one of the 65 angular intra prediction modes. Once the intra prediction mode index for the current window is derived idx its amplitude in the entry of the HoG[ idx ] is updated by addition: (2) FIG. 9A -C shows an example of the HoG computed after applying the above operations on all pixel positions in the template. FIG. 9A An example of a template 920 selected for a current block 910 is shown. The template 920 contains T rows above the current block and T columns to the left of the current block. For the intra prediction of the current block, the region 930 above and to the left of the current block corresponds to the reconstructed region, while the region 940 below and to the right of the current block corresponds to the unavailable region. FIG. 9B An example for T=3 is shown, where the HoG is computed for the pixels 960 in the middle row and the pixels 962 in the middle column. For example, for the pixel 952, a 3x3 window 950 is used. Fig. 9C shows an example of the amplitudes (A ampl ) computed based on equation (2) for the angular intra prediction mode determined by equation (1).

[0040] Once the HoG is computed, the indices with the two highest histogram bins are selected as the two implicitly derived intra prediction modes for the block and further combined with the planar mode as the prediction of the DIMD mode. The prediction fusion is applied as a weighted average of the three predictors. For this, the weight of the planar mode is fixed to 21 / 64 (about 1 / 3). The remaining 43 / 64 (about 2 / 3) weight is shared by the two HoG IPMs in proportion to their HoG bin amplitudes. FIG. 10 An example of the blending process is shown. As shown in FIG. 10 two intra modes (M1 1012 and M2 1014) are selected according to the indices with the two highest bins in the histogram 1010. Three predictors (1040, 1042 and 1044) are used to form the blended prediction. The three predictors apply M1, M2 and the planar intra mode (1020, 1022 and 1024 respectively) to the reference pixels 1030 to form the respective predictors. The three predictors are weighted by the respective weighting factors (1, 2 and 3) 1050. The weighted predictors are added by the addition sub-module 1052 to generate the blended predictor 1060. It is noted that if there is only one mode in the histogram (i.e. single mode), no blending is needed and no second DIMD mode is needed.

[0041] In addition, the two implicitly derived intra modes are included in the MPM list so that the DIMD process is performed before the MPM list is constructed. The main derived intra mode of the DIMD block is stored with the block and used for the MPM list construction of the neighboring blocks.

[0042] DIMD chroma mode The DIMD chroma mode uses the DIMD derivation method to derive the chroma intra prediction mode of the current block based on the neighboring reconstructed Y, Cb and Cr samples in the second neighboring row and the second neighboring column, as shown in FIG. 11 In FIG. 11 , the regions 1110, 1120 and 1130 correspond to the collocated Y block, the current Cb block and the current Cr block respectively. The circles outside the regions 1110, 1120 and 1130 correspond to the neighboring reconstructed samples respectively. The gray circles represent the sample positions for which the gradients are determined for DIMD. Specifically, for each collocated reconstructed luma sample and the reconstructed Cb and Cr samples of the current chroma block, the horizontal and vertical gradients are computed to construct the HoG. The intra prediction mode with the largest histogram amplitude value is then adopted for the chroma intra prediction of the current chroma block.

[0043] When the intra prediction mode derived from the DIMD chroma mode is the same as the intra prediction mode derived from the DM mode, the second largest histogram amplitude value intra prediction mode is used as the DIMD chroma mode. A CU level flag is sent to indicate whether the suggested DIMD chroma mode is applied. Then, the N best derived DIMD modes in terms of histogram amplitude are blended to form the final predictor for the current block.

[0044] Fusion of chroma intra prediction modes The DM mode and the four default modes can be fused with the MMLM LT mode as follows: , wherein pred 0 is the predictor obtained by applying the non-LM mode, pred 1 is the predictor obtained by applying the MMLM LT mode, pred is the final predictor for the current chroma block. w 0 and w 1 two weights are determined by the intra prediction modes of the neighboring chroma blocks, and shift is set to 2. Specifically, when both the top and bottom neighboring blocks are coded with LM mode, w 0, w 1} = {1, 3}; when both the top and bottom neighboring blocks are coded with non-LM mode, w 0, w 1} = {3, 1}; otherwise, w 0, w 1} = {2, 2}.

[0045] For syntax design, if the non-LM mode is selected, a flag is sent to indicate whether the fusion is applied. This method is only applicable to I slice.

[0046] Template-based Intra Mode Derivation (TIMD) The Template-based Intra Mode Derivation (TIMD) mode implicitly derives the intra prediction mode of a CU at the encoder and the decoder using neighboring templates, instead of sending the intra prediction mode to the decoder. As FIG. 12As shown, the template prediction samples (1212 and 1214) of the current block 1210 are generated using the template reference samples (1120 and 1222) of each candidate mode. The cost is calculated as the sum of absolute transformed differences (SATD) between the prediction samples and the reconstructed samples of the template. The intra prediction mode with the minimum cost is selected as the DIMD mode and is used for the intra prediction of the CU. The candidate modes can be the 67 intra prediction modes in VVC, or can be extended to 131 intra prediction modes. Generally, the MPMs can provide a clue indicating the direction information of the CU. Therefore, in order to reduce the intra mode search space and take advantage of the characteristics of the CU, the intra prediction mode can be implicitly derived from the MPM list.

[0047] For each intra prediction mode in the MPM, the SATD between the prediction samples and the reconstructed samples of the template is calculated. The first two intra prediction modes with the minimum SATD are selected as the TIMD modes. The two TIMD modes are fused by weighting after applying the PDPC process, and the current CU is coded using the weighted intra prediction. The position dependent intra prediction combination (PDPC) is included in the derivation of the TIMD mode.

[0048] The costs of the two selected modes are compared with a threshold, and in the test, a cost factor 2 is applied as follows: costMode2<2 costMode1, If the condition is true, fusion is applied, otherwise only mode 1 is used (i.e. single mode case). The weights of the modes are calculated according to their SATD costs as follows: weight1 = costMode2 / (costMode1+ costMode2), weight2 = 1 - weight1, Intra Sub-Partition (ISP) Intra sub-partitioning (ISP) divides a luma intra prediction block vertically or horizontally into 2 or 4 sub-partitions depending on the block size. For example, the minimum block size for ISP is 4x8 (or 8x4). If the block size is larger than 4x8 (or 8x4), the corresponding block will be divided into 4 sub-partitions. Notably, Mx128 (M<64) and 128xN (N<64) ISP blocks can potentially cause issues with 64x64 VDPU (virtual decoder pipeline unit). For example, an Mx128 CU in the single tree case has one Mx128 luma TB and two corresponding M / 2x64 chroma TBs. If the CU uses ISP, the luma TB will be divided into four Mx32 TBs (horizontal partitioning only is supported), each of which is smaller than a 64x64 block. However, in the current ISP design, the chroma blocks are not divided. Thus, both chroma components will be larger than a 32x32 block. Similarly, a similar situation occurs for a 128xN CU using ISP. Thus, both of these cases are problematic for a 64x64 decoder pipeline. Therefore, the CU size restriction for which ISP can be used is limited to a maximum of 64x64. FIGS. 13A and 13B illustrate examples of both possibilities. All sub-partitions satisfy the condition of having at least 16 samples.

[0049] In ISP, 1xN and 2xN subblock prediction is not allowed to depend on the reconstructed values of previously decoded 1xN and 2xN subblocks of the coded block, such that the minimum width of subblock prediction is four samples. For example, an 8xN (N>4) coded block encoded using ISP vertical partitioning is divided into two prediction regions, each region size of 4xN and four transform of size 2xN. In addition, a 4xN coded block encoded using ISP vertical partitioning is predicted using a full 4xN block; four 1xN transforms. While 1xN and 2xN transform sizes are allowed, it is asserted that the transforms of these blocks in the 4xN region can be done in parallel. For example, when a 4xN prediction region contains four 1xN transforms, there is no transform in the horizontal direction; the transform in the vertical direction can be done as a single 4xN transform in the vertical direction. Similarly, when a 4xN prediction region contains two 2xN transform blocks, the transform operations of the two 2xN blocks in each direction (horizontal and vertical) can be done in parallel. Thus, there is no added latency in processing these smaller blocks compared to processing a 4x4 regular coded intra block.

[0050]

[0051] For each sub-partition, the reconstructed samples are obtained by adding the residual signal to the prediction signal. Here, the residual signal is generated by processing such as entropy decoding, inverse quantization, and inverse transform. Thus, the reconstructed sample values of each sub-partition can be used to generate the prediction of the next sub-partition, and each sub-partition is processed consecutively. Furthermore, the first sub-partition to be processed is the one containing the top-left sample of the CU, and then it continues either down (horizontal split) or right (vertical split). Thus, the reference samples used to generate the sub-partition prediction signal are only located on the left and on the top side of the line. All sub-partitions share the same intra mode.

[0052] Template matching prediction (TMP) Template matching prediction (TMP) is a special intra prediction mode that copies the best prediction block from the reconstructed part of the current frame, whose L-shaped template matches the current one. As shown in FIG. 14, block 1420 corresponds to the current block in current picture 1410, and block 1430 corresponds to the reference block. Region 1422 corresponds to the template of current block 1420, and region 1432 corresponds to the template of reference block 1430. For a predetermined search range 1440, the encoder searches for the template that is most similar to the current template in the reconstructed part of the current frame, and uses the corresponding block as the prediction block (as the reference block). Then, the encoder sends the use of this mode, and the inverse operation is performed at the decoder side. FIG. 14

[0053] Cross component linear model (CCLM) The main idea of the CCLM mode (sometimes abbreviated as LM mode) is as follows: the chroma components of a block can be predicted from the co-located reconstructed luma samples by a linear model, whose parameters are derived from neighboring already reconstructed luma and chroma samples of the block.

[0054] In VVC, the CCLM mode exploits the inter-channel dependency by predicting the chroma samples from the reconstructed luma samples. This prediction is done by a linear model of the form , (3) Here, denotes the predicted chroma samples in a CU, denotes the reconstructed luma samples of the same CU, down-sampled in case of non-4:4:4 color format. The model parameters and are derived based on the reconstructed neighboring luma and chroma samples at the encoder and decoder side, without the need for explicit signaling.

[0055] ​Three CCLM modes, i.e., CCLM_LT, CCLM_L, and CCLM_T, are specified in VVC. These three modes differ in the reference sample positions used for model parameter derivation. CCLM_T mode involves only samples from the top boundary, and CCLM_L mode involves only samples from the left boundary. In CCLM_LT mode, samples from both the top boundary and the left boundary are used.

[0056] Overall, the prediction process of CCLM mode includes three steps: 1) down-sampling the luma block and its neighboring reconstructed samples to match the size of the corresponding chroma block, 2) model parameter derivation based on the reconstructed neighboring samples, and 3) applying the model equation (1) to generate the chroma intra prediction samples.

[0057] Downsampling of luma component To match the chroma sample positions of 4:2:0 or 4:2:2 color format video sequences, two types of down-sampling filters can be applied to the luma samples, both of which have a 2-to-1 down-sampling ratio in both horizontal and vertical directions. These two filters correspond to “Type-0” and “Type-2” 4:2:0 chroma format content, respectively, and are given by the following equations , (4) Based on the SPS level flag information, a 2-dimensional 6-tap (i.e., f 2) or 5-tap (i.e., f 1) filter is applied to the luma samples within the current block and their neighboring luma samples. SPS level refers to sequence parameter set level. If the top row of the current block is a CTU boundary, a 1-dimensional filter is applied to the above neighboring luma samples to avoid using multiple luma rows above the CTU boundary.

[0058] Model parameter derivation process : Model parameters and are derived from equation (4) based on the reconstructed neighboring luma and chroma samples at the encoder and decoder side to avoid any transmission overhead. In the originally adopted version of CCLM mode, a linear minimum mean square error (LMMSE) estimator is used for parameter derivation. However, in the final design, only four samples are involved to reduce the computational complexity. FIG. 15 The relative sample positions of a “Type-0” content displaying chroma block 1510, the corresponding luma block 1520, and their neighboring samples (shown as filled circles and triangles).

[0059] In FIG. 15 the example, four samples used in the CCLM LT mode are shown, which are marked with triangles. They are located at the and positions of the top boundary, and the and positions of the left boundary. In the CCLM T and CCLM L modes, the top and left boundaries are extended to the size of (M+N) samples, and the four samples used for model parameter derivation are located at the and positions.

[0060] Once the four samples are selected, four comparison operations are used to determine the two minimum and two maximum luma sample values. Let denote the average of the two maximum luma sample values, and let denote the average of the two minimum luma sample values. Similarly, let and denote the average of the corresponding chroma sample values. Then, the linear model parameters are obtained according to the following equations: , , (5) In this equation, the division operation to compute the parameter is implemented by a look-up table. To reduce the memory required to store this table, diff values, i.e., the difference between the maximum and minimum values, and the parameter are represented in exponential notation. Here, diff values are approximated with a 4-bit significant part and an exponent. Thus, the table for 1 / diff contains only 16 elements. This helps to reduce both the computational complexity and the memory size required to store the table.

[0061] MMLM Overview As the name suggests, the original CCLM mode employs one linear model to predict the chroma samples of the entire CU, while in MMLM (multi-model CCLM), there can be two models. In MMLM, the neighboring luma samples and the neighboring chroma samples of the current block are divided into two groups, and each group is used as a training set to derive a linear model (i.e., a particular a and b are derived for a particular group). In addition, the samples of the current luma block are also classified according to the classification rule of the neighboring luma samples.

[0062] The threshold is computed as the average of the neighboring reconstructed luma samples. The neighboring samples Rec' L[x, y] <= threshold are classified into group 1; while the neighboring samples Rec' L[x, y] > threshold are classified into group 2.

[0063] Correspondingly, the chroma prediction is obtained using a linear model: .

[0064] Chroma Intra Mode Coding For chroma intra mode coding, a total of 8 chroma intra mode codings are allowed. These modes include five traditional intra modes and three cross-component linear model modes (CCLM, LM_A and LM_L). The chroma mode signaling and derivation process is shown below. The chroma mode coding directly depends on the intra prediction mode of the corresponding luma block. Since separate block partitioning structure for luma and chroma components is enabled in I slices, one chroma block can correspond to multiple luma blocks. Therefore, for chroma DM mode, the intra prediction mode of the corresponding luma block covering the center position of the current chroma block is directly inherited.

[0065]

[0066] As shown in Table 5, a single binarization table is used regardless of the value of sps_cclm_enabled_flag.

[0067]

[0068] The first bin indicates whether it is a regular mode (0) or an LM mode (1). If it is an LM mode, the next bin indicates whether it is an LM_CHROMA (0). If it is not an LM_CHROMA, the next 1 bin indicates whether it is an LM_L (0) or an LM_A (1). In this case, the first bin of the binarization table for the corresponding intra_chroma_pred_mode can be discarded before entropy coding when sps_cclm_enabled_flag is 0. In other words, the first bin is inferred to be 0 and thus not coded. This single binarization table is used for the case when sps_cclm_enabled_flag is equal to 0 and 1. The first two bins are context coded using its own context model, and the rest of the bins are bypass coded.

[0069] Convolutional Cross-Component Model (CCCM) In CCCM, a convolutional model is applied to improve the chroma prediction performance. The convolutional model has a 7-tap filter, which consists of a 5-tap plus sign-shaped spatial component, a non-linear term, and a bias term. The input of the spatial 5-tap component of the filter includes a center (C) luma sample, which is co-located with the chroma sample to be predicted, and its north / west (N), south (S), west (W), and east (E) neighbors, as shown in FIG. 16shown.

[0070] The non-linear term (denoted as P) is the square of the central luma sample C, scaled to the sample value range of the content: P = ( C C + midVal )>>bitDepth.

[0071] For example, for 10-bit content, the non-linear term is computed as: P = ( C C + 512 )>>10 The bias term (denoted as B) is the scalar offset between input and output (similar to the offset term in CCLM), and is set to the mid chroma value (512 for 10-bit content).

[0072] The output of the filter is computed by the convolution of the filter coefficients c i with the input values, and clipped to the range of valid chroma samples: predChromaVal = c0C + c1N + c2S + c3E + c4W + c5P + c6B The filter coefficients c i are computed by minimizing the MSE between the predicted and reconstructed chroma samples in the reference region. FIG. 17 An example of a reference region is shown, which consists of 6 rows of chroma samples above and to the left of the PU. The reference region is extended right by one PU width, and down by one PU height. The region is adjusted to only contain available samples. The extension of the region (denoted as "padding") is to support FIG. 16 "side samples" of the plus-shaped spatial filter, and to pad the unavailable regions.

[0073] The MSE minimization is performed by computing the auto-correlation matrix of the luma input and the cross-correlation vector between the luma input and the chroma output. The auto-correlation matrix is LDL-decomposed, and the final filter coefficients are computed by back-substitution. This process roughly follows the computation of the ALF filter coefficients in ECM, but LDL-decomposition is chosen instead of Cholesky decomposition to avoid the use of square root operations.

[0074] Similarly, like CCLM, CCCM has the option to use a single model or a multi-model variant. The multi-model variant uses two models, one for samples above the average luma reference value, and the other for the remaining samples (following the spirit of the CCLM design). For PUs with at least 128 reference samples available, the multi-model CCCM mode can be selected.

[0075] Gradient Linear Model (GLM) In contrast to CCLM, GLM utilizes the gradient of the luma samples to derive the linear model instead of down-sampled luma values. Specifically, when GLM is applied, the input to the CCLM process, i.e., the down-sampled luma samples , are replaced by the gradient of the luma samples . Other parts of CCLM (e.g., parameter derivation, prediction sample linear transformation) remain unchanged.

[0076]

[0077] In terms of signaling, when the current CU enables the CCLM mode, two flags are sent for Cb and Cr components, respectively, to indicate whether GLM is enabled for each component. If GLM is enabled for one component, one syntax element is further sent to select one of the 16 gradient filters (1810-1840 in FIG. 18 ) for gradient calculation. GLM can be used in combination with the existing CCLM by sending one extra flag in the bitstream. When this combination is applied, the filter coefficients for deriving the linear model input luma samples are calculated as the combination of the gradient filter selected by GLM and the down-sampling filter of CCLM.

[0078] Spatial GPM JVET-Z0124 (Fan Wang et al., “Non-EE2: Spatial GPM,” Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 26th Meeting, April 20-29, 2022, via teleconference, document JVET-Z0124) proposes to extend GPM to intra prediction. Similar to inter GPM, Spatial Geometric Partitioning Mode (SGPM) consists of one partition mode and two associated intra prediction modes, as shown in FIG. 19A . If these modes are sent directly in the bitstream, as shown in FIG. 19B , significant overhead bits will be incurred. To express the necessary partition and prediction information more efficiently in the bitstream, a candidate list is used, and only the candidate index is sent in the bitstream. Each candidate in the list can derive a combination of one partition mode and two intra prediction modes, as shown in FIG. 19C .

[0079] A template is used to generate this candidate list. The shape of the template is as shown in FIG. 20A prediction for a template is generated and the partition weights are extended to the template for each possible combination of one partition mode and two intra prediction modes, as shown in FIG. 20 The combinations are sorted in ascending order according to the SATD between the prediction and the template reconstruction. The length of the candidate list is set to 16, and these candidates are considered as the most probable SGPM combinations for the current block. Both the encoder and the decoder build the same candidate list based on the templates.

[0080] To reduce the complexity of building the candidate list, the number of possible partition modes and the number of possible intra prediction modes are pruned. In the following tests, 26 out of 64 partition modes are used, and only the MPMs out of 67 intra prediction modes are used.

[0081] The following shows more schemes to accelerate the SGPM encoding time and improve the SGPM gain. For more details on SGPM, see JVET-AA0118 (Fan Wang et al., “EE2-1.4: Spatial GPM,” Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 27th Meeting, Through Telephone Conference, 13-22 July 2022, Document JVET-AA0118).

[0082] Matrix weighted intra prediction The matrix weighted intra prediction (MIP) method is a newly added intra prediction technique in VVC. To predict the samples of a rectangular block with width and height , the matrix weighted intra prediction (MIP) takes as input one row of H reconstructed neighboring boundary samples on the left side of the block and one row of reconstructed neighboring boundary samples on the top of the block. If the reconstructed samples are not available, they are generated following the method of traditional intra prediction. The generation of the prediction signal is based on three steps, namely averaging, matrix-vector multiplication, and linear interpolation, as shown in FIG. 21 The one row of H reconstructed neighboring boundary samples 2112 on the left side of the block and the one row of reconstructed neighboring boundary samples 2110 on the top of the block are shown as small squares filled with dots. After the averaging process, the boundary samples are down-sampled to the top boundary line 2114 and the left boundary line 2116. The down-sampled samples are provided to the matrix-vector multiplication unit 2120 to generate the down-sampled prediction block 2130. Then the interpolation process is applied to generate the prediction block 2140.

[0083] Averaging neighboring samples In the border samples, four samples or eight samples are selected by averaging according to the block size and shape. Specifically, the input border and are reduced to smaller border and by averaging neighboring border samples according to a predetermined rule depending on the block size. and are then concatenated into one reduced border vector with size four for blocks of shape and size eight for all other block shapes. If refers to the MIP mode, this concatenation is defined as follows:

[0084] The matrix multiplication is followed by an addition of an offset, with the average samples as input. The result is a reduced prediction signal on a sub-sampled set of samples of the original block. From the reduced input vector a reduced prediction signal is generated, which is a signal on a down-sampled block of width and height . Here, and are defined as: , The reduced prediction signal is computed by computing the matrix vector product and adding an offset:

[0085] Here, is a matrix with rows and 4 columns for and 8 columns for all other cases. is a vector of size . The matrix and the offset vector are taken from one of the sets , , . An index is defined as follows: , Here, each coefficient of the matrix A is represented with 8-bit precision. The sets consist of 16 matrices each with 16 rows and 4 columns, and 16 offset vectors , each of size 16. The matrix and offset vectors of this set are used for blocks of size . The set is composed of 8 matrices , each of size 64 rows and 8 columns, and 8 offset vectors , each of size 64. The set is composed of 6 matrices , each of size

[0086] 64 rows and 8 columns, and 6 offset vectors , each of size 64.

[0087] Signaling of MIP mode and coordination with other coding tools For each coding unit (CU) in intra mode, a flag is sent to indicate whether MIP mode is applied or not. If MIP mode is to be applied, the MIP mode is sent. For MIP mode, a transposed flag is derived, which determines whether the mode is transposed or not, and the MIP mode Id ( ), which determines which matrix is used for a given MIP mode.

[0088] , MIP coding mode is coordinated with other coding tools by considering the following aspects: LFNST (low frequency non-separable transform) is enabled for MIP on large blocks. Here, the LFNST transform is used with the planar mode The reference sample derivation for MIP is exactly the same as for the conventional intra prediction modes In the upsampling step used in MIP prediction, the original reference samples are used instead of the downsampled samples Clipping is performed before upsampling, instead of after upsampling MIP allows up to 64x64, without considering the maximum transform size The number of MIP modes is 32 for sizeId=0, 16 for sizeId=1, and 12 for sizeId=2 Modification of LFNST for blocks coded via MIP In JVET-AB0067 (Junyan Huo, et. al., “EE2-4.1: Modification of LFNST for MIP coded block”, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 28th Meeting, Mainz, DE, 21–28 October 2022, Document: JVET-AB0067), it is proposed to derive the LFNST transform set and determine the LFNST transposition flag using DIMD.

[0089] In FIG. 22 , a matrix-vector multiplication 2210 is applied to the input data to generate a MIP prediction output 2220. A MIP prediction 2230 with upsampling is applied to the MIP prediction output 2220 to generate an upsampled MIP prediction 2240. The proposed method uses DIMD 2250 to derive the intra prediction mode of the current block based on the MIP prediction samples before upsampling. Specifically, a horizontal gradient and a vertical gradient are calculated for each prediction sample to construct a HoG 2260, as shown in FIG. 22 . The intra prediction mode with the largest histogram amplitude is then used to determine the LFNST transform set and the LFNST transposition flag.

[0090] Furthermore, LFNST is enabled for MIP coded blocks with width and height greater than or equal to 4.

[0091] EE2-4.1: Modification of LFNST for MIP coded block In ECM6.0 (Muhammed Coban, et al., “Algorithm description of Enhanced Compression Model 6 (ECM 6)”, Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 27th Meeting, by teleconference, 13–22 July 2022, Document JVET-AA2025), both the LFNST transform set and the LFNST transposition flag are determined by the intra prediction mode predModeIntra of the current transform block. Using predModeIntra, the following operations are performed: If the current block is a MIP coded block, predModeIntra is mapped to PLANAR; If the current block is a CCLM coded block, predModeIntra is mapped to the collocated luma intra prediction mode; predModeIntra is then further derived from the wide angle intra prediction mapping, ranging from [-14, 83]. Selection of LFNST transform set In ECM 6.0, there are 35 transform sets in LFNST and 3 non-separable transform matrices (kernels) in each transform set. The transform set index lfnstTrSetldx is defined according to the predModeIntra list in Table 6.

[0092]

[0093] Determination of LFNST transposed flag The LFNST transposed flag determines the scan order of the LFNST output (decoder). FIG. 23A B shows the scan order for different LFNST transposed flags FIG. 23A For the flag equal to 0, FIG. 23B For the flag equal to 1).

[0094] The LFNST transposed flag is determined by predModeIntra as follows: If predModeIntra is less than or equal to 34, the LFNST transposed flag is set to 0; Otherwise, the LFNST transposed flag is set to 1.

[0095] For MIP coded blocks, it is mapped to the PLANAR mode, the LFNST transform set 0 is used, and the LFNST transposed flag is always equal to 0.

[0096] In ECM 6.0, LFNST is enabled for MIP coded blocks with width and height greater than or equal to 16.

[0097] Prediction process for MIP As mentioned previously, the matrix weighted intra prediction (MIP) takes as input a row of H reconstructed neighboring boundary samples to the left of the block and a row of W reconstructed neighboring boundary samples above the block. The generation of the predicted samples is based on the following three steps: the input 2410 includes the boundary samples (shown as darker squares) around the current block provided to the boundary down-sampling module 2420; it is then processed by the matrix vector multiplication module 2430 to generate the MIP prediction 2440; and it is further processed by the MIP prediction up-sampling module 2450 to generate the up-sampled output 2460, as shown. FIG. 24 ​

[0098] In particular, MIP first down-samples the reference samples, then multiplies the down-sampled reference samples with a prediction matrix to generate partial prediction samples. Finally, it up-samples to generate prediction samples at the remaining positions.

[0099] Chroma DM mode For the chroma DM mode, the intra prediction mode of the corresponding (collocated) luma block covering the center position of the current chroma block is directly inherited.

[0100] Intra block copy Intra block copy (IBC) is a tool adopted in the screen content coding (SCC) in HEVC extension. It is well known that it significantly improves the coding efficiency of screen content material. Since IBC mode is implemented as a block-level coding mode, block matching (BM) is performed in the encoder to find the best block vector (or motion vector) for each CU. Here, the block vector is used to indicate the displacement from the current block to a reference block, which has been reconstructed within the current picture. The luma block vector of an IBC-coded CU is integer precision. The chroma block vector is also rounded to integer precision. When used in combination with adaptive motion vector resolution (AMVR), the IBC mode can switch between 1-pel and 4-pel motion vector precision. An IBC-coded CU is considered as a third prediction mode in addition to the intra or inter prediction modes. IBC mode is applicable to a CU whose width and height are both less than or equal to 64 luma samples.

[0101] At the encoder side, hash-based motion estimation is performed for IBC. The encoder performs RD check for blocks whose width or height is not larger than 16 luma samples. For non-merge mode, block vector search is first performed using hash-based search. If hash search does not return a valid candidate, a local search based on block matching will be performed.

[0102] In hash-based search, the hash key (32-bit CRC) between the current block and the reference block is extended to all allowed block sizes. The hash key computation for each position in the current picture is based on 4x4 sub-blocks. For larger size of the current block, a hash key is determined to match the hash key of the reference block when the hash keys of all 4x4 sub-blocks match the hash keys of the corresponding reference positions. If multiple reference blocks are found whose hash keys match the hash key of the current block, the block vector cost of each matching reference is computed and the one with the lowest cost is selected.

[0103] In block matching search, the search range is set to cover the previous and current CTU.

[0104] At the CU level, IBC mode is sent via flags, and can be sent as IBC Advanced Motion Vector Prediction (AMVP) mode or IBC skip / merge mode, as shown below: IBC Skip / Merge Mode: Uses merge candidate indices to indicate which block vector to use to predict the current block from a list of neighboring candidate IBC codec blocks. The merge list consists of space, history-based motion vector prediction (HMVP), and paired candidates.

[0105] IBC AMVP mode: Block vector interpolation is encoded and decoded in the same way as motion vector interpolation. The block vector prediction method uses two candidates as predictors, one from the left neighbor and one from the upper neighbor (if it is IBC encoding and decoding). When either neighbor is unavailable, a default block vector is used as the predictor. A flag is sent to indicate the block vector predictor index.

[0106] IBC Reference Area To reduce memory consumption and decoder complexity, IBC in VVC only allows a portion of the predetermined region to be reconstructed, including the region of the current CTU and certain regions of the left CTU. FIG. 25 This displays a reference area for the IBC mode, where each block represents a 64x64 lumen sample unit. The following rules apply based on the current position of the codec CU within the current CTU: If the current block is located in the top left 64x64 block of the current CTU ( FIG. 25 In case 2510, in addition to the samples already reconstructed in the current CTU, reference samples in the lower right 64x64 block of the left CTU can also be referenced using the current picture referencing (CPR) mode. (More details about CPR can be found in JVET-T2002 (Jianle Chen et al., “Algorithm description of Versatile video codec and test model 11 (VTM 11),” Joint Video Experts Group (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 20th meeting, via teleconference, 7-16 October 2020, document: JVET-T2002). The current block can also use the CPR mode to reference reference samples in the lower left 64x64 block and the upper right 64x64 block of the left CTU.

[0107] If the current block is located in the upper right 64x64 block of the current CTU ( FIG. 25In case 2520, in addition to the samples that have been reconstructed in the current CTU, if the brightness position (0, 64) relative to the current CTU has not been reconstructed, the current block can also use CPR mode to reference the reference samples in the lower left 64x64 block and the lower right 64x64 block of the left CTU; otherwise, the current block can also reference the reference samples in the lower right 64x64 block of the left CTU.

[0108] If the current block is located in the lower left 64x64 block of the current CTU ( FIG. 25 In case 2530, besides the samples already reconstructed in the current CTU, if the brightness position (64, 0) relative to the current CTU has not yet been reconstructed, the current block can also use CPR mode to reference reference samples in the upper right 64x64 block and the lower right 64x64 block of the left CTU. Otherwise, the current block can also use CPR mode to reference reference samples in the lower right 64x64 block of the left CTU.

[0109] If the current block is located in the bottom right 64x64 block of the current CTU ( FIG. 25 In case 2540, only the CPR mode can be used to reference samples that have already been reconstructed in the current CTU.

[0110] This restriction allows the IBC mode to be implemented in hardware using local on-chip memory.

[0111] Inter-frame prediction overview According to Section 3.4 of JVET-T2002 (Jianle Chen et al., “Algorithm Description of Versatile Video Coding and Test Model 11 (VTM 11)”, Joint Video Experts Group (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 20th Meeting, via Teleconference, 7-16 October 2020, Document: JVET-T2002), for each CU for inter-frame prediction, motion parameters include motion vectors, reference image indexes, and reference image list usage indexes, as well as additional information for new VVC coding and decoding features used for inter-frame prediction sample generation. Motion parameters can be signaled explicitly or implicitly. When a CU is encoded in skip mode, the CU is associated with a PU and has no significant residual coefficients, no encoded motion vector increments, and no reference image indexes. A merge mode is specified where the motion parameters of the current CU are obtained from neighboring CUs, including spatial and temporal candidates, and additional plans introduced in the VVC. The merge mode can be applied to any inter-frame predicted CU, not just the skip mode. An alternative to the merge mode is to explicitly transmit motion parameters, where the motion vector of each CU, the corresponding reference image index, and the list of reference images are explicitly sent using flags and other necessary information.

[0112] In addition to the inter-frame encoding and decoding features in HEVC, VVC also includes some new and improved inter-frame prediction encoding and decoding tools, listed below: Extended merge forecast Merge mode with MMVD Symmetric MVD (SMVD) transmission Affine Motion Compensation Prediction Subblock-based temporal motion vector prediction (SbTMVP) Adaptive motion vector resolution (AMVR) Motion field storage: MV storage with a precision of 1 / 16 luminance samples and 8x8 motion field compression Bi-prediction with CU-level weight (BCW) Bidirectional optical flow (BDOF) Decoder-side motion vector refinement (DMVR) Geometric partitioning mode (GPM) Combined inter and intra prediction (CIIP) The following description provides detailed information about the inter-frame prediction methods specified in VVC.

[0113] Extended merge forecast In VVC, the merge candidate list is constructed by sequentially including the following five types of candidates: 1) Spatial MVP from spatially adjacent CUs 2) Time MVP from the same CU 3) Historical MVP from FIFO table 4) Paired average MVP 5) Zero MV.

[0114] The size of the merge list is sent in the sequence parameter set (SPS) header, and the maximum allowed size of the merge list is 6. For each coding unit (CU) encoding and decoding in merge mode, the index of the best merge candidate is encoded using unary binarization (TU). The first bit of the merge index is encoded using context encoding, and the remaining bits are encoded using bypass encoding.

[0115] This section provides the derivation process for the merge candidates for each category. Similar to HEVC, VVC also supports parallel derivation of the merge candidate list (or merge candidate list) for all CUs within a given region.

[0116] Spatial candidate derivation The derivation of spatial merge candidates in VVC is the same as in HEVC, except that the positions of the first two merge candidates are swapped. Currently, the maximum of four merge candidates (B0, A0, B1, and A1) for CU 2610 are derived from... FIG. 26 The candidate positions are selected from those depicted in the image. The derivation order is B0, A0, B1, A1, and B2. Position B2 is considered only if one or more adjacent CUs of positions B0, A0, B1, and A1 are unavailable (e.g., belonging to another slice or tile) or if it is an intra-frame encoding / decoding operation. After the candidate for position A1 is added, the addition of the remaining candidates requires a redundancy check to ensure that candidates with the same motion information are excluded from the list, thereby improving encoding / decoding efficiency. To reduce computational complexity, not all possible candidate pairs are considered in the above redundancy check. Instead, only... FIG. 27 The pairs are linked by arrows, and a candidate is added to the list only if the corresponding candidate used for redundancy checking does not have the same motion information.

[0117] Merge Mode with Motion Vector Interpolation (MMVD) In addition to the merge mode, where implicitly derived motion information is directly used to generate prediction samples for the current CU, VVC introduces a merge mode with Motion Vector Difference (MMVD). After sending the regular merge flag, the MMVD flag is sent to specify whether the MMVD mode is used for the CU.

[0118] In MMVD, after selecting a merge candidate (referred to herein as the base merge candidate), further optimization is performed using the MVDs information sent. This further information includes a merge candidate flag, an index specifying the motion amplitude, and an index indicating the motion direction. In MMVD mode, one of the first two candidates in the merge list is selected as the MV base. The MMVD candidate flag is sent to specify which merge candidate to use between the first and second merge candidates.

[0119] The distance index specifies motion amplitude information and indicates a predetermined offset of the starting points (2812 and 2822) of L0 reference block 2810 and L1 reference block 2820. For example... FIG. 28 As shown, offsets are added to the horizontal or vertical component of the starting MV, with different styles of small circles corresponding to different offsets from the center. The relationship between the distance index and the predetermined offset is specified in Table 7.

[0120]

[0121] The direction index indicates the direction of the MVD relative to the starting point. The direction index can represent the four directions shown in Table 8. It is important to note that the meaning of the MVD symbol may vary depending on the information of the starting MV. When the starting MV is an unpredictable MV or a double-predictable MV and both lists point to the same side of the current image (i.e., both reference POCs are greater than or less than the current image's POC), the symbol in Table 8 specifies the sign of the MV offset added to the starting MV. When the starting MV is a double-predictable MV and the two MVs point to different sides of the current image (i.e., one reference POC is greater than the current image's POC, and the other reference POC is less than the current image's POC), and the POC difference in list 0 is greater than the difference in list 1, the symbol in Table 8 specifies the sign of the MV offset added to the list 0 MV component of the starting MV, while the sign of the list 1 MV has the opposite value. Otherwise, if the POC difference in list 1 is greater than that in list 0, the symbol in Table 8 specifies the sign of the MV offset added to the list 1 MV component of the starting MV, while the sign of the list 0 MV has the opposite value.

[0122] MVD scales based on the POC difference in each direction. If the POC differences in two lists are the same, no scaling is needed. Otherwise, if the POC difference in list 0 is greater than the difference in list 1, then MVD for list 1 is scaled, by defining the POC difference of L0 as td and the POC difference of L1 as tb, as follows. FIG. 28 As stated above. If the POC difference of L1 is greater than that of L0, then the MVD of list 0 is scaled in the same way. If the initial MV is a one-way prediction, then the MVD is added to the available MV.

[0123]

[0124] Affine Motion Compensation Prediction In HEVC, only a translational motion model is used for motion compensation prediction (MCP). However, in the real world, there are many types of motion, such as scaling, rotation, perspective motion, and other irregular motions. In VVC, block-based affine transformation motion compensation prediction is used. For example... FIG. 29A As shown in -B, the affine motion field of the current block is... FIG. 29A Two control points (4 parameters) or FIG. 29B The motion information description of the motion vectors (6 parameters) of the three control points in the model.

[0125] For the 4-parameter affine motion model, the motion vector of the sample position (x, y) in the block is derived as follows: (6) For the 6-parameter affine motion model, the motion vector of the sample position (x, y) in the block is derived as follows: (7) Where (mv0x, mv0y) is the motion vector of the top left control point, (mv1x, mv1y) is the motion vector of the top right control point, and (mv2x, mv2y) is the motion vector of the bottom left control point.

[0126] To simplify motion-compensated prediction, a block-based affine transformation prediction was applied. To derive the motion vector for each 4×4 luma sub-block, the motion vector of the center sample of each sub-block was calculated according to the above equation and rounded to 1 / 16 fractional precision. Then, a motion-compensated interpolation filter was applied to generate predictions for each sub-block with the derived motion vectors. The sub-block size for the chroma components was also set to 4×4. The MV of the 4×4 chroma sub-block was calculated as the average of the MVs of the upper-left and lower-right luma sub-blocks in the corresponding 8x8 luma region.

[0127] Similar to translational motion inter-frame prediction, there are two affine motion inter-frame prediction modes: affine merging mode and affine AMVP mode.

[0128] Affine Merging Prediction The AF_MERGE mode can be applied to coding units (CUs) with a width and height both greater than or equal to 8. In this mode, the control point motion vector (CPMV) of the current CU is generated based on the motion information of spatially adjacent CUs. There can be a maximum of five CPMV prediction candidates, indicated by an index sent to the candidate for the current CU. The following three types of CPMV candidates are used to form the affine merging candidate list: Inherited affine merge candidates derived from the CPMVs of adjacent CUs Constructing affine merging candidate CPMVPs derived using translational motion information of adjacent CUs Zero motion vector In VVC, there are at most two inherited affine candidates, which are derived from the affine motion models of neighboring blocks: one from the left-neighboring CU and one from the upper-neighboring CU. Candidate blocks and FIG. 26 The same applies as shown. For the left predictor, the scan order is A0->A1, and for the upper predictor, the scan order is B0->B1->B2. Only the first successor candidate is selected on each side. No pruning check is performed between two successor candidates. When an adjacent affine CU is identified, its control point motion vector is used to derive the CPMVP candidate in the affine merging list of the current CU. FIG. 31 As shown, if the lower-left adjacent block A of the current block 3110 is encoded in affine mode, then the motion vectors of the upper-left, upper-right, and lower-left corners of the CU 3120 containing block A are obtained. , and When block A is encoded and decoded using a 4-parameter affine model, the two CPMVs of the current CU (i.e. and )according to and Calculation. If block A is encoded and decoded using a 6-parameter affine model, then the three CPMVs of the current CU are calculated according to... , and calculate.

[0129] Constructing affine candidates means that candidates are built by combining the translational motion information of adjacent control points. The motion information of the control points comes from... FIG. 32 The diagram shows the derivation of the specified spatial and temporal neighbors for the current block 3210. CPMVk (k=1, 2, 3, 4) represents the k-th control point. For CPMV1, the B2->B3->A2 block is checked, and the motion vector of the first available block is used. For CPMV2, the B1->B0 block is checked, and for CPMV3, the A1->A0 block is checked. If available, the TMVP is used as CPMV4.

[0130] After obtaining the motion vectors of the four control points, affine merging candidates are constructed based on the motion information. These are constructed sequentially using the following combinations of control point motion vectors: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4},{CPMV2, CPMV3, CPMV4}, { CPMV1, CPMV2}, { CPMV1, CPMV3} Combining three CPMVs constructs a 6-parameter affine merging candidate, and combining two CPMVs constructs a 4-parameter affine merging candidate. To avoid motion scaling, if the reference indices of the control points are different, the associated control point motion vector combinations are discarded.

[0131] After checking the inherited affine merge candidates and the constructed affine merge candidates, if the list is still not full, insert a zero motion vector at the end of the list.

[0132] Affine AMVP Prediction The affine AMVP mode can be applied to CUs with a width and height both greater than or equal to 16. An affine flag is sent to the bitstream at the CU level to indicate whether the affine AMVP mode is used, followed by another flag to indicate whether a 4-parameter affine or a 6-parameter affine is used. In this mode, the difference between the current CU's CPVM and its predicted sub-CPVM is sent in the bitstream. The affine AVMP candidate list is 2 in size and is generated sequentially using the following four types of CPVM candidates: Inherited Affine AMVP Candidates Derived from Adjacent CUs' CPMVs A candidate CPMVP for constructing an affine AMVP is derived using translational motion information from adjacent CUs. Translational motion vector from adjacent CU Zero motion vector The checking order for inherited affine AMVP candidates is the same as that for inherited affine merge candidates. The only difference is that, for AVMP candidates, only affine CUs with the same reference image as the current block are considered. No pruning is applied when inserting inherited affine motion predictors into the candidate list.

[0133] Constructing AMVP candidates from FIG. 32 The specified spatial neighbor derivation is shown. The same checking order as in the affine merge candidate construction is used. Additionally, the reference image indices of adjacent blocks are checked. In the checking order, the first block that is inter-coded and has the same reference image as the current CU is used. This is done when the current CU is encoded and decoded in 4-parameter affine mode, and... and If all three CPMVs are available, they are added as a candidate to the affine AMVP list. If the current CU is encoding / decoding in 6-parameter affine mode and all three CPMVs are available, they are added as a candidate to the affine AMVP list. Otherwise, the constructed AMVP candidates are set to unavailable.

[0134] If, after inserting valid inherited affine AMVP candidates and constructing AMVP candidates, the number of candidates in the affine AMVP list is still less than 2, then , and This will be added as a translational motion vector to predict the motion vectors of all control points in the current CU (if available). Finally, if the affine AMVP list is still not full, it will be filled with a zero motion vector.

[0135] Geometric Partitioning Mode (GPM) In VVC, geometric partitioning mode for inter-frame prediction is supported. Geometric partitioning mode uses CU-level flags for sending as a merging mode; other merging modes include regular merging mode, MMVD mode, CIIP mode, and sub-block merging mode. A total of 64 partitions are supported for each possible CU size. ,in Exclude 8x64 and 64x8.

[0136] When using this mode, the CU is divided into two parts by a geometrically positioned straight line (e.g., FIG. 33 (As shown). The position of the dividing line is mathematically derived from the angle and offset parameters of a specific partition. Each part of the geometric partition in the CU uses its own motion for inter-frame prediction; each partition only allows unidirectional prediction, i.e., each part has one motion vector and one reference index. Unidirectional prediction motion constraints are applied to ensure that, as with traditional bidirectional prediction, each CU only requires two motion-compensated predictions. The unidirectional prediction motion for each partition is derived.

[0137] If the current CU uses a geometric partitioning pattern, a geometric partition index indicating the partitioning pattern (angle and offset) is further sent, along with two merge indices (one for each partition). The number of maximum GPM candidate sizes is explicitly sent in the SPS, specifying the syntax binary of the GPM merge index. After predicting each part of the geometric partition, sample values ​​along the edges of the geometric partition are adjusted using a blending process with adaptive weights. This is the predicted signal for the entire CU, and the transform and quantization processes are applied to the entire CU as with other prediction patterns. Finally, the motion field of the CU predicted using the geometric partitioning pattern is stored.

[0138] One-way prediction candidate list construction The unidirectional prediction candidate list is directly derived from the merge candidate list constructed based on the extended merge prediction process. Let n be the index of the unidirectional prediction motion in the geometric unidirectional prediction candidate list. The LX motion vector (X = 0 or 1, i.e., LX = L0 or L1) of the nth extended merge candidate, where X equals the parity of n, is used as the nth unidirectional prediction motion vector of the geometric partitioning pattern. These motion vectors are... FIG. 34 The value is marked as "x". If the corresponding LX motion vector of the nth extended merge candidate does not exist, the L(1 – X) motion vector of the same candidate is used as the unidirectional predicted motion vector of the geometric partitioning pattern.

[0139] Blending along the edges of geometric partitions After using each part of its own motion prediction geometric partition, a blending process is applied to combine the two prediction signals to derive samples around the edges of the geometric partition. The blending weights at each location of the CU are derived based on the distance between the individual location and the partition edge, such as... FIG. 35 As shown.

[0140] Combined Inter and Intra Prediction (CIIP) In VVC, when a CU is encoding / decoding in merged mode, if the CU contains at least 64 luma samples (i.e., CU width multiplied by CU height equals or greater than 64), and if both the CU width and CU height are less than 128 luma samples, an additional flag is sent to indicate whether to apply the Combined Inter-Frame and Intra-Frame Prediction (CIIP) mode to the current CU. As its name suggests, CIIP prediction combines inter-frame prediction signals and intra-frame prediction signals. The inter-frame prediction signal in CIIP mode... The intra-frame prediction signal is derived using the same inter-frame prediction processing applied to the regular merging mode. The intra-frame prediction is derived using a planar pattern through conventional intra-frame prediction processing. Then, the intra-frame and inter-frame prediction signals are combined using a weighted average, where the weight value wt is based on the top and left adjacent blocks of the current CU 3610 (e.g., ...). FIG. 36 The encoding / decoding mode (as shown) is calculated as follows: If the top neighbor is available and intra-frame encoding / decoding is used, set isIntraTop to 1; otherwise, set isIntraTop to 0. If the left neighbor is available and intra-frame encoding / decoding is used, set isIntraLeft to 1; otherwise, set isIntraLeft to 0. If (isIntraLeft + isIntraTop) equals 2, then set wt to 3; Otherwise, if (isIntraLeft + isIntraTop) equals 1, then set wt to 2; Otherwise, set wt to 1.

[0141] The CIIP predictions are as follows: (8) GPM extension Several variants of the GPM mode were proposed (JVET-W0097 (Zhipin Deng et al., “AEE2-related: Combination of EE2-3.3, EE2-3.4 and EE2-3.5”, Joint Video Experts Group (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 23rd meeting, via teleconference, July 7-16, 2021, document: JVET-W0097) and JVET-Y0065 (Yoshitaka Kidani et al., “EE2-3.1: GPM with inter and intra-frame prediction (JVET-X0166)”, ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29). The Joint Video Experts Group (JVET), 29th meeting, held via teleconference from January 12th to 21st, 2022 (document: JVET-Y0065), aimed to improve the encoding and decoding efficiency of GPM mode in VVC. These methods were incorporated into the exploratory experiment (EE2) for further evaluation, and their main technical aspects are described below: EE2-3.3 Regarding GPM and MMVD (GPM-MMVD): 1) Add an additional MVD to the existing GPM merge candidate; 2) The MVD is sent in the same way as the MMVD in VVC, i.e., a distance index plus a direction index; 3) Send two flags to control whether the MMVD is applied to each GPM partition.

[0142] EE2-3.4-3.5 Regarding GPM and Template Matching (GPM-TM): 1) Template matching is extended to GPM mode by refining GPM MVs based on the left and top adjacent samples of the current CU; 2) Template samples are selected according to the GPM segmentation direction; 3) A single flag is sent to jointly control whether template matching is applied to the MVs of two GPM partitions.

[0143] JVET-W0097 proposes a combination of EE2-3.3, EE2-3.4, and EE2-3.5 to further improve the encoding and decoding efficiency of GPM mode. Specifically, in the proposed combination, the existing designs in EE2-3.3, EE2-3.4, and EE2-3.5 remain unchanged, while the following modifications are further applied to reconcile the two encoding and decoding tools: 1) GPM-MMVD and GPM-TM are exclusively enabled on a single GPM CU. This is accomplished by first sending the GPM-MMVD syntax. When both GPM-MMVD control flags are equal to false (i.e., GPM-MMVD is disabled for both GPM partitions), the GPM-TM flag is sent to indicate whether template matching is applied to both GPM partitions. Otherwise (if at least one GPM-MMVD flag is equal to true), the value of the GPM-TM flag is inferred to be false.

[0144] 2) The GPM merging candidate list generation methods in EE2-3.3 and EE2-3.4-3.5 are directly combined. The MV pruning scheme in EE2-3.4-3.5 (where the MV pruning threshold is adjusted according to the current CU size) is used to replace the default MV pruning scheme applied in EE2-3.3. In addition, as in EE2-3.4-3.5, multiple zero MVs are added until the GPM candidate list is completely filled.

[0145] In JVET-Y0065, in GPM and inter-frame and intra-frame prediction (or GPM intra-frame), the final prediction samples are generated by weighting the inter-frame and intra-frame prediction samples for each GPM-separated region. Inter-frame prediction samples are derived using the same scheme as the GPM in the current ECM, while intra-frame prediction samples are derived using an intra-prediction mode (IPM) candidate list and an index sent from the encoder. The IPM candidate list size is predefined as 3. Available IPM candidates are the angular mode parallel to the GPM block boundary (parallel mode), the angular mode perpendicular to the GPM block boundary (vertical mode), and the planar mode, such as... FIG. 37A -C is shown. Furthermore... FIG. 37D The GPM and intra-frame prediction shown are constrained in the proposed method to reduce the signaling overhead of IPM and avoid increasing the size of the intra-frame prediction circuitry on the hardware decoder. Furthermore, direct motion vectors and IPM storage on the GPM mixing region are introduced to further improve encoding and decoding performance.

[0146] This invention discloses a method for improving the transformation process. Summary of the Invention

[0147] A method and apparatus for video encoding and decoding are disclosed. According to the method, input data associated with a current block is received, wherein the input data includes residual data to be processed at an encoder end or encoded / decoded transform data associated with the current block to be decoded at a decoder end. Transform processing is applied to the input data, wherein the transform processing determines a target transform for the current block, and the target transform is selected from a transform candidate set containing at least one additional transform candidate relative to the original transform candidate set, or the target transform depends on an intra-prediction mode associated with the current block, a reference region, or one or more neighboring blocks. A transformed output is provided, wherein the transformed output includes transform coefficients of the current block at the encoder end or recovered residual data at the decoder end.

[0148] In one embodiment, the transform process includes a primary transform at the encoder end, or an inverse primary transform at the decoder end, and the transform process includes a secondary transform at the encoder end, or an inverse secondary transform at the decoder end. In one embodiment, the target transform corresponds to a separable or non-separable primary or secondary transform.

[0149] In one embodiment, the current block is encoded using IntraTMP (Intra-Template Matching Prediction) mode, Matrix Weighted Intra Prediction (MIP) mode, Spatial Geometric Partitioning Mode (SGPM), Direct Block Vector (DBV) mode, or a horizontal / vertical plane mode. In another embodiment, the current block is encoded using a cross-component mode. For example, the current block is encoded using a fusion of CCLM, MMLM, CCCM, GLM, any LM variant, and / or chroma intra-prediction modes. In yet another embodiment, the current block is encoded using inter-frame, intra-frame, or IBC modes. In one embodiment, the current block corresponds to a luma block or a chroma block.

[0150] In one embodiment, the functional control for enabling at least one additional transform candidate is sent or parsed in a Sequence Parameter Set (SPS) or a Picture Parameter Set (PPS). In one embodiment, an explicit flag is used to indicate whether at least one additional transform candidate for a chroma block is supported. In one embodiment, one or more explicit syntax elements are used to indicate the maximum number of transform candidates for at least one additional transform candidate.

[0151] In one embodiment, the original transform candidate set consists of DCT-II, and at least one additional transform candidate corresponds to one or more Multi Transform Selection (MTS) candidates. In one embodiment, on / off control is used at the block level to determine whether to use at least one additional transform candidate for the current block. In one embodiment, on / off control is indicated in each chroma transform unit (TU) or chroma transform block (TB). In one embodiment, if on / off control indicates support for one or more MTS candidates for the current block, the MTS type of the chroma TU or chroma TB of the current block is inherited from the luma TU or luma TB of the current block. In one embodiment, if on / off control indicates support for one or more MTS candidates for the current block, the MTS type of the chroma TU of the current block is indicated by the transmitted or resolved MTS index, or inferred as a predefined MTS type.

[0152] In one embodiment, when the current block uses the Joint Coding of Chroma Residual (JCCR) mode, the Cr and Cb components of the current block share the same transform selection.

[0153] In one embodiment, at least one additional transform candidate is enabled when one or more conditions are met. In one embodiment, for chroma TU or chroma TB, the one or more conditions include chroma block size, one or more chroma transform coefficients, or both. In another embodiment, the one or more conditions are uniform for both chroma and luminance components.

[0154] In one embodiment, the target transformation depends on an intra-prediction mode derived using decoder-side derivation or template-based derivation techniques, and is based on the current block or one or more neighboring blocks. In another embodiment, the target transformation depends on an intra-prediction mode indicated by a block vector or motion vector. In yet another embodiment, the intra-prediction mode is derived based on mode information in a reference region, which includes the corresponding reference block and / or neighboring regions of the corresponding reference block.

[0155] In one embodiment, the target transformation depends on an intra-prediction mode derived based on an inference technique, and the inference technique varies depending on the block width, block height, block area, or explicit syntax of the current block. Attached Figure Description

[0156] FIG. 1A An adaptive Inter / Intra video coding system incorporating loop processing is illustrated as an example.

[0157] FIG. 1B Examples illustrate FIG. 1A The encoder and its corresponding decoder.

[0158] FIG. 2 Examples of multiple tree structures corresponding to vertical binary partitions (SPLIT_BT_VER), horizontal binary partitions (SPLIT_BT_HOR), vertical ternary partitions (SPLIT_TT_VER), and horizontal ternary partitions (SPLIT_TT_HOR) are provided.

[0159] FIG. 3 An example of a signaling mechanism for segmenting information in a nested multi-type tree encoding and decoding tree structure in a quadtree is provided.

[0160] FIG. 4 An example is provided illustrating how a CTU is segmented into multiple CUs, using a quadtree and a nested multi-type tree codec block structure, where bold block edges represent quadtree segmentations and remaining edges represent multi-type tree segmentations.

[0161] FIG. 5 Examples are provided illustrating some instances where TT segmentation is prohibited when the width or height of the luma codec block is greater than 64.

[0162] FIG. 6 This example illustrates the intra-frame prediction mode used in the VVC video codec standard.

[0163] FIG. 7 An example is provided to illustrate the positions of adjacent blocks (L, A, BL, AR, AL) used to derive a general MPM list.

[0164] FIG. 8A -B provides an example illustrating wide-angle intra-frame prediction, where FIG. 8A For blocks whose width is greater than their height, FIG. 8B A block whose height is greater than its width.

[0165] FIG. 9A An example of a selected template is provided, wherein the template includes T rows above the current block and T columns to the left of the current block.

[0166] FIG. 9B An example is illustrated with T=3, and HoGs (gradient histograms) are calculated for the pixels in the middle row and the middle column.

[0167] FIG. 9C An example of the amplitude (ampl) of the angular intra-frame prediction mode is illustrated.

[0168] FIG. 10An example of the blending process is illustrated, in which two intra-frame modes (M1 and M2) are selected based on the two highest bars of the histogram.

[0169] FIG. 11 An example is provided for using adjacent reconstructed Y, Cb, and Cr samples to derive the DIMD gradient.

[0170] FIG. 12 An example of a template-based intra-frame mode derivation (TIMD) mode is illustrated, where TIMD implicitly derives the intra-frame prediction mode of the CU at the encoder and decoder using adjacent templates.

[0171] FIG. 13A An example of intra-frame sub-partitioning (ISP) is illustrated, in which a block is divided into two sub-blocks horizontally or vertically.

[0172] FIG. 13B An example of an intra-frame sub-partition (ISP) is illustrated, in which a block is divided into four sub-blocks horizontally or vertically.

[0173] FIG. 14 An example of template matching prediction is illustrated below.

[0174] FIG. 15 The example illustrates the relative sample position of the chroma block, the corresponding luminance block, and the "Type-0" content of its adjacent samples (displayed as filled circles and triangles).

[0175] FIG. 16 An example of the spatial portion of a convolutional filter is illustrated below.

[0176] FIG. 17 An example of a padded reference region for deriving filter coefficients is illustrated.

[0177] FIG. 18 The following examples illustrate 16 gradient modes of the Gradient Linear Model (GLM).

[0178] FIG. 19A -C provides an example illustrating spatial GPM and related signaling.

[0179] FIG. 20 An example template for Spatial GPM (SGPM) is provided.

[0180] FIG. 21 An example of the processing flow for matrix-weighted intra-frame prediction (MIP) is provided.

[0181] FIG. 22 An example of LFNST modification of a MIP coded block is illustrated, which uses DIMD to derive the LFNST transform set and determine the LFNST transpose flag.

[0182] FIG. 23A -B provides an example illustrating the LFNST output scan order for different LFNST transpose flags, where FIG. 23A The flag is equal to 0. FIG. 23B The flag is set to 1.

[0183] FIG. 24 An example of the processing flow for matrix-weighted intra-frame prediction (MIP) is provided.

[0184] FIG. 25 An example of an IBC reference region is illustrated below.

[0185] FIG. 26 An example is provided for deriving adjacent blocks as candidates for VVC space merging.

[0186] FIG. 27 This example illustrates possible candidate pairs for considering redundancy checks in VVC.

[0187] FIG. 28 An example illustrates the distance offset from the starting MV in the horizontal and vertical directions according to the merge mode with MVD (MMVD).

[0188] FIG. 29A An example of an affine motion field for a block described by motion information from two control points (4 parameters) is provided.

[0189] FIG. 29B An example of an affine motion field for a block described by motion information of three control point motion vectors (6 parameters) is provided.

[0190] FIG. 30 An example of block-based affine transformation prediction is illustrated, where the motion vector of each 4×4 luminance subblock is derived from the control point MV.

[0191] FIG. 31 An example is provided illustrating the derivation of inheritance affine candidates based on the control point MV of adjacent blocks.

[0192] FIG. 32 An example is provided illustrating how to construct an affine candidate by combining translational motion information of each control point in terms of spatial neighbors and time.

[0193] FIG. 33 An example of GPM segmentation grouped by the same angle is illustrated.

[0194] FIG. 34 An example of single-predict MV selection for a geometric partitioning pattern is illustrated.

[0195] FIG. 35An example illustrating a hybrid weighting scheme using a geometric partitioning pattern is provided.

[0196] FIG. 36 An example of combined inter-frame and intra-frame prediction (CIIP) weight values ​​derived from the encoding / decoding modes of the top and left adjacent blocks is illustrated.

[0197] FIG. 37A -C provides an example of available IPM candidates: an angled pattern parallel to the GPM block boundary (parallel pattern). FIG. 37A ), and the angle mode perpendicular to the GPM block boundary (vertical mode, FIG. 37B ) and planar mode ( FIG. 37C ).

[0198] FIG. 37D An example of GPM with intra-frame and intra-frame prediction is illustrated, where intra-frame prediction is limited to reduce the signaling overhead and hardware decoder cost of IPM.

[0199] FIG. 38A An example illustrating the application of histogram analysis to spatially adjacent reconstructed samples is provided.

[0200] FIG. 38B An example illustrating the application of template analysis to spatially adjacent reconstructed samples is provided.

[0201] FIG. 39A The example illustrates a template that includes a top template, a left template, and a top-left template.

[0202] FIG. 39B An example template is provided that includes an extended top template and an extended left template.

[0203] FIG. 40 An example is provided illustrating DIMD / TIMD as an intra-prediction mode of transformation on the current predictor.

[0204] FIG. 41 An example is illustrated below of DIMD / TIMD as an intra-prediction mode of transformation in neighboring reconstruction.

[0205] FIG. 42 A flowchart illustrating an exemplary video encoding / decoding system with an improved transformation process according to an embodiment of the present invention is provided. Detailed Implementation

[0206] It will be readily understood that the components of the present invention, as described and illustrated in the figures, can be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of embodiments of the systems and methods of the present invention, as shown in the figures, is not intended to limit the scope of the claimed invention, but is merely representative of selected embodiments of the invention. References throughout the specification to “an embodiment,” “one embodiment,” or similar language mean that a particular feature, structure, or characteristic associated with that embodiment may be included in at least one embodiment of the invention. Therefore, phrases such as “in one embodiment” or “in one embodiment” appearing throughout the specification do not necessarily refer to the same embodiment.

[0207] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. However, those skilled in the art will recognize that the invention can be practiced without one or more specific details, or other methods, components, etc., can be used. In other instances, well-known structures or operations have not been shown or described in detail to avoid obscuring aspects of the invention. Embodiments of the invention will be best understood by referring to the accompanying drawings, in which similar parts are designated with similar numerals throughout. The following description is by way of example only and merely illustrates certain selected embodiments of devices and methods consistent with the invention claimed herein.

[0208] This invention discloses various improved transformation processing techniques.

[0209] This invention proposes a novel transform processing mechanism to improve residual encoding / decoding of the current block by supporting more transform candidates for the target transform mode. Part I introduces the general concepts of the transform process of our proposed mechanism. Part II describes the implicit / explicit rules for using additional transform candidates on the current block. Rules related to function control, on / off control, and candidate selection are described in Sections II.1 to II.3 of Part II, respectively. Finally, Part III defines the interactions for transform selection related to specific modes of our proposed mechanism. Note that the methods proposed in Part III are not limited to use in the transform phase and / or can be used in any processing / phase mentioned in Part III. In one embodiment, the current block can be either chroma or luma. In another embodiment, the current block can only be chroma. In yet another embodiment, the current block can only be luma.

[0210] I. General Concepts of Transform Processing In the encoder's general transform process, the residual of the current block is processed (or transformed) through the primary transform mode, becoming the first temporary transform coefficients. These temporary transform coefficients can be further processed through the secondary transform mode, becoming the second temporary transform coefficients. After the remaining operations of residual encoding and decoding, the bitstream containing the transform coefficient information is transmitted to the decoder. In the decoder's general transform process, if the secondary transform is used, the second temporary transform coefficients are obtained from the received bitstream. If the secondary transform is used, the second temporary transform coefficients may be processed through the secondary transform mode (inverse transform), then yielding the first temporary transform coefficients. The first temporary transform coefficients are processed through the first transform mode, then the residual of the current block is obtained. In the example above, at most two transform modes (primary transform mode and secondary transform mode) are used for the current block. Note that this is not limited to this case.

[0211] In one embodiment, the current block uses more than two transform modes. That is, in the encoder, more transform modes are applied after the next transform process; in the decoder, more transform modes are applied before the next transform process.

[0212] In another embodiment, the current block uses only one transform mode (e.g., the primary transform mode). For example, when NSPT is used as the primary transform, the current block uses only the primary transform (no secondary transforms).

[0213] In another embodiment, whether additional transform modes are used on the current block depends on predetermined rules. For example, the predetermined rules may be implicit. Implicit rules may relate to block width, block height, block area, and / or adjacent codec information. For another example, the predetermined rules may be explicit. Explicit rules may depend on block, SPS, PPS, tile, slice, image, and / or sequence-level flags. In one case, an explicit flag indicates the maximum number of transform modes for the current block. If the maximum number of transform modes is 2, then the primary transform mode and secondary transform modes are used for the current block. In another case, an explicit flag indicates whether more transform modes are used on the current block. For yet another example, the rules depend on block attributes such as the number of significant coefficients, the location of significant coefficients, and / or the distribution of significant coefficients. Rules may use syntax related to the location of the last significant coefficient, syntax related to whether the codec group within the current block contains any significant coefficients, and / or the location of the codec group within the current block.

[0214] In another embodiment, the proposed mechanism is to support more transform candidates for the target transform mode. The target transform mode refers to the primary transform mode, secondary transform mode, and / or any transform mode (e.g., any more than two transform modes).

[0215] In another embodiment, if the transformation mode refers to a separable transformation mode, then the transformation mode includes the transformation type of the horizontal transformation (e.g., matrix or kernel) and the transformation type of the vertical transformation; otherwise (if the transformation mode refers to a non-separable transformation mode), a joint transformation type is used. For example, the primary transformation mode is a separable transformation mode. For example, the primary transformation mode is a non-separable transformation mode. For example, the secondary transformation mode is a separable (or separable) transformation mode. For example, the secondary transformation mode is a non-separable transformation mode. An example of a separable transformation mode is setting {horizontal transformation, vertical transformation} to {DCT-II, DCT-II}, {any transformation type of MTS, any transformation type of MTS}, {DCT-II, any transformation type of MTS}, or {any transformation type of MTS, DCT-II}. An example of a non-separable transformation is LFSNT or NSPT (non-separable primary transformation). For NSPT (a primary transform mode similar to LFNST, but with a different matrix), each NSPT type (i.e., NPST4x4, NPST4x8, NPST8x4, and NPST8x8) uses 35 transform sets, determined by the intra-frame mode. Each NSPT transform set includes three transform kernels, specified by the explicitly sent NSPT index. For examples using DCT-II or MTS as the primary transform and LFNST as the secondary transform (i.e., secondary transforms cannot be used when MTS is used), NSPT is used to replace DCT-II+LFNST with block sizes of 4x4, 4x8, 8x4, and 8x8. Therefore, LFNST4 and LFNST8 will not be tested for these block sizes. However, they are still used for larger block sizes. NSPT can be extended or modified here as follows: Additional non-separable transforms are applied to intra-blocks of 4x4, 4x8, 4x16, 8x4, 8x8, 8x16, 16x4, 16x8, and 16x16. In addition to the existing DCT-II + LFNST and adaptive MTS, additional non-separable transformations are applied. Based on different intra-frame prediction modes, 67 transform kernels are derived for each block size. Send an additional flag (e.g., after the existing MTS flag) to indicate whether the non-separable master transform is applied. If the flag is true, the syntax of the separable master transform (e.g., MTS index) is not sent and / or NSPT is used instead (e.g., send the syntax of NSPT). Otherwise, the syntax of the separable master transform (e.g., MTS index) is sent further.

[0216] In another embodiment, nullification can be used in transform mode. Whether to parse transform mode signaling at the decoder will check if any significant coefficients are located in the nullification region. This check can be performed using syntax associated with the position of the last significant coefficient, a significant flag of the codec group (i.e., whether the codec group contains any significant coefficients), and / or the position of the codec group in the current block. If so, for example, the check indicates that any significant coefficients are located in the nullification region, the transform mode signaling is bypassed or always indicates that it is disabled (bitstream consistency definition may be required); otherwise, transform mode signaling can be sent and / or parsed at the encoder and / or decoder.

[0217] In another embodiment, the current block is encoded and decoded via intra-frame, inter-frame, or IBC methods. For example, the prediction type for the current block is MODE_INTRA, MODE_INTER, or MODE_IBC. For an example of an intra-frame encoded and decoded current block, at the encoder, the residual is generated based on a prediction based on spatially nearby reference samples and the intra-frame prediction mode; at the decoder, the residual is reconstructed based on the transform process and the prediction recovered. For an example of an inter-frame encoded and decoded current block, at the encoder, the residual is generated based on a prediction based on reference samples from previously encoded and decoded images and inter-frame motion information; at the decoder, the residual is reconstructed based on the transform process and the prediction recovered. For an example of an IBC encoded and decoded current block, at the encoder, the residual is generated based on a prediction based on reference samples from the same image and block vector information; at the decoder, the residual is reconstructed based on the transform process and the prediction recovered.

[0218] II. Signaling / Use of the Proposed Mechanism This section discloses the signaling and usage of additional transform candidates. An additional transform candidate refers to any additional transform candidate allowed beyond those in the original transform candidate set. For example, in conventional video codec systems, only chroma DCT-II is supported. Therefore, the original chroma transform set consists only of DCT-II.

[0219] II.1. Functional Signaling In another embodiment, the functional control of the proposed mechanism depends on predetermined rules. For example, the predetermined rules are implicitly defined based on sequence resolution, QP, sequence attributes (e.g., whether it is screen content), configuration (e.g., AI / RA / LB), image type (e.g., LDB), and / or slice type (e.g., inter-frame or intra-frame slice). For example, the predetermined rules are explicitly defined based on block, CTU, sequence, slice, tile, image, SPS, PPS level syntax. For example, explicit flags indicate whether the functionality of the proposed mechanism is supported. For example, explicit flags indicate the maximum number of candidates or the maximum number of candidates for the target transform mode of the current block.

[0220] II.2. On / Off Control In another embodiment, whether the proposed mechanism is applied to the current block depends on predetermined rules. When the predetermined rules are met, the proposed mechanism is applied to the current block. For an example where the target transform mode in the proposed mechanism is the primary transform mode, more candidates for the primary transform mode of the current block are supported (in addition to the default candidate such as DCT-II).

[0221] In another sub-implementation, when the current block is a chroma block, the transform type of the target transform mode is derived from one or more corresponding luma blocks. The current chroma block can find its corresponding luma block according to note A (described later) and inherit the proposed transform type used by the predetermined corresponding luma block or the majority of corresponding luma blocks. The target transform mode is used as the primary transform mode in the following example. Note that the proposed method can be used for any other transform mode. If the predetermined or majority of corresponding luma blocks use an MTS type transform candidate as the primary transform mode, then the current chroma block will use an MTS type transform candidate (either the same as luma or derived from luma) as the primary transform mode.

[0222] In one sub-implementation, the predetermined rules are implicit. Implicit rules relate to block width, block height, block area, and / or adjacent codec information. For an example using implicit rules to determine whether to support a proposed mechanism, if the implicit rules are met, a candidate from the proposed mechanism is applied to the current block; otherwise, the original default candidate / rule (an additional candidate not belonging to the proposed mechanism) is applied to the current block. The following is an example of the MTS as the main transform for the current block. For larger resolution sequences (width > 1080), the maximum CU size used by the inter-frame MTS is set to 32 (i.e., the inter-frame MTS is used for CUs with width <= 32 and height <= 32), and for the remaining sequences (smaller resolutions), it is set to 16.

[0223] In another sub-implementation, the predetermined rules are explicit. For example, explicit rules depend on block, SPS, PPS, tile, slice, image, and / or sequence-level flags. For instance, explicit flags are sent to indicate whether the proposed mechanism (multiple candidates for the target transformation mode) for the current block is supported. Explicit signaling depends on the enabling conditions of the proposed mechanism.

[0224] - Explicit signaling is sent / parsed in the bitstream if all or any of the following predetermined subset conditions are met.

[0225] Condition 1: The block width (or height) is greater than or equal to a predetermined threshold (K1), where K1 is 2, 4, 8, 16, 32, 64 or any predetermined value in the standard.

[0226] Condition 2: The block width (or height) is less than a predetermined threshold (K2), where K2 is 32, 64, CTB size, maximum transform block size, or any predetermined value in the standard.

[0227] Condition 3: The block area is greater than or equal to a predetermined threshold (K3), where K3 is 2, 4, 8, 16, 32, 64 or any predetermined value in the standard.

[0228] Condition 4: The block area is less than a predetermined threshold (K4), where K4 is 16, 64, or the CTB size. CTB size, maximum transform block size Maximum transform block size, or any predetermined value in the standard.

[0229] Condition 5: The number of significance coefficients is greater than the predetermined threshold.

[0230] Condition 6: The sum of the absolute values ​​of the transformation coefficients of the current block is within the predetermined interval [K5, K6] or greater than the predetermined threshold.

[0231] Condition 7: The position of the last significant coefficient of the current transform block is greater than a predetermined threshold K7, where K7 = 0, 1, 2, and codec group (CG) size.

[0232] --- K7 = 0 indicates that the current transform block contains AC significance coefficients.

[0233] --- K7 = CG size indicates that the current transform block contains significant coefficients in non-first CGs.

[0234] Condition 8: The transform skip flag of the current block indicates that transform skip mode is disabled.

[0235] Further conditions can be designed based on the JCCR syntax, the distribution of significant coefficients, the syntax associated with whether the current block contains a codec group with significant coefficients, and / or the position of the codec group in the current block. For an example of a current block using JCCR, chroma residuals are jointly encoded and decoded (the codec transform block is jointly associated with CBFs). The use (activation) of the JCCR mode is indicated by the TU-level flag tu_joint_cbcr_residual_flag, and the selected mode is implicitly indicated by the chroma CBFs. The flag tu_joint_cbcr_residual_flag exists if one or both chroma CBFs of the TU are equal to 1. In another embodiment, the proposed mechanism can be used for one or more color components. For example, the proposed mechanism only allows use for the luma component (Y). In another example, the proposed mechanism only allows use for the chroma component (containing Cb, Cr, or both). In yet another example, the proposed mechanism allows use for both luma and chroma components. In yet another example, the use of the proposed mechanism depends on tree splitting.

[0236] In one case: -- If the proposed mechanism is allowed for use with the luma component, then the proposed mechanism is allowed for use with blocks in a luma dual-tree and / or the proposed mechanism is allowed for use with luma blocks in a single-tree; otherwise, the proposed mechanism is not allowed for use with blocks in a luma dual-tree and / or the proposed mechanism is not allowed for use with luma blocks in a single-tree.

[0237] -- If the proposed mechanism allows use for chroma components, then the proposed mechanism allows use for blocks in a chroma dual tree and / or the proposed mechanism allows use for chroma blocks in a single tree; otherwise, the proposed mechanism does not allow use for blocks in a chroma dual tree and / or the proposed mechanism does not allow use for chroma blocks in a single tree.

[0238] In another case: in a single tree, the proposed mechanism is only allowed for the luminance component.

[0239] In another case: in a single tree, the proposed mechanism is only allowed for use with the chromaticity component.

[0240] In another embodiment, the explicit signaling described above in the proposed mechanism uses a context codec bin. Several examples are used to select the context. For example, luminance and each chrominance component use separable contexts. That is, Y, Cb, and Cr have their own contexts. For example, each chrominance component shares the same context (but is independent of the luminance context). For example, luminance and chrominance components share the same context.

[0241] In one sub-implementation, the choice of context depends on adjacent codec information. For the example where the target transform mode is the primary transform, if both the upper and left adjacent blocks use non-default transform candidates, a single context is used for transmission. Otherwise, if only one adjacent block uses a non-default transform candidate, a different context is used for transmission. Otherwise (i.e., neither the upper nor left adjacent blocks use non-default transform candidates), a different context is used for transmission.

[0242] In another sub-implementation, the choice of context depends on one or more corresponding luma blocks. For the example where the target transform mode is the primary transform, if the predetermined corresponding luma block uses a non-default transform candidate, then one context is used for transmission. Otherwise, another context is used for transmission.

[0243] In another sub-implementation, the choice of context depends on the tree segmentation (luminance dual-tree, chrominance dual-tree, or single-tree), the signaling of the current block such as JCCR, or the mode type (intra-frame or inter-frame).

[0244] In another embodiment, the signaling or enable conditions for any subset of the proposed mechanism are uniform across the luminance and chrominance components. For example, one or more thresholds used in a particular enable condition are the same for both luminance and chrominance.

[0245] The proposed mechanism being permitted means that an explicit syntax or implicit rule associated with the proposed mechanism is required to determine whether the proposed mechanism should be applied to the current block or which candidate (supported in the proposed mechanism) should be applied to the current block. The proposed mechanism being prohibited means that no explicit syntax or implicit rule is required for the current block, and the proposed mechanism is determined to be unsuitable for the current block by inference without using the syntax or by definition in bitstream consistency.

[0246] II.3. Transforming Candidate Selection In another embodiment, in response to applying the proposed mechanism to the current block, which candidate is used or how the candidate is derived depends on predetermined rules. For example, a candidate is indicated to be applied to the current block according to predetermined rules. For example, the transformation matrix of the candidate is derived according to predetermined rules.

[0247] In one sub-implementation, if only one candidate is supported, then the unique candidate is inferred to be used for the current block. The condition described above, "if only one candidate is supported, then the unique candidate is inferred to be used for the current block," is referred to in this disclosure as Note B.

[0248] For example, the unique candidate transformation matrix is ​​predefined in the standard. In another example where the current block is a chroma block, the unique candidate transformation matrix is ​​derived from one or more corresponding luma blocks. The corresponding luma blocks can be any one or more of the following definitions, referred to in this disclosure as Note A: The top-left pixel address of the corresponding luminance block region refers to the same position as the top-left pixel address of the current chroma block.

[0249] The predetermined pixel address of the corresponding luminance block region is at a luminance sample distance of half the height of the current codec block to the upper boundary of the corresponding luminance block region, and at a luminance sample distance of half the width of the current codec block to the left boundary of the corresponding luminance block region. Here, {half the width and half the height} can refer to {width / 2, height / 2}, {(width+1) / 2, (height+1) / 2}, {(width+1) / 2, height / 2} and / or {width / 2, (height+1) / 2}.

[0250] In one case: the chroma transformation matrix can be set as a reference transformation matrix from a predefined co-luminance block among the aforementioned co-luminance blocks. If the reference transformation matrix is ​​unavailable (e.g., the co-luminance block does not use any transformation candidate for the target transformation mode supported in the proposed mechanism), the predefined transformation matrix is ​​used. The predefined method is determined in the standard or by block width, block height, block area and / or adjacent codec information, and / or more as described in Note C (described later).

[0251] In another case: the order in which the aforementioned co-luminance blocks are examined is defined. The chroma transform matrix is ​​set to the first or most available reference transform matrix from the co-luminance block. If none of the reference transform matrices are available (e.g., none of the co-luminance blocks use any transform candidates for the target transform mode supported in the proposed mechanism), a predetermined transform matrix is ​​used. The predetermined method is determined in the standard or by block width, block height, block area, adjacent codec information, and / or more in Note C.

[0252] In another example, the only candidate transform matrix is ​​predetermined in the standard or depends on predetermined implicit rules, such as block width, block height, block area, and / or adjacent encoding / decoding information. Note C shows some methods as follows: Note C: The candidate set is determined using block width, block height, or intra / inter / IBC prediction mode, and the first candidate (with the smallest index) is used.

[0253] Taking the target transform as the primary transform and the proposed mechanism supporting additional MTS type transform candidates as an example. In the current VVC design, only DST7 and DCT8 transform kernels are used for MTS. Additional MTS types include DCT5, DST4, DST1, and Identity Transform (IDT). The MTS set is determined based on the transform block size and intra-frame mode information. A total of 16 different transform block sizes are considered, and for each transform block size, 5 different categories are considered based on intra-frame mode information. Furthermore, the MTS set depends on the TU size and intra-frame mode information (e.g., the representative prediction mode in Section 3). For each category, 1, 4, or 6 different transform pairs are considered. The number of intra-frame MTS candidates is adaptively selected based on the sum of the absolute values ​​of the transform coefficients (between 1, 4, and 6 MTS candidates). The total number of allowed MTS candidates is determined by comparing the sum with two fixed thresholds: One candidate if the sum is less than or equal to th0. There are 4 candidates. If th0 < sum <= th1 Six candidates, if the sum > th1 Note that while a total of 80 different categories were considered, some of these categories often share the exact same transform set. Therefore, there are 58 unique entries in the resulting LUT (fewer than 80). For angular modes, joint symmetry of the TU shape and intra-prediction was considered. For wide-angle modes (not within the range of the regular intra-prediction mode index, e.g., not in the range of 0 to 66), the nearest regular angular mode was used to determine the transform set. For example, mode 2 was used for all modes between -2 and -14. Similarly, mode 66 was used for modes 67 through 80.

[0254] Another example of a target transform being the primary transform and the proposed mechanism supporting additional MTS-type transform candidates. Four candidates: {(DST7, DST7), (DST7, DCT8), (DCT8, DST7), (DCT8, DCT8)} are used for each CU. For 4-point, 8-point, and 16-point transforms, the current transform cores (or kernels), namely DST-7 and DCT-8, are replaced by separable KLTs. The luma and chroma components may have uniform or different rules.

[0255] In another sub-implementation, the predetermined rules are implicit. Implicit rules refer to block width, block height, block area, and / or adjacent codec information. For the example using implicit rules to determine which candidate (supported in the proposed mechanism) is used, the candidate indicated by the implicit rule is applied to the current block. For the example using implicit rules to derive the transformation matrix of a candidate, the transformation matrix used in the candidate is set according to the implicit rules. Further methods can be found in note B.

[0256] In another sub-implementation, the predefined rules are explicit. Explicit rules rely on block, SPS, PPS, tile, slice, image, and / or sequence-level flags. For example, an explicit index indicates which candidate (supported in the proposed mechanism) is used for the current block.

[0257] In another embodiment, when the current block is a chroma block, one or more transform candidates for the target transform mode are derived based on one or more co-occurring luma blocks in Note A. In one sub-implementation, the transform candidate derived from luma is the first candidate in the candidate set. If a transform candidate derived from luma is redundant with any other transform candidate in the set, the redundant candidate (belonging to the other transform candidates mentioned above) is replaced with a predefined transform matrix. The predefined method is determined in the standard or by block width, block height, block area and / or adjacent codec information, and / or more in Note C.

[0258] In another embodiment, when the proposed mechanism allows for use with the chroma component, signaling (in sections II.2 and / or II.3) is shared for both the CB and CR components. That is, an on / off syntax and / or a transform candidate selection syntax are sent and used for both the CB and CR components.

[0259] In another embodiment, when the proposed mechanism allows for use with chrominance components, signaling (in sections II.2 and / or II.3) is used for the Cb and Cr components, respectively. That is, an on / off syntax and / or a transform candidate selection syntax is sent and used for Cb, and another on / off syntax and / or another transform candidate selection syntax is sent and used for Cr. When using JCCR (Joint Codec Residual / Coefficients of Cb and Cr), an on / off syntax and / or a transform candidate selection syntax is sent for the joint codec of Cb and Cr.

[0260] In another embodiment, the signaling of the proposed mechanism (in sections II.2 and / or II.3) is at the transform block level, transform unit level, codec block level, and / or codec unit level.

[0261] In another embodiment, when the proposed mechanism is applied to the primary transform (as the target transform), the secondary transform (as a disable) is inferred based on the on / off signaling of the proposed mechanism.

[0262] In another embodiment, when the subtransform is applied to the current block, the proposed mechanism is inferred to be applied to the main transform (as a disable) based on the on / off signaling of the subtransform.

[0263] In another embodiment, when the proposed mechanism is applied to the secondary transformation (as the target transformation), the primary transformation (as the default) is inferred from the on / off signaling of the secondary transformation.

[0264] In another embodiment, when the primary transform (as NSPT) is applied to the current block, the proposed mechanism is inferred to be applied to the secondary transform (as disabled by default) based on the on / off signaling of the NSPT.

[0265] III. Representative Prediction Models in Transformation A scheme for determining representative prediction modes for regular intra-frame modes, special intra-frame modes, non-intra-frame modes, and / or any mode (in note D) is proposed.

[0266] A conventional intra-frame mode refers to generating the predicted value for the current block using one or more conventional intra-frame prediction modes and spatially adjacent reference samples (located in adjacent or non-adjacent reference lines of the current block). Conventional intra-frame modes can be used for the luma and / or chroma components. The conventional intra-frame prediction modes used can be indicated by syntax elements and / or by predefined implicit derivation methods (e.g., DIMD suggests one or more intra-frame prediction modes through histogram analysis, and / or TIMD suggests one or more intra-frame prediction modes through template analysis).

[0267] A special intra-frame mode refers to a mode that, when encoding and decoding the current block using a special intra-frame mode, does not use a traditional intra-frame prediction mode (e.g., one of the 67 intra-frame prediction modes) to generate the predicted value for the current block. Instead, it applies an alternative scheme (e.g., a matrix-based scheme and / or cross-component information) to spatially adjacent reference samples and / or reference samples from the corresponding luma block to generate the predicted value for the current block. For example, a special intra-frame mode refers to matrix-based intra-frame prediction (MIP) or any intra-frame mode that uses a derived or predetermined matrix and input reference samples to generate the predicted value for the current block. Special intra-frame modes can be used for luma and / or chroma components. For example, a special intra-frame mode for chroma refers to any cross-component mode, meaning that the current chroma block references one or more predicted / reconstructed samples from corresponding luma blocks or any predetermined information to generate the predicted value for the current chroma block. An example of a cross-component mode is an LM mode (e.g., CCLM and / or MMLM) and / or any LM variant (e.g., CCCM and / or GLM). For example, a special intra-frame mode refers to the fusion of CCLM, MMLM, CCCM, GLM, any LM variant, or chroma intra-frame prediction mode. For example, a special intra-frame mode refers to TMP.

[0268] Non-intra-frame mode refers to IBC, inter-frame mode, and / or any mode whose mode type is not equal to the intra-frame mode type (MODE_TYPE_INTRA). Non-intra-frame mode can be used for the luma component and / or chroma component.

[0269] During the prediction phase, to improve prediction efficiency, different modes may apply adjustments when generating predicted values ​​or afterward. For regular intra-frame modes, the final predicted value may be a multi-hypothesis combination based on predictions from multiple traditional intra-frame prediction modes. Therefore, it is difficult to directly find a representative prediction mode among multiple traditional intra-frame prediction modes. For special intra-frame modes and / or non-intra-frame modes and / or any mode (in note D), the predicted value is not generated based on (traditional) intra-frame prediction modes. However, in some cases, the intra-frame prediction mode for the current block is necessary.

[0270] The intra-prediction mode of the current block will be used in the following encoding / decoding stages after / before / in parallel with the prediction processing.

[0271] During the transform / inverse transform phase, the intra-frame prediction mode is used by the secondary transform, where the transform mode can be any transform mode mentioned in Section I. For example, this might correspond to selecting a transform set and transpose flag for the secondary transform (e.g., LFNST). The secondary transform in this invention is not limited to LFNST and can be any predefined separable / non-separable and / or low-frequency / non-low-frequency transform that is performed in the encoder (or decoder) after (or before) the primary transform and applied to the primary transform (or received inverse quantization) coefficients in the encoder (or decoder).

[0272] During the transform / inverse transform phase, the intra-frame prediction mode can be used by the primary transform, where the transform mode can be any transform mode mentioned in Section I. For example, this might correspond to selecting a transform kernel for the primary transform (e.g., default primary transforms such as DCT-II and / or MTS and / or NSPT). The primary transform in this invention can be any predefined transform performed on the residuals (or inverse transform coefficients or received inverse quantized transform coefficients) in the encoder (or decoder).

[0273] During the transform / inverse transform phase, the intra-frame prediction mode can be used by any transform mode (i.e., it can be referenced by the primary transform, secondary transform, or any of the two or more transform modes mentioned in Section I).

[0274] For subsequent codec blocks, intra-prediction mode information from one or more adjacent or previous codec blocks is used to derive the MPM list or any scheme that may reference intra-prediction mode information from one or more adjacent or previous codec blocks.

[0275] If the current block is a luma block, the corresponding chroma block may require an intra-prediction mode from the luma (e.g., used to determine the chroma DM).

[0276] Therefore, when one of the above patterns is applied to the current block, the proposed new mechanism can be used to determine a representative prediction pattern for use in one or more of the above cases.

[0277] In one embodiment, the representative prediction pattern is any of the DIMD derivation patterns (e.g., applying histogram analysis to spatially adjacent or non-adjacent reconstructed samples of the current block), the TIMD derivation pattern (e.g., applying template analysis to spatially adjacent or non-adjacent reconstructed samples of the current block), DC, planar, horizontal, vertical, diagonal, or any predefined pattern from available intra-prediction patterns. For example, consider a current block encoded via MIP (which can be replaced by any regular intra-prediction pattern, special intra-prediction pattern, and / or non-intra-prediction pattern, and / or any pattern (in note D)) and a predefined intra-prediction pattern as the DIMD derivation pattern. The DIMD derivation pattern is stored for MIP and / or can be used for any one or more of the following: Subsequent blocks use a scheme to derive the MPM list or any intra-prediction mode information that may reference one or more adjacent or previous codec blocks. If the current block is a luma block, the corresponding chroma block determines its intra-frame prediction mode (e.g., deriving the chroma DM). The current block determines the transform kernel for the transform phase. An example of applying histogram analysis to spatially adjacent reconstructed samples is as follows: FIG. 38A As shown, region 3820 corresponds to the adjacent region used as the template for the current block 3820. Window 3822 is applied to each or any subset of the adjacent regions (templates). The center of the window is located on the center sample of each or any subset of the adjacent regions. The center sample is displayed as a circle 3824 filled with dots.

[0278] An example of applying template analysis to spatially adjacent reconstructed samples is as follows: FIG. 38B As shown. The TIMD cost (based on template) is calculated as the distortion between the reconstructed sample and the predicted sample in the neighboring region 3820 (i.e., the template) of the current block 3810, where the predicted sample in the neighboring region (i.e., the template) is generated according to the candidate prediction pattern.

[0279] In one sub-implementation, when using the template from the above example, the template may include, for example: FIG. 39A The top template 3920, left template 3930, and upper left template 3940 shown are used for the current block 3910, or as follows: FIG. 39BThis includes any extensions from the top template 3950 and / or any extensions from the left template 3960 and / or only any subset thereof. The size of the top template is predetermined as block width x H1, the size of the left template is predetermined as W1 x block height, and the size of the top-left template is predetermined as W1 x H1, where W1 and / or H1 are 1, 2, 3, 4, …, or any predetermined value. In one example, H1 and W1 are the same. In another example, H1 and W1 are different. In yet another example, H1 and / or W1 vary with the block size. If the block height is greater than a predetermined threshold, H1 increases. If the block width is greater than a predetermined threshold, W1 increases. Some extensions of the templates are as follows. The width of the extended top template is increased to any predetermined value greater than the block width, such as block width + block height or k. Block width. The height of the extended left template increases to any predetermined value greater than the block height, such as block height + block width or k. Block height.

[0280] In another embodiment, the representative prediction mode is determined by applying a predetermined process to prediction samples of all or any subset of the current block. The predetermined process will suggest an intra-frame prediction mode as the representative prediction mode for the current block. In one sub-embodiment, the predetermined process refers to DIMD and / or TIMD. In one sub-embodiment, when the current block is a chroma block and is encoded / decoded using cross-component mode or chroma intra-frame prediction mode fusion, the predetermined process is applied to reconstructed / predicted samples of all or any subset of one or more corresponding luma blocks. Taking DIMD as an example, the window used in DIMD is applied to the predictors within the current block. In another sub-embodiment, the prediction samples for which the predetermined process is applied refer to downsampled or reduced prediction samples, and the predetermined process is applied to all or any subset of the downsampled or reduced prediction samples. For example, the original size of the current block is 2Mx2N, and by downsampling from 2Mx2N, the reduced or downsampled size of the current block is MxN. In another sub-embodiment, as... FIG. 40 As shown, the center of the window is applied to samples (gray areas) within the current (or reduced) block, but not to the boundaries of the current (or reduced) block. In another sub-implementation, the center of the window is applied to each sample within the current (or reduced) block. If the window requires any samples outside the current (or reduced) block, filling from the boundaries is used instead of referencing samples outside the current (or reduced) block. Taking TIMD as an example, the TIMD cost of a candidate prediction pattern is calculated as the distortion between the final predicted sample (e.g., if the current block is encoded and decoded via IBC, the final predictor is generated based on the block vector) and the predicted samples (from the candidate prediction patterns) of all or any subset of the current (or reduced) block.

[0281] Take, for example, the representative prediction mode of the current block encoded and decoded via MIP (which can be changed using any regular intra mode, special intra mode, and / or non-intra mode, and / or any mode (in note D)) and the DIMD from the current block. The representative prediction mode is stored for MIP and / or can be used for any one or more of the following. Subsequent blocks derive the MPM list or any other method that might reference intra-prediction mode information from one or more adjacent or previously encoded blocks. (If the current block is luma) the corresponding chroma block to determine its intra-frame prediction mode (e.g., deriving chroma DM). The current block determines the transform kernel for the transform phase. In another embodiment, a representative prediction mode is predefined as a default mode. The default mode can be planar, DC, or any predefined intra-prediction mode in the standard. In one sub-implementation, the default mode is fixed to an intra-prediction mode. In another sub-implementation, the default mode is selected based on the codec mode of the current block. For example, the default mode for MIP is planar; the default mode for intra-TMP is planar; the default mode for SGPM / GPM / any GPM variant is a mode based on partition angle or any mode in the IPM list; the default mode for chroma across component modes is chroma DM; the default mode for horizontal / vertical planes is horizontal / vertical; the default mode for DBV is chroma DM or DC or the same mode as the default mode for IBC; the default mode for CIIP is TIMD or planar or any intra-prediction mode used in CIIP; and / or the default mode for IBC is DC or planar. For example, the default mode for SGPM is a mode based on partition angle; for example, a mapping table is created to map from partition angles to the default mode, and the size of the mapping table depends on the number of candidate partition angles. If the number of candidate partition angles is 32, the size of the mapping table is 32. The following shows a possible mapping table, where the default mode is one of the 67-domain intra-prediction modes: {50, 0, 44, 41, 34, 27, 0, 0, 18, 0, 0, 9, 66, 59, 56, 0, 50, 0, 44,41, 34, 27, 0, 0, 18, 0, 0, 9, 66, 59, 56, 0}.

[0282] The numbers in {} refer to the default mode. An angle equal to 0 (i.e., the vertical partitioning direction) refers to the default mode 50 (i.e., the vertical intra prediction mode). The default mode numbers are symmetric. The first half of the table is the same as the second half of the table. In another example, an angle equal to 0 (i.e., the vertical partitioning direction) refers to the default mode 18 (i.e., the horizontal intra prediction mode). By grouping N candidate angles into multiple (<N) groups and using the group index to find the default mode from the mapping table, the size of the mapping table can be reduced. The size of the mapping table is the number of groups. For example, using angle symmetry (e.g., angles 0 and 16 are symmetric and both belong to the vertical partitioning direction), the symmetric angles are put into the same group.

[0283] In another embodiment, when the representative prediction mode is a DIMD-derived mode (applying histogram analysis to the reconstructed samples that are spatially adjacent or non-adjacent to the current block), a TIMD-derived mode (applying template analysis to the reconstructed samples that are spatially adjacent or non-adjacent to the current block), or any mode that obtains the optimal suggestion (e.g., minimum cost, maximum histogram, etc.) among all candidate modes, or when the representative prediction mode is determined by applying a predetermined process to all or any subset of the prediction samples on the current block, where the predetermined process (e.g., DIMD, TIMD, and / or any predetermined process) suggests an intra prediction mode as the representative prediction mode of the current block, some embodiments are intended to determine whether the suggestion is credible. If it is credible, the suggested mode is used as the representative prediction mode of the current block; otherwise, the default method is used to decide the representative prediction mode of the current block.

[0284] In a sub-embodiment, when using DIMD to suggest the representative prediction mode, whether the suggestion is credible depends on the distribution of the histogram. For example, when the maximum histogram is greater than (T the second-largest histogram), the suggestion is credible, where T is predetermined to be any value, such as 1.25, 1.5, 2.0, greater than 1. In implementation, a shift operation is used instead of multiplying by the floating value (T). For example, (2.0 the second-largest histogram) is implemented as (the second-largest histogram << 1). Another example, (1 .25 the second-largest histogram) is implemented as (the second-largest histogram + the second-largest histogram >> 2). Another example, (1.5 the second-largest histogram) is implemented as (the second-largest histogram + the second-largest histogram >> 1). In another implementation, a lookup table is used instead of multiplying by the floating value (T). By using the lookup table, T is approximated.

[0285] In another sub-embodiment, when using TIMD to suggest the representative prediction mode, whether the suggestion is credible depends on the distribution of the TIMD cost. For example, when T2 The recommendation is considered credible when the minimum cost is less than the second smallest histogram, where T2 is predefined as any value, such as 1.25, 1.5, 2.0, or any value greater than 1. In the implementation, a shift operation is used instead of multiplying by the float value (T2). For example, (2.0...) The minimum cost is achieved as (minimum cost << 1). Another example is (1.25). The minimum cost is realized as (minimum cost + minimum cost >> 2). Another example is (1.5). The minimum cost is implemented as (minimum cost + minimum cost >> 1). In another implementation, a lookup table is used instead of multiplying by the floating value (T2). By using the lookup table, T2 is approximated.

[0286] In another implementation, when the representative prediction pattern is a DIMD-derived pattern (i.e., applying histogram analysis to reconstructed samples that are spatially adjacent or not adjacent to the current block), a TIMD-derived pattern (i.e., applying template analysis to reconstructed samples that are spatially adjacent or not adjacent to the current block), or any pattern that yields the best recommendation among all candidate patterns (e.g., minimum cost, maximum histogram, etc.), or when the representative prediction pattern is determined by applying a predetermined process to prediction samples of all or any subset of the current block, wherein the predetermined process (e.g., DIMD, TIMD, and / or any predetermined process) recommends an intra-frame prediction pattern as the representative prediction pattern for the current block, some implementations aim to generate new transformation matrices / sets based on multiple (e.g., two) existing transformation matrices / sets selected from multiple (e.g., two) recommended representative patterns, and / or the new transformation matrices / sets are used for the current block. When using DIMD to recommend a representative intra-frame prediction pattern and the histograms of the candidate patterns are not easily distinguishable (e.g., the maximum histogram is similar to the second largest histogram), multiple representative patterns (e.g., two) are introduced. When using TIMD to suggest representative intra-prediction patterns and the costs of candidate patterns are not easily distinguishable (e.g., the minimum cost is similar to the second minimum cost), multiple representative patterns (e.g., 2) are introduced.

[0287] In one implementation, a new transformation matrix / set is generated by blending multiple existing transformation matrices / sets. The following is an example of blending two existing transformation matrices / sets into a new transformation matrix. First, an existing transformation matrix / set is selected according to one proposed pattern, and another existing transformation matrix / set is selected according to another proposed pattern. Then, a new matrix / set is generated by either weighted averaging or blending the transformation matrix coefficient values ​​of all or any subsets of the two existing transformation matrices, or by blending the transformation matrix members of all or any subsets of the two existing transformation sets into a new transformation set. The rules for weighted averaging or member selection may or may not depend on the histogram / cost in the predetermined process.

[0288] In another embodiment, one or more intra-frame prediction modes are pre-defined as preferred modes. When the representative prediction mode is a DIMD-derived mode (applying histogram analysis to reconstructed samples that are spatially adjacent or not adjacent to the current block), a TIMD-derived mode (i.e., applying template analysis to reconstructed samples that are spatially adjacent or not adjacent to the current block), or any mode that receives the best recommendation among all candidate modes (e.g., minimum cost, maximum histogram, ...), or when the representative prediction mode is determined by applying a pre-defined process to prediction samples of all or any subset of the current block, where the pre-defined process (e.g., DIMD, TIMD, and / or any pre-defined) recommends an intra-frame prediction mode as the representative prediction mode for the current block, some embodiments are designed to give the preferred mode higher priority in the pre-defined process and then compare it with other non-preferred modes. For example, when using DIMD to recommend a representative prediction mode, the histogram of the preferred mode is adjusted by multiplying its histogram by T3, where T3 = 1.25, 1.5, 2.0, or any value greater than 1. In implementations, a shift operation is used instead of multiplying by a floating-point value (T3). For example, (2.0) The histogram is implemented as (histogram << 1). Another example is (1.25). The histogram is implemented as (histogram + histogram >> 2). Another example is (1.5). The histogram is implemented as (histogram + histogram >> 1). In another implementation, a lookup table is used instead of multiplying by a floating-point value (T3). By using a lookup table, T3 is approximated. Another example is when using TIMD to suggest a representative prediction model, the cost of the preferred model is adjusted by multiplying its cost by T4, where T4 = 0.25, 0.5, or any value less than 1. In the implementation, a shift operation is used instead of multiplying by a floating-point value (T4) or dividing by 1 / T4. For example, (0.25... The cost is realized as (cost >> 2). Another example is (0.5). The cost is implemented as (cost >> 1). In another implementation, a lookup table is used instead of multiplying by a floating-point value (T4) or dividing by 1 / T4. By using a lookup table, T4 is approximated.

[0289] In one sub-implementation, different codec modes may have different preferred modes. For example, when the current block is encoded using the SGPM codec mode, the preferred mode is defined according to the partition angle of the current block. Another example is when the current block is encoded using the MIP codec mode, the preferred mode is defined as planar, DC, or any predetermined intra-prediction mode. Yet another example is when the current block is encoded using IBC or inter-frame codec modes, the preferred mode is defined as planar, DC, any intra-prediction mode based on mode information, for any predetermined subset of prediction units (or codec block or any predetermined region) in a reference region containing a corresponding reference block indicated by a block vector or motion vector and / or the neighboring template of the corresponding reference block, or any predefined intra-prediction mode. Yet another example is when the current block is encoded using a cross-component mode, the preferred mode is defined as chroma DM, planar, DC, or any predefined intra-prediction mode. Another example: when the current block is encoded using the DBV encoding / decoding mode, the preferred mode is defined as chroma DM, planar, DC, horizontal, vertical, any intra-prediction mode based on mode information, for any predetermined subset of prediction units (or encoding / decoding block or any predetermined region) in the reference region, the reference region containing the corresponding reference block indicated by the block vector and / or the adjacent template of the corresponding reference block, or any predefined intra-prediction mode. Another example: when the current block is encoded using the horizontal planar encoding / decoding mode, the preferred mode is defined as planar, DC, horizontal, vertical, or any predetermined intra-prediction mode. Another example: when the current block is encoded using the vertical planar encoding / decoding mode, the preferred mode is defined as planar, DC, horizontal, vertical, or any predefined intra-prediction mode. Another example is when the current block is encoded using an intra-TMP encoding / decoding mode, the preferred mode is defined as any intra-prediction mode based on mode information, whether planar, DC, horizontal, vertical, or for any predetermined subset of prediction units (or codec blocks or any predetermined region) in the reference region, which includes the corresponding reference block indicated by the block vector and / or the neighboring template of the corresponding reference block, or any predetermined intra-prediction mode. Yet another example is that the preferred mode refers to the default mode.

[0290] In another sub-implementation, when using DIMD to recommend representative prediction patterns, the preferred patterns are the top N patterns recommended by TIMD, where N = 1, 2, ... or any positive integer.

[0291] In another sub-implementation, when using TIMD to recommend representative prediction patterns, the preferred patterns are the first N2 patterns recommended by DIMD, where N2 = 1, 2, ... or any positive integer.

[0292] In another manner, mode information from a reference region is used to derive a representative prediction mode for regular intra-frame modes, special intra-frame modes, and / or non-intra-frame modes, and / or any mode (in note D). Compared to performing texture analysis (DIMD or TIMD) on predictors within the current block or spatially adjacent samples, the proposed alternative is simpler and / or can output the derived representative intra-frame prediction mode earlier without waiting for a reconstruction or prediction phase to understand the required prediction or reconstruction samples. Some of the following embodiments focus on the use of IBC on the current block. Note that the proposed method is not limited to application to IBC blocks and can be used for the current block encoded by any other mode (e.g., intra-TMP or inter-frame blocks). When the current block is encoded using an intra-TMP, a reference block is found through template matching. When the current block is encoded using an inter-frame mode, the reference block can be in a reference image or a predetermined image indicated by the motion information of the current block and / or predetermined in a previously encoded image different from the current image. For example, for the motion information of the current block being bidirectional prediction, a reference image indicated by a reference index from list 0 and / or a reference index from list 1 is used. Another example is when the motion information for the current block is unidirectionally predicted, using a reference image indicated by the reference index of either list 0 or list 1. If multiple reference images are used, an order is used to define which reference image is used first. One possible order is to use the reference image that is closer to the current image first (i.e., the smaller POC). Another possible order is to use reference images from a predetermined list (list 0 or list 1) first.

[0293] In one embodiment, the representative prediction mode is derived based on mode information of any predetermined subset of prediction units (or codec blocks or any predefined regions) in a reference region (containing the corresponding reference block and / or adjacent templates of the corresponding reference block). The mode information includes mode type, intra-frame prediction mode, motion information, block width, block height, block area, block shape, block ratio, residual information, transform information, partitioning information, and / or any subset / extension of the above.

[0294] In one sub-implementation, if a representative prediction mode cannot be found from a predetermined subset of prediction units, a default prediction mode is used as the representative prediction mode for the current block. The default prediction mode can be any available intra-prediction mode, such as a planar or normal DIMD mode, which is always derived at the decoder end.

[0295] In another sub-implementation of IBC encoding / decoding for the current block, a reference block is indicated by a block vector, and one or more mode information (e.g., intra-prediction mode) stored in the reference block is used to derive the representative prediction mode for the current block. If the reference block is any particular intra-prediction mode and / or non-intra-prediction mode, the default intra-prediction mode is used as the representative prediction mode; otherwise, the intra-prediction mode of the reference block is used as the representative prediction mode for the current block.

[0296] In another sub-implementation of IBC encoding / decoding for the current block, one or more prediction units in the reference region (potentially containing the reference block and / or one or more spatially adjacent / non-adjacent prediction units) are predetermined, and a scan order is applied to the predetermined prediction units. An example of a scan order is shown below: The intra-prediction mode is selected in prediction units P1 to P5 in a predetermined order: P1, P2, P3, (P4, P5), where the positions of P1-P5 are as follows: FIG. 41 As shown.

[0297] P1 covers the middle position in the reference block.

[0298] P2 covers the bottom right position in the reference block.

[0299] P3 covers the top left position in the reference block.

[0300] P4 covers the predefined position (e.g., the middle position) above the outer neighbor of the reference block.

[0301] P5 covers the predetermined position (e.g., the middle position) adjacent to the left outer side of the reference block.

[0302] If the block's height is greater than its width, check P5 first.

[0303] Otherwise, check P4 first.

[0304] Following the scan order, the intra-prediction mode in the predetermined prediction unit can be used as the representative prediction mode. For example, the first mode with an intra-prediction mode is used as the representative prediction mode. Another example is sending / resolving an explicit index to indicate that the intra-prediction mode in the predetermined prediction unit is the representative prediction mode. When a prediction unit in the predetermined prediction unit is not a valid mode, invalid prediction units are skipped during the scan or a default prediction mode is set for invalid prediction units. For example, non-intra-frame (e.g., IBC), intra-frame TMP, inter-frame and / or MIP and / or any of the modes mentioned in note D are not valid.

[0305] In another sub-implementation of IBC encoding / decoding for the current block, one or more prediction units (potentially comprising the reference block and / or one or more spatially adjacent / non-adjacent prediction units) in the reference region are predetermined, and a voting method is applied to the predetermined prediction units. The voting method (e.g., a majority-based counting method) means using the most popular prediction mode as the representative prediction mode for the current block. When a prediction unit in the predetermined prediction units is not a valid mode, invalid prediction units are skipped during counting, or a default prediction mode is set for the counted invalid prediction units. For example, non-intra-frame (e.g., IBC), intra-frame TMP, inter-frame and / or MIP, and / or any of the modes mentioned in note D are not valid.

[0306] In another embodiment, when the representative prediction mode (intra-prediction mode) is used to determine the transform kernel of the MTS, the MTS can be implicit or explicit, and the current block can be a regular intra-mode, a special intra-mode, a non-intra-mode, and / or any of the modes mentioned in note D. An explicit MTS (e.g., an enhanced MTS for intra-codec) means that after selecting an MTS set for the current block, an MTS index is sent to find a candidate (transform pair) from the MTS set. An implicit MTS means that implicit mapping rules (independent of syntax elements) are used to find candidates. With the proposed method, for explicit MTS, the selection of the MTS set depends on the representative prediction mode; for implicit MTS, the candidates depend on the representative prediction mode (following the mapping rules). Tables 9-11 show examples of mapping rules.

[0307]

[0308] In another embodiment, the proposed method can be used for a 4:4:4 color format. An example is shown below. When the color format is 4:4:4, the MIP can be used for chroma. For the chroma MIP block, a representative prediction mode can be used for the primary and / or secondary transforms to select the transform set and / or transpose flag.

[0309] In another embodiment, when the current block is encoded or decoded using a special intra-frame mode, a non-intra-frame mode, and / or any of the modes mentioned in note D, a representative prediction mode is stored in an intra-frame prediction mode buffer. This buffer is readily available in subsequent processes should any procedure require access to it.

[0310] In another embodiment, the representative prediction pattern may implicitly vary with block width, block height, and block area, or vary according to explicit rules (e.g., block, tile, slice, image, SPS, or PPS level syntax). For example, a flag is sent at the block level to indicate which proposed method is used to determine the transformation set of the primary and / or secondary transformations of the current block. If the flag indicates 1, a predetermined process (e.g., DIMD of the current block) is performed within the current block to determine the transformation set; otherwise (if the flag indicates 0), a predetermined process (e.g., a derived DIMD pattern) is performed in the spatially adjacent region of the current block to determine the transformation set.

[0311] In another embodiment, any of the proposed methods or any combination of proposed methods can be applied to any intra-frame mode, such as WAIP (Wide-Angle Intra-Frame Prediction), intra-frame angle mode, ISP, MIP, or any intra-frame mode specified in VVC or HEVC or related technologies, or to more modes developed below (referred to as Note D in this disclosure): SGPM HOR / VER PL.

[0312] In JVET-AA0104 (Xinwei Li et al., “Non-EE2: On planar horizontal mode and planar vertical mode”, ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 Joint Video Exploration Team (JVET), 27th meeting, via teleconference, July 13-22, 2022, document JVET-AA0104), for the horizontal planar mode, horizontal linear interpolation is performed only based on the left reference sample and the upper right reference sample to predict the current sample as: , For the vertical plane pattern, vertical linear interpolation is performed only based on the upper reference sample and the lower left reference sample to predict the current sample as follows: , When HOR / VER PL is used for the current block, a decoder-side derivation method is proposed to determine which directional plane mode to use. Specifically, the horizontal gradient of each sample in the adjacent rows of the current block and the vertical gradient of each sample in the adjacent columns of the current block are calculated. If the sum of the absolute values ​​of the horizontal gradients is greater than the sum of the absolute values ​​of the vertical gradients, the vertical plane mode is used; otherwise, the horizontal plane mode is used.

[0313] In the transform / inverse transform stage and / or subsequent codec blocks, the planar mode is not used directly, but rather (1) the horizontal planar mode and the vertical planar mode are modified to vertical mode and horizontal mode respectively, or (2) any method proposed in this invention. In another embodiment, the choice between (1) and (2) depends on the width / height / area of ​​the block. For example, if the width / height / area of ​​the block is less than a predefined threshold (such as 2, 4, ... or any predefined value), then (1) is used; otherwise, (2) is used. For example, if the width / height / area of ​​the block is greater than a predefined threshold (such as 2, 4, ... or any predefined value), then (1) is used; otherwise, (2) is used.

[0314] DBV (Direct Block Vector): A chromaticity prediction mode Variant 1: Send a bin to indicate whether to use DBV mode. When the chroma dual tree is activated in an intra-frame slice, for a chroma CU encoded in DBV mode, if a corresponding luma block is encoded in IBC mode or IntraTmp mode (intra-template matching prediction), its block vector bvL is used to derive the chroma block vector bvC. The bv scaling process is determined based on template matching.

[0315] By using the current chroma block's position at (xCb, yCb) and its bvC, the corresponding offset position (xCb+bvC[0], yCb + bvC[1]) is determined, and block copy prediction is performed.

[0316] Variant 2: When the chroma dual tree is activated in an intra-frame slice, for a chroma CU encoded in DM mode, if the corresponding luma block is encoded in IBC mode or IntraTMP mode, its block vector bvL is used to derive the chroma block vector bvC. The bv scaling process is determined based on template matching.

[0317] By using the current chroma block's position at (xCb, yCb) and its bvC, the corresponding offset position (xCb+bvC[0], yCb + bvC[1]) is determined, and block copy prediction is performed.

[0318] In the transform / inverse transform phase and / or subsequent codec blocks, a planar mode is not used; instead, (1) chroma DM or (2) any method proposed in this invention is used. In another embodiment, the choice between (1) and (2) depends on the width / height / area of ​​the block. For example, if the width / height / area of ​​the block is less than a predetermined threshold (such as 2, 4, ... or any predetermined value), then (1) is used; otherwise, (2) is used. For example, if the width / height / area of ​​the block is greater than a predetermined threshold (such as 2, 4, ... or any predetermined value), then (1) is used; otherwise, (2) is used.

[0319] Hybrid modes (e.g., CIIP / GPM) In another embodiment, in order to reduce the delay caused by the method of proposing a representative prediction mode for the current block, in the encoder, when performing the main transform, the proposed method is used only to determine the prediction mode for the vertical transform (not for the horizontal transform); in the decoder, the proposed method is used only to determine the prediction mode for the inverse vertical transform (not for the inverse horizontal transform).

[0320] In another embodiment, in order to reduce the delay caused by the method of proposing a representative prediction mode for the current block, in the encoder, when performing the main transform, the proposed method is used only to determine the prediction mode for the horizontal transform (not for the vertical transform); in the decoder, the proposed method is used only to determine the prediction mode for the inverse horizontal transform (not for the inverse vertical transform).

[0321] In another embodiment, in order to reduce the latency caused by the method of proposing a representative prediction mode for the current block, the proposed method is used only for determining the prediction mode of separable transform encoding / decoding (e.g., the primary transform), and not for non-separable transforms (e.g., the secondary transform).

[0322] In another embodiment, in order to reduce the delay caused by the method of proposing a representative prediction mode for the current block, the proposed method is not used to determine the prediction mode for the sub-transformation.

[0323] In another embodiment, in order to reduce the latency caused by the method of proposing a representative prediction mode for the current block, some enable conditions are designed for the proposed method, and the proposed method is only allowed to be used when the enable conditions are met.

[0324] In one sub-implementation, the enabling condition includes mode setting. Mode setting means first checking the mode of the current block; if the current block's mode is a complex mode, the proposed method cannot be applied. For example, a complex mode refers to a hybrid mode that forms the final predictor through multiple prediction hypotheses. A hybrid mode could be a CIIP / GPM / MHP / TIMD / DIMD codec mode. Another example is a complex mode that refers to a refinement mode that refines the motion / predictor through multiple passes or decoder-side derivation. A refinement mode could be a DMVR / intra-TMP codec mode.

[0325] In another sub-implementation, the enabling conditions include size settings related to block width, block height, block area, or block shape. Size settings mean first checking the size of the current block; if the current block size is unsuitable for the hardware or detrimental to the computational cost of DIMD histogram calculation or TIMD calculation, the proposed method cannot be applied. For example, the proposed method cannot be used when the width or height of the current block is greater than a predetermined threshold. Another example is when the width or height of the current block is less than a predetermined threshold. Yet another example is if the area of ​​the current block is greater than a predetermined threshold, the proposed method cannot be used. Yet another example is if the area of ​​the current block is less than a predetermined threshold, the proposed method cannot be used. Yet another example is if the long side of the current block is significantly larger than the short side of the current block, the proposed method cannot be used. The predetermined threshold can be any integer, such as 2, 4, 8, 16, ..., or any predetermined value. When the proposed method cannot be used, a default method is used to determine the representative prediction mode.

[0326] In another embodiment, in order to reduce the latency caused by the proposed method in determining the representative prediction mode of the current block, some preprocessing operations are designed for the proposed method, and the proposed method is applied after the preprocessing operations.

[0327] In another sub-implementation, the preprocessing operation includes block setting. For example, block setting refers to subsampling (e.g., downsampling) the used prediction / reconstruction samples (within the current block and / or the spatial neighborhood region of the current block and / or the reference region of the current block), and then applying the proposed method to the subsampled samples. Another example is that block setting refers to applying the proposed method only to a subset of the used prediction / reconstruction samples (within the current block and / or the spatial neighborhood region of the current block and / or the reference region of the current block). The subset of used samples can be the first N rows or columns of the used samples, where N can be any predefined integer, such as 4, 8, 16, etc. The subset can also be the first M samples in the used samples, where M can be any predetermined integer, such as 4, 8, 16, etc.

[0328] In another sub - embodiment, the pre - processing operation includes a partitioning setting. The partitioning setting refers to dividing the prediction / reconstruction samples to be used (within the current block and / or in the spatial neighborhood region of the current block and / or in the reference region of the current block) into K sub - blocks, and applying the proposed method to each sub - block separately, where K can be any predetermined integer, such as 4, 16, etc.

[0329] The proposed method in the present invention can be enabled and / or disabled according to implicit rules (such as block width, height or area) or according to explicit rules (such as syntax at the block, tile, slice, image, SPS or PPS level). For example, the proposed method is applied when the block area is less than / greater than a threshold.

[0330] In the present invention, the term "block" can refer to TU / TB, CU / CB, PU / PB, a predetermined region or CTU / CTB.

[0331] Any combination of the proposed methods in the present invention can be applied.

[0332] The following is an example of how chrominance uses the proposed mechanism by allowing more primary transform candidates. Assume that N MTS candidates (for intra - frame MTS) are supported for chrominance, where N is set to 6, 4, 2 or 1. When the enabling condition is met, each chrominance TB can signal an index to select its mtsIdx (mtsIdxCb and mtsIdxCr). The enabling conditions for more primary transform candidates may include: Enabling condition (1): The current CU is intra - frame and has a chrominance component Enabling condition (2): 4 <= chrominance width < T and 4 <= chrominance height < T T is set to 32, 64, 128 or the maximum TB size in the standard Enabling condition (3): For a chrominance TB, MTS is allowed only if there are AC significant coefficients or significant coefficients in a predetermined region The position of the last significant coefficient is greater than 0 (DC position) The position of the last significant coefficient is greater than any predetermined value When using JCCR, only the coded / decoded TBs will send an mtsIdx When "any one of mtsIdxCb and mtsIdxCr">0, LFNST is disabled For each chrominance TB, if mtsIdx == 0, DCT2 is used as the original; otherwise, MTS is used as an additional transform option.

[0333] In one embodiment, the N candidates are selected from the set of transform candidates in Note C.

[0334] In another embodiment, the N MTS candidates for chromaticity (inherited from luminance) can be determined as follows: If luminance is not MTS, then set a transform candidate as the MTS candidate for chrominance. Transform candidates are determined by the chroma TB size as implicit intra-frame MTS Any of the proposed improved transform processing methods can be implemented in the encoder and / or decoder. For example, any proposed method can be implemented in the inter-frame / intra-frame / IBC / prediction / transform module of the encoder, and / or in the inter-frame / intra-frame / IBC / prediction / transform module of the decoder. Alternatively, any proposed method can be implemented as circuitry connected to the inter-frame / intra-frame / IBC / prediction / transform module of the encoder and / or the inter-frame / intra-frame / IBC / prediction / transform module of the decoder to provide the information required by the inter-frame / intra-frame / IBC / prediction / transform module. References FIG. 1A and FIG. 1B In the encoder and decoder, any proposed method can be implemented in the encoder's inter-frame / intra-frame / prediction / transform module (e.g. FIG. 1A Implemented in intra-frame prediction (110) in the decoder, and / or in the inter-frame / intra-frame / prediction / transform module (e.g. FIG. 1B Intra-frame prediction (150) is implemented in the framework.

[0335] FIG. 42 A flowchart of an exemplary video encoding / decoding system is shown, which incorporates improved transform processing according to an embodiment of the invention. The steps shown in the flowchart can be executed as program code on one or more processors (e.g., one or more central processing units) at the encoder and / or decoder ends. The steps shown in the flowchart can also be implemented in hardware, such as one or more electronic devices or processors arranged to perform the steps in the flowchart. According to the method, in step 4210, input data associated with the current block is received, wherein the input data includes residual data to be processed at the encoder end or encoded / decoded transform data associated with the current block to be decoded at the decoder end. In step 4220, transform processing is applied to the input data, wherein the transform process determines a target transform for the current block, and the target transform is selected from a transform candidate set containing at least one additional transform candidate relative to the original transform candidate set, or the target transform depends on an intra-prediction mode associated with the current block, a reference region, or one or more neighboring blocks. In step 4230, a transformed output is provided, wherein the transformed output includes the transform coefficients of the current block at the encoder end or the recovered residual data at the decoder end.

[0336] The flowchart shown is intended to illustrate an example of video encoding and decoding according to the present invention. Those skilled in the art can modify each step, rearrange the steps, split a step, or combine the steps to practice the invention without departing from its spirit. Specific syntax and semantics are used in this disclosure to illustrate examples of implementing embodiments of the invention. Those skilled in the art can practice the invention by using equivalent syntactic and semantic substitutions without departing from its spirit.

[0337] The foregoing description is intended to enable those skilled in the art to practice the invention in the context of specific applications and their requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments. Therefore, the invention is not intended to be limited to the specific embodiments shown and described, but should be given the broadest scope in accordance with the principles and novel features disclosed herein. Various specific details have been shown in the foregoing detailed description to provide a thorough understanding of the invention. However, those skilled in the art will understand that the invention can be practiced.

[0338] The embodiments of the present invention described above can be implemented in various hardware, software code, or combinations thereof. For example, one embodiment of the invention may be one or more circuits integrated into a video compression chip, or program code integrated into video compression software to perform the processes described herein. Another embodiment of the invention may be program code to be executed on a digital signal processor (DSP) to perform the processes described herein. The invention may also relate to several functions performed by a computer processor, digital signal processor, microprocessor, or field-programmable gate array (FPGA). These processors can be configured to perform specific tasks according to the invention by executing machine-readable software code or firmware code that defines specific methods embodied by the invention. The software code or firmware code can be developed in different programming languages ​​and different formats or styles. The software code can also be compiled for different target platforms. However, different code formats, styles, and languages ​​of the software code, as well as other configuration codes to perform methods consistent with the tasks of the invention, do not depart from the spirit and scope of the invention.

[0339] This invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples are to be regarded as illustrative rather than restrictive in all respects. Therefore, the scope of the invention is indicated by the appended claims rather than the foregoing description. All variations within the meaning and equivalence of the claims should be included within its scope.

Claims

1. A video encoding / decoding method, the method comprising: Receive input data associated with the current block, wherein the input data includes residual data to be processed at the encoder end or encoded transform data associated with the current block to be decoded at the decoder end; A transform process is applied to the input data, wherein the transform process determines a target transform for the current block, and the target transform is selected from a transform candidate set containing at least one additional transform candidate relative to the original transform candidate set, or the target transform depends on an intra-prediction mode associated with the current block, a reference region, or one or more neighboring blocks. Provides a transformed output, wherein the transformed output includes multiple transform coefficients of the current block at the encoder end or recovery residual data at the decoder end.

2. The video encoding and decoding method as described in claim 1, characterized in that, The transformation process includes a primary transformation at the encoder end, or the transformation process includes an inverse primary transformation at the decoder end, and wherein the transformation process includes a secondary transformation at the encoder end, or the transformation process includes an inverse secondary transformation at the decoder end.

3. The video encoding and decoding method as described in claim 2, characterized in that, The target transformation corresponds to a separable or non-separable primary or secondary transformation.

4. The video encoding and decoding method as described in claim 1, characterized in that, The current block is encoded and decoded using IntraTMP (Intra-Temporal Matching Prediction), Matrix Weighted IntraPrediction (MIP), Spatial Geometry Partitioning (SGPM), Direct Block Vector (DBV), or Horizontal / Vertical Plane mode.

5. The video encoding and decoding method as described in claim 1, characterized in that, The current block is encoded and decoded using cross-component mode.

6. The video encoding and decoding method as described in claim 1, characterized in that, The current block is encoded and decoded using inter-frame, intra-frame, or IBC modes.

7. The video encoding and decoding method as described in claim 1, characterized in that, The current block corresponds to either the luminance block or the chrominance block.

8. The video encoding and decoding method as described in claim 1, characterized in that, Functional controls for enabling the at least one additional transform candidate are sent or parsed in the Sequence Parameter Set (SPS) or Image Parameter Set (PPS).

9. The video encoding and decoding method as described in claim 8, characterized in that, Use explicit flags to indicate whether the at least one additional transform candidate for the chroma block is supported.

10. The video encoding and decoding method as described in claim 8, characterized in that, Use one or more explicit syntax elements to indicate the maximum number of transformation candidates for the at least one additional transformation candidate.

11. The video encoding and decoding method as described in claim 1, characterized in that, The original transform candidate set consists of DCT-II, and the at least one additional transform candidate corresponds to one or more multiple transform selection (MTS) candidates.

12. The video encoding and decoding method as described in claim 11, characterized in that, At the block level, on / off control is used to determine whether to use the at least one additional transformation candidate for the current block.

13. The video encoding and decoding method as described in claim 12, characterized in that, The on / off control is indicated in each chromaticity conversion unit (TU) or chromaticity conversion block (TB).

14. The video encoding and decoding method as described in claim 13, characterized in that, If the on / off control indication supports one or more MTS candidates for the current block, then the MTS type of the current block's chroma TU or chroma TB is inherited from the current block's luminance TU or luminance TB.

15. The video encoding and decoding method as described in claim 13, characterized in that, If the on / off control indication supports one or more MTS candidates for the current block, the MTS type of the chroma TU of the current block is indicated by the sent or parsed MTS index, or inferred to be a predefined MTS type.

16. The video encoding and decoding method as described in claim 1, characterized in that, When the current block uses the Joint Codec for Chromatic Residuals (JCCR) mode, the Cr and Cb components of the current block share the same transform selection.

17. The video encoding and decoding method as described in claim 1, characterized in that, The at least one additional transformation candidate is enabled when one or more conditions are met.

18. The video encoding and decoding method as described in claim 17, characterized in that, For chromaticity TU or chromaticity TB, the one or more conditions include chromaticity block size, one or more chromaticity transformation coefficients, or both.

19. The video encoding and decoding method as described in claim 17, characterized in that, The one or more conditions mentioned above are consistent for both the chromaticity and luminance components.

20. The video encoding and decoding method as described in claim 1, characterized in that, The target transformation depends on the intra-frame prediction mode derived using decoder-side derivation techniques or template-based derivation techniques, and is based on the current block or one or more neighboring blocks.

21. The video encoding and decoding method as described in claim 1, characterized in that, The target transformation depends on the intra-frame prediction mode indicated by the block vector or motion vector.

22. The video encoding and decoding method as described in claim 1, characterized in that, The intra-frame prediction mode is derived based on mode information in a reference region, which includes the corresponding reference block and / or the adjacent regions of the corresponding reference block.

23. The video encoding and decoding method as described in claim 1, characterized in that, The target transformation depends on the intra-prediction mode derived from the derivation technique, which varies depending on the block width, block height, block area, or explicit syntax of the current block.

24. A video encoding / decoding apparatus, the apparatus comprising one or more electronic devices or processors configured to: Receive input data related to the current block, wherein the input data includes residual data processed at the encoder end or encoded transform data related to the current block to be decoded at the decoder end; A transform process is applied to the input data, wherein the transform process determines a target transform for the current block, and the target transform is selected from a transform candidate set containing at least one additional transform candidate relative to the original transform candidate set, or the target transform depends on an intra-prediction mode associated with the current block, a reference region, or one or more neighboring blocks. Provide a transformed output, wherein the transformed output includes multiple transform coefficients of the current block at the encoder end or residual data recovered at the decoder end.

Images