Method and apparatus for intra prediction using decoder derivation in video coding system
By using BDPCM and inter-intra-intra-mixed GPM in the video encoding and decoding system, the prediction direction is determined using the statistical data or histogram derived from the template, the problem of insufficient intra-prediction data transmission is solved, and the encoding efficiency and compression performance are improved.
Patent Information
- Application Number
- CN202380030515.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-08
- Filing Date
- 2023-04-07
- Publication Date
- 2024-11-15
AI Technical Summary
In video encoding and decoding systems, there is a problem of insufficient bit savings in data transmission and encoding and decoding related to intra prediction, especially in inter-intra-mixed mode, resulting in low encoding efficiency.
Using block differential pulse codec modulation (BDPCM) and inter-intra-mixed geometric partition mode (GPM), the prediction direction is determined by using the statistical data or histograms derived from the template on the encoder and decoder side, and in the decoder side. Intra-mode export is performed on the side to reduce the bit stream of data transmission.
The data transmission amount related to intra prediction is effectively reduced, and the encoding efficiency is improved. Especially in the inter-intra-mix mode, the compression performance of video encoding and decoding is improved.
Smart Images

Figure CN118975247A_ABST
Abstract
Description
[0001]
Cross reference
[0002] The present invention is part of a non-provisional application claiming priority to U.S. Provisional Patent Application No. 63 / 328,766 filed on April 8, 2022. The above application is incorporated herein by reference. [Technical field]
[0003] The present invention relates to intra-frame prediction in video coding systems, and in particular to bit saving of coding parameters associated with Block Differential Pulse Coded Modulation (BDPCM) and inter-intra mixed Geometric Partition Mode (GPM). [Background technology]
[0004] Versatile Video Codec (VVC) is the latest international video codec standard (MPEG) developed by the Joint Video Experts Group (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group. The standard has been published as an ISO standard: ISO / IEC 23090-3:2021, Information technology-Coded representation of immersive media-Part 3:Versatile video coding, released in February 2021. VVC was developed by adding more codec tools to improve codec efficiency and handle various types of video sources, including 3D video signals, based on its predecessor HEVC (High Efficiency Video coding).
[0005] Figure 1AAn exemplary adaptive Inter / Intra video codec system including loop processing is described. For intra prediction, prediction data is derived from previously coded video data in the current picture (hereinafter also referred to as a picture). For inter prediction 112, motion estimation (abbreviated as ME) is performed on the encoder side and motion compensation (abbreviated as MC) is performed based on the result of ME to provide prediction data derived from other pictures and motion data. Switch 114 selects intra prediction 110 or inter prediction 112 and the selected prediction data is provided to adder 116 to form a prediction error, also referred to as a residual. The prediction error is then processed by transform (T) 118 and subsequent quantization (Q) 120. The transformed and quantized residual is then encoded by entropy encoder 122 to be included in the video bitstream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packaged with side information (e.g., motion and decoding modes associated with intra-frame prediction and inter-frame prediction) and other information (e.g., parameters associated with the loop filter applied to the underlying image area). Side information associated with intra-frame prediction 110, inter-frame prediction 112, and loop filter 130 is provided to entropy encoder 122, as shown in FIG. Figure 1A As shown. When using inter-frame prediction mode, one or more reference pictures must also be reconstructed at the encoder end. Therefore, the transformed and quantized residual is processed by inverse quantization (IQ) 124 and inverse transform (IT) 126 to recover the residual. The residual is then added back to the prediction data 136 at reconstruction (REC) 128 to reconstruct the video data. The reconstructed video data can be stored in a reference picture buffer 134 and used to predict other frames.
[0006] like Figure 1AAs shown, the input video data undergoes a series of processes in the encoding system. Due to the series of processes, the reconstructed video data from REC 128 may be subject to various impairments. Therefore, the loop filter 130 is often applied to the reconstructed video data before it is stored in the reference picture buffer 134 to improve the video quality. For example, a deblocking filter (DF), a sample adaptive offset (SAO), and an adaptive loop filter (ALF) may be used. It may be necessary to merge the loop filter information into the bitstream so that the decoder can correctly recover the required information. Therefore, the loop filter information is also provided to the entropy encoder 122 to be merged into the bitstream. Figure 1A In , a loop filter 130 is applied to the reconstructed video before the reconstructed samples are stored in a reference picture buffer 134 . Figure 1A The system in is intended to illustrate an exemplary structure of a typical video encoder. It may correspond to the High Efficiency Video Codec (HEVC) system, VP8, VP9, H.264, or VVC.
[0007] like Figure 1B As shown, except for transform 118 and quantization 120, the decoder can use similar or identical functional blocks as the encoder, because the decoder only needs inverse quantization 124 and inverse transform 126. Instead of entropy encoder 122, the decoder uses entropy decoder 140 to decode the video bitstream into quantized transform coefficients and required coding information (such as ILPF information, intra-frame prediction information, and inter-frame prediction information). Intra-frame prediction 150 on the decoder side does not need to perform mode search. Instead, the decoder only needs to generate intra-frame prediction based on the intra-frame prediction information received from entropy decoder 140. In addition, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) based on the inter-frame prediction information received from entropy decoder 140 without motion estimation.
[0008] According to VVC, similar to HEVC, the input picture is partitioned into non-overlapping square block areas called CTUs (Codec Tree Units). Each CTU can be divided into one or more smaller Codec Units (CUs). The generated CU partitions can be square or rectangular. In addition, VVC divides CTUs into Prediction Units (PUs) as units for applying prediction processing, such as inter-frame prediction, intra-frame prediction, etc.
[0009] The VVC standard combines various new coding and decoding tools to further improve the coding and decoding efficiency based on the HEVC standard. Among the various new coding and decoding tools, some coding and decoding tools related to the present invention are summarized as follows.
[0010] Use tree structure to divide CTU
[0011] In HEVC, CTUs are divided into CUs to accommodate various local features using a quaternary-tree (QT) structure represented as a codec tree. The decision to encode and decode a picture area using inter-picture (temporal) or intra-picture (spatial) prediction is made at the leaf-CU level. Each leaf CU can be further split into one, two, or four PUs according to the PU splitting type. Within a PU, the same prediction process is applied, and relevant information is transmitted to the decoder on a PU basis. After obtaining the residual block by applying the prediction process based on the PU splitting type, the leaf CU can be divided into transform units (TUs) according to another quadtree structure similar to the codec tree of the CU. One of the key features of the HEVC structure is that it has multiple partition concepts, including CU, PU, and TU.
[0012] In VVC, a quadtree with a nested multi-type tree using a binary and ternary splitssegmentation structure replaces the concept of multiple partition unit types, that is, it removes the separation of the concepts of CU, PU, and TU (except for CUs that are too large for the maximum transform length), and supports more flexible CU partition shapes. In the decoding tree structure, a CU can be a square or a rectangle. A codec tree unit (CTU) is first divided by a quadtree (also known as a quadtree) structure. Then the quadtree leaf nodes can be further divided into multi-type tree structures. Such as Figure 2 As shown, there are four types of splits in the multi-type tree structure, vertical binary split (SPLIT_BT_VER 210), horizontal binary split (SPLIT_BT_HOR 220), vertical ternary split (SPLIT_TT_VER 230), and horizontal ternary split (SPLIT_TT_HOR240). The multi-type tree leaf node is called a codec unit (CU), and this segmentation is used for prediction and transform processing without further division unless the CU is too large for the maximum transform length. This means that in most cases, CUs, PUs, and TUs have the same block size in a quadtree with a nested multi-type tree block structure. An exception occurs when the maximum supported transform length is less than the width or height of the CU color component.
[0013] Figure 3 The signaling mechanism for partitioning split information in a quadtree of a nested multi-type tree codec tree structure is described. The codec tree unit (CTU) is regarded as the root of the quadtree and is first partitioned by the quadtree structure. Each quadtree leaf node (when large enough to allow it) is then further partitioned by the multi-type tree structure. In the multi-type tree structure, a first flag (mtt_split_cu_flag) is signaled to indicate whether the node is further partitioned; when a node is further partitioned, a second flag (mtt_split_cu_vertical_flag) is signaled to indicate the partition direction, and then a third flag (mtt_split_cu_binary_flag) is signaled to indicate whether the partition is a binary partition or a ternary partition. Based on the values of mtt_split_cu_vertical_flag and mtt_split_cu_binary_flag, the multi-type tree partition mode (MttSplitMode) of the CU is derived, as shown in Table 1.
[0014] Table 1 - Derivation of MttSplitMode based on multi-type tree syntax element
[0015]
[0016] Figure 4 The partitioning of a CTU into multiple CUs using a quadtree and nested multi-type tree codec block structure is shown, where bold block edges represent quadtree partitions and the remaining edges represent multi-type tree partitions. The quadtree with nested multi-type tree partitions provides a content-adaptive codec tree structure consisting of CUs. The size of a CU can be as large as a CTU or as small as 4×4 in units of luma samples. For the 4:2:0 chroma format, the maximum chroma CB size is 64×64 and the minimum size chroma CB consists of 16 chroma samples.
[0017] In VVC, the maximum supported luma transform size is 64 × 64, and the maximum supported chroma transform size is 32 × 32. When the width or height of the CB is larger than the maximum transform width or height, the CB is automatically split horizontally and / or vertically to meet the transform size limit in that direction.
[0018] The following parameters are defined and specified by the SPS syntax elements for quadtree with nested multi-type tree codec tree scheme.
[0019] CTU size: The root node size of the quadtree
[0020] MinQTSize: The minimum allowed size of a quadtree leaf node
[0021] MaxBtSize: Maximum allowed binary tree root node size
[0022] MaxTtSize: Maximum allowed ternary tree root node size
[0023] MaxMttDepth: The maximum allowed hierarchy depth of the multi-type tree split from the leaves of the quad tree
[0024] MinBtSize: The minimum allowed binary tree leaf node size
[0025] MinTtSize: The minimum allowed ternary tree leaf node size
[0026] In an example of a quadtree with a nested multi-type tree codec tree structure, the CTU size is set to 128×128 luma samples and two corresponding 64×64 blocks of 4:2:0 chroma samples, MinQTSize is set to 16×16, MaxBtSize is set to 128×128, MaxTtSize is set to 64×64, MinBtSize and MinTtSize (width and height) are set to 4×4, and MaxMttDepth is set to 4. Quadtree partitioning is first applied to the CTU to generate quadtree leaf nodes. The size of the quadtree leaf node can range from 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If the leaf QT node is 128×128, the binary tree will not be further split because the size exceeds MaxBtSize and MaxTtSize (i.e., 64×64). Otherwise, the leaf qdtree node may be further split by the multi-type tree. Therefore, the quadtree leaf node is also the root node of the multi-type tree, and its multi-type tree depth (mttDepth) is 0. When the multi-type tree depth reaches MaxMttDepth (i.e. 4), no further splitting is considered. When the width of the multi-type tree node is equal to MinBtSize and less than or equal to 2*MinTtSize, no further horizontal splitting is considered. Similarly, when the height of the multi-type tree node is equal to MinBtSize and less than or equal to 2*MinTtSize, no further vertical splitting is considered.
[0027] In order to allow 64×64 luma blocks and 32×32 chroma pipeline design in the VVC hardware decoder, TT splitting is prohibited when the width or height of the luma codec block is greater than 64, such as Figure 5As shown. Wherein block 500 corresponds to a 128x128 luma CU. The CU can be split using vertical binary splitting (510) or horizontal binary splitting (520). The block is split into 4 CUs, each CU is 64x64 in size, and the CU can be further split using partitions including TT. For example, the 64x64 CU in the upper left corner is partitioned using vertical ternary splitting (530) or horizontal ternary splitting (540). TT splitting is also prohibited when the width or height of the chroma codec block is greater than 32.
[0028] In VVC, the codec tree scheme supports the ability to have separate block tree structures for luma and chroma. For P and B slices, the luma and chroma CTBs in a CTU must share the same codec tree structure. However, for I slices, luma and chroma can have separate block tree structures. When separate block tree mode is applied, the luma CTB is divided into CUs by one codec tree structure, and the chroma CTB is divided into chroma CUs by another codec tree structure. This means that a CU in an I slice may consist of a codec block for the luma component or a codec block for two chroma components, while a CU in a P or B slice always consists of codec blocks for all three color components unless the video is monochrome.
[0029] Virtual Pipeline Data Unit (VPDU)
[0030] A virtual pipeline data unit (VPDU) is defined as a non-overlapping unit in a picture. In a hardware decoder, consecutive VPDUs are processed simultaneously by multiple pipeline stages. In most pipeline stages, the VPDU size is roughly proportional to the buffer size, so it is important to keep the VPDU size small. In most hardware decoders, the VPDU size can be set to the maximum transform block (TB) size. However, in VVC, ternary tree (TT) and binary tree (BT) partitioning may cause the VPDU size to increase.
[0031] To keep the VPDU size to 64x64 luma samples, the following canonical partitioning restrictions (with syntax signaling modifications) are applied in VTM, as Figure 7 As shown:
[0032] – For CUs with width or height equal to 128 or width and height equal to 128, TT split is not allowed (such as Figure 7” in the figure).
[0033] – For 128xN CUs with N ≤ 64 (i.e., width equals 128 and height less than 128), horizontal BT is not allowed.
[0034] For Nx128CUs with N≤64 (i.e., height equal to 128 and width less than 128), vertical BT is not allowed. Figure 6 In the example, the luminance block size is 128x128. The dotted line indicates a block size of 64x64. According to the above restrictions, examples where splitting is not allowed are indicated by "X", such as Figure 6 As shown in various examples (610-680).
[0035] Intra-mode codec with 67 intra-prediction modes
[0036] To capture arbitrary edge directions present in natural videos, the number of directional intra modes in VVC is expanded to 65 from the 33 used in HEVC. Figure 7 New directional modes not found in HEVC are shown in Figure 1 (shown as red dotted arrows), while planar and DC modes remain unchanged. These more dense directional intra prediction modes apply to all block sizes and luma and chroma intra prediction.
[0037] In VVC, for non-square blocks, several traditional angular intra prediction modes are adaptively replaced with wide-angle intra prediction modes.
[0038] In HEVC, each intra-coded block has a square shape and the length of each of its sides is a power of 2. Therefore, no division operation is required to generate the intra-predictor using the DC mode. In VVC, blocks can have a rectangular shape, which in general requires the use of a division operation for each block. To avoid the division operation for DC prediction, only the longer sides are used to calculate the average value for non-square blocks.
[0039] In order to keep the complexity of most probable mode (MPM) list generation low, an intra mode codec with 6 MPMs is used by considering two available neighboring intra modes. The following three aspects are considered when constructing the MPM list:
[0040] - Preset intra-frame mode
[0041] –Neighborhood Intra Mode
[0042] – Intra mode for export.
[0043] A unified 6-MPM list is used for intra blocks, regardless of whether the MRL and ISP codecs are applied. The MPM list is constructed based on the intra modes of the left and upper neighboring blocks. Assuming the mode on the left is denoted as Left and the mode of the upper block is denoted as Above, the unified MPM list is constructed as follows:
[0044] – When a neighborhood block is not available, its intra mode preset is set to Planar.
[0045] – If both Left and Above modes are non-angular mode:
[0046] –MPM list → {Planar,DC,V,H,V-4,V+4}
[0047] – If one of Left and Above modes is in angular mode and the other is in non-angular mode:
[0048] – Set mode Max to the larger mode between Left and Above
[0049] –MPM list → {Planar,Max,DC,Max-1,Max+1,Max-2}
[0050] – If Left and Above are both angled and they are different:
[0051] – Set mode Max to the larger mode between Left and Above
[0052] – If the difference between the modes Left and Above is in the range of 2 to 62, inclusive
[0053] MPM list → {Planar,Left,Above,DC,Max-1,Max+1}
[0054] otherwise
[0055] MPM list → {Planar,Left,Above,DC,Max-2,Max+2}
[0056] – If Left and Above are both angled and they are the same:
[0057] –MPM list→{Planar,Left,Left-1,Left+1,DC,Left-2}
[0058] In addition, the first bit bin of the MPM index codeword is CABAC context coded. A total of three contexts are used, corresponding to whether the current intra block is MRL enabled, ISP enabled, or a normal intra block.
[0059] In the 6MPM list generation process, pruning is used to remove repeated patterns so that only unique patterns can be included in the MPM list. For the entropy coding and decoding of the 61 non-MPM patterns, truncated binary coding (TBC) is used.
[0060] In the present invention, a method and apparatus for further reducing data associated with intra-frame prediction are disclosed. [Summary of the invention]
[0061] A method and apparatus for video encoding and decoding are disclosed. According to the method, pixel data associated with a current block is received at an encoder side or encoded data associated with a current block to be decoded is received at a decoder side. A prediction direction for the current block is determined between vertical prediction and horizontal prediction based on a template of the current block or based on a decoder-side intra-mode derivation using statistics or histograms of angular fields derived from the template of the current block. The current block is encoded or decoded using BDPCM (block differential pulse code modulation) in the prediction direction. In one embodiment, the template includes one or more sampling lines in a neighboring area of the current block.
[0062] According to another method, the current block is segmented into a first region and a second region based on region segmentation. The first region is encoded or decoded based on an inter-frame codec. The second region is encoded or decoded based on an intra-frame codec. For this method, at least a portion of the region separation parameters, a portion of the inter-frame codec parameters, or a portion of the intra-frame codec parameters are determined based on a template of the current block or based on a decoder-side intra-frame mode derivation using statistics or histograms of angular fields derived from the template of the current block. In one embodiment, the template includes one or more sampling lines in a neighboring area of the current block.
[0063] In one embodiment, a template of the current block is used to derive a motion vector for inter-coding.
[0064] In one embodiment, the intra prediction angles for intra coding are derived using a template or decoder-side intra mode derivation of the current block.
[0065] In one embodiment, the partition boundary offsets associated with the region segmentation are derived using a template of the current block. In one embodiment, the information of the partition boundary slopes associated with the region segmentation is signaled in the bitstream at the encoder side. In one embodiment, the information of the partition boundary slopes associated with the region segmentation is parsed from the bitstream at the decoder side.
Brief Description of the Drawings
[0066] Figure 1A An exemplary adaptive inter / intra video coding system incorporating loop processing is described.
[0067] Figure 1B Picture shows Figure 1A The corresponding decoder of the encoder in .
[0068] Figure 2 Examples of multi-type tree structures corresponding to vertical binary segmentation (SPLIT_BT_VER), horizontal binary segmentation (SPLIT_BT_HOR), vertical ternary segmentation (SPLIT_TT_VER), and horizontal ternary segmentation (SPLIT_TT_HOR) are shown.
[0069] Figure 3 An example of a signaling mechanism for partition split information in a quadtree with a nested multi-type tree codec tree structure is shown.
[0070] Figure 4 An example of a CTU being partitioned into multiple CUs with quadtree and nested multi-type tree codec block structures is shown, where bold block edges represent quadtree partitions and the remaining edges represent multi-type tree partitions.
[0071] Figure 5 An example showing disabling TT splitting when the width or height of the luma codec block is greater than 64.
[0072] Figure 6 Some examples are shown for disabling TT splitting when the width or height of the luma codec block is greater than 64.
[0073] Figure 7 The figure shows the intra-frame prediction mode adopted by the VVC video coding standard.
[0074] Fig. 8A -B shows a block that is wider than it is tall ( Fig. 8A ) and blocks that are taller than they are wide ( Figure 8B ) is an example of wide-angle intra prediction.
[0075] Fig. 9 An example of two vertically adjacent prediction samples using two non-adjacent reference samples in the case of wide-angle intra prediction is shown.
[0076] Fig. 10A An example of a template selected for a current block is shown, where the template includes T rows above the current block and T columns to the left of the current block.
[0077] Fig. 10B An example of T=3 is shown, and the HoG (Histogram of Gradients) is calculated for pixels in the middle row and pixels in the middle column.
[0078] Fig. 10C An example of the amplitude (ampl) of the angular intra prediction mode is shown.
[0079] Fig.11 An example of a hybrid process is shown, where two intra modes (M1 and M2) and the planar mode are selected according to the index of the histogram bins having the two highest bins.
[0080] Fig.12 An example of a Template-based Intra Mode Derivation (TIMD) mode is shown, where TIMD implicitly derives the intra prediction mode of a CU using neighboring templates at both the encoder and the decoder.
[0081] Fig.13 An example of 64 partitions used in the VVC standard is shown, where the partitions are grouped according to their angles and the dashed lines represent redundant partitions.
[0082] Fig.14 An example of unidirectional prediction MV selection for geometric partition mode is shown.
[0083] Fig.15 An example of the warping weight ω0 using a geometric partitioning pattern is shown.
[0084] Fig.16A An example of an inter-intra hybrid GPM mode is shown, where an occluded object emerges from behind another object.
[0085] Fig. 16B An example of an inter-intra hybrid GPM mode applied to a current block is shown.
[0086] Fig.17A An example of an inter-intra hybrid GPM according to an embodiment of the present invention is shown, where a template is used to derive information related to a partition region, an inter-codec or an intra-codec.
[0087] Fig. 17B Examples of templates for deriving parameters for inter-frame coding and intra-frame coding are shown.
[0088] Fig.18 An exemplary process of inter-frame and intra-frame mixing of GPMs according to an embodiment of the present invention is shown.
[0089] Fig.19 An example of determining between vertical binary partitioning and horizontal binary partitioning using TIMD or DIMD according to an embodiment of the present invention is shown.
[0090] Fig. 20 A flowchart of an exemplary video encoding and decoding system for deriving a partition mode of BDPCM using TIMD / DIMD according to an embodiment of the present invention is shown.
[0091] Fig.21 A flowchart of an exemplary video encoding and decoding system for deriving encoding and decoding parameters related to inter-frame GPM using TIMD / DIMD according to an embodiment of the present invention is shown. [Specific implementation method]
[0092] It is readily understood that the elements of the present invention as generally described and illustrated in the accompanying drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of the embodiments of the system and method of the present invention as shown in the figures is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. References throughout this specification to "an embodiment", "one embodiment" or similar language mean that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment" or "in one embodiment" that appear throughout this specification do not necessarily all refer to the same embodiment.
[0093] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. However, those skilled in the relevant art will recognize that the present invention may be practiced without one or more specific details, or using other methods, elements, etc. In other cases, well-known structures or detailed descriptions of operations are not shown or are not shown to avoid obscuring aspects of the present invention. The illustrated embodiments of the present invention will be best understood with reference to the accompanying drawings, in which like parts are represented by like digits throughout. The following description is intended to be exemplary only, and simply illustrates certain selected embodiments of the apparatus and methods consistent with the present invention as claimed herein.
[0094] Wide-Angle Intra Prediction for Non-square Blocks
[0095] The conventional angular intra prediction direction is defined as clockwise from 45 degrees to -135 degrees. In VVC, several traditional angular intra prediction modes are adaptively replaced with wide-angle intra prediction modes for non-square blocks. The replaced modes are signaled using the original mode index, which is remapped to the index of the wide-angle mode after parsing. The total number of intra prediction modes remains unchanged, i.e. 67, and the intra mode encoding and decoding method remains unchanged.
[0096] To support these prediction directions, the top reference of length 2W+1 and the left reference of length 2H+1 are respectively as follows Fig. 8A and Figure 8B is defined.
[0097] The number of alternative modes in wide-angle direction mode depends on the aspect ratio of the block. The alternative intra prediction modes are shown in Table 2.
[0098] Table 2 - Intra-prediction modes replaced by wide-angle mode
[0099] Aspect Ratio Alternative intra prediction modes W / H==16 Mode 12,13,14,15 W / H==8 Mode 12,13 W / H==4 Mode 2,3,4,5,6,7,8,9,10,11 W / H==2 Mode 2,3,4,5,6,7, W / H==1 none W / H==1 / 2 Mode 61,62,63,64,65,66 W / H==1 / 4 Mode 57,58,59,60,61,62,63,64,65,66 W / H==1 / 8 Mode 55,56 W / H==1 / 16 Mode 53,54,55,56
[0100] like Fig. 9 As shown, in the case of wide-angle intra prediction, two vertically adjacent prediction samples (samples 910 and 912) may use two non-adjacent reference samples (samples 920 and 922). Therefore, a low-pass reference sample filter and side smoothing are applied to wide-angle prediction to reduce the negative impact of the increased gap Δpα. If the wide-angle mode represents a non-fractional offset. There are 8 modes in the wide-angle mode that meet this condition, namely [-14, -12, -10, -6, 72, 76, 78, 80]. When predicting blocks through these modes, the samples in the reference buffer are directly copied without applying any interpolation. With this modification, the number of samples that need to be smoothed is reduced. In addition, it aligns the design of non-fractional modes in traditional prediction mode and wide-angle mode.
[0101] In VVC, 4:2:2 and 4:4:4 as well as 4:2:0 chroma formats are supported. The chroma derived mode (DM) derivation table for the 4:2:2 chroma format was originally ported from HEVC, expanding the number of entries from 35 to 67 to be consistent with the extension of intra prediction modes. Since the HEVC specification does not support prediction angles below -135° and above 45°, the luma intra prediction modes from 2 to 5 are mapped to 2. Therefore, the chroma DM derivation table for the 4:2:2 chroma format is updated by replacing some values of the mapping table entries to more accurately convert the prediction angles of the chroma blocks.
[0102] 4-tap interpolation filter (IF) and reference sample smoothing
[0103] Four-tap intra-frame interpolation filters are used to improve directional intra prediction accuracy. In HEVC, two-tap linear interpolation filters have been used to generate intra prediction blocks in directional prediction mode (i.e., excluding planar and DC predictors). In VVC, two sets of 4-tap IFs replace the low-precision linear interpolation in HEVC, one set is a DCT-based interpolation filter (DCTIF) and the other is a 4-tap smoothing interpolation filter (SIF). DCTIF is constructed in the same way as used for chroma component motion compensation in HEVC and VVC. SIF is obtained by convolving a 2-tap linear interpolation filter with a [1 2 1] / 4 filter.
[0104] Depending on the intra prediction mode, the following reference sampling processing is performed:
[0105] Directional intra prediction modes fall into one of the following groups:
[0106] – Group A: vertical or horizontal mode (HOR_IDX, VER_IDX),
[0107] – Group B: Directional modes representing non-fractional angles (-14, -12, -10, -6, 2, 34, 66, 72, 76, 78, 80,) and Planar mode.
[0108] – Group C: remaining directional patterns;
[0109] If the directional intra prediction mode belongs to group A, no filter is applied to the reference samples to generate the predicted samples;
[0110] Otherwise, if the mode belongs to group B and the mode is a directional mode and all of the following conditions are true, then a [1,2,1] reference sample filter (depending on the MDIS condition) may be applied to the reference samples to further copy these filtered values to the intra predictor according to the selected direction, but no interpolation filter is applied:
[0111] –refIdx equals 0 (no MRL)
[0112] –TU size greater than 32
[0113] – Brightness (luma)
[0114] – No ISP Block
[0115] Otherwise, if the mode is classified as belonging to group C, the MRL index is equal to 0, and the current block is not an ISP block, only the intra reference sample interpolation filter is applied to the reference samples to generate prediction samples that fall into fractional or integer positions between the reference samples according to the selected direction (no reference sample filtering is performed). The interpolation filter type is determined as follows:
[0116] – Set minDistVerHor equal to Min(Abs(predModeIntra-50),Abs(predModeIntra-18))
[0117] – Set nTbS equal to (Log2(W)+Log2(H))>>1
[0118] – Set intraHorVerDistThres[nTbS] as specified below:
[0119]
[0120] – If minDistVerHor is greater than intraHorVerDistThres[nTbS], use SIF for interpolation
[0121] – Otherwise, use DCTIF for interpolation
[0122] Decoder Side Intra Mode Derivation (DIMD)
[0123] When DIMD is applied, two intra modes are derived from the reconstructed neighbor samples and these two predictors are combined with the planar mode predictor with weights derived from the gradients. DIMD mode is used as an alternative prediction mode and is always checked in high complexity RDO mode.
[0124] To implicitly derive the intra prediction mode of a block, texture gradient analysis is performed on both the encoder and decoder sides. This process starts with an empty Histogram of Gradients (HoG) containing 65 entries corresponding to 65 angular modes. The magnitudes of these entries are determined during texture gradient analysis.
[0125] In the first step, DIMD selects T=3 columns and lines of templates from the left and top of the current block, respectively. This area is used as a reference for gradient-based intra prediction mode derivation.
[0126] In the second step, the horizontal and vertical Sobel filters are applied to all 3×3 window positions, centered on the pixels in the middle line of the template. At each window position, the Sobel filter calculates the pure horizontal and vertical intensities as G x and G y . The texture angle of the window is then calculated as follows:
[0127] angle=arctan(G x / G y ), (1)
[0128] It can be converted into one of 65 angular intra prediction modes. Once the intra prediction mode index of the current window is derived as idx, the magnitude of its entry in HoG[idx] is updated by adding:
[0129] ampl=|G x |+|G y | (2)
[0130] Fig. 10A -C shows an example of the HoG calculated after applying the above operation to all pixel positions in the template. Fig. 10A An example of a selected template 1020 for a current block 1010 is shown. The template 1020 includes T rows above the current block and T columns to the left of the current block. For intra prediction of the current block, an area 1030 above and to the left of the current block corresponds to a reconstructed area, while an area 1040 below and to the right of the block corresponds to an unavailable area. Fig. 10B An example of T=3 is shown, and the HoG is calculated for a pixel 1060 in the middle row and a pixel 1062 in the middle column. For example, for pixel 1052, a 3x3 window 1050 is used. Fig. 10C An example of the amplitude (ampl) calculated based on equation (2) is shown, where the angular intra prediction mode is determined according to equation (1).
[0131] Once the HoG is calculated, the indices with the two highest histogram bins are selected as the two implicitly derived intra prediction modes for the block and further combined with the planar mode as the prediction for DIMD mode. Prediction fusion is applied as a weighted average of the above three predictors. For this purpose, the weight of the plane is fixed to 21 / 64 (~1 / 3). Then, the remaining weight 43 / 64 (~2 / 3) is distributed between the two HoG IPMs, proportional to the amplitude of their HoG bins. Fig.11 An example of a mixed process is shown. Fig.11As shown, two intra modes (M1 1112 and M2 1114) are selected according to the index of the two highest bars in the histogram bar 1110. Three predictors (1140, 1142, and 1144) are used to form a mixed prediction. The three predictors correspond to applying M1, M2, and planar intra modes (1120, 1122, and 1124, respectively) to the reference pixel 1130 to form the corresponding predictor. The three predictors are weighted by respective weighting factors (ω1, ω2, and ω3) 1150. The weighted predictors are summed using an adder 1152 to generate a mixed predictor 1160.
[0132] Furthermore, two implicitly derived intra modes are included in the MPM list so that the DIMD process is performed before constructing the MPM list. The primary derived intra mode of a DIMD block is stored with the block and used for MPM list construction for neighboring blocks.
[0133] Template-based Intra Mode Derivation (TIMD)
[0134] The template-based intra-mode derivation mode implicitly derives the intra-prediction mode of a CU using neighboring templates at both the encoder and the decoder, instead of signaling the intra-prediction mode to the decoder. Fig.12 As shown, the reference samples (1220 and 1222) of the template of each candidate mode are used to generate the prediction samples (1212 and 1214) of the template of the current block 1210. The cost is calculated as the sum of the absolute transformed differences (SATD) between the predicted samples and the reconstructed samples of the template. The intra prediction mode with the smallest cost is selected as the DIMD mode for intra prediction of the CU. The candidate mode can be 67 intra prediction modes as in VVC or extended to 131 intra prediction modes. In general, MPM can provide clues indicating the directional information of the CU. Therefore, in order to reduce the intra mode search space and utilize the characteristics of the CU, the intra prediction mode can be implicitly derived from the MPM list.
[0135] For each intra prediction mode in MPM, the SATD between the predicted and reconstructed samples of the template is calculated. The first two intra prediction modes with the smallest SATD are selected as TIMD modes. These two TIMD modes are fused with weights after applying the PDPC process, and this weighted intra prediction is used to encode and decode the current CU. The position dependent intra prediction combination (PDPC) is included in the derivation of the TIMD mode.
[0136] The costs of the two selected modes are compared to the threshold, and in the test, a cost factor of 2 is applied as follows:
[0137] costMode2<2*costMode1.
[0138] If this condition is true, fusion is applied, otherwise only mode 1 is used. The weight of the mode is calculated based on its SATD cost as follows:
[0139] weight1=costMode2 / (costMode1+costMode2)
[0140] weight2=1-weight1.
[0141] Block Differential Pulse Coded Modulation (BDPCM)
[0142] VVC supports Block Differential Pulse Codec Modulation (BDPCM) for screen content encoding and decoding. At the sequence level, the BDPCM enable flag is signaled in the SPS; this flag signal is only signaled when the transform skip mode is enabled in the SPS.
[0143] When BDPCM is enabled, if the CU size in terms of luma samples is less than or equal to MaxTsSize multiplied by MaxTsSize, and if the CU is intra-coded, a flag is transmitted at the CU level, where MaxTsSize is the maximum block size allowed for transform skip mode. The flag indicates whether regular intra-codec or BDPCM is used. If BDPCM is used, the BDPCM prediction direction flag is sent to indicate whether the prediction is horizontal or vertical. The block is then predicted using the regular horizontal or vertical intra prediction process and unfiltered reference samples. The residuals are quantized and the difference between each quantized residual and its predictor (i.e., the residual of the previously coded horizontal or vertical (depending on the BDPCM prediction direction) neighboring position) is coded.
[0144] For a block of size M (height) × N (width), let r i,j ,0≤i≤M-1,0≤j≤N-1 is the prediction residual. Let Q(r i,j ), 0≤i≤M-1,0≤j≤N-1 represents the residual r i,j Applying BDPCM to the quantized residual values produces a quantized version of The modified M×N array in It is predicted based on the quantized residual value of its neighbors. For the vertical BDPCM prediction mode, for 0≤j≤(N-1), the following formula is used to derive
[0145]
[0146] For the horizontal BDPCM prediction mode, for 0≤i≤(M-1), the following formula is used to derive
[0147]
[0148] At the decoder side, the above process is reversed to calculate Q(r i,j ), 0≤i≤M-1,0≤j≤N-1, as follows:
[0149]
[0150] The residual of inverse quantization, Q -1 (Q(r i,j )), is added to the intra block prediction value to generate the reconstructed sample value.
[0151] The predicted quantized residual value is converted to Sent to the decoder. For lossless codec, if slice_ts_residual_coding_disabled_flag is set to 1, the quantized residual values are sent to the decoder using the regular transform residual codec. For MPM modes for future intra mode codecs, if the BDPCM prediction direction is horizontal or vertical, store the CU for which the horizontal or vertical prediction mode is used for the BDPCM codec respectively. For deblocking, if both blocks on either side of a block boundary are using the BDPCM codec, that particular block boundary will not be deblocked.
[0152] Geometric Partitioning Mode (GPM)
[0153] In VVC, geometric partition mode (GPM) is supported for inter-frame prediction, as described in JVET-W2002 (Adrian Browne et al., Algorithm description for Versatile Video Coding and Test Model 14 (VTM14), ITU-T / ISO / IEC Joint Video Exploration Team (JVET), 23rd meeting, by teleconference, July 7-16, 2021, document: document JVET-M2002). The geometric partition mode is signaled using a CU-level flag as a merge mode. Other merge modes include normal merge mode, MMVD mode, CIIP mode, and sub-block merge mode. For each possible CU size, the geometric partition mode supports a total of 64 partitions, w×h=2 m ×2 n , where m,n∈{3...6} does not include 8x64 and 64x8. The GPM mode can be applied to skip or merge CUs with sizes within the above limits and with at least two normal merge modes.
[0154] When using this mode, the CU is split into two parts by a geometrically positioned straight line at a certain angle. In VVC, GPM uses a total of 20 angles and 4 offset distances, which is reduced from 24 angles in earlier drafts. The position of the split line is mathematically derived based on the angle and offset parameters of the specific partition. In VVC, there are a total of 64 partitions, such as Fig.13 As shown in Figure 1, the partitions are grouped according to angle, and the dotted lines indicate redundant partitions. Each part of the geometric partition in the CU uses its own motion for inter-frame prediction; each partition only allows uni-prediction, that is, each part has a motion vector and a reference index. Fig.13 , each row corresponds to the boundary of a partition. The partitions are grouped according to their angles. For example, partition group 1310 consists of three vertical GPM partitions (i.e., 90°). Partition group 1320 consists of four inclined GPM partitions with a small angle to the vertical direction. Moreover, partition group 1330 consists of three vertical GPM partitions similar to group 1310, but with the opposite direction (i.e., 270°). A unidirectional prediction motion constraint is applied to ensure that each CU only requires two motion compensated predictions, the same as traditional bidirectional prediction. The unidirectional prediction motion for each partition is derived using the process described later.
[0155] If geometry partition mode is used for the current CU, a geometry partition index indicating the selected partition mode (angle and offset) of the geometry partition as well as two merge indices (one for each partition) are further signaled. The number of maximum GPM candidate sizes is explicitly signaled in the SPS (sequence parameter set) and the syntax binarization of the GPM merge indices is specified. After predicting each part of the geometry partition, the sample values along the edges of the geometry partition are adjusted using a hybrid process with adaptive weights using the process described later. This is the prediction signal for the entire CU and, as with other prediction modes, the transform and quantization process will be applied to the entire CU. Finally, the motion field of the CU predicted using the geometry partition mode is stored using the process described later.
[0156] Uni-Prediction Candidate List Construction
[0157] The unidirectional prediction candidate list is derived directly from the merge candidate list constructed according to the extended merge prediction process. Denote n as the index of the unidirectional prediction motion in the geometric unidirectional prediction candidate list. The LX motion vector of the nth extended merge candidate (X=0 or 1, i.e. LX=L0 or L1) is used as the nth unidirectional prediction motion vector of the geometric partition mode, where X is equal to the parity of n. These motion vectors are Fig.14 If the corresponding LX motion vector of the n-th extended merge candidate does not exist, the L(1-X) motion vector of the same candidate is used as the unidirectional prediction motion vector of the geometry partition mode.
[0158] Blending along geometric partition edges
[0159] After predicting each part of the geometric partition using its own motion, blending is applied to the two prediction signals to derive samples around the edges of the geometric partitions. The blending weights for each position of the CU are derived based on the distance between each position and the partition edge.
[0160] The distance from position (x, y) to the edge of the partition is derived as follows:
[0161]
[0162] where i,j are the angle and offset indices of the geometry partition, which depend on the signaled geometry partition index. x,j and ρ y,j The sign of depends on the angle index i.
[0163] The weights of each part of the geometric partition are calculated as follows:
[0164] wIdxL(x,y)=partIdx? 32+d(x,y):32-d(x,y) (11)
[0165]
[0166] w1(x,y)=1-w0(x,y) (13)
[0167] partIdx depends on the angle index i. Fig.15 An example of weight w0 is shown in FIG. 1 , where the angle is indicated for GPM index i. and offset ρI 1520, and point 1530 corresponds to the center of the block.
[0168] Motion field storage in geometric partitioning mode
[0169] Mv1 from the first part of the geometry partition, Mv2 from the second part of the geometry partition, and the combined MV of Mv1 and Mv2 are stored in the motion field of the CU coded in geometry partition mode.
[0170] The type of motion vector stored for each individual position in the motion field is determined as:
[0171] sType=abs(motionIdx)<32?2:
[0172] (motionIdx≤0?(1-partIdx):partIdx) (14)
[0173] where motionIdx is equal to d(4x+2,4y+2), which is recalculated according to equation (2). partIdx depends on the angle index i.
[0174] If sType is equal to 0 or 1, then Mv0 or Mv1 is stored in the corresponding motion field, otherwise if sType is equal to 2, the combined MV from Mv0 and Mv2 is stored. The combined Mv is generated using the following process:
[0175] If Mv1 and Mv2 are from different reference picture lists (one from L0 and the other from L1), then Mv1 and Mv2 are simply combined to form a bi-directional prediction motion vector.
[0176] Otherwise, if Mv1 and Mv2 are from the same list, only the unidirectional predicted motion Mv2 is stored.
[0177] Proposed Method A: Implicit Signaling of BDPCM
[0178] It is proposed to apply DIMD or TIMD to BDPCM to implement implicit signaling of BDPCM, which is one of the codec tools in the VVC standard.
[0179] In BDPCM, there are two modes: horizontal mode and vertical mode. In one embodiment of the present invention, we can use DIMD or TIMD to estimate the mode to be used for the current block and there is no need to signal the BDPCM direction flag.
[0180] Since BDPCM only supports two directions, it is easy to estimate the direction based on DIMD (or TIMD). It may have significant benefits for screen content compression, because the overhead associated with the flag (i.e., for BDPCM direction) can be large. Therefore, saving a flag can have significant benefits in improving compression efficiency.
[0181] Proposed method B: Template-based inter-intra mixed GPM mode
[0182] Fig.16A 1 shows an example of an inter-intra hybrid GPM mode, where scene 1600 shows an exemplary scene in a reference picture and scene 1620 shows a corresponding scene in a current picture. Object 1610 (shown as a triangle) corresponds to an object in front, and object 1612 (shown as a cloud shape) corresponds to a moving object behind object 1610. Block 1614 is a current block in the current picture. Fig. 16B An intra-frame inter-frame hybrid GPM process of a current block 1614 is shown, where a partition 1644 of the current block 1614 corresponds to a stationary portion of the object 1610, and another partition 1642 of the current block 1614 corresponds to an uncovered area from a moving object. A partition line 1618 between the two portions corresponds to an edge of the object 1610. In order to efficiently encode the current block, it is necessary to apply intra-frame codec to the uncovered partition (i.e., partition 1642) and inter-frame codec to partition 1644. The reason for the intra-frame codec portion is that the content of this portion cannot find any corresponding content in the reference picture due to occlusion.
[0183] The Inter-Intra hybrid GPM mode is similar to the VVC GPM mode. However, in the VVC GPM mode, both partitions are coded in Inter mode. In the Inter-Intra hybrid GPM mode, one partition is coded in Intra mode and the other partition is coded in Inter mode.
[0184] Since occlusion is very common in moving objects, the occlusion-resolving codec mode will greatly increase the codec gain, that is, the intra-frame and inter-frame hybrid GPM mode will be of great benefit for such content.
[0185] For the inter-intra hybrid GPM mode, the encoder needs to send side-information of the inter-coding part (e.g., candidate index, MVD, etc.) and side-information of the intra-coding part (e.g., prediction angle, intra-mode, etc.). In order to save syntax overhead, we propose a TIMD / DIMD-based method for the inter-intra hybrid GPM mode according to an embodiment of the present invention.
[0186] In the proposed method, it only sends the region segmentation information (similar to the GPM syntax), and uses the L-template based method to derive the MV of the inter-codec part. For the intra-codec part, the DIMD / TIMD based method can be used to derive the intra-frame prediction angle. An example is shown in Figures 17A-B, where the intra-frame angle can be predicted at the decoder side to determine the intra-frame prediction angle of the intra-frame codec partition. For the intra-frame codec partition 1642, the L-shaped template (1710 and 1712) may not be reliable because they may correspond to the object in front (i.e., object 1610). Therefore, only a part of the top template (as shown in the dashed box 1720) is used to derive the intra-frame prediction angle of the intra-frame codec partition 1642, as shown in Figure 17A-B. Fig.17A As shown. For the inter-frame codec partition 1644, a portion of the top template (i.e., template 1720) corresponds to the uncovered portion of the moving object, which may not provide a reliable reference to derive the MV. Therefore, only the portion of the top template above the inter-frame codec partition 1644 (i.e., Fig. 17B The template 1730 in the figure is used together with the left template 1710 to export the MV.
[0187] In another embodiment, we can reduce the overhead associated with the partition information of the intra-frame inter-frame hybrid GPM mode by using inter-L-template matching (e.g., comparing the current L-neighboring reconstructed sample with the reference L-neighboring sample). As shown in Figures 17A-B, the encoder only needs to send the partition boundary slope (i.e., the angle index in VVC GPM) without sending the partition boundary offset (i.e., the distance index in VVC GPM). In other words, only the information related to the partition boundary slope is signaled in the bitstream on the encoder side or parsed from the bitstream on the decoder side. The decoder can derive the partition boundary offset by inter-L-template matching. For example, in Fig. 17B In the example, some areas on the top-neighbouring part may be blocked (e.g., area 1720), which results in Fig. 17B Therefore, the decoder can observe this and decide the partition offset.
[0188] In another embodiment, we can provide more surrounding reconstructed pixels (or predicted pixels) for the intra-frame codec partition in the intra-frame inter-frame hybrid GPM mode. Fig.18 An example of the proposed method is shown in Fig.18 In the embodiment, inter-codec partition 1644 is first applied to generate a reconstructed inter-codec partition 1844. After the inter-codec partition is reconstructed, intra-codec is then applied to intra-codec partition 1642. When intra-codec is applied to an intra-codec partition, neighboring reconstructed (or predicted) pixels in a neighboring region 1846 within the inter-codec partition 1844 can be used for intra-prediction. Therefore, we have an extended template (i.e., region 1846) for DIMD, which can be used to derive parameters related to intra-codec. In another embodiment, intra-codec can refer to the results of the inter-codec partition (prediction samples or reconstructed samples) used for intra-prediction. In another embodiment, TIMD or DIMD can be applied to the inter-codec region to help the intra-codec region obtain a more accurate angle. In this technique, the transform kernel needs to be appropriately designed according to the new residual distribution.
[0189] Proposed Method C: Apply DIMD to Save Split Flag
[0190] In the proposed method, we can use DIMD or TIMD to estimate the partition direction at the decoder side. The decoder can assume different tree partitioning versions and apply DIMD or TIMD to calculate the associated distortion and guess the decided partitioning pattern based on the distortion.
[0191] Take BT (binary tree) as an example. On the decoder side, it can be assumed that the partition is one of HBT (horizontal BT) or VBT (vertical BT), and based on this assumption, it has two sub-CUs. Next, by applying DIMD or TIMD on the two sub-CUs, the determined DIMD or TIMD angles can further help construct "outer predicted samples" (that is, predicted samples in the L neighboring area outside the current CU). By comparing the "outer predicted samples" with the L neighboring reconstructed samples, we can determine the distortion. By comparing the distortion of the HBT hypothesis and the VBT hypothesis, the decoder can derive the split direction (without receiving the split direction flag from the encoder).
[0192] The same method can be applied to other segmentation methods, such as QT (quadtree), TT (ternary tree), ABT (asymmetric BT), etc.
[0193] In another embodiment, the decoder can guess the partition direction by analyzing the content of L-neighboring reconstructed samples. Fig.19 As shown, there are two object boundaries (1910 and 1920) crossing the top and bottom edges of the current block 1900. Through content analysis (on the decoder side), the decoder can determine that the VBT (as shown by the dotted line 1930) is a better partition. Therefore, the decoder can implicitly determine the BT as a VBT instead of a HBT.
[0194] Proposed Method D: Joint-Deblocking Based DIMD
[0195] In the proposed method, deblocking is used to make DIMD more accurate. Pixels of L-shaped neighboring samples and intra-CU samples may have blocking artifacts. In order to improve the accuracy of DIMD angle prediction, it is proposed to apply deblocking across CU boundaries.
[0196] An example of this process is shown below:
[0197] Step 1: Get the angle using DIMD, then apply intra prediction to the intra-CU samples.
[0198] Step 2: Add the residual to the intra-CU samples to generate some fake reconstructed samples.
[0199] Step 3: Deblocking is performed across CU boundaries (between external L-shaped reconstruction samples and internal CU pseudo-reconstruction samples).
[0200] Step 4: Perform DIMD again to get a more accurate angle
[0201] Proposed Method E: TIMD / DIMD for Neighboring CU Deblocking
[0202] In the proposed method, deblocking may be pre-applied to the L-neighboring area (ie, outside the current CU) to make DIMD or TIMD more accurate.
[0203] Before performing DIMD or TIMD, the L neighboring area (ie, the area outside the current CU) will first be filtered by a deblocking filter.
[0204] The basic idea behind this method is that the top / left neighbor may have multiple CUs, where there may be multiple boundary effects, thus making DIMD / TIMD less accurate. Accordingly, deblocking the neighboring CU will make the surrounding pixels smoother, thus improving the accuracy of DIMD / TIMD.
[0205] Proposed method F: implicitly selecting different edge filters
[0206] In the DIMD process, an edge filter is used to detect the angle field (or angle histogram) in the L-shaped neighborhood (i.e., outside the current CU). In traditional DIMD, the size of the edge filter is fixed. According to an embodiment of the present invention, more edge filter kernels are defined. It can implicitly select between these predefined edge filter kernels by analyzing (at the decoder side) the L-shaped neighborhood samples. In one example, the decoder can calculate the pixel variance of neighboring pixels.
[0207] If the difference is small, then it means the content is smooth. So for this case, choose a larger kernel for the edge filter.
[0208] If the difference is large, then it means the content is not smooth. So for this case, choose a smaller kernel for the edge filter.
[0209] Proposed Method G: Edge Filter Selection
[0210] In the DIMD process, an edge filter is used to detect the angle field (or angle histogram) in the L-shaped neighboring area (ie, outside the current CU). In traditional DIMD, the size of the edge filter is fixed. According to an embodiment of the present invention, more edge filter kernels are defined.
[0211] In another embodiment, the encoder will find the best edge filter kernel and send a signal to the decoder to indicate the best edge filter kernel.
[0212] In another embodiment, some CUs inside the current CTU will receive edge filter selection (from a signal sent by the encoder); for other CUs, it can use some (merge mode-similar) inheritance-based method to inherit the edge filter selection of neighboring CUs.
[0213] Proposed Method H: Multi-Hypothesis
[0214] The MH (Multiple Hypothesis) concept is to first make at least two predictors (from the same or different encoding and decoding methods) and then blend these predictors together to get a more accurate predictor.
[0215] In this new method, we apply MH to DIMD and / or TIMD. In one embodiment, it can apply MH between the angle predictors sent by one or more encoders (e.g., intra-frame prediction angles determined from signals sent by the encoder) and the predictors generated by one or more DIMDs (and / or TIMDs).
[0216] In another embodiment, it may apply MH between one or more TIMD generated predictors and one or more DIMD generated predictors.
[0217] In another embodiment, MH can be applied between one or more angle predictors sent by the encoder (e.g., intra-frame prediction angles determined from signals sent by the encoder) and one or more "DIMD / TIMD refined angle predictors" (defined as: first receiving the intra-angle from a signal sent by the encoder; and refining the angle derived by DIMD or TIMD).
[0218] Proposed Method I: MH for Two Angles
[0219] In the proposed method, we can apply MH to intra prediction associated with DIMD / TIMD.
[0220] In one embodiment, we can apply MH to one predictor from the explicitly sent intra angles and another predictor for angles derived using DIMD.
[0221] In another embodiment, we can apply MH to one predictor from TIMD-derived angles and another predictor using DIMD-derived angles.
[0222] Proposed method J: Generate more reference templates based on neighboring CUs
[0223] In the proposed method, the goal is to perform MH processing on L-neighborhood samples to make the TIMD / DIMD angles more accurate.
[0224] The basic concept is that in addition to the original L-neighborhood sampling, we can apply MH to the L-shaped region and remove some of the noise in the L-neighborhood sampling (by finding another L-shaped sampling from somewhere else). As a result, the angle prediction of TIMD / DIMD will be more accurate.
[0225] In one embodiment, we can search for another L-shaped area sample in the current picture. This embodiment uses the original L neighborhood (i.e., around the current CU) sample as a template and uses the template to search in the current picture to find the best match. After obtaining the best match (designated as L'), we can apply MH to L (the original L neighborhood sample around the current CU) and L'. Finally, the MH results are combined into a new L shape, and DIMD / TIMD is applied to the new L shape.
[0226] The exemplary steps of the above process are as follows:
[0227] Step 1: Use L-Shape (L) to search the current image.
[0228] Step 2: Find the best match of the L shape (in the current image), denote the best match as L'.
[0229] Step 3: Apply MH on these two L-shapes (L and L') to form a new L-shaped area.
[0230] Step 4: Perform TIMD / DIMD based on the new L-shaped area.
[0231] In another embodiment, we can search for L' in the reference picture instead of searching the current picture. In other words, the process is the same as the previous embodiment except that L' is found in the reference picture.
[0232] Any of the aforementioned proposed methods using BDPCM (Block Differential Pulse Codec Modulation) can be implemented in the encoder and / or decoder. For example, any of the proposed methods can be implemented within a frame (e.g. Figure 1B Intra-frame 150) / inter-frame coding and decoding module, motion compensation module (e.g. Figure 1B Alternatively, any of the proposed methods may be implemented as a coupled intra-frame to the encoder (e.g., Figure 1A Intra-frame 110) / inter-frame coding module and / or motion compensation module (eg, Figure 1B MC 112 in), a circuit of a merge candidate derivation module of an encoder to determine a prediction direction of a current block based on a template of the current block or based on vertical prediction and horizontal prediction statistics of an angular field derived from a template of the current block or an intra-mode histogram on a decoder side.
[0233] Fig. 20A flowchart of an exemplary video codec system for deriving a partition mode of BDPCM using TIMD / DIMD according to an embodiment of the present invention is shown. The steps shown in the flowchart can be implemented as executable program code on one or more processors (e.g., one or more CPUs) on the encoder side. The steps shown in the flowchart can also be implemented based on hardware, such as one or more electronic devices or processors arranged to perform the steps in the flowchart. According to the method, pixel data associated with a current block on the encoder side or encoded data associated with a current block to be decoded on the decoder side is received in step 2010. In step 2020, a prediction direction for the current block is determined between vertical prediction and horizontal prediction based on a template of the current block or based on a decoder-side intra-frame mode derived using statistics or histograms of an angular field derived from a template of the current block. In step 2030, the current block is encoded or decoded using BDPCM in the prediction direction.
[0234] Fig.21 A flowchart of an exemplary video codec system for deriving codec parameters related to inter-frame GPM using TIMD / DIMD according to an embodiment of the present invention is shown. According to the method, pixel data associated with a current block on the encoder side or encoded data associated with a current block to be decoded on the decoder side is received in step 2110. In step 2120, the current block is divided into a first region and a second region according to region segmentation. In step 2130, the first region is encoded or decoded based on inter-frame codec, and in step 2140, the second region is encoded or decoded according to intra-frame codec. In step 2150, for the above-mentioned block partitioning, inter-frame codec and intra-frame codec, at least a portion of the region segmentation parameters, a portion of the inter-frame codec parameters or a portion of the intra-frame codec parameters are determined based on a template of the current block or based on a decoder-side intra-frame mode derived using statistics or histograms of corner fields derived from a template of the current block.
[0235] The flowchart shown is intended to illustrate an example of video coding according to the present invention. Without departing from the spirit of the present invention, those skilled in the art may modify each step, rearrange the steps, split the steps, or combine the steps to implement the present invention. In the present disclosure, specific syntax and semantics have been used to illustrate examples to implement embodiments of the present invention. Without departing from the spirit of the present invention, those skilled in the art may implement the present invention by replacing syntax and semantics with equivalent syntax and semantics.
[0236] The above description is provided to enable those skilled in the art to practice the present invention provided in the context of a specific application and its requirements. Various modifications to the described embodiments will be apparent to those with ordinary knowledge in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the specific embodiments shown and described, but to the widest range consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details are illustrated to provide a thorough understanding of the present invention. However, those skilled in the art will appreciate that the present invention may be implemented.
[0237] The embodiments of the present invention as described above can be implemented in various hardware, software codes or a combination of the two. For example, one embodiment of the present invention can be one or more circuits integrated into a video compression chip or a program code integrated into the video compression software to perform the processing described herein. The embodiments of the present invention can also be a program code to be executed on a digital signal processor (DSP) to perform the processing described herein. The present invention can also be related to many functions performed by a computer processor, a digital signal processor, a microprocessor or a field programmable gate array (FPGA). These processors can be configured to perform specific tasks according to the present invention by executing machine-readable software codes or firmware codes that define the specific methods embodied by the present invention. The software code or firmware code can be developed in different programming languages and different formats or styles. The software code can also be compiled for different target platforms. However, the different code formats, styles and languages of the software code and other ways of configuring the code to perform tasks according to the present invention will not depart from the spirit and scope of the present invention.
[0238] The invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples should be considered in all respects as illustrative and not restrictive. Therefore, the scope of the invention is indicated by the appended claims rather than by the foregoing description. All changes that fall within the meaning and scope of the equivalents of the claims should be included within their scope.
Claims
1. A video encoding and decoding method, the method comprising: Receiving pixel data associated with a current block at an encoder side or receiving encoded data to be decoded associated with the current block at a decoder side; determining a prediction direction for the current block between vertical prediction and horizontal prediction based on a template for the current block or based on a decoder-side intra-mode derivation using statistics or histograms of angular fields derived from the template for the current block; and The current block is encoded or decoded using block differential pulse codec modulation in the prediction direction.
2. The method according to claim 1, characterized in that The template includes one or more sampling lines in a neighborhood of the current block.
3. A video encoding and decoding method, the method comprising: Receiving pixel data associated with the current block at the encoder side or receiving encoded data to be decoded associated with the current block at the decoder side; The current block is divided into a first region and a second region according to region division; Encoding or decoding the first region based on inter-frame coding; as well as Encoding or decoding the second area according to intra-frame coding and decoding; as well as In which, at least a part of the region segmentation parameters, a part of the inter-frame coding and decoding parameters or a part of the intra-frame coding and decoding parameters are determined based on the template of the current block or based on the decoder-side intra-frame mode export using the statistical data or histogram of the angle field derived from the template of the current block.
4. The method according to claim 3, characterized in that The template includes one or more sampling lines in a neighborhood of the current block.
5. The method according to claim 3, characterized in that: The template of the current block is used to derive a motion vector for the inter-frame codec.
6. The method according to claim 3, characterized in that The template or the decoder-side intra mode derivation of the current block is used to derive intra prediction angles for the intra codec.
7. The method according to claim 3, characterized in that The template of the current block is used to derive partition boundary offsets associated with the region segmentation.
8. The method according to claim 7, characterized in that Information on the slopes of partition boundaries associated with the region partitioning is signaled in the bitstream at the encoder side.
9. The method according to claim 7, characterized in that: The information of partition boundary slopes related to the region segmentation is parsed from the bitstream at the decoder side.
10. An apparatus for video encoding and decoding, the apparatus comprising one or more electronic devices or processors, arranged to: Receiving pixel data associated with a current block at an encoder side or receiving encoded data to be decoded associated with the current block at a decoder side; determining a prediction direction for the current block between vertical prediction and horizontal prediction based on a template for the current block or based on a decoder-side intra-mode derivation using statistics or histograms of angular fields derived from the template for the current block; and The current block is encoded or decoded using block differential pulse codec modulation in the prediction direction.
11. An apparatus for video encoding and decoding, the apparatus comprising one or more electronic devices or processors, arranged to: Receiving pixel data associated with the current block at the encoder side or receiving encoded data to be decoded associated with the current block at the decoder side; The current block is divided into a first region and a second region according to region division; Encoding or decoding the first region based on inter-frame coding; as well as Encoding or decoding the second area according to intra-frame coding and decoding; as well as In which, at least a part of the region segmentation parameters, a part of the inter-frame coding and decoding parameters or a part of the intra-frame coding and decoding parameters are determined based on the template of the current block or based on the decoder-side intra-frame mode export using the statistical data or histogram of the angle field derived from the template of the current block.
