Template-based intra mode derivation and prediction
By adopting the template-based intra mode export method in video encoding and decoder-side mode export, the problem of low intra mode encoding and decoder-side mode export in the prior art is solved, and more efficient is achieved video codec performance.
Patent Information
- Application Number
- CN202380034095.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-15
- Filing Date
- 2023-04-14
- Publication Date
- 2024-11-29
AI Technical Summary
The existing video encoding and decoding technology has problems with inefficiency in intra-mode encoding and decoding, especially when processing complex video signals, it is difficult to effectively utilize the intra-prediction mode to reduce bit rate and improve encoding and decoding performance.
The template-based intra mode export (TIMD) method is adopted to determine the most likely set of intra prediction modes by encoding and decoding the adjacent blocks of the current block, and on this basis, the intra prediction mode, including the most likely mode ( MPM) and its adjacent modes to generate intra predictions and encode or decode.
Improve the efficiency and performance of video encoding and decoding, reduce the bit rate through fine intra prediction mode export, and use the decoder-side intra mode export (DIMD) and template-based TIMD method under appropriate conditions to improve the frame accuracy and efficiency of intraprediction.
Smart Images

Figure CN119054284A_ABST
Abstract
Description
[0001] Cross-references
[0002] This disclosure is part of a non-provisional application claiming priority to U.S. Provisional Patent Application No. 63 / 331,349, filed on April 15, 2022. The contents of the above application are incorporated herein by reference. Technical Field
[0003] The present invention relates to video coding and decoding. More specifically, the present invention relates to intra-mode coding and decoding of pixel blocks. Background Art
[0004] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims listed later and are not admitted to be prior art included in this section.
[0005] High-Efficiency Video Coding (HEVC) is an international video coding standard developed by the Joint Collaboration on Video Coding (JCT-VC). HEVC is based on a hybrid block-based motion compensation DCT-like transform codec architecture. The basic unit of compression is called a codec unit (CU), which is a 2Nx2N square block of pixels. Each CU can be recursively split into four smaller CUs until a predefined minimum size is reached. Each CU contains one or more prediction units (PUs).
[0006] Versatile video coding (VVC) is the latest international video codec standard developed by the Joint Video Experts Group (JVET) of ITU-T SG16WP3 and ISO / IEC JTC1 / SC29 / WG11. The input video signal is predicted from the reconstructed signal, which is derived from the coded image region. The prediction residual signal is processed by block transform. The transform coefficients are quantized and entropy coded along with other side information in the bitstream. The reconstructed signal is generated from the prediction signal and the reconstructed residual signal after inverse transforming the dequantized transform coefficients. The reconstructed signal is further processed by loop filtering to remove codec artifacts. The decoded pictures are stored in a frame buffer and used to predict future pictures in the input video signal.
[0007] In VVC, a coded picture is divided into non-overlapping square block areas represented by associated codec tree units (CTUs). The leaf nodes of the codec tree correspond to codec units (CUs). A coded picture can be represented by a set of slices, each containing an integer number of CTUs. Each CTU in a slice is processed in raster scan order. Bi-predictive (abbreviated as B) slices can be decoded using intra prediction or inter prediction with up to two motion vectors and reference indices to predict the sample values of each block. Predictive (abbreviated as P) slices are decoded using intra prediction or inter prediction with up to one motion vector and reference index to predict the sample values of each block. Intra (abbreviated as I) slices are decoded using only intra prediction.
[0008] A CTU can be partitioned into one or more non-overlapping codec units (CUs) using a quadtree (QT) with a nested multi-type tree (MTT) structure to accommodate various local motion and texture characteristics. A CU can be further partitioned into smaller CUs using one of five partition types: quadtree partitioning, vertical binary tree partitioning, horizontal binary tree partitioning, vertical center-side ternary tree partitioning, and horizontal center-side ternary tree partitioning.
[0009] Each CU contains one or more prediction units (PUs). The prediction unit, together with the associated CU syntax, is used as a basic unit for signaling prediction sub-information. A specified prediction process is used to predict the values of relevant pixel samples within the PU. Each CU may contain one or more transform units (TUs) for representing prediction residual blocks. A transform unit (TU) consists of a transform block (TB) of luminance samples and two corresponding transform blocks of chrominance samples, and each TB corresponds to a residual block of samples from one color component. An integer transform is applied to the transform block. The level value of the quantization coefficient is entropy encoded and decoded in the bitstream together with other auxiliary information. The terms codec tree block (CTB), codec block (CB), prediction block (PB), and transform block (TB) are defined as 2-D sample arrays specifying one color component associated with a CTU, CU, PU, and TU, respectively. Therefore, a CTU consists of a luminance CTB, two chrominance CTBs, and associated syntax elements. Similar relationships are also valid for CUs, PUs, and TUs.
[0010] For each inter-predicted CU, motion parameters consisting of motion vector, reference picture index and reference picture list usage index and additional information are used for inter-predicted sample generation. Motion parameters can be signaled explicitly or implicitly. When a CU is encoded and decoded using skip mode, the CU is associated with one PU and has no significant residual coefficients, no coded motion vector increments or reference picture indices. A merge mode is specified, whereby the motion parameters of the current CU are obtained from neighboring CUs, including spatial and temporal candidates and the additional scheduling introduced in VVC. Merge mode can be applied to any inter-predicted CU. An alternative to merge mode is the explicit transmission of motion parameters, where the motion vector, the corresponding reference picture index and reference picture list usage flag for each reference picture list and other required information are explicitly signaled for each CU. Summary of the invention
[0011] The following summary is illustrative only and is not intended to be limiting in any way. That is, the following summary is provided to introduce the concepts, highlights, benefits, and advantages of the novel and non-obvious technology described herein. Selected, but not all, implementations are further described below in the detailed description. Therefore, the following summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter.
[0012] Some embodiments of the present disclosure provide a method for performing template-based intra-frame mode derivation (TIMD) using enhanced candidate intra-frame prediction modes. A video codec determines a most probable mode (MPM) set for intra-frame encoding and decoding of a current block based on an intra-frame prediction mode for encoding and decoding neighboring blocks of the current block. The video codec determines a template of the current block among reconstructed pixels adjacent to the current block. The video codec determines a set of candidate intra-frame prediction modes to include (i) the set of MPMs and (ii) one or more intra-frame prediction modes adjacent to the MPMs in the set of MPMs. The video codec derives an intra-frame prediction mode based on the determined template and the set of candidate intra-frame prediction modes. The video encoder generates an intra-frame prediction by using the derived intra-frame prediction mode to encode or decode the current block.
[0013] In some embodiments, the video implicitly determines whether to perform TIMD to derive intra-frame prediction based on whether the statistical variation of the MPM set is greater than a threshold. In some embodiments, if the statistical variation of the MPM is less than the threshold, the video codec will not perform TIMD.
[0014] In some embodiments, when the current block is larger than a threshold size, one or more intra-frame prediction modes adjacent to the MPM are included in the set of candidate intra-frame prediction modes. In some embodiments, when the current block is smaller than a threshold size, one or more intra-frame prediction modes adjacent to the MPM are not included in the set of candidate intra-frame prediction modes. In some embodiments, when the statistical variation of the set of MPMs is less than a threshold, one or more intra-frame prediction modes adjacent to the MPM are included in the set of candidate intra-frame prediction modes. In some embodiments, the video codec performs DIMD as pre-processing to identify candidate intra-frame prediction modes for TIMD.
[0015] In some embodiments, the set of candidate intra prediction modes is constrained to exclude one or more intra prediction modes based on prediction modes of neighboring blocks of the current block. In some embodiments, the set of candidate intra prediction modes is restricted to a predefined range.
[0016] Intra-frame prediction can be for luma or chroma components. In some embodiments, the video codec refines the derived intra-frame prediction mode and generates the intra-frame prediction using the refined intra-frame prediction mode. In some embodiments, the video codec refines the generated intra-frame prediction based on the gradient of the reconstructed samples adjacent to the current block. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated into and constitute a part of the present disclosure. The accompanying drawings illustrate embodiments of the present disclosure and, together with the description, are used to explain the principles of the present disclosure. It should be understood that the drawings are not necessarily drawn to scale, as some components may be shown out of proportion to the dimensions in actual implementation in order to clearly illustrate the concepts of the present disclosure.
[0018] Figure 1 Different directional intra prediction modes are shown.
[0019] Figure 2 It is shown that the intra prediction mode of the current block is implicitly derived using decoder-side intra mode derivation (DIMD).
[0020] Figure 3 It is shown that the intra prediction mode of the current block is implicitly derived using template-based intra mode derivation (TIMD).
[0021] Figure 4 The TIMD process of adaptively selecting from neighboring intra-modes of the most probable mode (MPM) is conceptually shown.
[0022] Figure 5 Conceptually illustrated is the refinement of TIMD intra prediction by gradients of neighboring reconstructed samples.
[0023] Fig. 6A -B shows an example of computing the cost of candidate intra-modes for TIMD using different lines of the template.
[0024] Figure 7 An example video encoder supporting TIMD intra prediction is shown.
[0025] Figure 8 Portions of a video encoder implementing TIMD with an enhanced list of candidate intra-prediction modes are shown.
[0026] Fig. 9 A process for performing TIMD using an enhanced list of candidate intra-prediction modes is conceptually illustrated.
[0027] Fig.10 An example video decoder supporting TIMD intra prediction is shown.
[0028] Fig.11 Part of a video decoder implementing TIMD with an enhanced list of candidate intra-prediction modes is shown.
[0029] Fig.12 A process for performing TIMD using an enhanced list of candidate intra-prediction modes is conceptually illustrated.
[0030] Fig.13 An electronic system for implementing some embodiments of the present disclosure is conceptually illustrated. DETAILED DESCRIPTION
[0031] In the detailed description below, many specific details are set forth by way of example to provide a thorough understanding of the relevant teachings. Any variation, derivation and / or extension based on the teachings described herein are within the scope of protection of the present disclosure. In some cases, well-known methods, processes, components and / or circuits associated with one or more example implementations disclosed herein may be described at a relatively high level without details to avoid unnecessarily obscuring the various aspects of the teachings of the present disclosure.
[0032] I. Intra-frame prediction
[0033] The intra prediction method uses a reference tier adjacent to the current prediction unit (PU) and an intra prediction mode to generate a prediction sub-unit for the current PU. The intra prediction direction can be selected from a mode set containing multiple prediction directions. For each PU decoded by intra prediction, an index will be used and encoded to select one of the intra prediction modes. The corresponding prediction will be generated, and the residual can then be derived and transformed.
[0034] Figure 1Intra-frame prediction modes in different directions are shown. These intra-frame prediction modes are called directional modes, excluding DC mode or planar mode. As shown in the figure, there are 33 directional modes (V: vertical direction; H: horizontal direction), so H, H+1~H+8, H-1~H-7, V, V+1~V+8, V-1~V-8 are used. In general, the directional mode can be expressed as H+k or V+k mode, where k=±1, ±2,..., ±8. Each of such intra-frame prediction modes can also be called an intra-frame prediction angle. In order to capture arbitrary edge directions presented in natural videos, the number of directional intra-frame modes can be extended from the 33 directional modes used in HEVC to 65 directional modes, so that k ranges from ±1 to ±16. These denser directional intra-frame prediction modes are applicable to all block sizes and luminance and chrominance intra-frame predictions. By including DC and planar modes, the number of intra-frame prediction modes is 35 (or 67).
[0035] Among the 35 (or 67) intra prediction modes, some modes (for example, 3 or 5 modes) are identified as a set of most probable modes (MPMs) for the intra prediction mode in the current prediction block. The encoder can reduce the bit rate by signaling the index of selecting one of the MPMs instead of selecting the index of one of the 35 (or 67) intra prediction modes. For example, the intra prediction mode used in the left prediction block and the intra prediction mode used in the above prediction block are used as MPMs. When the intra prediction modes of two adjacent blocks use the same intra prediction mode, the intra prediction mode can be used as an MPM. When only one of the two adjacent blocks is available and is encoded and decoded in a directional mode, the two adjacent directions adjacent to the directional mode can be used as MPMs. DC mode and planar mode are also considered as MPMs to fill the available points in the MPM set, especially if the above or top adjacent block is not available or does not use intra prediction encoding and decoding, or if the intra prediction mode in the adjacent block is not a directional mode. If the intra prediction mode of the current prediction block is one of the modes in the MPM set, 1 or 2 bits are used to signal which one it is. Otherwise, the intra prediction mode of the current block is different from any entry in the MPM set, and the current block will be encoded as non-MPM mode. There are 32 such non-MPM modes in total, and a (5-bit) fixed-length codec method is used to signal the mode.
[0036] The MPM list is constructed based on the intra-frame modes of the left and upper neighboring blocks. Assuming that the mode of the left neighboring block is represented as Left and the mode of the upper neighboring block is represented as Above, the unified MPM list can be constructed as follows:
[0037] – When a neighboring block is not available, its intra mode is set to Planar by default.
[0038] – If both “Left” and “Above” modes are non-angle modes:
[0039] MPM list → {Planar,DC,V,H,V-4,V+4}
[0040] – If one of the "Left" and "Above" modes is an angular mode and the other is a non-angular mode
[0041] model:
[0042] ■ Set the Max mode to the larger of Left and Above
[0043] ■MPM list → {Planar,Max,Max-1,Max+1,Max––2,Max+2}
[0044] – If both “Left” and “Above” modes are angle modes and the two modes are different:
[0045] ■ Set the Max mode to the larger of Left and Above
[0046] ■ Set the Min mode to the smaller of Left and Above
[0047] ■If Max–Min is equal to 1:
[0048] –MPM list → {Planar,Left,Above,Min–1,Max+1,Min–
[0049] 2}
[0050] ■ Otherwise, if Max–Min is greater than or equal to 62:
[0051] –MPM list → {Planar,Left,Above,Min+1,Max–1,Min+2}
[0052] ■ Otherwise, if Max–Min is equal to 2:
[0053] –MPM list → {Planar,Left,Above,Min+1,Min–1,Max+1}
[0054] ■Otherwise:
[0055] –MPM list → {Planar,Left,Above,Min–1,Min+1,Max–1}
[0056] – If both “Left” and “Above” modes are angle modes and both modes are the same
[0057] ■MPM list→{Planar,Left,Left-1,Left+1,Left–2,Left+2}
[0058] II. Decoder-Side Intra Mode Derivation (DIMD)
[0059] Decoder-side intra mode derivation (DIMD) is a technique that derives two intra prediction modes / angles / directions from the reconstructed neighboring samples (template) of a block, and these two predictors are combined with a plane mode predictor with weights derived from gradients. DIMD mode is used as an alternative prediction mode and is always checked in high-complexity RDO mode. In order to implicitly derive the intra prediction mode of a block, texture gradient analysis is performed on both the encoder and decoder sides. The process starts with an empty gradient histogram (Histogram of Gradient, abbreviated as HoG) with 65 entries, corresponding to the 65 angle / direction intra prediction modes. The magnitudes of these entries are determined during texture gradient analysis.
[0060] A video codec that implements DIMD performs the following steps: In the first step, the video codec picks a template of T = 3 columns and rows from the left and above the current block, respectively. This region is used as a reference for gradient-based intra prediction mode derivation. In the second step, horizontal and vertical Sobel filters are applied to all 3×3 window locations, centered on the pixels in the template midline. At each window location, the Sobel filter computes the pure horizontal and vertical intensities as G x and G y . The texture angle of the window is then calculated as follows:
[0061] angle=arctan(G x / G y ),
[0062] It can be converted into one of 65 angular intra prediction modes. Once the intra prediction mode index of the current window is derived as idx, the magnitude of its entry in HoG[idx] is updated by addition:
[0063] ampl=|G x |+|G y |
[0064] Figure 2The use of decoder-side intra mode derivation (DIMD) to implicitly derive the intra prediction mode of the current block is shown. The figure shows an example gradient histogram (HoG) 210, which is calculated after applying the above operations to all pixel positions in the template 215 around the current block 200. Once the HoG is calculated, the indices of the two highest histogram bars (M1 and M2) are selected as the two implicitly derived intra prediction modes (IPMs) of the block. The predictions of the two IPMs are further combined with the plane mode as the prediction of the DIMD mode. The prediction fusion application is a weighted average of the above three predictors (M1 prediction, M2 prediction, and plane mode prediction). To this end, the weight of the plane can be set to 21 / 64 (~1 / 3). Then, the remaining weight 43 / 64 (~2 / 3) is distributed between the two HoGIPMs, proportional to the amplitude of their HoG bars. The prediction fusion or combined prediction of DIMD can be:
[0065] Pred DIMD =(43*(w1*pred M1 +w2*pred M2 )+21*pred planar )>>6
[0066] w1=amp M1 / (amp M1 +amp M2 )
[0067] w2=amp M2 / (amp M1 +amp M2 )
[0068] Additionally, two implicitly derived intra prediction modes are added to the most probable mode (MPM) list, so the DIMD process is performed before building the MPM list. The primary derived intramode of a DIMD block is stored with the block and used for MPM list construction for neighboring blocks.
[0069] III. Template-based Intra Mode Derivation (TIMD)
[0070] Template-based intra mode derivation (TIMD) is a coding approach in which the intra prediction mode of a CU is implicitly derived by using neighboring templates at the encoder and decoder, instead of the encoder signaling the exact intra prediction mode to the decoder.
[0071] Figure 3 The intra prediction mode of the current block 300 is implicitly derived using template-based intra mode derivation (TIMD). As shown, the neighborhood pixels of the current block 300 are used as template 310. For each candidate mode, the prediction samples of the template 310 are generated using the reference samples in the reference area 320 located above and to the left of the template 310. The cost is calculated based on the difference between the prediction samples and the reconstructed samples of the template (e.g., SATD). The intra prediction mode with the lowest cost (like the DIMD mode) is selected for intra prediction of the CU. The candidate mode can include 67 intra prediction modes (as in VVC) or be extended to 131 intra prediction modes. MPM can be used to indicate the directional information of the CU. Therefore, in order to reduce the intra mode search space and utilize the characteristics of the CU, the intra prediction mode is implicitly derived from the MPM list.
[0072] In some embodiments, for each intra prediction mode in the MPM list, the SATD between the predicted sample and the reconstructed sample of the template is calculated. The first two intra prediction modes with the smallest SATD are selected as TIMD modes. The two TIMD modes are fused with weights after applying the PDPC process, and this weighted intra prediction is used to encode and decode the current CU. Position dependent intra prediction combination (PDPC) is included in the derivation of TIMD mode.
[0073] The costs of the two selected modes (mode 1 and mode 2) are compared with the threshold value. In the test, a cost factor of 2 is applied as follows:
[0074] costMode2<2*costMode1
[0075] If this condition is true, prediction fusion is applied, otherwise only mode 1 is used. The weight of the mode is calculated based on its SATD cost as follows:
[0076] weight1=costMode2 / (costMode1+costMode2)
[0077] weight2=1-weight1
[0078] IV. Improved Intra Mode Export
[0079] Some embodiments provide methods for improved cost calculation and signaling TIMD syntax. In some embodiments, the SATD cost between the luma prediction and the reconstruction template (the template constructed from the reconstructed pixels) and the SATD cost between the chroma prediction and the reconstruction template are added together as the final cost for the intra mode used to derive TIMD.
[0080] A. Additional candidate intra prediction modes for TIMD
[0081] As mentioned, the TIMD process can check a limited set of candidate intra-frame prediction modes based on a template of the current block (e.g., the MPM of the current block). (The template can be a region of reconstructed samples adjacent to the current block.) In some embodiments, in addition to the MPM mode itself, the TIMD process can also adaptively select from neighboring intra-frame modes of the MPM mode (e.g., modes in MPM+k, or modes in MPM–k, where k>0).
[0082] Figure 4 The TIMD process of adaptively selecting from adjacent intra-frame modes of an MPM mode is conceptually illustrated. As shown, the video codec initially generates / identifies a set of MPM modes 410 for the current block 400, whose adjacent reconstructed pixels are considered as a template 420 of the current block. The MPM mode set 410 includes a planar mode and several intra-frame prediction angle modes 10, 50, 9, 11 and 49. The TIMD process 405 considers the MPM mode set 410 as a candidate intra-frame prediction mode. In addition to the MPM modes in the set 410, the TIMD process 405 can also check or consider adjacent modes of the MPM mode as candidate intra-frame prediction modes. For example, when the intra-frame prediction angle mode 50 is considered as a TIMD candidate, the TIMD process also considers intra-frame prediction angle modes 48, 49, 51 and 52 (assuming k=2) as TIMD candidates because these additional modes are adjacent / neighboring modes of the MPM intra-frame prediction mode 50.
[0083] The TIMD process 405 calculates the cost of each candidate intra prediction mode (including neighboring / adjacent modes of the MPM mode) based on the template from the template area 420 and a set of reference samples from the reference area 430. Based on the calculated costs, the TIMD process 405 can derive one (or more) final intra prediction modes and generate a TIMD predictor 450 (which is an intra prediction) for the current block 400 based on the identified final intra prediction mode.
[0084] In some embodiments, the number k of neighboring intra modes available for selection may depend on the current CU size. For example, if TIMD is applied to a small-sized CU (e.g., CU width + height or CU area is less than a threshold), only the modes in the MPM list are considered as candidate intra prediction modes for TIMD. On the other hand, if TIMD is applied to a large-sized CU (e.g., CU width + height or CU area is greater than a threshold), the neighboring intra modes of the modes in the MPM list are also considered as candidate intra prediction modes for TIMD.
[0085] B. Implicit TIMD
[0086] In some embodiments, TIMD is implicitly applied to the current block. In some embodiments, if the candidate intra-frame modes (e.g., MPM) examined or considered by the TIMD process are very similar (e.g., statistical variation of MPM < threshold), TIMD is implicitly disabled. On the other hand, if the candidate intra-frame modes (e.g., MPM) examined or considered by the TIMD process are not similar (e.g., statistical variation of MPM > threshold), TIMD is implicitly applied to identify the intra-frame prediction mode.
[0087] In some embodiments, the TIMD prediction or process is applied to the chroma CU to implicitly derive the intra-frame angle mode of the chroma CU. In some embodiments, if the candidate intra-frame chroma modes considered by the TIMD process include DC, vertical, horizontal, plane, and DM, the TIMD process is applied to derive the final intra-frame angle mode to determine the predictor of the current block. In some embodiments, a flag is used to indicate whether TIMD is used to derive the final intra-frame angle mode. In some embodiments, if the flag is true, the TIMD process is used to derive the final intra-frame angle mode, and the DC, vertical, horizontal, plane, and DM modes are excluded from the candidate intra-frame modes of the TIMD process.
[0088] In some embodiments, after deriving the intra-frame angle mode through TIMD, the video codec can perform a fine search around the derived intra-frame angle mode to refine the derived intra-frame angle mode. For example, the TIMD process can derive an intra-frame angle mode k as one of intra-directional modes 0 to 67, and the encoder can search for additional intra-frame modes (i.e., refine) between (k-1) and (k+1). The encoder can then signal a delta value to indicate the final intra-frame prediction angle mode.
[0089] C. Constraints on candidate intra modes for TIMD
[0090] In some embodiments, when performing TIMD or DIMD, the video codec may exclude or reduce gradients of neighboring inter-coded positions when computing gradient histograms, or may increase the cost between prediction and reconstruction of inter-coded templates.
[0091] The candidate intra-angle modes for the TIMD process may be further constrained by the prediction modes of the neighboring blocks. For example, in some embodiments, if the upper or top neighboring CU is inter-coded in skip mode, the intra-angle modes that are larger than the diagonal intra-angle mode (e.g., mode 66 among 131 intra-angle modes, or mode 34 among 67 intra-angle modes, or mode 18 among 34 intra-angle modes) are excluded from the candidate intra-angle modes of TIMD. For another example, in some embodiments, if the left neighboring CU is inter-coded in skip mode, the intra-angle modes that are smaller than the diagonal intra-angle mode (e.g., mode 66 among 131 intra-angle modes, mode 34 among 67 intra-angle modes, or mode 18 among 34 intra-angle modes) are excluded from the candidate intra-angle modes of TIMD.
[0092] In some embodiments, the candidate intra-angle modes of TIMD can be further constrained or reduced to a predefined range. Specifically, if the candidate intra-angle mode can be extracted from 67 intra-angle modes (i.e., 0, 1, 2, 3, ..., 67), the video codec can constrain the valid intra-angle mode candidates of TIMD to a subset of these 67 modes (i.e., candidate <67 modes). In other words, the constrained candidates may be {0, 1, 2, 4, 6, 8, ..., 66}, {0, 1, 3, 5, 7, 9, ..., 65}, {0, 1, 2, 3, 4, 5, ..., 34}, {34, 35, 36, 37, 38, ..., 67}, etc. The constraint can be signaled in the PPS, SPS, picture header, slice header, CTU level syntax, or implicitly derived depending on other syntax, or always applied. As another example, if the constraint is signaled, a CU encoded and decoded using the DIMD / TIMD process may check fewer candidate intra angle modes to derive a final intra angle mode for encoding and decoding the current block.
[0093] D. Refining TIMD intra prediction
[0094] In some embodiments, after generating the intra prediction based on the intra angle mode by TIMD, the intra prediction derived by TIMD is further refined by gradients of neighboring reconstructed samples.
[0095] Figure 5The TIMD intra prediction is conceptually illustrated to be refined by the gradient of the neighboring reconstructed samples. As shown in the figure, for the current block 500, if the current intra prediction comes from the left neighboring reconstructed sample, the current prediction at (x, y) is obtained by refining the top left corner sample (e.g., R -1,-1 ) and the current left neighboring sample (e.g., R -1,y ) to refine the gradient between them. Then, the refined prediction at (x,y) is
[0096] (w1×(R x,-1 +(R -1,-1 -R -1,y ))+w2×pred(x,y)) / (w1+w2).
[0097] For another example, if the current intra prediction comes from the upper neighboring reconstructed sample, the upper left corner sample (eg, R -1,-1 ) and the current upper neighboring sample (e.g., R x,-1 ) further refines the current prediction at (x, y). Then, the refined prediction at (x, y) is
[0098] (w1×(R -1,y +(R -1,-1 -R x,-1 ))+w2×pred(x,y)) / (w1+w2).
[0099] E. Using DIMD as a preprocessing for TIMD
[0100] In some embodiments, the DIMD and TIMD prediction processes are combined to derive a final intra-frame angle mode for encoding and decoding the current block. In some embodiments, the neighboring window positions of the current block are divided into multiple groups, and the DIMD process is performed to select one group. The TIMD process is then performed to select the final intra-frame angle mode from the group (neighboring window positions) selected by DIMD. More generally, in some embodiments, the DIMD process is performed to derive some possible intra-frame prediction modes, and then the TIMD process is performed to select the final intra-frame angle mode from the intra-frame prediction mode selected by DIMD or the neighboring mode of the intra-frame prediction mode selected by DIMD. In other words, DIMD is used to pre-process the neighboring reconstructed samples of the current block to speed up the TIMD process.
[0101] F. Template size reduction for TIMD
[0102] In order to reduce the template buffer size required in TIMD, the neighboring line (i.e., the sample line around the current block used to generate the template) can be related to the number of samples in the current block (e.g., block size, block shape), or determined based on the number of samples in the current block (e.g., block size, block shape). In some embodiments, the number of neighboring lines is determined based on the current block size or the side length of the current block. For example, if the current block size is less than or equal to k, the number of neighboring lines is n. For another example, if the current block is a rectangular block (i.e., the block width is not equal to the block height), the neighboring line of the long side of the current block is s, the neighboring line of the short side is t, and s is not equal to t. In some embodiments, the number of neighboring lines depends on the pixel smoothness of the top / left neighboring reconstructed pixels. For example, if the variance of the neighboring reconstructed pixels on the first side of the current block is higher than the variance of the neighboring reconstructed pixels on the second side of the current block, the number of neighboring lines of the first template on the first side can be greater than the number of neighboring lines of the second template on the second side.
[0103] In some embodiments, to reduce the template size of the TIMD, the reference samples of the template are not used (e.g., Figure 3 To calculate the cost of candidate intra modes for TIMD, the video codec uses the prediction angle to calculate the distortion / difference between the inner neighboring lines of the template and the outer neighboring lines of the template.
[0104] Fig. 6A -B shows an example of calculating the cost of candidate intra modes of TIMD using different rows of the template. The cost of each candidate intra mode is calculated by comparing the internal line samples with the interpolated samples of the external lines, where the interpolated samples are obtained from the candidate intra angle mode.
[0105] like Fig. 6A As shown, if the candidate intra mode A is greater than or equal to 34 and the upper template has 3 neighboring lines, the cost is the SAD between the interpolated samples in the 1st line (samples interpolated according to the candidate intra mode A) and the corresponding samples in the 0th line, and the sum of the SADs between the interpolated samples in the 2nd line and the corresponding samples in the 1st line.
[0106] like Figure 6B As shown, if the candidate intra mode B is less than 34 and the left panel has 3 neighboring lines, the cost is the SAD of the interpolated samples in the 1st line (the samples interpolated according to the candidate intra mode B) and the corresponding samples in the 0th line, and the sum of the SAD between the interpolated samples in the 2nd line and the corresponding samples in the 1st line.
[0107] Any of the aforementioned proposed methods may be implemented in an encoder and / or a decoder. For example, any of the proposed methods may be implemented in an inter-frame / intra-frame / prediction module of an encoder and / or an inter-frame / intra-frame / prediction module of a decoder. Alternatively, any of the proposed methods may be implemented as a circuit coupled to an inter-frame / intra-frame / prediction module of an encoder and / or an inter-frame / intra-frame / prediction module of a decoder to provide information required by the inter-frame / intra-frame / prediction module.
[0108] V. Example Video Encoder
[0109] Figure 7 An example video encoder 700 supporting TIMD intra prediction is shown. As shown, the video encoder 700 receives an input video signal from a video source 705 and encodes the signal into a bitstream 795. The video encoder 700 has several components or modules for encoding the signal from the video source 705, including at least some components selected from the following: a transform module 710, a quantization module 711, an inverse quantization module 714, an inverse transform module 715, an intra-picture estimation module 720, an intra-frame prediction module 725, a motion compensation module 730, a motion estimation module 735, a loop filter 745, a reconstructed picture buffer 750, an MV buffer 765, an MV prediction module 775, and an entropy encoder 790. The motion compensation module 730 and the motion estimation module 735 are part of the inter-frame prediction module 740.
[0110] In some embodiments, modules 710-790 are software instruction modules executed by one or more processing units (e.g., processors) of a computing device or electronic device. In some embodiments, modules 710-790 are hardware circuit modules implemented by one or more integrated circuits (ICs) of an electronic device. Although modules 710-790 are shown as separate modules, some modules can be combined into a single module.
[0111] The video source 705 provides a raw video signal that presents the pixel data of each video frame without compression. The subtractor 708 calculates the difference between the raw video pixel data of the video source 705 and the predicted pixel data 713 from the motion compensation module 730 or the intra-frame prediction module 725 as a prediction residual 709. The transform module 710 converts the difference (or residual pixel data or residual signal 708) into transform coefficients (e.g., by performing a discrete cosine transform, or DCT). The quantization module 711 quantizes the transform coefficients into quantized data (or quantized coefficients) 712, which are encoded into a bitstream 795 by the entropy encoder 790.
[0112] The inverse quantization module 714 inversely quantizes the quantized data (or quantized coefficients) 712 to obtain transform coefficients, and the inverse transform module 715 inversely transforms the transform coefficients to generate a reconstructed residual 719. The reconstructed residual 719 is added to the predicted pixel data 713 to generate reconstructed pixel data 717. In some embodiments, the reconstructed pixel data 717 is temporarily stored in a line buffer (not shown) for intra-frame prediction and spatial MV prediction. The reconstructed pixels are filtered by the loop filter 745 and stored in the reconstructed picture buffer 750. In some embodiments, the reconstructed picture buffer 750 is a memory external to the video encoder 700. In some embodiments, the reconstructed picture buffer 750 is an internal memory of the video encoder 700.
[0113] The intra-picture estimation module 720 performs intra-prediction based on the reconstructed pixel data 717 to generate intra-prediction data. The intra-prediction data is provided to the entropy encoder 790 to be encoded into a bitstream 795. The intra-prediction data is also used by the intra-prediction module 725 to generate predicted pixel data 713.
[0114] The motion estimation module 735 performs inter-frame prediction by generating MVs to refer to pixel data of a previously decoded frame stored in the reconstructed picture buffer 750. These MVs are provided to the motion compensation module 730 to generate predicted pixel data.
[0115] Instead of encoding the complete actual MV in the bitstream, the video encoder 700 uses MV prediction to generate a predicted MV, and the difference between the MV used for motion compensation and the predicted MV is encoded as residual motion data and stored in the bitstream 795.
[0116] The MV prediction module 775 generates a predicted MV based on a reference MV generated for encoding a previous video frame, i.e., a motion compensated MV for performing motion compensation. The MV prediction module 775 retrieves a reference MV from a previous video frame from the MV buffer 765. The video encoder 700 stores the MV generated for the current video frame in the MV buffer 765 as a reference MV for generating the predicted MV.
[0117] The MV prediction module 775 uses the reference MV to create a predicted MV. The predicted MV can be calculated by spatial MV prediction or temporal MV prediction. The entropy encoder 790 encodes the difference (residual motion data) between the predicted MV and the motion compensation MV (MC MV) of the current frame into a bitstream 795.
[0118] The entropy encoder 790 encodes various parameters and data into a bitstream 795 by using an entropy coding technique such as context adaptive binary arithmetic coding and decoding (CABAC) or Huffman coding. The entropy encoder 790 encodes various header elements, flags along with quantized transform coefficients 712 and residual motion data as syntax elements into the bitstream 795. The bitstream 795 is in turn stored in a storage device or transmitted to a decoder through a communication medium such as a network.
[0119] The loop filter 745 performs filtering or smoothing operations on the reconstructed pixel data 717 to reduce encoding and decoding artifacts, particularly at the boundaries of pixel blocks. In some embodiments, the filtering operations performed include sample adaptive offset (SAO). In some embodiments, the filtering operations include adaptive loop filtering (ALF).
[0120] Figure 8 The portion of the video encoder 700 that implements TIMD using an enhanced list of candidate intra-prediction modes is shown. Specifically, the figure shows the components of the intra-prediction module 725 of the video encoder 700. As shown, the intra-prediction module 725 includes a template matching module 810, a TIMD candidate identification module 820, an intra-mode history buffer 830, and an intra-prediction generation module 840. The intra-prediction module 725 can use these modules to perform TIMD intra-prediction on both luma and chroma components.
[0121] As shown, the template matching module 810 constructs a template (e.g., template region 420) of the current block based on the template samples provided by the coded picture buffer 750. The template is matched with reference samples (e.g., reference region 430) provided by the coded picture buffer 750 in different candidate intra prediction modes 825. (In some embodiments, the internal template samples are matched with the external template samples in different candidate intra prediction modes.) The template matching module 810 calculates the matching cost for each candidate intra prediction mode based on the template and the reference samples, and then identifies one or more final intra prediction modes 815 for the TIMD process.
[0122] The candidate intra prediction modes 825 are specified by the TIMD candidate identification module 820, which can identify a set of MPMs based on the intra prediction modes used by neighboring blocks (which can be stored in the intra mode history buffer 830). In some embodiments, the TIMD candidate identification module 820 can perform a DIMD process to identify at least some of the candidate intra prediction modes 825 (e.g., based on DIMD histograms of different intra modes). The TIMD candidate identification module 820 can also add additional intra prediction modes (e.g., adjacent modes of the MPM mode) to the candidate intra prediction mode 825. The TIMD candidate identification module 820 can also constrain the candidate intra prediction modes, such as by limiting the candidate intra prediction modes 825 to certain direction / angle ranges or excluding certain intra prediction modes. In some embodiments, the TIMD candidate identification module 820 can also check the variance of the candidate intra prediction modes 825 to determine whether to enable the template matching module 810.
[0123] The final intra prediction mode 815 of the TIMD process is used by the intra prediction generation module 840 to generate a TIMD intra prediction 845 (based on the contents of the reconstructed picture buffer 750) to be used as the predicted pixel data 713 for the current block. The intra prediction generation module 840 may refine the final intra prediction mode 815 before using it to generate the predicted pixel data 713. The intra prediction generation module 840 may also refine the generated TIMD intra prediction 845 based on gradients of neighboring blocks. The intra prediction generation module 840 may mix / fuse multiple predictions / predictors based on multiple final intra prediction modes into one TIMD intra prediction 845 for the current block.
[0124] Fig. 9 A process 900 for performing TIMD with an enhanced list of candidate intra-prediction modes is conceptually illustrated. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing the encoder 700 perform the process 900 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing the encoder 700 performs the process 900.
[0125] The encoder receives (at step 910) data for a current block of pixels in a current picture to be encoded as a video.
[0126] The encoder (at step 920) determines a most probable mode (MPM) set for intra-coding the current block based on the intra-prediction mode used to code the neighboring blocks (e.g., above and to the left) of the current block. The encoder (at step 930) determines a template for the current block among the reconstructed pixels adjacent to the current block.
[0127] In some embodiments, the video implicitly determines whether to perform TIMD to derive intra-frame prediction based on whether the statistical variation of the MPM set is greater than a threshold. In some embodiments, if the statistical variation of the MPM is less than the threshold, the encoder will not perform TIMD (thus ending process 900).
[0128] The encoder determines (at step 940) a set of candidate intra-prediction modes to include (i) the set of MPMs and (ii) one or more intra-prediction modes adjacent to an MPM in the set of MPMs. In some embodiments, when the current block is larger than a threshold size, one or more intra-prediction modes adjacent to the MPM are included in the set of candidate intra-prediction modes. In some embodiments, when the current block is smaller than a threshold size, one or more intra-prediction modes adjacent to the MPM are not included in the set of candidate intra-prediction modes. In some embodiments, when the statistical variation of the set of MPMs is less than a threshold, one of the multiple intra-prediction modes adjacent to the MPM is included in the set of candidate intra-prediction modes. In some embodiments, the encoder derives a gradient histogram for different intra-prediction angles based on gradient magnitudes at different pixel positions along reconstructed pixels adjacent to the current block, and the set of candidate intra-prediction modes includes a mode identified by using the derived histogram. (In other words, the video encoder performs DIMD as preprocessing to identify candidate intra-prediction modes for TIMD.)
[0129] In some embodiments, the set of candidate intra prediction modes is constrained to exclude one or more intra prediction modes based on prediction modes of neighboring blocks of the current block. In some embodiments, the set of candidate intra prediction modes is restricted to a predefined range.
[0130] The encoder derives (at step 950) an intra prediction mode based on the determined template and a set of candidate intra prediction modes that specify an intra prediction direction.
[0131] The encoder (at step 960) encodes the current block using the derived intra prediction mode to generate an intra prediction. The intra prediction can be for luma or chroma components. In some embodiments, the video encoder refines the derived intra prediction mode and generates the intra prediction using the refined intra prediction mode. In some embodiments, the video encoder refines the generated intra prediction based on the gradient of the reconstructed samples adjacent to the current block.
[0132] VI. Example Video Decoder
[0133] In some embodiments, an encoder may signal (or generate) one or more syntax elements in a bitstream such that a decoder may parse the one or more syntax elements from the bitstream.
[0134] Fig.10An example video decoder 1000 supporting TIMD intra prediction is illustrated. As shown, the video decoder 1000 is an image decoding or video decoding circuit that receives a bitstream 1095 and decodes the content of the bitstream into pixel data of a video frame for display. The video decoder 1000 has several components or modules for decoding the bitstream 1095, including some components selected from an inverse quantization module 1011, an inverse transform module 1010, an intra prediction module 1025, a motion compensation module 1030, a loop filter 1045, a decoded picture buffer 1050, an MV buffer 1065, an MV prediction module 1075, and a parser 1090. The motion compensation module 1030 is part of the inter prediction module 1040.
[0135] In some embodiments, modules 1010-1090 are software instruction modules executed by one or more processing units (e.g., processors) of a computing device. In some embodiments, modules 1010-1090 are hardware circuit modules implemented by one or more ICs of an electronic device. Although modules 1010-1090 are illustrated as separate modules, some modules may be combined into a single module.
[0136] The parser 1090 (or entropy decoder) receives the bitstream 1095 and performs initial parsing according to the syntax defined by the video codec or image codec standard. The parsed syntax elements include various header elements, flags, and quantized data (or quantized coefficients) 1012. The parser 1090 parses out the various syntax elements by using entropy coding techniques such as context adaptive binary algorithm (CABAC) or Huffman coding.
[0137] The inverse quantization module 1011 dequantizes the quantized data (or quantized coefficients) 1012 to obtain transform coefficients, and the inverse transform module 1010 performs inverse transform on the transform coefficients 1016 to generate a reconstructed residual signal 1019. The reconstructed residual signal 1019 is added to the predicted pixel data 1013 from the intra prediction module 1025 or the motion compensation module 1030 to generate decoded pixel data 1017. The decoded pixel data is filtered by the loop filter 1045 and stored in the decoded picture buffer 1050. In some embodiments, the decoded picture buffer 1050 is a storage component external to the video decoder 1000. In some embodiments, the decoded picture buffer 1050 is a component internal to the video decoder 1000.
[0138] The intra prediction module 1025 receives intra prediction data from the bitstream 1095 and generates predicted pixel data 1013 from the decoded pixel data 1017 stored in the decoded picture buffer 1050. In some embodiments, the decoded pixel data 1017 is also stored in a line buffer (not shown) for intra picture prediction and spatial MV prediction.
[0139] In some embodiments, the contents of decoded picture buffer 1050 are used for display. Display device 1055 either fetches the contents of decoded picture buffer 1050 for direct display or fetches the contents of decoded picture buffer to a display buffer. In some embodiments, the display device receives pixel values from decoded picture buffer 1050 via pixel transfer.
[0140] The motion compensation module 1030 generates predicted pixel data 1013 from the decoded pixel data 1017 stored in the decoded picture buffer 1050 according to the motion compensation MV (MC MV). These motion compensation MVs are decoded by adding the residual motion data received from the bitstream 1095 and the predicted MV received from the MV prediction module 1075.
[0141] The MV prediction module 1075 generates a predicted MV based on a reference MV generated for decoding a previous video frame, for example, a motion compensated MV for performing motion compensation. The MV prediction module 1075 retrieves the reference MV of the previous video frame from the MV buffer 1065. The video decoder 1000 stores the motion compensated MV generated for decoding the current video frame in the MV buffer 1065 as a reference MV for generating the predicted MV.
[0142] The loop filter 1045 performs filtering or smoothing operations on the decoded pixel data 1017 to reduce encoding and decoding artifacts, particularly at the boundaries of pixel blocks. In some embodiments, the filtering operations performed include sample adaptive offset (SAO). In some embodiments, the filtering operations include adaptive loop filtering (ALF).
[0143] Fig.11 The portion of the video decoder 1000 that implements TIMD using an enhanced list of candidate intra-prediction modes is shown. Specifically, the figure shows the components of the intra-prediction module 1025 of the video decoder 1000. As shown, the intra-prediction module 1025 includes a template matching module 1110, a TIMD candidate identification module 1120, an intra-mode history buffer 1130, and an intra-prediction generation module 1140. The intra-prediction module 1025 can use these modules to perform TIMD intra-prediction on both luma and chroma components.
[0144] As shown, the template matching module 1110 constructs a template (e.g., template region 420) of the current block based on the template samples provided by the decoded picture buffer 1050. The template is matched with reference samples (e.g., reference region 430) provided by the decoded picture buffer 1050 in different candidate intra prediction modes 1125. (In some embodiments, internal template samples are matched with external template samples in different candidate intra prediction modes.) The template matching module 1110 calculates the matching cost for each candidate intra prediction mode based on the template and the reference samples, and then identifies one or more final intra prediction modes 1115 for the TIMD process.
[0145] The candidate intra prediction modes 1125 are specified by the TIMD candidate identification module 1120, which can identify a set of MPMs based on the intra prediction modes used by neighboring blocks (which can be stored in the intra mode history buffer 1130). In some embodiments, the TIMD candidate identification module 1120 can perform a DIMD process to identify at least some of the candidate intra prediction modes 1125 (e.g., based on DIMD histograms of different intra modes). The TIMD candidate identification module 1120 can also add additional intra prediction modes (e.g., neighboring modes of the MPM mode) to the candidate intra prediction mode 1125. The TIMD candidate identification module 1120 can also constrain the candidate intra prediction modes, such as by limiting the candidate intra prediction modes 1125 to certain direction / angle ranges or excluding certain intra prediction modes. In some embodiments, the TIMD candidate identification module 1120 can also check the variance of the candidate intra prediction mode 1125 to determine whether to enable the template matching module 1110.
[0146] The final intra prediction mode 1115 of the TIMD process is used by the intra prediction generation module 1140 to generate a TIMD intra prediction 1145 (based on the contents of the reconstructed picture buffer 1050) to be used as the predicted pixel data 1013 for the current block. The intra prediction generation module 1140 may refine the final intra prediction mode 1115 before using it to generate the predicted pixel data 1013. The intra prediction generation module 1140 may also refine the generated TIMD intra prediction 1145 based on gradients of neighboring blocks. The intra prediction generation module 1140 may mix / fuse multiple predictions / predictors based on multiple final intra prediction modes into one TIMD intra prediction 1145 for the current block.
[0147] Fig.12A process 1200 for performing TIMD using an enhanced list of candidate intra-prediction modes is conceptually illustrated. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing the decoder 1000 perform the process 1200 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing the decoder 700 performs the process 1200.
[0148] The decoder receives (at step 1210) data for a current block of pixels in a current picture of a video to be decoded.
[0149] The decoder (at step 1220) determines a most probable mode (MPM) set for intra-coding the current block based on the intra-prediction mode used to code the neighboring blocks (e.g., above and to the left) of the current block. The decoder (at step 1230) determines a template for the current block among the reconstructed pixels adjacent to the current block.
[0150] In some embodiments, the video implicitly determines whether to perform TIMD to derive intra-frame prediction based on whether the statistical variation of the MPM set is greater than a threshold. In some embodiments, if the statistical variation of the MPM is less than the threshold, the decoder will not perform TIMD (thus ending process 1200).
[0151] The decoder determines (at step 1240) a set of candidate intra-prediction modes to include (i) the set of MPMs and (ii) one or more intra-prediction modes adjacent to an MPM in the set of MPMs. In some embodiments, when the current block is larger than a threshold size, one or more intra-prediction modes adjacent to the MPM are included in the set of candidate intra-prediction modes. In some embodiments, when the current block is smaller than a threshold size, one or more intra-prediction modes adjacent to the MPM are not included in the set of candidate intra-prediction modes. In some embodiments, when the statistical variation of the set of MPMs is less than a threshold, one of the plurality of intra-prediction modes adjacent to the MPM is included in the set of candidate intra-prediction modes. In some embodiments, the decoder derives a gradient histogram for different intra-prediction angles based on gradient magnitudes at different pixel positions along reconstructed pixels adjacent to the current block, and the set of candidate intra-prediction modes includes a mode identified by using the derived histogram. (In other words, the video decoder performs DIMD as preprocessing to identify candidate intra-prediction modes for TIMD.)
[0152] In some embodiments, the set of candidate intra prediction modes is constrained to exclude one or more intra prediction modes based on prediction modes of neighboring blocks of the current block. In some embodiments, the set of candidate intra prediction modes is restricted to a predefined range.
[0153] The decoder derives (at step 1250) an intra prediction mode based on the determined template and a set of candidate intra prediction modes that specify an intra prediction direction.
[0154] The decoder (at step 1260) generates an intra prediction to reconstruct the current block by using the derived intra prediction mode. The intra prediction can be for luminance or chrominance components. In some embodiments, the video decoder refines the derived intra prediction mode and generates the intra prediction using the refined intra prediction mode. In some embodiments, the video decoder refines the generated intra prediction according to the gradient of the reconstructed samples adjacent to the current block. The decoder can then provide the reconstructed current block for display as part of the reconstructed current picture.
[0155] VII. Example Electronic Systems
[0156] Many of the above features and applications are implemented as software processes, which are specified as a set of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When these instructions are executed by one or more computing or processing units (e.g., one or more processors, processor cores, or other processing units), they cause the processing units to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, random access memory (RAM) chips, hard drives, erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), etc. Computer-readable media do not include carrier waves and electronic signals transmitted wirelessly or via wired connections.
[0157] In this specification, the term "software" is meant to include firmware residing in a read-only memory or an application stored in a magnetic storage device that can be read into the memory for processing by the processor. In addition, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while retaining different software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that implement the software inventions described herein together are within the scope of this disclosure. In some embodiments, when the software program is installed to run on one or more electronic systems, one or more specific machine implementations for executing and performing the operations of the software program are defined.
[0158] Fig.13An electronic system 1300 is conceptually illustrated to implement some embodiments of the present disclosure. The electronic system 1300 may be a computer (e.g., a desktop computer, a personal computer, a tablet computer, etc.), a phone, a PDA, or any other kind of electronic device. Such an electronic system includes various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 1300 includes a bus 1305, a processing unit 1310, a graphics processing unit (GPU) 1315, a system memory 1320, a network 1325, a read-only memory 1330, a permanent storage device 1335, an input device 1340, and an output device 1345.
[0159] Bus 1305 collectively represents all system, peripheral, and chipset buses that communicatively couple the numerous internal devices of electronic system 1300. For example, bus 1305 communicatively couples processing unit 1310 with GPU 1315, read-only memory 1330, system memory 1320, and permanent storage device 1335.
[0160] From these various memory units, the processing unit 1310 retrieves instructions to be executed and data to be processed in order to perform the processes of the present disclosure. In different embodiments, the processing unit may be a single processor or a multi-core processor. Some instructions are passed to and executed by the GPU 1315. The GPU 1315 may offload various calculations or supplement the image processing provided by the processing unit 1310.
[0161] Read-only memory (ROM) 1330 stores static data and instructions used by processing unit 1310 and other modules of the electronic system. On the other hand, permanent storage device 1335 is a read-write storage device. This device is a non-volatile storage unit that stores instructions and data even when electronic system 1300 is turned off. Some embodiments of the present disclosure use a mass storage device (such as a magnetic disk or optical disc and its corresponding disk drive) as permanent storage device 1335.
[0162] Other embodiments use removable storage devices (e.g., floppy disks, flash memory devices, etc., and their corresponding disk drives) as permanent storage devices. Like permanent storage device 1335, system memory 1320 is a read-write storage device. However, unlike storage device 1335, system memory 1320 is a volatile read-write memory, such as random access memory. System memory 1320 stores some instructions and data used by the processor at runtime. In some embodiments, processes according to the present disclosure are stored in system memory 1320, permanent storage device 1335 and / or read-only memory 1330. For example, various memory units include instructions for processing multimedia clips and some embodiments. From these different memory units, processing unit 1310 retrieves instructions to be executed and data to be processed in order to perform the processes of some embodiments.
[0163] The bus 1305 is also connected to input and output devices 1340 and 1345. The input device 1340 enables a user to transmit information and select commands to the electronic system. The input device 1340 includes an alphanumeric keyboard and a pointing device (also known as a "mouse control device"), a camera (e.g., a webcam), a microphone or a similar device for receiving voice commands, etc. The output device 1345 displays images generated by the electronic system or otherwise outputs data. The output device 1345 includes a printer and a display device, such as a cathode ray tube (CRT) or a liquid crystal display (LCD), and a speaker or similar audio output device. Some embodiments include devices that function as both input and output devices, such as a touch screen.
[0164] Finally, if Fig.13 As shown, bus 1305 also couples electronic system 1300 to a network 1325 via a network adapter (not shown). In this manner, the computer may be part of a computer network (e.g., a local area network ("LAN"), a wide area network ("WAN"), or an intranet, or a network of networks. For example, any or all components of electronic system 1300 may be used in conjunction with the present disclosure.
[0165] Some embodiments include electronic components, such as microprocessors, storage devices, and memories, which store computer program instructions in machine-readable or computer-readable media (or alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, compact disc-read only (CD-ROM), compact disc-recordable (CD-R), compact disc-rewritable (CD-RW), digital versatile discs (e.g., DVD-ROM, double-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini-SD card, micro SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable CDs, ultra-density CDs, any other optical or magnetic media, and floppy disks. Computer-readable media can store a computer program that is executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as that generated by a compiler, and files including high-level code that is executed by a computer, electronic component, or microprocessor using an interpreter.
[0166] Although the above discussion primarily involves microprocessors or multi-core processors executing software, many of the above features and applications are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuits themselves. In addition, some embodiments execute software stored in programmable logic devices (PLDs), ROM, or RAM devices.
[0167] As used in this specification and any claims of this application, the terms "computer", "server", "processor" and "memory" refer to electronic or other technical devices. These terms do not include people or groups of people. For the purposes of this specification, the terms display or display means display on an electronic device. As used in this specification and any claims of this application, the terms "computer-readable medium", "computer-readable storage medium" and "machine-readable medium" are entirely limited to tangible physical objects that store information in a readable form. These terms do not include any wireless signals, wired download signals, and any other temporary signals.
[0168] Although the present disclosure has been described with reference to many specific details, those skilled in the art will recognize that the present disclosure may be implemented in other specific forms without departing from the spirit of the present disclosure. Fig. 9 and Fig.12) conceptually illustrate the processes. The specific operations of these processes may not be performed in the exact order shown and described. Specific operations may not be performed in a continuous series of operations, and different specific operations may be performed in different embodiments. In addition, the process can be implemented using multiple sub-processes or as part of a larger macro process. Therefore, those skilled in the art will understand that the present disclosure is not limited by the foregoing illustrative details, but is defined by the appended claims.
[0169] The subject matter described herein sometimes illustrates that different components are contained within or connected to different other components. It should be understood that the architectures depicted in this way are merely examples, and that many other architectures that implement the same functionality can actually be implemented. Conceptually, any arrangement of components that implement the same functionality is effectively "associated" so that the desired functionality is implemented. Therefore, any two components combined here to implement a specific functionality can be considered to be "associated" with each other so that the desired functionality is implemented, regardless of the architecture or intermediate components. Similarly, any two components so associated can also be considered to be "operably connected" or "operably coupled" to each other to achieve the desired functionality, and any two components that can be so associated can also be considered to be "operably connected" or "coupled" to each other to achieve the desired functionality. Specific examples of operably coupled include, but are not limited to, components that can physically cooperate and / or physically interact and / or components that can wirelessly interact and / or wirelessly interact and / or components that logically interact and / or logically interact.
[0170] In addition, with respect to the use of substantially any plural and / or singular terms herein, those skilled in the art can translate from the plural to the singular and / or from the singular to the plural and / or apply according to the context. For clarity, various singular / plural arrangements may be explicitly set forth herein.
[0171] In addition, those skilled in the art will understand that, in general, the terms used herein, especially in the appended claims, such as the bodies of the appended claims, are generally intended as "open" terms, for example, the word "including" should be interpreted as "including but not limited to", the word "having" should be interpreted as "at least having", the word "including" should be interpreted as "including but not limited to", etc. Those skilled in the art will further understand that if a specific number of claim recitations is intended to be introduced, such intent will be explicitly stated in the claim, and in the absence of such statement, such intent is absent. For example, to aid understanding, the following appended claims may contain the use of the introductory phrases "at least one" and "one or more" to introduce claim recitations. However, the use of such phrases should not be interpreted as implying that a claim introduced by the indefinite article "a" or "an" limits any particular claim containing such introduced claim to implementations containing only one such statement, even when the same claim includes the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" or "an", for example, "a" and / or "an" should be interpreted as "at least" one or "one or more"; the same applies to the use of definite articles to introduce claim statements. In addition, even if a specific number of an introduced claim recitation is explicitly cited, a person skilled in the art will recognize that such recitation should be interpreted as indicating at least the number of citations, for example, "two recitations", without other modifiers, means at least two citations, or two or more citations. In addition, in those cases where the convention is similar to "at least one of A, B, and C, etc.", generally, such construction is intended to be understood by those skilled in the art in the sense of the convention, for example, "a system having at least one of A, B, and C" will include but is not limited to such systems having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc. In those cases where the convention is similar to "at least one" is used, generally, such construction is intended to be understood by those skilled in the art in the sense of the convention, for example, "a system having at least one of A, B, or C" will include but is not limited to such systems having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc. Those skilled in the art will further understand that in fact, whether in the specification, claims, or drawings, any disjunctive words and / or phrases that appear with two or more alternative terms should be understood to contemplate the possibility of including one term, one term, or both terms. For example, the phrase "A or B" will be understood to include the possibility of "A" or "B" or "A and B".
[0172] It can be understood from the above that various embodiments of the present disclosure have been described herein for illustrative purposes, and various modifications can be made without departing from the scope and spirit of the present disclosure. Therefore, the various embodiments disclosed herein are not intended to be limiting, and the true scope and spirit are indicated by the appended claims.
Claims
1. A video encoding and decoding method, comprising: Receiving data of a pixel block to be encoded or decoded as a current block of a current picture of a video; Determine a most likely mode set for intra-frame coding and decoding of the current block based on the intra-frame prediction modes used for coding and decoding neighboring blocks of the current block; Determine a template of the current block among reconstructed pixels adjacent to the current block; determining a set of candidate intra prediction modes to include (i) the MPM set and (ii) one or more intra prediction modes adjacent to a MPM in the MPM set; deriving an intra prediction mode based on the determined template and the set of candidate intra prediction modes; as well as An intra prediction is generated using the derived intra prediction mode to encode or decode the current block.
2. The video encoding and decoding method according to claim 1, further comprising determining whether to use the template to derive the intra-frame prediction mode based on whether the statistical variation of the most likely mode set is greater than a threshold.
3. The video encoding and decoding method according to claim 1, characterized in that: When the current block is larger than a threshold size, the one or more intra prediction modes neighboring a most probable mode are included in the set of candidate intra prediction modes.
4. The video encoding and decoding method according to claim 3, characterized in that: When the current block is smaller than the threshold size, the one or more intra prediction modes adjacent to a most probable mode are not included in the set of candidate intra prediction modes.
5. The video encoding and decoding method according to claim 1, characterized in that: When the statistical variation of the set of most probable modes is less than a threshold, the one or more intra prediction modes adjacent to a most probable mode are included in the set of candidate intra prediction modes.
6. The video encoding and decoding method according to claim 1 also includes deriving gradient histograms of different intra-frame prediction angles based on gradient magnitudes at different pixel positions along reconstructed pixels adjacent to the current block, wherein the set of candidate intra-frame prediction modes includes a mode determined by using the derived histogram.
7. The video encoding and decoding method according to claim 1, characterized in that: The current block is a block of chroma samples. 8 . The video encoding and decoding method according to claim 1 , further comprising refining the derived intra-frame prediction mode, and using the refined intra-frame prediction mode to generate intra-frame prediction to encode or decode the current block.
9. The video encoding and decoding method according to claim 1, characterized in that: The set of candidate intra prediction modes is restricted to exclude one or more intra prediction modes based on prediction modes of neighboring blocks of the current block.
10. The video encoding and decoding method according to claim 1, characterized in that: The set of candidate intra prediction modes is restricted to a predefined range.
11. The video encoding and decoding method according to claim 1, further comprising refining the generated intra prediction according to the gradient of the reconstructed samples adjacent to the current block.
12. An electronic device comprising: The video codec circuit is configured to perform operations including: Receiving data of a pixel block to be encoded or decoded as a current block of a current picture of a video; Determine a most likely mode set for intra-frame coding and decoding of the current block based on the intra-frame prediction modes used for coding and decoding neighboring blocks of the current block; Determine a template of the current block among reconstructed pixels adjacent to the current block; determining a set of candidate intra prediction modes to include (i) the MPM set and (ii) one or more intra prediction modes adjacent to a MPM in the MPM set; deriving an intra prediction mode based on the determined template and the set of candidate intra prediction modes; as well as An intra prediction is generated using the derived intra prediction mode to encode or decode the current block.
13. A video decoding method, comprising: Receiving data of a pixel block of a current block of a current picture to be decoded as a video; Determine a most likely mode set for intra-frame coding and decoding of the current block based on the intra-frame prediction modes used for coding and decoding neighboring blocks of the current block; Determine a template of the current block among reconstructed pixels adjacent to the current block; determining a set of candidate intra prediction modes to include (i) the MPM set and (ii) one or more intra prediction modes adjacent to a MPM in the MPM set; deriving an intra prediction mode based on the determined template and the set of candidate intra prediction modes; as well as An intra prediction is generated using the derived intra prediction mode to reconstruct the current block.
14. A video encoding method, comprising: Receiving data of a pixel block to be encoded as a current block of a current picture of a video; Determine a most likely mode set for intra-frame coding and decoding of the current block based on the intra-frame prediction modes used for coding and decoding neighboring blocks of the current block; Determine a template of the current block among reconstructed pixels adjacent to the current block; determining a set of candidate intra prediction modes to include (i) the MPM set and (ii) one or more intra prediction modes adjacent to a MPM in the MPM set; deriving an intra prediction mode based on the determined template and the set of candidate intra prediction modes; as well as An intra prediction is generated using the derived intra prediction mode to encode the current block.
Citation Information
Cited By
Processing method, processing device and storage medium
WO2026001799A3
Methods and apparatus of intra prediction with neural network-based intra prediction
WO2026119070A1
Fusion of intra prediction modes in video coding
WO2026119093A1
Video decoding method and apparatus, video encoding method and apparatus, storage medium, and program product
WO2026194941A1
