Method and device for predicting by using plurality of reference lines

Abstract

The invention provides a method for predictive coding and decoding by using a plurality of reference lines. The video codec receives data of a pixel block of a current block of a current picture to be encoded or decoded as a video. A video codec receives or signals selection of a first reference line and a second reference line from a plurality of reference lines adjacent to a current block. The video codec mixes the first reference line and the second reference line into a fused reference line. A video codec generates a prediction of a current block by using a plurality of samples of a fused reference line. The video codec encodes or decodes the current block using the generated prediction.

Description

[Technical field]

[0001] The present invention generally relates to video coding and decoding. In particular, the present invention relates to methods for coding and decoding pixel blocks using a plurality of reference lines through intra prediction and / or cross-component prediction. [Background technology]

[0002] Unless otherwise indicated herein, the methods described in this section are not prior art in the scope of the following patent claims and are not admitted to be prior art by inclusion of this section.

[0003] High-Efficiency Video Coding (HEVC) is an international video coding standard developed by the Joint Collaborative Team on Video Coding (JCT-VC). HEVC is based on a hybrid block motion compensated DCT-like transform coding and decoding architecture. The basic unit of compression is called a coding unit (CU), which is a 2N×2N square pixel block. Each CU can be recursively divided into four smaller CUs until a predefined minimum size is reached. Each CU contains one or more prediction units (PUs).

[0004] Versatile Video Coding (VVC) is the latest international video coding standard developed by ITU-T SG16WP3 and ISO / IEC JTC1 / SC29 / WG11 Joint Video Expert Team (JVET). The input video signal is predicted from the reconstructed signal, which is derived from the coded picture area. The prediction residual signal is processed by block transform. The transform coefficients are quantized and entropy coded together with other side information in the bitstream. The reconstructed signal is generated from the prediction signal and the reconstructed residual signal after inverse transforming the dequantized transform coefficients. The reconstructed signal is also processed by loop filtering to remove coding artifacts. The decoded pictures are stored in a buffer and used to predict future pictures in the input video signal.

[0005] In VVC, a coded picture is divided into non-overlapping square block areas represented by related coding tree units (CTUs). The leaf nodes of the coding tree correspond to coding units (CUs). A coded picture can be represented by a set of slices, each slice comprising an integer number of CTUs. Each CTU in a slice is processed in raster scan order. Using intra-frame prediction or inter-frame prediction of at most two motion vectors and reference indices, a dual-prediction (B) slice can be decoded to predict the sample values ​​of each block. Using intra-frame prediction or inter-frame prediction of at most one motion vector and reference index, a prediction (P) slice is decoded to predict the sample values ​​of each block. Intra-frame (I) slices are decoded using only intra-frame prediction.

[0006] Using a quadtree (QT) with a nested multi-type-tree (MTT) structure, a CTU can be split into one or more non-overlapping coding units (CUs) to accommodate various local motion and texture features. The CU is further divided into smaller CUs using one of five partition types: quadtree partition, vertical binary tree partition, horizontal binary tree partition, vertical center side ternary tree partition, and horizontal center side ternary tree partition.

[0007] Each CU contains one or more prediction units (PUs). The prediction unit, together with the associated CU syntax, serves as a basic unit for signaling prediction sub-information. The specified prediction process is used to predict the values ​​of the relevant pixel samples within the PU. Each CU may contain one or more transform units (TUs) representing prediction residual blocks. The transform unit (TU) includes a transform block (TB) of luminance samples and two corresponding transform blocks of chrominance samples, each TB corresponding to a residual block sample of a color component. An integer transform is applied to the transform block. The layer values ​​of the quantized coefficients are entropy encoded in the bitstream together with other side information. The terms Coding Tree Block (CTB), Coding Block (CB), Prediction Block (PB), and Transform Block (TB) are defined to specify a 2D sample array of a color component associated with a CTU, CU, PU, ​​or TU, respectively. Therefore, a CTU includes a luminance CTB, two chrominance CTBs, and related syntax elements. Similar relationships also apply to CU, PU, ​​and TU.

[0008] For each inter-predicted CU, motion parameters including motion vector, reference picture index and reference picture list usage index and additional information are used to generate inter-predicted samples. Motion parameters can be signaled in an explicit or implicit way. When the CU is encoded using skip mode, the CU is associated with one PU and has no valid residual coefficients, no coded motion vector delta or reference picture index. A merge mode is specified, in which the motion parameters of the current CU are obtained from neighboring CUs, including spatial candidates and temporal candidates, as well as the additional schedule introduced in VVC. Merge mode can be applied to any inter-predicted CU. An alternative to merge mode is the explicit transmission of motion parameters, in which motion vectors, corresponding reference picture indexes for each reference picture list and reference picture list usage flags, as well as other required information, are explicitly signaled for each CU. [Summary of the invention]

[0009] The following summary of the invention is provided for illustration only and is not intended to be limiting in any way. That is, the following summary of the invention is provided to introduce the concepts, highlights, benefits and advantages of the novel and non-obvious technologies described herein. Selected embodiments will be further described in the following specific examples. Therefore, the following summary of the invention is not intended to identify the essential characteristics of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter.

[0010] Some embodiments of the present invention provide a method for predictive coding using a plurality of reference lines. A video codec receives data of a pixel block of a current block of a current picture to be encoded or decoded as a video. The video codec receives or signals to select a first reference line and a second reference line from a plurality of reference lines adjacent to the current block. The video codec blends the first reference line and the second reference line into a fused reference line. The video codec generates a prediction of the current block using a plurality of samples of the fused reference line. The video codec encodes or decodes the current block using the generated prediction.

[0011] Each reference line includes a set of pixel samples forming an L shape near the current block. The plurality of reference lines includes a reference line adjacent to the current block and two or more reference lines not adjacent to the current block. For example, the first reference line is adjacent to the current block, or the first reference line and the second reference line are not adjacent to the current block.

[0012] In some embodiments, the selection of the first reference line and the second reference line includes: an index representing a combination including the first reference line and the second reference line, wherein different combinations of two or more reference lines are represented by different indices. The different indices representing different reference line combinations are determined based on different combinations (e.g., the different combinations are sorted based on cost). In some embodiments, each combination further specifies an intra-frame prediction mode, wherein a prediction of the current block is generated based on the fused reference line through the intra-frame prediction mode. In some embodiments, the selection of the first reference line and the second reference line includes a first index and a second index. The first index determines the first reference line, and the second index is an offset to be added to the first index to determine the second reference line.

[0013] In some embodiments, the video codec may perform decoder side intra-mode derivation (DIMD) based on the fused reference line. Specifically, the video codec derives a gradient histogram (HoG), the gradient histogram including a plurality of data entries corresponding to different intra-prediction angles, wherein when a gradient calculated based on the fused reference line indicates a specific intra-prediction angle corresponding to a data entry, the entry is assigned to the data entry. The video codec may determine two or more intra-prediction modes based on the gradient histogram, and generate a prediction of the current block based on the determined two or more intra-prediction modes.

[0014] In some embodiments, the video codec may perform cross-component prediction based on the fused reference line. For example, the video codec may derive a linear model based on a plurality of luma component samples and chroma component samples of the fused reference line, and the prediction of the current block is a chroma prediction generated by applying the derived linear model to a plurality of luma samples of the current block.

Brief Description of the Drawings

[0015] Figure 1 Intra prediction modes in different directions are shown.

[0016] Figure 2A -B schematically illustrates the top reference template and the left reference template with extended lengths for supporting wide-angle directional modes for non-square blocks of different aspect ratios.

[0017] Figure 3 The use of decoder-side intra mode derivation (DIMD) to implicitly derive the intra prediction mode of the current block is described.

[0018] Figure 4It describes the use of template-based intra mode derivation (TIMD) to implicitly derive the intra prediction mode of the current block.

[0019] Figure 5 The chrominance and luma samples used to derive the linear model parameters are schematically illustrated.

[0020] Figure 6 An example of classifying adjacent samples into groups is shown.

[0021] Figure 7 Reconstructed luma samples and reconstructed chroma samples for DIMD chroma intra prediction are illustrated.

[0022] Fig. 8A -C describes the blocks that are used to generate multiple intra predictions and are adjacent to the current block.

[0023] Fig. 9 It demonstrates the refinement of intra prediction via gradients of neighboring reconstructed samples.

[0024] Fig.10 The most recent complex L-shapes used for HoG accumulation are shown.

[0025] Fig.11 The pixels near the block boundary used to calculate the boundary matching (BM) cost are illustrated.

[0026] Fig. 12A -B illustrates the fusion of pixels adjacent to a coding unit (CU).

[0027] Fig.13A -D illustrates several different types of HoGs with different characteristics.

[0028] Fig.14 The adjacent reference lines and non-adjacent reference lines and reference samples of the current block are illustrated.

[0029] Fig.15A -F shows the corner elimination of the L-shaped reference line of the DIMD.

[0030] Fig.16 Reference line blending based on intra prediction mode is shown.

[0031] Fig.17 Various luma sample phases and chroma sample phases are described.

[0032] Fig.18A -B illustrates that multiple adjacent reference lines are combined into one line for deriving model parameters in CCLM / MMLM.

[0033] Fig.19 An exemplary video encoder that can use a plurality of reference lines when encoding a block of pixels is described.

[0034] Fig. 20A -C illustrates the portion of a video encoder that implements prediction via a plurality of reference lines.

[0035] Fig.21 Schematically illustrates the process of using multiple reference lines to generate predictions when encoding a pixel block.

[0036] Fig. 22 An exemplary video decoder that can use a plurality of reference lines when decoding a pixel block is described.

[0037] Fig.23A -C illustrates the portion of the video decoder that implements prediction via a plurality of reference lines.

[0038] Fig.24 The process of using a plurality of reference lines to generate predictions when decoding a pixel block is schematically illustrated.

[0039] Fig.25 An electronic system for implementing some embodiments of the present application is schematically shown. [Specific implementation method]

[0040] In the detailed description below, many specific details are set forth by way of example to provide a comprehensive understanding of the relevant teachings. Any variants, derivatives and / or extensions based on the teachings of the present application are within the scope of protection of the present application. In some cases, in order to avoid unnecessary confusion of various aspects of the teachings of the present application, well-known methods, processes, components and / or circuits related to one or more example embodiments disclosed herein may be described at a relatively high level without details.

[0041] I. Intra-frame prediction mode

[0042] The intra prediction method generates a predictor for the current prediction unit (PU) by using a reference tier adjacent to the current PU and an intra prediction mode. The intra prediction direction can be selected from a set of modes containing multiple prediction directions. For each PU coded by intra prediction, an index is used and encoded to select one of the intra prediction modes. The corresponding prediction is generated, and then the residual is derived and transformed.

[0043] Figure 1Intra-frame prediction modes in different directions are shown. These intra-frame prediction modes are called directional modes, excluding DC mode or planar mode. As shown in the figure, there are 33 directional modes (V: vertical direction; H: horizontal direction), so H, H+1~H+8, H-1~H-7, V, V+1~V+8, V-1~V-8 are used. In general, the directional mode can be represented as H+k mode or V+k mode, where k=±1, ±2,..., ±8. Each such intra-frame prediction mode can also be called an intra-frame prediction angle. In order to capture arbitrary edge directions appearing in natural video, the number of directional intra-frame modes can be extended from 33 used in HEVC to 65 directional modes, so that k ranges from ±1 to ±16. These more dense directional intra-frame prediction modes are applicable to all block sizes and are applicable to both luminance intra-frame prediction and chrominance intra-frame prediction. By including DC mode and planar mode, the number of intra-frame prediction modes is 35 (or 67).

[0044] Among the 35 (or 67) intra prediction modes, some modes (such as 3 or 5) are determined as the most probable mode (MPM) set for intra prediction in the current prediction block. By selecting an MPM through a signaling index instead of selecting one from 35 (or 67) through a signaling index, the encoder can reduce the bit rate. For example, the intra prediction mode used in the left prediction block and the intra prediction mode used in the upper prediction block are used as MPMs. When the intra prediction modes in two adjacent blocks use the same intra prediction mode, the intra prediction mode can be used as an MPM. When only one of the two adjacent blocks is available and encoded in a directional mode, the two adjacent directions adjacent to the directional mode can be used as MPMs. DC mode and planar mode can also be considered as MPMs to fill available positions in the MPM set, especially when the upper adjacent block or the top adjacent block is not available or is not encoded in intra prediction, or when the intra prediction mode of the adjacent block is not a directional mode. If the intra prediction mode of the current prediction block is one of the ones in the MPM set, 1 or 2 bits are used to signal which mode it is. Otherwise, the intra prediction mode of the current block is different from any entry in the MPM set, and the current block will be encoded as a non-MPM mode. There are 32 such non-MPM modes, and a (5-bit) fixed-length encoding method is used to signal this mode.

[0045] The MPM list is constructed based on the intra-frame modes of the left neighboring block and the upper neighboring block. Assuming that the mode of the left neighboring block is recorded as Left and the mode of the upper neighboring block is recorded as Above, the unified MPM list can be constructed as follows:

[0046] - When a neighboring block is not available, its intra mode is set to planar mode by default.

[0047] - If both mode Left and mode Above are non-angle modes, then:

[0048] ○MPM list {Flat, DC, V, H, V-4, V+4}

[0049] - If one of the modes Left and Above is an angle mode and the other is a non-angle mode,

[0050] but:

[0051] ○ Set a mode Max in Left and Above as the larger mode

[0052] ○MPM list → {Plane, Max, Max-1, Max+1, Max-2, Max+2}

[0053] -If Left and Above are both angles and are different, then:

[0054] ○ Set a mode Max in Left and Above as the larger mode

[0055] ○ Set a mode Min in Left and Above as the smaller mode

[0056] ○ If Max-Min is equal to 1, then:

[0057] MPM list → {Flat, Left, Above, Min-1, Max+1, Min-2}

[0058] ○ Otherwise, if Max-Min is greater than or equal to 62, then:

[0059] MPM list → {Flat, Left, Above, Min+1, Max-1, Min+2}

[0060] ○ Otherwise, if Max-Min is equal to 2, then:

[0061] MPM list → {Plane, Left, Above, Min+1, Min-1, Max+1}

[0062] Otherwise:

[0063] MPM list → {Plane, Left, Above, Min-1, Min+1, Max-1}

[0064] -If Left and Above are both angles and are the same, then:

[0065] ○MPM list → {Plane, Left, Left-1, Left+1, Left-2, Left+2}

[0066] The conventional angular intra prediction direction is defined as from 45 degrees to -135 degrees in the clockwise direction. In VVC, some conventional angular intra prediction modes are adaptively replaced by wide-angle intra prediction modes for non-square blocks. The replaced mode is signaled using the original mode index, which is remapped to the index of the wide-angle mode after parsing.

[0067] In some embodiments, the total number of intra prediction modes remains unchanged, i.e., 67, and the intra mode coding method remains unchanged. To support these prediction directions, a top reference template of length 2W+1 and a left reference template of length 2H+1 are defined. Figure 2A -B schematically illustrates the top reference template and the left reference template with extended lengths for supporting wide-angle directional modes for non-square blocks of different aspect ratios.

[0068] The number of substituted modes in the wide angle direction mode depends on the aspect ratio of the block. Table 1 below lists the substituted intra prediction modes for different blocks with different aspect ratios.

[0069] Table 1: Intra prediction modes replaced by wide mode

[0070]

[0071]

[0072] II. Decoder Side Intra Mode Derivation (DIMD)

[0073] Decoder Side Intra Mode Derivation (DIMD) is a technique that derives two intra prediction modes / angles / directions from the reconstructed neighboring samples (template) of a block and combines these two predictors with a planar mode predictor with weights derived from gradients. DIMD mode is always checked in high complexity RDO mode as an alternative prediction mode. In order to implicitly derive the intra prediction mode of a block, texture gradient analysis is performed both at the encoder and decoder sides. This process starts with an empty Histogram of Gradient (HoG) with 65 entries corresponding to 65 angle / direction intra prediction modes. During the texture gradient analysis, the magnitudes of these entries are determined.

[0074] The video codec that implements DIMD performs the following steps: In the first step, the video codec selects T=3 columns and T=3 rows of templates from the left and above the current block respectively. This area will be used as a reference for gradient-based intra-frame prediction mode derivation. In the second step, a horizontal Sobel filter and a vertical Sobel filter are applied to all 3×3 window positions centered on the pixels in the middle row of the template. At each window position, the Sobel filter calculates the intensity in the pure horizontal direction and the intensity in the pure vertical direction as G_x and G_y, respectively. Then, the texture angle of the window is calculated as

[0075] angle=arctan(G x / G y ),

[0076] It can be converted to one of 65 angular intra prediction modes. Once the intra prediction mode index of the current window is derived as idx, the magnitude of its entry in HoG[idx] is updated by the following addition.

[0077] ampl=|G x |+|G y |

[0078] Figure 3 The use of decoder-side intra mode derivation (DIMD) to implicitly generate the intra prediction mode of the current block is illustrated. The figure shows an example of a Histogram of Gradient (HoG) 310, which is calculated after applying the above operation to all pixel positions in a template 315 including adjacent rows of pixel samples around the current block 300. Once the HoG is calculated, the indices of the two highest histogram bins (M1 and M2) are selected as the two implicitly derived intra prediction modes (IPMs) for the block. The predictions of these two IPMs are further combined with the planar mode as the prediction of the DIMD mode. The prediction fusion is applied as a weighted average of the above three predictions (M1 prediction, M2 prediction and planar mode prediction). For this purpose, the weight of the planar mode can be set to 21 / 64 (~1 / 3). The remaining weight 43 / 64 (~2 / 3) is shared between the two HoG IPMs, proportional to the amplitude of their HoG bins. The prediction fusion or combined prediction of DIMD can be:

[0079] Pred DIMD =(43*(w1*pred M1 +w2*pred M2 )+21*pred planar )>>6

[0080] w1=amp M1 / (amp M1 +amp M2 )

[0081] w2=amp M2 / (amp M1 +amp M2 )

[0082] In addition, the two implicitly derived intra prediction modes are added to the most probable mode (MPM) list, so the DIMD process is performed before building the MPM list. The main derived intra mode of the DIMD block is stored with the block and used to build the MPM list of neighboring blocks.

[0083] III. Template-based Intra Mode Derivation (TIMD)

[0084] For mode selection, template matching methods can be applied by computing the cost between reconstructed samples and predicted samples. One example is template-based intra mode derivation (TIMD). TIMD is a coding method in which the intra prediction mode of a CU is implicitly derived by using neighboring templates at the encoder and decoder, instead of the encoder signaling the exact intra prediction mode to the decoder.

[0085] Figure 4 The use of template-based intra mode derivation (TIMD) to implicitly derive the intra prediction mode of the current block 400 is described. As shown in the figure, the neighboring pixels of the current block 400 are used as templates 410. For each candidate intra mode, the reference samples in the L-shaped reference area 420 located on the upper left of the template 410 are used to generate the prediction samples of the template 410. The TM cost of the candidate intra mode cost is calculated based on the difference (such as SATD) between the reconstructed samples of the template and the prediction samples of the template generated by the candidate intra mode. The candidate intra prediction mode with the smallest cost is selected (such as in DIMD mode) and used for intra prediction of the CU. The candidate mode may include 67 intra prediction modes (such as in VVC) or be extended to 131 intra prediction modes. MPM can be used to indicate the directional information of the CU. Therefore, in order to reduce the intra mode search space and by virtue of the characteristics of the CU, the intra prediction mode is implicitly derived from the MPM list.

[0086] In some embodiments, for each intra prediction mode in the MPM list, the SATD between the predicted sample and the reconstructed sample of the template is calculated as the TM cost of the intra mode. First, the two intra prediction modes with the smallest SATD are selected as TIMD modes. These two TIMD modes are fused with weights after applying the PDPC process, and this weighted intra prediction is used to encode the current CU. The derivation of the TIMD mode includes a position dependent intra prediction combination (PDPC for short).

[0087] The costs of the two selected modes (mode 1 and mode 2) are compared with the threshold, and in the test a cost factor of 2 is applied as follows:

[0088] costMode2<2*costMode1

[0089] If this condition is true, prediction fusion is applied, otherwise only mode 1 is used. The weight of a mode is calculated from its SATD cost as follows:

[0090] weight1=costMode2 / (costMode1+costMode2)

[0091] weight2=1-weight1

[0092] IV. Cross Component Linear Model (CCLM)

[0093] The CCLM or Linear Model (LM) mode is a cross-component prediction mode in which the chrominance components of a block are predicted from the co-located reconstructed luma samples using a linear model. The parameters of the linear model (such as scaling and offset) are derived from the reconstructed luma samples and reconstructed chroma samples adjacent to the block. For example, in VVC, the CCLM mode predicts chroma samples from reconstructed luma samples using correlations between channel frames. This prediction is performed using a linear model of the following form:

[0094] P(i,j)=α·rec′ L (i,j)+β (1)

[0095] In procedure (1), P(i,j) represents the predicted chroma sample in the CU (or the predicted chroma sample of the current CU), rec′ L (i, j) represents the downsampled reconstructed luma samples of the same CU (or the corresponding reconstructed luma samples of the current CU).

[0096] The CCLM model parameters (scaling parameter) α and (offset parameter) β are derived based on the most complex four adjacent chroma samples and their corresponding downsampled luma samples. In LM_A mode (also called LM-T mode), only the upper adjacent template or the top adjacent template is used to calculate the linear model coefficients. In LM_L mode (also called LM-L mode), only the left template is used to calculate the linear model coefficients. In LM-LA mode (also called LM-LT mode), both the left template and the upper template are used to calculate the linear model coefficients.

[0097] Figure 5 The chrominance samples and luma samples used to derive linear model parameters are schematically illustrated. The figure shows a current block 500 with luma component samples and chroma component samples in a 4:2:0 format. The luma samples and chroma samples adjacent to the current block are reconstructed samples. These reconstructed samples are used to derive the cross-component linear model (parameters and parameters). Since the current block is in a 4:2:0 format, the luma samples are first downsampled before being used for linear model derivation. In this example, there are 16 pairs of reconstructed luma (downsampled) samples and chroma samples adjacent to the current block. These 16 pairs of luma values ​​and chroma values ​​are used to derive linear model parameters.

[0098] Assuming the size of the current chroma block is W×H, W' and H' are set to:

[0099] -When using LM-LT mode, W'=W, H'=H;

[0100] -When using LM-T mode, W'=W+H;

[0101] - When the LM-L mode is adopted, H'=H+W.

[0102] The upper adjacent position is represented as S[0,-1]...S[W'-1,-1], and the left adjacent position is represented as S[-1,0]...S[-1,H'-1]. Then, four samples are selected as:

[0103] -S[W' / 4,-1],S[3*W' / 4,-1],S[-1,H' / 4],S[-1,3*H' / 4] when LM mode is used (both upper and left adjacent samples are available);

[0104] -S[W' / 8,-1],S[3*W' / 8,-1],S[5*W' / 8,-1],S[7*W' / 8,-1] when using LMT mode (only upper adjacent samples are available);

[0105] -S[-1,H' / 8],S[-1,3*H' / 8],S[-1,5*H' / 8],S[-1,7*H' / 8] when LM-L mode is used (only left adjacent samples are available);

[0106] The four adjacent luma samples at the selected position are downsampled and compared four times to find the two larger values: x0A and x1A, and the two smaller values: x0B and x1B. Their corresponding chroma sample values ​​are recorded as y0A, y1A, y0B, and y1B, respectively. Then, XA, XB, YA, and YB are derived as:

[0107] X a =(x 0 A +x 1 A +1)>>1;X b =(x 0 B +x 1 B +1)>>1; (2)

[0108] Y a =(y 0 A +y 1 A +1)>>1;Y b =(y 0 B +y 1 B +1)>>1(3)

[0109] The linear model parameters and are obtained according to the following procedure:

[0110]

[0111] β=Y b -α·X b (5)

[0112] The operation of calculating the sum of the parameters according to procedures (4) and (5) can be implemented by a lookup table. In some embodiments, to reduce the memory required to store the lookup table, the diff value (the difference between the maximum value and the minimum value) and the parameter are represented in exponential notation. For example, diff is represented by a 4-bit significant part and an exponent approximation. Therefore, the table of 1 / diff is reduced to 16 elements with 16 significant values, as shown below:

[0113] DivTable[]={0,7,6,5,5,4,4,3,3,2,2,1,1,1,1,0}(6)

[0114] This not only reduces the computational complexity, but also reduces the amount of memory required to store the required tables.

[0115] In some embodiments, in order to obtain more complex samples for calculating CCLM model parameters and, the upper template is expanded to include (W+H) samples for LM-T mode; the left template is expanded to include (H+W) samples for LM-L mode. For LM-LT mode, both the expanded left template and the expanded upper template are used to calculate the linear model coefficients.

[0116] To match the chroma sample positions of a 4:2:0 video sequence, two downsampling filters are applied to luma samples to achieve a 2:1 downsampling ratio in the horizontal and vertical directions. The choice of downsampling filter is specified by a sequence parameter set (SPS) level flag. The two downsampling filters corresponding to "type-0" and "type-2" content are as follows.

[0117] rec L '(i,j)=[rec L (2i-1,2j-1)+2*rec L (2i-1,2j-1)+rec L (2i+1,2j-1)+rec L (2i-1,2j)

[0118] +2*rec L (2i,2j)+rec L (2i+1,2j)+4]>>3(7)

[0119] rec L '(i,j)=[rec L (2i,2j-1)+rec L (2i-1,2j)+4*rec L (2i,2j)+rec L (2i+1,2j)

[0120] +rec L (2i,2j+1)+4]>>3(8)

[0121] In some embodiments, when the upper reference line is located at a CTU boundary, only one luma line (common line buffer in intra prediction) is used to make the downsampled luma samples.

[0122] In some embodiments, the calculation of the parameters and is performed as part of the decoding process, rather than just an encoder search operation. Therefore, there is no need to use syntax to pass the and values ​​to the decoder.

[0123] For chroma intra mode coding, a total of 8 intra modes are allowed. These modes include five normal intra modes and three cross-component linear model modes (LM_LA, LM_A and LM_L). Chroma intra mode coding can directly depend on the intra prediction mode of the corresponding luma block. Chroma intra mode signaling and the corresponding luma intra prediction mode are shown in the following table:

[0124]

[0125]

[0126] Since the independent block partitioning structure of chroma components and luma components is enabled in the I slice, a chroma block can correspond to multiple luma blocks. Therefore, for the derived mode (DM) mode, the intra prediction mode of the corresponding luma block covering the center position of the current chroma block is directly inherited.

[0127] A single unified binary coding table (mapped to data bin strings) is used for chroma intra prediction modes according to the following table:

[0128]

[0129]

[0130] In this table, the first binary code indicates whether it is normal mode (0) or LM mode (1). If it is LM mode, the next binary code indicates whether it is LM_CHROMA (0). If it is not LM_CHROMA, the next binary code indicates whether it is LM_L (0) or LM_A (1). In this case, when sps_cclm_enabled_flag is 0, the first binary code of the binary coding table of the corresponding intra_chroma_pred_mode can be discarded before entropy coding. Or, in other words, the first binary code is inferred to be 0, so no coding is performed. This single binary coding table is used for the case where the value of sps_cclm_enabled_flag is equal to 0 and the case where it is equal to 1. The first two binary codes in the table are context coded and decoded using their own context models, and the remaining binary codes are bypass coded and decoded.

[0131] In addition, to reduce luma-chroma latency in dual trees, when the 64×64 luma codec tree node is not split (and ISP is not used for 64×64 CUs) or is split with QT, the chroma CUs in the 32×32 / 32×16 chroma codec tree nodes are allowed to use CCLM as follows:

[0132] - If a 32×32 chroma node is not split or is split with QT, all chroma CUs in the 32×32 node can use CCLM;

[0133] - If a 32×32 chroma node is split using horizontal BT and the 32×16 child nodes have no split or use vertical BT split, all chroma CUs in the 32×16 chroma node can use CCLM.

[0134] - Under all other luma and chroma codec tree partitioning conditions, chroma CUs are not allowed to use CCLM.

[0135] V. Multi-Model CCLM (MMLM)

[0136] The MMLM mode uses two models for predicting chroma samples from the luma samples of the entire CU. Similar to CCLM, three complex model CCLM modes (MMLM_LA, MMLM_A, and MMLM_L) are used to indicate whether to use both the upper and left neighboring samples, only the upper neighboring samples, or only the left neighboring samples in deriving model parameters.

[0137] In MMLM, the adjacent luma samples and adjacent chroma samples of the current block are classified into two groups, each of which is used as a training set to derive a linear model (i.e., for a specific group, a specific α and β are derived). In addition, the samples of the current luma block are also classified based on the same rules as those for the classification of the adjacent luma samples.

[0138] Figure 6 An example of classifying neighboring samples into two groups is shown. Threshold is calculated as the average of neighboring reconstructed luma samples. Neighboring samples at [x,y] where Rec′L[x,y]<=Threshold are classified into group 1; and neighboring samples at [x,y] where Rec′L[x,y]>Threshold are classified into group 2. Therefore, the complex model CCLM prediction of the chroma samples is:

[0139] Pred c [x,y]=α1×Recˊ L [x,y]+β1if Rec′ L [x,y]≤Threshold

[0140] Pred c [x,y]=α2×Recˊ L [x,y]+β2if Rec′ L [x,y]>Threshold

[0141] VI.DIMD Chroma Mode

[0142] The DIMD chroma mode uses the DIMD derivation method to derive the chroma intra prediction mode for the current block based on the adjacent reconstructed Y samples, the adjacent reconstructed Cb samples, and the adjacent reconstructed Cr samples in the second adjacent rows and columns. Figure 7 The reconstructed luma samples and reconstructed chroma (Y, Cb and Cr) samples for DIMD chroma intra prediction are described, especially the luma samples and chroma samples in the second adjacent row and column. The horizontal gradient and the vertical gradient are calculated for each co-located reconstructed luma sample and reconstructed Cb sample and reconstructed Cr sample of the current chroma block to establish HoG. Then, the intra prediction mode with the largest histogram amplitude value is used to perform chroma intra prediction on the current chroma block.

[0143] When the intra prediction mode derived from the DIMD chroma mode is the same as the intra prediction mode derived from the derived mode (DM), the intra prediction mode with the second largest histogram amplitude value is used as the DIMD chroma mode. (For the chroma DM mode, the intra prediction mode of the corresponding or co-located luminance block covering the center position of the current chroma block is directly inherited.) A CU-level flag can be signaled to indicate whether the proposed DIMD chroma mode is applied.

[0144] VII. Fusion of Chroma Intra Prediction Modes

[0145] In some embodiments, the predictor generated by the MMLM_LT mode can be fused with the predictors generated by other non-LM modes (such as the DM mode, the four default modes, etc.) according to the following:

[0146] pred=(w0*pred0+w1*pred1+(1<<(shift-1)))>>shift

[0147] Among them, pred0 is the predictor obtained using the non-LM mode, pred1 is the predictor obtained using the MMLM_LT mode, and pred is the final predictor of the current chroma block. The two weights w0 and w1 are determined by the intra-frame prediction mode of the adjacent chroma block, and shift is set to 2. Specifically, when the upper adjacent block and the left adjacent block are both encoded and coded using the LM mode, {w0, w1} = {1, 3}; when the upper adjacent block and the left adjacent block are both encoded and coded using the non-LM mode, {w0, w1} = {3, 1}; otherwise, {w0, w1} = {2, 2}. If the non-LM mode is selected, a flag can be signaled to indicate whether fusion is applied. In some embodiments, fusion of chroma prediction modes is only applied to I slices.

[0148] In some embodiments, the fusion of DIMD chroma mode and intra chroma prediction mode can be combined. Specifically, the DIMD chroma mode described in Section VI above is applied. I slices, DM mode, four default modes and DIMD chroma mode can be fused with MMLM_LT mode using the described weights. In some embodiments, for non-I slices, only DIMD chroma mode can be fused with MMLM_LT mode using equal weights.

[0149] VIII. Combining Multiple Intra Predictors

[0150] In some embodiments, a final intra prediction for the current block is generated by combining a plurality of intra predictions. The plurality of intra predictions may be from intra angular prediction, intra DC prediction, intra planar prediction, or other intra prediction tools. In some embodiments, one of the plurality of intra predictions (denoted as P1) may be derived from an intra angular mode that is implicitly derived from gradients of neighboring reconstructed samples (e.g., via DIMD) and has the highest gradient histogram bin, and another of the plurality of intra predictions (denoted as P2) may be derived implicitly by template matching (e.g., via TIMD), the most frequently selected intra prediction mode of neighboring 4×4 blocks, the selected intra mode after excluding high texture areas, or an explicitly signaled angular mode, or explicitly signaled and derived from one of a plurality of MPMs. In some embodiments, P1 may be an intra-frame angle mode implicitly derived from the gradient of adjacent reconstructed samples (e.g., through DIMD), and the intra-frame mode angle is greater than or equal to the diagonal intra-frame angle (e.g., mode 34 among 67 intra-frame mode angles, mode 66 among 131 intra-frame mode angles), and P2 may be implicitly derived from DIMD, and the intra-frame mode angle is less than the diagonal intra-frame angle. In other embodiments, P1 may be an intra-frame angle mode implicitly derived from DIMD, and P2 may be implicitly derived from adjacent blocks.

[0151] Fig. 8A -C describes the blocks that are used to generate multiple intra predictions and are adjacent to the current block. Fig. 8A P2 of the top area and the left area (as indicated by the diagonal line area) of the current block is shown, which is derived according to the intra prediction mode of the adjacent 4×4 block.

[0152] In some embodiments, P1 may be an intra-frame angle mode implicitly derived from DIMD, and P2 may be a plane prediction, which refers to any smooth intra-frame prediction method using a plurality of reference samples at the corners of the current block, such as the plane prediction defined in HEVC / VVC, or other modifications or variations of the plane prediction. In some embodiments, the final intra-frame prediction of the current block is calculated as follows:

[0153] weight1×P1+weight2×P2,

[0154] (P1+P2+1)>>1, or

[0155] Max(P1,P2)=(P1+P2+abs(P1-P2))>>1.

[0156] In some embodiments, Figure 8B As shown, the neighboring window positions of the current block are divided into many groups (such as G1, G2, G3 and G4), each group selects an intra-frame angle mode, and the final intra-frame prediction is a fusion of the selected intra-frame angle predictions and weights.

[0157] In some embodiments, Figure 8C As shown, the final intra prediction can be divided into many regions, and the intra prediction of each region can depend on the adjacent window positions. For example, the intra prediction of the R1 region is a fusion of the intra predictions derived from G2 and G3, the intra prediction of the R2 region is a fusion of the intra predictions derived from G1 and G3, the intra prediction of the R3 region is a fusion of the intra predictions derived from G2 and G4, and / or, the intra prediction of the R4 region is a fusion of the intra predictions derived from G1 and G4.

[0158] In some embodiments, when the gradient magnitude after applying the Sobel filter is less than a threshold value (which varies with block size), all derived DIMD modes are set to planar mode, or the current prediction is set to planar prediction. In other embodiments, when the sum of the HoG accumulated gradient magnitudes after applying the Sobel filter is greater than a threshold value (which varies with block size) or the accumulated gradient magnitude of the first DIMD mode after applying the Sobel filter is greater than a threshold value (which varies with block size), the current intra prediction is set to the prediction from the first DIMD mode (without mixing with planar prediction).

[0159] In some embodiments, during the DIMD process, the boundary smoothness between the candidate intra-frame angle mode prediction and the adjacent reconstructed samples will be further considered when deriving the final intra-frame angle mode prediction. For example, assuming N intra-frame mode candidates derived through DIMD, the SAD between the top / left prediction sample and the adjacent samples of each intra-frame mode candidate will be considered when determining the final intra-frame angle mode prediction.

[0160] In some embodiments, in order to improve the encoding and decoding performance of DIMD, the delta angle is signaled to the decoder end. The final intra-frame angle mode is the intra-frame mode derived through DIMD plus the delta angle. In some embodiments, the encoder end can use the original samples to estimate the best intra-frame angle mode. To reduce the mode signaling overhead, DIMD is applied to implicitly derive the intra-frame angle mode, and then the delta angle between the best intra-frame angle mode and the intra-frame angle mode derived by DIMD is signaled to the decoder end. The delta angle may include the syntax of the magnitude of the delta angle and the syntax of the sign of the delta angle. The final intra-frame angle mode at the decoder end is the intra-frame angle mode derived by DIMD plus the delta angle.

[0161] To simplify the DIMD process, the HoG calculation comes from some of the selected adjacent window positions to reduce the amount of calculation. In some embodiments, the DIMD process can select the upper middle, upper right, middle left, and lower left adjacent window positions to apply the Sobel filter to establish the HoG. Alternatively, even or odd adjacent window positions can be selected to apply the Sobel filter to construct the HoG. In other embodiments, an angle mode is implicitly derived by applying a Sobel filter to an upper selected window position (e.g., an upper adjacent window position between 0, ..., the current block width - 1, an upper adjacent window position between 0, ..., 2× the current block width - 1, or an upper adjacent window position between 0, ..., the current block width + the current block height - 1), and another angle mode is implicitly derived by applying a Sobel filter to a left selected window position (e.g., a left adjacent position between 0, ..., the current block height - 1, a left adjacent position between 0, ..., 2× the current block height - 1, or a left adjacent position between 0, ..., the current block width + the current block height - 1), and then, no HoG calculation is required because only one position is selected and therefore no HoG needs to be established.

[0162] In some embodiments, in order to improve the coding performance of DIMD, DIMD prediction is applied to the chroma CU to implicitly derive the intra-frame angle mode. In one embodiment, if the candidate intra-frame chroma modes are DC, vertical, horizontal, planar, and DM, DIMD prediction is applied to derive the final intra-frame angle mode. In another embodiment, a flag is used to indicate whether to use DIMD to derive the final intra-frame angle mode. If the flag is true, DIMD implicitly derives the final intra-frame angle mode and excludes the DC mode, vertical mode, horizontal mode, planar mode, and DM mode in the candidate intra-frame mode list.

[0163] In some embodiments, after the intra angle mode is derived through DIMD, a refined search can be performed around the derived intra angle mode. In some embodiments, DIMD derives the intra angle mode from 2 to 67 modes. Assuming that intra angle mode k is derived, the encoder can insert more complex intra mode searches between (k-1) and (k+1) and signal a delta value to indicate the final intra prediction angle mode.

[0164] In some embodiments, when deriving intra angle modes via DIMD, the video codec may exclude or reduce the gradients of adjacent inter-coded positions when calculating the gradient histogram, or increase the cost between prediction and reconstruction of the inter-coded template.

[0165] In order to reduce the comparisons required for DIMD, the candidate intra-angle modes in DIMD may depend on the block size or the prediction mode of the neighboring blocks. In some embodiments, the candidate intra-angle modes in DIMD for smaller CUs (e.g., CU width + height or CU area is less than a threshold) are less than the candidate intra-angle modes in DIMD for larger CUs. For example, the number of candidate intra-angle modes in DIMD for smaller CUs is 34, while the number of candidate intra-angle modes in DIMD for larger CUs is 67. In other embodiments, the candidate intra-angle modes in DIMD may also be constrained or reduced to a predetermined range. For example, if the current intra-angle mode can support a plurality of up to 67 modes (i.e., 0, 1, 2, 3, ..., 67), a subset of these 67 modes (i.e., candidate modes < 67 modes) may be used to constrain the candidate intra-angle modes in DIMD. The constrained candidate modes can be {0,1,2,4,6,8,…,66}, {0,1,3,5,7,9,…,65}, {0,1,2,3,4,5,…,34} or {34,35,36,37,38,…,67}. This constraint can be signaled in the PPS, SPS, picture header, slice header, CTU level syntax, or can be implicitly derived based on other syntax, or always applied. For example, if the constraint is signaled, a CU that only uses DIMD encoding uses fewer candidate intra-frame angle modes to derive the final intra-frame angle mode.

[0166] In some embodiments, the candidate intra-frame angle modes in DIMD may also be constrained by the prediction modes of the neighboring blocks. For example, if the top neighboring CU is inter-coded in skip mode, the intra-frame angle modes that are larger than the diagonal intra-frame angle mode (e.g., mode 66 among 131 intra-frame angle modes, mode 34 among 67 intra-frame angle modes, mode 18 among 34 intra-frame angle modes) will be excluded from the candidate intra-frame angle modes in DIMD. If the left neighboring CU is inter-coded in skip mode, the intra-frame angle modes that are smaller than the diagonal intra-frame angle mode (e.g., mode 66 among 131 intra-frame angle modes, mode 34 among 67 intra-frame angle modes, mode 18 among 34 intra-frame angle modes) will be excluded from the candidate intra-frame angle modes in DIMD.

[0167] In some embodiments, the number of neighboring lines for calculating the HoG in DIMD may be signaled in the PPS, SPS, picture header, slice header, CTU level syntax, or may be implicitly derived from other syntaxes. For example, when the current block size is smaller or larger than a threshold, the video codec may use a greater number of neighboring lines to calculate the HoG in DIMD.

[0168] After generating the intra-frame angle mode prediction through DIMD, the intra-frame prediction is further refined through the gradient of the adjacent reconstructed samples. In some embodiments, the intra-frame prediction is refined through the gradient of the adjacent reconstructed samples. Fig. 9 This illustrates the refinement of intra predictions through the gradients of neighboring reconstructed samples. As shown in the figure, if the current intra prediction comes from the left neighboring reconstructed sample, the current prediction at (x, y) is further refined through the gradient between the top left corner sample (such as R-1, -1) and the current left neighboring sample (such as R-1, y). Subsequently, the refined prediction at (x, y) is (w1×(R x,-1 +(R -1,-1 -R -1,y ))+w2×pred(x,y)) / (w1+w2). In another example, if the current intra prediction comes from the upper adjacent reconstructed sample, the current prediction at (x,y) is further refined through the gradient between the upper left corner sample (such as R-1,-1) and the current upper adjacent sample (such as R x,-1). Subsequently, the refined prediction value at (x,y) is (w1×(R -1,y +(R -1,-1 -R x,-1 ))+w2×pred(x,y)) / (w1+w2).

[0169] In some embodiments, when the current block is a narrow block (eg, width << height) or a wide block (eg, width >> height), the horizontal Sobel filter and the vertical Sobel filter are replaced by the following two matrices to map support for wide-angle intra mode.

[0170]

[0171] or

[0172]

[0173] If the mapped intra angle mode is greater than 135 (such as mode 66) or less than -45 (such as mode 2), the mapped intra angle mode is converted to the intra mode at the other end. For example, if the mapped intra angle mode is greater than mode 66, the converted intra prediction mode is set to be equal to the mapped intra angle mode of the original mode -65. For another example, if the mapped intra angle mode is less than mode 2, the converted intra prediction mode is set to be equal to the mapped intra angle mode of the original mode +67.

[0174] In some embodiments, non-adjacent HoG accumulation may be applied to DIMD.Explicit signaling for selecting one of the nearest N L shapes for HoG accumulation may be applied. Fig.10 The closest plurality of L shapes used for HoG accumulation are shown. As shown, the L shape with index equal to zero is the L shape originally used by DIMD. The L shapes with index greater than 1 are non-adjacent L shapes. If the gradient statistics of the farther L shapes are more representative of the intra-frame direction of the CU, additional coding gain is achieved by using non-adjacent L shapes for HoG accumulation. In addition to explicit signaling, boundary matching costs can also be used for implicit L shape selection. In some embodiments, boundary matching (BM) can be used as a cost function to evaluate discontinuities across block boundaries. Fig.11 The pixels near the block boundary used to calculate the BM cost are illustrated. In the figure, Reco (or R) refers to the reconstructed samples adjacent to the current block. Pred (or P) refers to the predicted samples of the current block. The calculation procedure of the BM cost is:

[0175]

[0176] IX. DIMD based on multiple reference lines

[0177] In some embodiments, to calculate the cost, a prediction of the CU is generated by using one of the N candidate L-shape reference lines for HoG accumulation. The boundary matching cost is calculated by procedure (9) with the predicted samples and reconstructed samples around the CU boundary. In some embodiments, for this extended DIMD method, multiple adjacent reference L-shapes are used to determine the DIMD intra mode. When the current block uses DIMD, the video codec can implicitly derive the reference L-shape at the encoder and decoder ends, or explicitly indicate the reference L-shape in the bitstream.

[0178] In some embodiments, when DIMD is used and N neighboring reference L shapes are available for the current block, a candidate intra prediction mode is derived by performing a statistical analysis (e.g., HoG) on neighboring reconstructed samples. The prediction of the candidate intra prediction mode is then combined with the prediction of the planar mode to produce a final intra prediction. When generating the prediction of the candidate intra prediction mode or the prediction of the planar mode, the video codec may use one of the N neighboring reference L shapes and explicitly indicate the used neighboring reference L shape in the bitstream.

[0179] In some embodiments, one of the N adjacent reference L shapes is implicitly derived through boundary matching. When performing boundary matching, the boundary matching cost of the candidate mode refers to a discontinuity measure (e.g., including top boundary matching and / or left boundary matching) between the current prediction (e.g., a predicted sample within the current block generated from the currently selected L shape) and an adjacent reconstruction (e.g., a reconstructed sample within one or more adjacent blocks). Top boundary matching refers to a comparison between the current top prediction sample and the adjacent top reconstructed sample, while left boundary matching refers to a comparison between the current left prediction sample and the adjacent left reconstructed sample. The L shape candidate with the smallest boundary matching cost is selected to generate a derived DIMD intra-frame angle mode for the current block.

[0180] In some embodiments, the boundary matching cost is calculated using a predefined subset of the current prediction. N lines of the top boundary in the current block and / or M lines of the left boundary in the current block are used. In addition, M and N may also be determined based on the current block size. In some embodiments, the boundary matching cost is calculated based on:

[0181]

[0182] The weights (a,b,c,d,e,f,g,h,i,j,k,l) ​​can be any positive integer or equal to 0. The following are examples of possible weights:

[0183] a=2,b=1,c=1,d=2,e=1,f=1,g=2,h=1,i=1,j=2,k=1,l=1

[0184] a=2,b=1,c=1,d=0,e=0,f=0,g=2,h=1,i=1,j=0,k=0,l=0

[0185] a=0,b=0,c=0,d=2,e=1,f=1,g=0,h=0,i=0,j=2,k=1,l=1

[0186] a=1,b=0,c=1,d=0,e=0,f=0,g=1,h=0,i=1,j=0,k=0,l=0

[0187] a=2,b=1,c=1,d=2,e=1,f=1,g=1,h=0,i=1,j=0,k=0,l=0

[0188] a=1, b=0, c=1, d=0, e=0, f=0, g=2, h=1, i=1, j=2, k=1, l=1.

[0190] In some embodiments, a plurality of L shapes may be used for HoG accumulation. The plurality of L shapes may be explicitly selected through signaling syntax elements to select the plurality of L shapes. An implicit method for selecting the plurality of L shapes through boundary matching may also be used. For example, if three L shapes are selected, the three L shapes with the lowest, second lowest, and third lowest boundary matching costs may be selected for HoG accumulation.

[0191] In some embodiments, the DIMD HoG accumulation and intra-frame prediction generation processes are orthogonal. DIMD HoG accumulation can be performed using the nearest L shape of the CU, while the intra-frame prediction generation can refer to multiple L shapes. If both the DIMD HoG accumulation and the intra-frame prediction generation use multiple L shapes, one or more indices of the selected L shape can be transferred from the HoG accumulation process to the prediction generation process. In the prediction generation process, the generated prediction corresponding to the selected index of the L shape can be reused. In some embodiments, if the index selected in the first intra-frame mode generation process is K, the index used to generate the prediction can have a predefined relationship with K. For example, the index used to generate the prediction can be one of the following indices: K-1, K, K+1.

[0192] In some embodiments, rather than applying a Sobel filter on a plurality of L-shapes to accumulate the HoG, neighboring pixels are initially fused together to generate a new representative L-shape. Fig. 12A -B illustrates the fusion of pixels adjacent to a coding unit (CU). Fig. 12A Pixels neighboring the CU are shown, numbered 1 to 18. Fig. 12B The fused pixels are shown as 12' and 18'. The fused pixels can be generated as follows:

[0193] 12'=(10+11+12) / 3

[0194] 12'=(4+8+12) / 3

[0195] 12'=(16+14+12) / 3

[0196] 18'=(1+4+8+11+15+18)

[0197] A fused L-shaped reference line can be generated by applying a filtering process to each neighboring position of a pixel. This filtering process can reduce noise and enhance the intensity along a specific direction. The original HoG accumulation process described in Section II above can be used to derive two DIMD intra-frame modes based on at least one fused reference line (or up to three fused reference lines).

[0198] When DIMD is used for the current block, the two intra-frame modes with the highest gradient values ​​are determined from the reconstructed adjacent samples, and the predictions of these two intra-frame modes are also combined with the weighted planar mode predictor to produce the final intra-frame predictor. When determining the first two intra-frame modes, the gradient of each intra-frame mode is compared with the current best intra-frame mode and the second best intra-frame mode. However, if the gradient of the current candidate intra-frame mode is the same as the current best intra-frame mode and / or the second best intra-frame mode or is very close within a threshold, the TIMD cost of the current intra-frame prediction mode can also be compared with the TIMD cost of the current best intra-frame mode and / or the second best intra-frame mode. For example, if the gradient amplitude of the current candidate intra-frame mode is the same as the gradient amplitude of the current best intra-frame mode and / or the second best intra-frame mode or is very close within a threshold, the template cost of the candidate intra-frame mode is calculated as the SATD between the predicted sample and the reconstructed sample of the template. If the template cost of the current candidate intra-frame mode is lower than the current best intra-frame mode and / or the second best intra-frame mode, the current candidate intra-frame mode is selected as the current best intra-frame mode or the second best intra-frame mode.

[0199] In some embodiments, after establishing the DIMD HoG, the K candidate intra-modes with the highest gradient values ​​are selected. The video codec then applies TM to the candidate intra-modes to determine the final two intra-modes as the DIMD intra-modes. For example, K can be set to 5, and the TM cost is calculated using the five candidates with the highest gradient values. If the number of non-zero data entries in the HoG is less than K, TM is calculated for the available non-zero data entries. In some embodiments, a fixed number K is used to simplify the hardware implementation for detailed design.

[0200] In some embodiments, the DIMD process described in Section II above may be used for the first pass of selection to generate two angle mode candidates with the highest and second highest gradient values. TM is then applied to the second pass of selection to refine the intra-frame mode. Assume that the two intra-frame modes from the DIMD process are M and N. TM is then applied to the intra-frame modes {M-1, M, M+1} and {N-1, N, N+1} to refine the two intra-frame modes of DIMD. After refinement, the two DIMD intra-frame modes may become the same mode. In order to keep the number of modes at 2, a predefined rule may be applied to select the second intra-frame mode. For example, a second DIMD intra-frame mode that is different from the refined first DIMD intra-frame mode is selected from the list {M, N, M-1, M+1, N-1, N+1}.

[0201] In some embodiments, the HoG data sub-item value and the TM cost are fused as a final evaluation measure to select the DIMD intra mode. A possible fusion procedure is shown below:

[0202] Final evaluation value = HoG data item value + clamp ((1 / TMcost)*S, C)

[0203] where clamp(V, C) = V>C? ​​C:V; S is the scaling factor

[0204] The final evaluation value contains the original HoG data segment value and the clamped value proportional to the scaled value inverted from the TM cost. With this approach, the DIMD intra mode is generated by jointly considering the HoG data segment value and the TM cost.

[0205] In some embodiments, the characteristics of the HoG may be used to modify the intra-MPM construction process. Fig.13A -D illustrates several different types of HoGs with different characteristics. Fig.13A A HoG with a horizontal threshold (TH) of data item values ​​is marked. For a conventional HoG, some data item values ​​are greater than TH, while some data item values ​​are less than TH. For this type of HoG, the final two DIMD modes come from the wide-span intra-frame mode. Figures 13B-D are three special cases of HoG, where the pattern diversity of the HoG is constrained. Fig. 13B In this case, when constructing the MPM, the DIMD intra-frame modes in the MPM list can be reduced from two to one, or from two to zero. In this way, other intra-frame modes can be included in the MPM list, thereby obtaining additional coding gain. Fig. 13CIn , half of the data items of HoG are zero or almost zero, so the MPM list can be changed with less DIMD intra mode usage. Fig.13D In the example, there is only one main data item of HoG. When constructing MPM, only this main DIMD intra-frame mode is retained, reducing the DIMD intra-frame modes from two to one. In addition to modifying the number of DIMD intra-frame modes when constructing MPM according to special HoG characteristics, the remaining intra-frame modes used to fill the intra-frame MPM mode list can also be specially selected to further improve the coding and decoding gain. The following Table 2-1 shows an example for filling the intra-frame MPM list.

[0206] Table 2-1:

[0207]

[0208]

[0209] In some embodiments, three HoGs (left HoG, top HoG, and original top left HoG) are established. Two DIMD modes are derived from each HoG. In some embodiments, in order to determine the final DIMD mode, these three mode combinations (DIMD first mode and second mode of 3 HoGs) are sent to the high complexity RDO process to calculate the cost. Additional syntax elements are added to the bitstream to indicate which side (left / top / top left) is used for HoG accumulation, thereby deriving the DIMD intra-frame mode. In some other embodiments, in order to determine the final DIMD mode, the TM cost of the three mode combinations (DIMD first mode and second mode of 3 HoGs) is evaluated to decode which side (left / top / top left) is used for HoG accumulation, thereby deriving the DIMD intra-frame mode. For example, the TM cost of the first mode (of 3 HoGs) can be used to select the end with the lowest TM cost. Other cost evaluation methods can be used, such as the TM cost of the (weighted) sum of the first mode and the second mode of the three HoGs, to determine the final selection from the 3 HoGs used for HoG accumulation. In this way, additional coding gain is achieved without requiring additional syntax elements.

[0210] In some embodiments, through the above explicit HOG end or implicit HOG end selection, the selectable DIMD modes associated with different HOG ends are different. Table 2-2 shows an example implementation:

[0211] Table 2-2:

[0212] DIMD HoG end Selectable intra mode Left + Top 2，3，4，5，……，64，65，66 Left 2，3，4，5，……，33，34，35 Above 33，34，35，……，64，65，66

[0213] Through the mode selection constraints, the probability that the DIMD mode derived from the left HoG is different from the DIMD mode derived from the upper HoG is higher. In Table 2-2, there is some overlap between the selectable modes on the left and the selectable modes on the upper. Therefore, there is still a chance that the left HoG and the upper HoG can derive the same DIMD intra-frame mode. In some embodiments, using Table 2-3 below, the intra-frame modes derived from the left and the upper are different.

[0214] Table 2-3:

[0215] DIMD HoG end Selectable intra mode Left + Top 2，3，4，5，……，64，65，66 Left 2，3，4，5，……，32，33，34 Above 35，36，37，……，64，65，66

[0216] To decide which table to use, in some embodiments, the video codec can compress the video database to check the coding gain and select a predefined table for coding. In some embodiments, a set of possible tables is defined to be used by the video codec for signaling syntax elements to select the best table for coding.

[0217] In some embodiments, the selectable DIMD modes associated with each end are different through the explicit HoG end or implicit HoG end selection described above. Tables 2-4 show an example implementation.

[0218] Table 2-4:

[0219] DIMD HoG end 131 Intra modes selectable in the Angle Mode field Left 2，6，10，14，18，22，…，130 Above 4，8，12，16，20，24，…，128 Left + Top 3，5，7，9，11，13，…，131

[0220] In Table 2-4, for DIMD applied in the 131 intra-frame angle mode domain, the left HoG is used to derive two intra-frame modes only from (4×i+2) intra-frame modes, where i=0…32, the right HoG is used to derive two intra-frame modes only from (4×j+4) intra-frame modes, where j=0…32, and the left+top HoG is used to derive two intra-frame modes only from (2×k+3) intra-frame modes, where k=0…64.

[0221] In some embodiments, through the above explicit HoG terminal or implicit HoG terminal selection, the selectable DIMD mode associated with each terminal is different. Table 2-5 shows an example implementation:

[0222] Table 2-5:

[0223] DIMD HoG end 67 selectable intra modes in the angle mode field Left 2，6，10，14，18，22，…，66 Above 4，8，12，16，20，24，…，64 Left + Top 3，5，7，9，11，13，…，65

[0224] In some embodiments, if the HoG data item values ​​are all zero, the default DIMD intra-frame mode is assigned to the planar mode in the current DIMD algorithm. To further improve the codec gain, the default mode can be changed to the DC mode. In some embodiments, if the HoG data item values ​​are all zero, the default DIMD intra-frame mode is assigned to the planar mode in the current DIMD algorithm. To further improve the codec gain, the default mode can be switched to the DC mode according to the control flag signaled in the SPS or PPS or PH or SH.

[0225] In some embodiments, if the HoG data item values ​​are all zero, the default DIMD intra mode is assigned to the planar mode in the current DIMD algorithm. To further improve the coding gain, the default mode can be switched between the planar mode and the DC mode based on the most recently reconstructed picture or the reconstructed neighboring pixels without additional signaling. The decoder has the ability to decide whether to switch the default mode between the planar mode and the DC mode based on the reconstructed picture or the reconstructed neighboring pixels.

[0226] In some embodiments, the two DIMD intra modes are also used for the generation of the intra MPM list. If the TM process is used to determine the DIMD intra mode, the computational burden of the decoder may be greatly increased. Therefore, in some embodiments, in order to reduce the complexity of the decoder, when the DIMD mode derivation is applied to construct the MPM list, the complex derivation process, such as TM, non-adjacent L-shape selection, etc., is discarded. The complex DIMD derivation process is enabled only when the CU is encoded in DIMD mode.

[0227] Any of the above-mentioned methods can be implemented in an encoder and / or a decoder. For example, any of the above-mentioned methods can be implemented in an inter-frame / intra-frame / prediction module of an encoder and / or an inter-frame / intra-frame / prediction module of a decoder. Alternatively, any of the above-mentioned methods can be implemented as a circuit coupled to an inter-frame / intra-frame / prediction module of an encoder and / or an inter-frame / intra-frame / prediction module of a decoder, thereby providing information required by the inter-frame / intra-frame / prediction module.

[0228] X. Prediction based on multiple reference lines (MRL)

[0229] A. Use non-adjacent reference lines

[0230] In a prediction scheme, reference samples of a current block (e.g., neighboring prediction samples and / or neighboring reconstructed samples of the current block) are adjacent to a left boundary and / or a top boundary of the current block. Some embodiments provide methods for improving the accuracy of cross-component prediction and / or intra / inter prediction by using non-neighboring reference samples (neighboring prediction samples and / or neighboring reconstructed samples that are not adjacent to a boundary of the current block) as (i) reference samples to generate a prediction of the current block and / or (ii) reference samples to determine an intra prediction mode of the current block.

[0231] Fig.14 The adjacent reference lines and non-adjacent reference lines and reference samples of the current block are illustrated. Line 0 is a reference line having reference samples adjacent to the current block. Line 1 and Line 2 are non-adjacent reference lines having reference samples that are not adjacent to the current block. In some embodiments, the non-adjacent reference samples are not limited to Line 1 and Line 2. In some embodiments, the non-adjacent reference samples may be any extended non-adjacent reference lines (such as Line n, where n is a positive integer, such as 1, 2, 3, 4 and / or 5). In some embodiments, the non-adjacent reference samples may be any sample subset in each of the selected one or more non-adjacent reference lines.

[0232] In some embodiments, a flag (e.g., an SPS flag) is signaled to indicate whether one or more non-adjacent reference lines are allowed as candidate reference lines for the current block in addition to the adjacent reference lines (used for conventional prediction). In some embodiments, the candidate reference lines for the current block may include adjacent reference lines (e.g., line 0) and one or more non-adjacent reference lines (e.g., lines 1 to N). In some embodiments, the candidate reference lines for the current block include only one or more non-adjacent reference lines, and do not include adjacent reference lines.

[0233] In some embodiments, an implicit rule is used to indicate whether one or more non-adjacent reference lines are allowed as candidate reference lines for the current block in addition to the adjacent reference lines. In some embodiments, the implicit rule may depend on block width, height, area, pattern information from other color components, or pattern information of adjacent blocks. In some embodiments, when the area of ​​the current block is less than a predefined threshold, only adjacent reference lines may be used to generate intra-frame predictions for the current block. In some embodiments, when most of the adjacent blocks (such as the top adjacent block and the left adjacent block) use one or more non-adjacent reference lines, in addition to the adjacent reference lines, the non-adjacent reference lines may also be used as candidate reference lines for the current block. In some embodiments, when most of the adjacent blocks (such as the top adjacent block and the left adjacent block) use one or more non-adjacent reference lines, only non-adjacent reference lines may be used as candidate reference lines for the current block. In some embodiments, reference line selection for the current color component is based on reference line selection for other color components.

[0234] In some embodiments, when the current encoding / decoding component is a chroma component (such as Cb, Cr), the non-adjacent reference lines may refer to only line 1 and line 2 or any subset of only line 1 and line 2, only line 1 to 5 or any subset of line 1 to line 5, or any subset of line 1 to line n, where n is a positive integer. In other words, for chroma components, except line 0 (adjacent reference line), any combination or subset of non-adjacent reference lines may be used as candidate reference lines for the current block.

[0235] B. Chroma prediction based on multiple reference lines

[0236] A chroma block may refer to a chroma CB belonging to a CU, which includes a luminance and / or chroma CB. A chroma block may be located in an intra slice / tile. A chroma block may be partitioned from a dual-tree partition. In some embodiments, in addition to using a plurality of candidate reference lines for chroma prediction based on an LM mode, a non-LM (linear model-independent) chroma prediction method may also be used using a plurality of candidate reference lines.

[0237] In some embodiments, when the current block is encoded in an intra prediction mode, one or more reference lines are selected from a plurality of candidate reference lines. (The intra prediction mode may be a DIMD chroma mode, a chroma DM, an intra chroma mode in a candidate list of chroma MRL, a DC, a plane or an angle mode, or may be those selected from 67 intra prediction modes, or those selected from an extended 67 intra prediction modes (e.g., 131 intra prediction modes). For the chroma DM mode, the intra prediction mode of the corresponding (co-located) luminance block covering the center position of the current chroma block is directly inherited.

[0238] In some embodiments, the candidate list of chroma MRL includes planar mode, vertical mode, horizontal mode, DC mode, LM mode, chroma DM, DIMD chroma mode, diagonal (DIA) mode, vertical diagonal (VDIA) mode (mode 66 of 67 intra-frame prediction modes), or any subset of the above modes. For example, the candidate list of chroma MRL may include planar mode (which is changed to VDIA when copied with chroma DM), vertical mode (which is changed to VDIA when copied with chroma DM), horizontal mode (which is changed to VDIA when copied with chroma DM), DC mode (which is changed to VDIA when copied with chroma DM), 6 LM modes, chroma DM. For another example, the candidate list of chroma MRL includes planar mode (which is changed to VDIA when copied with chroma DM), vertical mode (which is changed to VDIA when copied with chroma DM), horizontal mode (which is changed to VDIA when copied with chroma DM), DC mode (which is changed to VDIA when copied with chroma DM), chroma DM. For another example, the candidate list of chroma MRL includes 6 LM modes and chroma DM.

[0239] In some embodiments, when the current block is a chroma block, and the current block uses the DIMD chroma mode (as described in Section VI above), Fig.14 Reference lines 0 / 1 / 2 are used to calculate HoG (line 1 is the centerline for calculating HoG). In some embodiments, an indication is used to determine the centerline from one or more candidate centerlines. For example, if the indication specifies that the centerline is line 2, then reference lines 1, 2, and 3 are used to calculate HoG.

[0240] In some embodiments, the indication is explicitly signaled in the bitstream. The indication can be encoded using a truncated unary codeword to select among candidate centerlines 2, 3, or 4. For example, line 2 is represented by codeword 0, line 3 is represented by codeword 10, and line 4 is represented by codeword 11. In some embodiments, the candidate centerlines always include a default centerline (such as reference line 1). In some embodiments, the candidate centerlines are predefined based on explicit signaling. For example, a flag is signaled to determine whether to use the default centerline. If the flag indicates that the default centerline is not to be used, an index is also signaled to select a centerline from one or more candidate centerlines (excluding the default centerline).

[0241] In some embodiments, the candidate center lines are predefined according to implicit rules. For example, when the current block is larger than a threshold, the candidate center lines include line k, where k is larger than the line number of the default center line.

[0242] In some embodiments, when the current block is a chroma block and one or more non-adjacent lines are used for the current block to generate a prediction, a flag is signaled to indicate whether the DIMD chroma mode is used for the current block. If the flag indicates that the DIMD chroma mode is used for the current block, an index is also signaled to select a center line (e.g., used to calculate HoG; for the DIMD chroma mode, the center line is line 1) from one or more candidate center lines (excluding the default center line). In some embodiments, the default center line refers to the center line used for the DIMD chroma mode when the neighboring lines are used for the current block to generate a prediction. In some embodiments, the center line used for the DIMD chroma mode to calculate the HoG affects the reference line used to generate the prediction of the current block. In some embodiments, the center line is used as a reference line to generate the prediction of the current block.

[0243] In some embodiments, a line located at an offset from the center line position is used as a reference line to generate a prediction for the current block. For example, if the center line is line 2 and the offset is 1, then line 3 is used as the reference line. For another example, if the center line is line 2 and the offset is -1, then line 1 is used as the reference line. For another example, for the DIMD chroma mode described in Section VI, the center line is line 1.

[0244] In some embodiments, the center line used for calculating the HoG in the DIMD chroma mode does not affect the reference line used to generate the prediction of the current block. In some embodiments, when the center line is different from the default center line (such as line 1), the reference line used to generate the prediction of the current block is fixed by using a predefined reference line. For example, the predefined reference line is a neighboring reference line.

[0245] In some embodiments, a plurality of DIMD chrominance modes are generated by varying the centerline of the HoG generation. Template matching costs for the intra modes are calculated. The intra mode with the lowest template matching cost is used to determine the final centerline of the HoG generation for the encoding process and the decoding process. In some of these embodiments, the syntax element for selecting a non-default DIMD chrominance centerline for the HoG generation is not signaled.

[0246] In some embodiments, the original process of DIMD chroma mode derivation uses both luma information and chroma information for HoG generation. For HoG generation using a non-default centerline, the contribution from luma is eliminated and only chroma is used to derive HoG generation for the DIMD chroma mode. In some embodiments, in addition to eliminating the contribution from luma to HoG when deriving the DIMD chroma mode using a non-default centerline, the contribution from luma to HoG is also eliminated when deriving the DIMD chroma mode using a default centerline.

[0247] In some embodiments, the corners of the L shape are removed for generating the HoG. In some embodiments, in order to reduce the amount of computation of the HoG, the gradients from the top corner position and the left corner position are eliminated from the HoG accumulation. This elimination can be applied based on some judgment of the block size, or it can be applied all the time. For example, if the current block width plus the current block height or the current block area is greater than a predefined threshold, the gradients from these corner positions are discarded, that is, only the gradients from the top and the left are included in the HoG calculation.

[0248] Fig.15A -F shows the elimination of corners of the L-shaped reference line of the DIMD HoG calculation. Figures 15A-B show one type of corner elimination (Type 0). Fig.15A Corner elimination when the center line is line 1 is shown. Fig. 15B Corner elimination is shown when the center line is line 2. This can reduce the complexity of HoG generation and maintain the coding gain without severe degradation.

[0249] In some embodiments, more than two corner positions are removed from the HoG generation (indicated as Type 1 in FIGS. 15C-D ). The gradient calculation of the HoG generation process involves two 3×3 Sobel filters. The gradient calculation requires 3×3 neighboring pixels. By also removing the additional two positions, the HoG generation depends only on the pixels of the CU above and the CU to the left. The dependency on the top left CU pixels is removed. With this modification, the implementation of DIMD chroma mode derivation is simplified, especially in the case of hardware implementation.

[0250] In some embodiments, pixel padding is used when calculating gradients for locations adjacent to removed locations. Fig.15E The HoG centerline is shown as 1. Fig.15F It shows that the HoG centerline is 2, and the pixels denoted by "p" are padded to the left before the gradient calculation. The pixels denoted by "q" are padded to the top before the gradient calculation. In this way, the number of removed positions generated by HoG is the same as that of type 0 corner removal, and similar to type 1 corner removal, the HoG correlation to the top left CU is also removed.

[0251] The above mentioned corner removal types can be applied to both luma HoG generation and chroma HoG generation.Corner removal can also be applied to single components, ie to luma only or chroma only.

[0252] In some embodiments, after determining the intra mode derived through the DIMD chroma intra mode derivation process with a specific non-default center line, a prediction is generated using the derived intra mode, wherein the selected line used to generate the prediction is either related to or unrelated to the center line generated by HoG. This prediction can also be mixed with other intra coding methods that also generate a prediction of the current CU. A flag can be signaled to indicate whether the mixing process is activated. In addition, if the fusion process is activated, the mixing weights of the involved predictions are signaled. For the simplest case, only two predictions are involved in the mixing process. For more complex cases, more than two predictions may be involved in the mixing process. In some embodiments, the mixing process is always enabled, so that the flag used to indicate the activation of the mixing is eliminated. In some embodiments, the activation of the mixing process can be implicitly inferred through available coding information and / or pixels of neighboring CUs. Therefore, the flag used to indicate the activation of the mixing is eliminated. In some embodiments, the weights of the mixing process are predefined or can be implicitly inferred through available coding information and / or pixels of neighboring CUs. Therefore, the signaling of the mixing weights is eliminated.

[0253] In some embodiments, when generating intra-frame prediction with reference to adjacent samples, only one reference line is used, and this reference line is selected from a plurality of candidate reference lines. In some embodiments, when generating intra-frame prediction with reference to adjacent samples, a plurality of reference lines are used. In some embodiments, whether to use only one reference line or a plurality of reference lines depends on an implicit rule. The implicit rule depends on the block width, block height or block area. For example, when the block area, block width or block height is less than a predefined threshold, only one reference line is used when generating intra-frame prediction. The threshold can be any positive integer, such as 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024..., or it can be the maximum transform size. For another example, the implicit rule may depend on the mode information of the current block, such as the intra-frame prediction mode of the current block. For example, when the current intra-frame prediction mode has non-integer prediction samples (referring to samples located at non-integer positions), a plurality of reference lines are used to generate intra-frame prediction. As another example, when the current intra prediction mode may require an intra interpolation filter to generate an intra prediction, one or more prediction samples may fall into fractional positions or integer positions between reference samples depending on the selected direction (through the current intra prediction mode).

[0254] For another example, the implicit rule depends on the mode information of the previous coded block, such as the intra prediction mode of the neighboring block, the selected reference line of the neighboring block, etc. In some embodiments, the video codec can use only one reference line or multiple reference lines according to the explicit syntax. In some embodiments, multiple reference lines are used when generating intra predictions with reference to neighboring samples.

[0255] C. Mix multiple reference lines

[0256] In some embodiments, the MRL blending process is used when multiple reference lines are used to generate an intra prediction for the current block. In some embodiments ("First Version"), each used reference line is used to generate a prediction, and then a blending process is applied to blend multiple prediction hypotheses from the multiple used reference lines. Alternatively, in some embodiments ("Second Version"), a blending process is used to blend each used reference line, and the blended reference lines are used to generate predictions. In some embodiments, when the MRL blending process is applied, fusion of chroma intra prediction modes is disabled (or inferred to be disabled). In some embodiments, when chroma intra prediction mode fusion is applied, the MRL fusion process is disabled.

[0257] In some embodiments, intra prediction modes are taken into account when blending reference lines. Fig.16 Reference line mixing based on intra prediction mode is shown. As shown in the figure, when the intra prediction mode is an angle mode, the position (x) of the sample to be mixed (r1) on one reference line and the corresponding position (x') of the sample to be mixed (r2) on another reference line are determined according to the intra prediction mode. When the intra prediction mode is not an angle mode, such as a DC or planar mode, the position (x) of the sample to be mixed (r1) on one reference line and the corresponding position (x') of the sample to be mixed (r2) on another reference line are the same.

[0258] Alternatively, in some embodiments, the intra prediction mode is not considered when mixing reference lines. When the intra prediction mode is the angle mode, the position (x) of the sample to be mixed (r1) on one reference line is the same as the corresponding position (x') of the sample to be mixed (r2) on another reference line.

[0259] For example, when two reference lines are used, the final prediction of the current block may be formed by a weighted average of a first prediction from a first reference line and a second prediction from a second reference line. (Alternatively, the final prediction of the current block may be formed by a weighted average of predictions from the first reference line and the second reference line).

[0260] In some embodiments, the weights of the MRL mixing are predefined through implicit rules. For example, the weights may be fixed to any one of (w1, w2) = (3, 1), (3, 1), (2, 2), where w1 is the weight of the first reference line and w2 is the weight of the second reference line. Since the sum of w1 and w2 is 4, a right shift of 2 is applied after adding the predicted samples from the first reference line and the second reference line. (If the MRL mixing process is performed by weighted averaging of the reference lines, the process may be right shifted by 2 after adding the first reference line and the second reference line). In some embodiments, the implicit rules may depend on the width, height, area, pattern information of the neighboring blocks, or the width or height of the current block. In some embodiments, the weights may be indicated using explicit syntax.

[0261] D. Signaling MRL combination selection for intra prediction

[0262] Some embodiments of the present invention provide an MRL combination selection / signaling method for intra-frame prediction. In some embodiments, an index is signaled to indicate a selected combination of a current block, the combination referring to an intra-frame prediction mode, a first reference line, and a second reference line. (According to the previous section, the first reference line and the second reference line in the signaled combination can be used to generate a fused or mixed reference line). In some embodiments, the index is signaled using a truncated unary codeword. In some embodiments, the index is signaled using a context. In some embodiments, the mapping from the index to the selected combination is based on boundary / template matching by calculating the boundary / template matching cost of each candidate combination according to the following steps:

[0263] Step 0: If boundary matching is used, then for each candidate combination, the prediction of the current block is a hybrid prediction from a plurality of prediction hypotheses through a first reference line (based on an intra prediction mode) and a second reference line (based on an intra prediction mode). In some embodiments, for each candidate combination, the prediction of the current block is a hybrid or fused reference line prediction from the first reference line and the second reference line (based on an intra prediction mode). If template matching is used, then for each candidate combination, the prediction about the template is a hybrid prediction from a plurality of prediction hypotheses through a first reference line (based on an intra prediction mode) and a second reference line (based on an intra prediction mode). If the candidate combination is line1 and line2, and the template width and height are both equal to 1, then line1 will be the reference line adjacent to the template, and line2 will be the reference line adjacent to line1. In some embodiments, for each candidate combination, the prediction of the template is a hybrid reference line prediction from the first reference line and the second reference line (based on an intra prediction mode).

[0264] Step 1: The signaling of each combination follows the cost order in step 0. An index equal to 0 is signaled with the shortest or most efficient codeword and mapped to the pair with the smallest boundary / template matching cost. The encoder and decoder can perform steps 0 and 1 to obtain the same mapping from the signaled index to the combination.

[0265] In some embodiments, the number of candidate combinations for signaling can be reduced from the total number of original candidate combinations to the top K candidate combinations with the lowest cost, and the codewords used for signaling of the selected combination can be reduced. When K is set to 1, the selected combination can be inferred to be the combination with the lowest cost without the need for a signaling index. For chroma MRL, candidate intra-frame prediction modes may include planar mode (which is changed to VDIA when copied with chroma DM), vertical mode (which is changed to VDIA when copied with chroma DM), horizontal mode (which is changed to VDIA when copied with chroma DM), DC mode (which is changed to VDIA when copied with chroma DM), chroma DM and 6 LM modes, and candidate reference lines include line 0, line 1, line 2 (the first reference line is line n, and the second reference line is line n+1). The total number of candidate combinations can be 11*3, and the video codec can only use the top K combinations with the lowest cost as candidate combinations for signaling, where K can be a positive integer such as 1, 2, 3 or 32. In some embodiments, when the boundary / template of the current block is not available, a default combination (e.g., any one of the candidate intra prediction modes, any pair of candidate reference lines) is defined and used. In some embodiments, when the boundary / template of the current block is not available, chroma MRL is inferred to be disabled.

[0266] In some embodiments, when a plurality of reference lines (which may include adjacent reference lines and / or one or more non-adjacent reference lines) are used to generate intra prediction of the current block, the above intra prediction MRL combined signaling scheme is applied. In some embodiments, whether to use the intra prediction MRL combined signaling scheme may depend on the signaling syntax or implicit rules to be enabled or disabled. When the intra prediction MRL combined signaling scheme is disabled, an optional signaling scheme for intra prediction and / or reference lines (e.g., specified by VVC) may be followed.

[0267] In some embodiments, an index is signaled to indicate a selected combination of the current block, the combination being the intra prediction mode, the first reference line, the second reference line, and the weight. In some embodiments, the index is signaled using a truncated unary codeword. In some embodiments, the index is signaled using a context. In some embodiments, the mapping from the index to the selected combination depends on boundary / template matching according to the following steps:

[0268] Step 0: Calculate the boundary / template matching cost for each candidate combination. If boundary matching is used, then for each candidate combination, the prediction of the current block is a hybrid prediction from multiple prediction hypotheses and weights through the first reference line and the second reference line (based on the intra prediction mode). If template matching is used, then for each candidate combination, the prediction about the template is a hybrid prediction from multiple prediction hypotheses and weights through the first reference line and the second reference line. If the selected candidate reference line pair is reference line 1 and reference line 2, and the template width and height are both equal to 1, then line 1 is the reference line adjacent to the template, and line 2 is the reference line adjacent to line 1.

[0269] Step 1: The signaling of each combination follows the cost order in step 0. An index equal to 0 is signaled with the shortest or most efficient codeword and mapped to the pair with the smallest boundary / template matching cost. The encoder and decoder can perform steps 0 and 1 to obtain the same mapping from the signaled index to the combination.

[0270] In some embodiments, the number of candidate combinations for signaling can be reduced from the total number of original candidate combinations to the top K candidate combinations with the lowest cost, and the codewords used for signaling of the selected combination can be reduced. When K is set to 1, the selected combination can be inferred to be the combination with the lowest cost without the need for a signaling index. For chroma MRL, candidate intra-frame prediction modes may include planar mode (which is changed to VDIA when copied with chroma DM), vertical mode (which is changed to VDIA when copied with chroma DM), horizontal mode (which is changed to VDIA when copied with chroma DM), DC mode (which is changed to VDIA when copied with chroma DM), chroma DM and 6 LM modes, and candidate reference lines include line 0, line 1, line 2 and candidate weights (w1, w2), such as (1, 3), (3, 1), (2, 2). The video encoder can use only the top K combinations with the lowest cost as candidate combinations for signaling, where K can be a positive integer, such as 1, 2, 3. In some embodiments, when the boundary / template of the current block is not available, a default combination (e.g., any one of the candidate intra prediction modes, any pair of candidate reference lines) is defined and used. In some embodiments, when the boundary / template of the current block is not available, chroma MRL is inferred to be disabled.

[0271] In some embodiments, a first index is signaled to indicate a first reference line, and a second index is signaled to indicate a second reference line. The signaling of the second index depends on the first reference line. (The total available candidate reference lines include line 0, line 1, and line 2.) In some embodiments, the first reference line = first index (ranging from 0 to 2), and the second reference line = second index (ranging from 0 to 1) + 1 + first index. In some embodiments, the first reference line cannot be the same as the second reference line. In some embodiments, the first index is signaled to indicate the first reference line. Based on the first reference line, the second reference line is inferred. In other words, the index is signaled to determine a pair of a first reference line and a second reference line. In some embodiments, the second reference line is a reference line adjacent to the first reference line.

[0272] Generally, the first reference line is line n and the second reference line is line n+1. In some embodiments, if line n+1 is beyond the farthest reference line allowed for the current block (e.g., line 2 when only lines 0, 1, and 2 are candidate reference lines for the current block), the second reference line cannot be used. In some embodiments, the first reference line is line n and the second reference line is line n-1. If n is equal to 0, the second reference line cannot be used. In some embodiments, the mapping from index to selected reference line pair depends on boundary / template matching according to the following steps:

[0273] Step 0: Calculate the boundary / template matching cost for each candidate reference line pair. If boundary matching is used, then for each candidate line pair, the prediction of the current block is a mixture of prediction hypotheses from the first reference line and the second reference line. If template matching is used, then for each candidate line pair, the prediction about the template is a mixture of prediction hypotheses from the first reference line and the second reference line. If a line pair equal to line1 and line2 is a candidate line pair, and the template width and height are both equal to 1, then line1 is the reference line adjacent to the template, and line2 is the reference line adjacent to line1.

[0274] Step 1: The signaling of each pair follows the cost order in step 0. An index equal to 0 is signaled with the shortest or most efficient codeword and mapped to a pair of reference lines with the lowest boundary / template matching cost. The encoder and decoder perform steps 0 and 1 to obtain the same mapping from the signaled index to the pair. The number of candidate pairs for signaling can be reduced to the top K candidate pairs with the lowest cost, and the codewords used to signal the selected pairs can be reduced. In some embodiments, when the boundary / template of the current block is not available, a default reference line pair (such as line 0 and line 2, line 0 and line 1, line 1 and line 2) is defined and used as the first reference line pair. In some embodiments, when the boundary / template of the current block is not available, a default reference line (such as line 0) is defined and used as the first reference line, and only the first reference line is used to generate intra-frame prediction.

[0275] In some embodiments, the first reference line is derived implicitly, and the second reference line is determined based on explicit signaling and the first reference line. For example, the first reference line can be inferred to be line 0, or a reference line with the lowest boundary matching or template matching cost. In some embodiments, when the boundary / template of the current block is not available, a default reference line (such as line 0) is defined and used as the first reference line. In some embodiments, when the boundary / template of the current block is not available, a default reference line is defined and used as the first reference line, and only the first reference line is used to generate intra-frame prediction.

[0276] In some embodiments, the first reference line and the second reference line are implicitly derived. In some embodiments, the selected reference line pair is determined based on boundary / template matching according to the following steps:

[0277] Step 0: Calculate the boundary / template matching cost for each candidate reference line pair. If boundary matching is used, then for each candidate line pair, the prediction of the current block is a mixture of prediction hypotheses from the first reference line and the second reference line. If template matching is used, then for each candidate pair, the prediction of the template is a mixture of prediction hypotheses from the first reference line and the second reference line. If line1 and line2 are a candidate line pair, and the template width and height are both equal to 1, then line1 can be a reference line adjacent to the template, and line2 can be a reference line adjacent to line1.

[0278] Step 1: The selected reference line pair is inferred as the pair with the minimum boundary / template matching cost. The encoder and decoder can both perform step 0 and step 1 to obtain the same selected pair.

[0279] In some embodiments, when the boundary / template of the current block is not available, a default reference line pair (e.g., line 0 and line 2, line 0 and line 1, line 1 and line 2) is defined and used as the first reference line pair. In some embodiments, when the boundary / template of the current block is not available, a default reference line (e.g., line 0) is defined and used as the first reference line, and only the first reference line is used to generate intra-frame prediction. More generally, for a candidate reference line pair, if the first reference line is line n, the second reference line is line n+1, and line n+1 exceeds the farthest reference line allowed for the current block (e.g., line 2 when only lines 0 to 2 are allowed), the second reference line cannot be used. For a candidate reference line pair, if the first reference line is line n, the second reference line is line n-1.

[0280] In some embodiments, when only one reference line is used to generate an intra prediction for the current block, the selection of the reference line is based on an implicit rule. For example, in some embodiments, the selected reference line (among the candidate reference lines) is the reference line with the smallest boundary matching cost or the smallest template matching cost. In some embodiments, when the boundary / template of the current block is not available, a default reference line (e.g., line 0) is defined and used as the first reference line. In some sub-embodiments, an index is signaled to indicate the selected reference line for the current block. In some embodiments, an index is signaled to indicate the selected combination for the current block (the combination refers to the intra prediction mode and the reference line).

[0281] In some embodiments, the index is signaled using truncated unary codewords. In some embodiments, the index is signaled using context. In some embodiments, the mapping from the index to the selected combination depends on boundary / template matching according to the following steps:

[0282] Step 0: Calculate the boundary / template matching cost for each candidate combination. If boundary matching is used, then for each candidate combination, the prediction for the current block is the prediction from the reference line (based on intra prediction mode). If template matching is used, then for each candidate combination, the prediction about the template is the prediction from the reference line (based on intra prediction mode). If the reference line equal to line1 is the candidate reference line, and the template width and height are both equal to 1, then line1 will be the reference line adjacent to the template, and line2 will be the reference line adjacent to line1.

[0283] Step 1: The signaling of each combination follows the cost order in step 0. An index equal to 0 is signaled with the shortest or most efficient codeword and mapped to the pair with the smallest boundary / template matching cost. The encoder and decoder can both perform steps 0 and 1 to obtain the same mapping from the signaled index to the combination.

[0284] In some embodiments, the number of candidate combinations for signaling can be reduced from the total number of original candidate combinations to the top K candidate combinations with the lowest cost, and the codewords for signaling the selected combination can be reduced. When K is set to 1, the selected combination can be inferred to be the combination with the lowest cost without the need for a signaling index. For chroma MRL, candidate intra-frame prediction modes may include a planar mode (which is changed to VDIA when copied with chroma DM), a vertical mode (which is changed to VDIA when copied with chroma DM), a horizontal mode (which is changed to VDIA when copied with chroma DM), a DC mode (which is changed to VDIA when copied with chroma DM), chroma DM, and 6 LM modes, and candidate reference lines include line 0, line 1, and line 2 (the first reference line is line n, and the second reference line is line n+1).

[0285] The total number of candidate combinations can be 11*3 (for n=0, n=1 and n=2), but the video codec can only use the top K combinations with the lowest cost as candidate combinations for signaling, where K can be a positive integer, such as 1, 2, 3 or 22 or 33. In some embodiments, when the boundary / template of the current block is not available, a default combination (e.g., any one of the candidate intra prediction modes, any pair from the candidate reference lines) is defined and used. In some embodiments, when the boundary / template of the current block is not available, the chroma MRL is inferred to be disabled.

[0286] In some embodiments, when a plurality of reference lines (which may include adjacent reference lines and / or one or more non-adjacent reference lines) are used to generate intra-frame prediction of the current block, the above-mentioned intra-frame prediction MRL combined signaling scheme is applied. In some embodiments, the above-mentioned intra-frame prediction MRL combined signaling scheme is applied only when the intra-frame prediction of the current block is generated using non-adjacent reference lines. In some embodiments, whether to use the intra-frame prediction MRL combined signaling scheme may depend on the signaling syntax or implicit rules to be enabled or disabled. When the intra-frame prediction MRL combined signaling scheme is disabled, an optional signaling scheme for intra-frame prediction and / or reference lines (e.g., specified by VVC) may be followed.

[0287] In some embodiments, a candidate boundary matching cost is calculated. The boundary matching cost of the candidate mode may refer to a discontinuity measure (including top boundary matching and / or left boundary matching) between the current prediction (prediction sample within the current block) and the adjacent reconstruction (reconstructed samples within one or more adjacent blocks) generated from the candidate mode. Top boundary matching refers to the comparison between the current top prediction sample and the adjacent top reconstructed sample, while left boundary matching refers to the comparison between the current left prediction sample and the adjacent left reconstructed sample. See above for boundary matching cost. Fig.11 .

[0288] In some embodiments, a predefined subset of the current prediction is used to calculate the boundary matching cost, for example, n lines of the top boundary within the current block and / or m lines of the left boundary within the current block may be used. (In addition, n2 lines of the top adjacent reconstruction and / or m2 lines of the left adjacent reconstruction are used.) The above program (10) provides an example of boundary matching cost calculation. Another example of calculating the boundary matching cost (n=2, m=2, n2=1, m2=1) is provided as follows:

[0289]

[0290] The weights (a, b, c, g, h, i) can be any positive integers, such as a = 2, b = 1, c = 1, g = 2, h = 1, i = 1. Another example of calculating the boundary matching cost. (n = 1, m = 1, n2 = 2, m2 = 2):

[0291]

[0292] The weights (d, e, f, j, k, l) can be any positive integers, such as d = 2, e = 1, f = 1, j = 2, k = 1, l = 1. Another example of calculating the boundary matching cost. (n = 1, m = 1, n2 = 1, m2 = 1):

[0293]

[0294] The weights (a, c, g, i) can be any positive integers, such as a = 1, c = 1, g = 1, i = 1. Another example of calculating the boundary matching cost. (n = 2, m = 1, n2 = 2, m2 = 1):

[0295]

[0296] The weights (a, b, c, d, e, f, g, i) can be any positive integers, such as a = 2, b = 1, c = 1, d = 2, e = 1, f = 1, g = 1, i = 1. Another example of calculating the cost of boundary matching. (n = 1, m = 2, n2 = 1, m2 = 2):

[0297]

[0298] The weights (a, c, g, h, i, j, k, l) can be any positive integers, such as a=1, c=1, g=2, h=1, i=1, j=2, k=1, l=1.

[0299] Other examples of n and m can also be used for n2 and m2; n can be any positive integer, such as 1, 2, 3, 4, etc., and m can be any positive integer, such as 1, 2, 3, 4, etc. In some embodiments, n and / or m vary with the width, height, or area of ​​the block. In some embodiments, for larger blocks (area greater than threshold 2=65, 128, or 256), m becomes larger (2 or 4, instead of 1 or 2).

[0300] In some cases, for larger blocks (area > threshold2=64, 128, or 256), n becomes larger (increased to 2 or 4 instead of 1 or 2). In some embodiments, for taller blocks (height > threshold2*width), m becomes larger and / or n becomes larger. Threshold2=1, 2, or 4. When height > threshold2*width, m is increased (e.g., to 2 or 4 instead of 1 or 2.) In some embodiments, for wider blocks (width > threshold2*height), n becomes larger and / or m becomes smaller. Threshold2=1, 2, or 4. When width is greater than threshold2*height, m ​​is increased (e.g., to 2 or 4 instead of 1 or 2.)

[0301] In some embodiments, the template matching cost of a candidate may refer to the distortion between the template prediction (prediction samples within the template) and the template reconstruction (reconstruction samples within the template) generated from the candidate (including top template matching and / or left template matching). Top template matching refers to the distortion between the top template prediction samples and the top template reconstruction samples, while left template matching refers to the distortion between the left template prediction samples and the left template reconstruction samples. The distortion can be SAD, SATD or any difference measurement matrix / method.

[0302] E. Inferring linear models based on multiple reference lines

[0303] If the chrominance component or luminance component of the current block has multiple adjacent reference lines (including adjacent reference lines and / or non-adjacent reference lines), the adjacent samples can be used to derive the model parameters of CCLM / MMLM. The derivation of this linear model can be adaptive through reference line selection. Fig.14 In the example of , line 0 corresponds to the first reference line, line 1 corresponds to the second reference line, line 2 corresponds to the third reference line, etc. Such multiple reference lines can be used for CCLM / MMLM model parameter derivation.

[0304] In some embodiments, if the current block has N adjacent reference chroma lines, the i-th adjacent reference line is selected for deriving model parameters in CCLM / MMLM, where N>1 and N≥i≥1. In some embodiments, if the current block has N adjacent reference lines, more than one reference line may be selected for deriving model parameters in CCLM / MMLM. Specifically, if the current block has N adjacent reference lines, the video encoder may select k (k≥2) from the N adjacent reference lines for deriving model parameters. The selected adjacent reference lines may include adjacent reference lines (the 1st reference line, or line 0) and / or non-adjacent reference lines (such as the 2nd reference line, the 3rd reference line, or line 1, line 2, line 3...). For example, if 2 are selected from the N adjacent reference lines, the 2 lines may be the 1st reference line and the 3rd reference line, the 2nd reference line and the 4th reference line, the 1st reference line and the 4th reference line..., and so on.

[0305] In some embodiments, if an adjacent chroma reference line is selected, the video codec may select another luma reference line. The luma reference line does not have to be the corresponding luma reference line of the selected chroma reference line. For example, if the i-th adjacent chroma reference line is selected, the video codec may select the j-th adjacent chroma reference line, where i and j may be different or the same. In addition, the video encoder may also use luma reference line samples without luma downsampling process to derive model parameters.

[0306] Fig.17Various luma sample phases and chroma sample phases are illustrated. Luma samples and chroma samples are in a 4:2:0 color subsampling format. As shown, if the corresponding luma sample associated with a chroma sample C corresponds to Y0, Y1, Y2, Y3, and Y'0, Y'1, Y'2, Y'3 are luma samples associated with adjacent chroma samples, the video encoder may select Y0, Y1, (Y0+Y2+1)>>1, (Y'2+(Y0<<1)+Y2+2)>>2, (Y0+(Y2<<1)+Y'0+2)>>2, or (Y0+Y2-Y'2) samples located at a specified adjacent luma line to derive model parameters. The video codec may also select every Y1, Y3, (Y1+Y3+1)>>1, (Y'3+(Y1<<1)+Y3+2)>>2, (Y1+(Y3<<1)+Y'1+2)>>2, or (Y1+Y3-Y'3) samples located at a specified adjacent luma line to derive model parameters.

[0307] In some embodiments, if a reference line among the plurality of reference lines is invalid due to unavailable neighboring samples or CTU row buffer size constraints, another valid reference line may be used to replace the invalid reference line. Fig.14 In the example of FIG. 1 , reference lines 0, 1, 2, ... n are shown. If reference line 2 is invalid, but reference lines 0 and 1 are valid, the video encoder may use reference lines 0 and 1 instead of reference line 2. In some embodiments, only valid reference lines may be used for cross-component model derivation. In other words, invalid reference lines are not used for cross-component model derivation.

[0308] If there are multiple adjacent reference lines for the chrominance component or the luma component of the current block, the video encoder may combine or merge the multiple adjacent reference lines into one line to derive model parameters in CCLM / MMLM. Fig.18A -B illustrates that multiple adjacent reference lines are combined into one line for generating model parameters in CCLM / MMLM.

[0309] In some embodiments, if three neighboring reference lines are available for the current block, the video codec may use a 3×3 window to combine the three neighboring reference lines into one line and use the combined line to derive cross-component model parameters. The combined result of the 3×3 window is processed as Where w_i can be positive or negative, or 0, and b is the offset value. Fig.18A The 3×3 window used to combine three adjacent reference lines is illustrated. Similarly, the video codec can use a 3×2 window to combine three adjacent reference lines. The combination result of the 3×2 window is Where w_i can be a positive or negative value and b is an offset value. Fig.18B A 3x2 window for combining three adjacent reference lines is illustrated.

[0310] In the above example, Ci can be adjacent luminance samples or adjacent chrominance samples. In another embodiment, the general procedure is Where Li and Ci are adjacent chroma samples and adjacent luma samples, S is the applied window size, wi can be positive or negative or 0, and b is an offset value.

[0311] In some embodiments, the model derivation of CCLM / MMLM is based on the selection of different adjacent reference lines, and the indication of the selected reference line of CCLM / MMLM is explicitly determined or implicitly derived. For example, if one or two reference lines are allowed to be used for the current block, and the selected line of CCLM / MMLM is explicitly determined, the first data item is used to indicate whether one line or two lines are used. Then, the second or multiple data items (encoded with truncated univocal codes or fixed-length codes) are used to indicate which reference line or which reference line combination is selected. For example, if one reference line is used, the signaling can indicate the selection from {first line, second line, third line...}. If two reference lines are used, the signaling can indicate the selection from {first line+second line, second line+third line, first line+third line...}.

[0312] By using decoder-side tools, for example, the selected lines of CCLM / MMLM can be derived implicitly by using template matching costs or boundary matching costs. For example, at the decoder side, the final line selection for the current block is the CCLM / MMLM with the line that minimizes the difference between the boundary samples of the current block and the neighboring samples of the current block. In some embodiments, at the decoder side, the final line selection for the current block is the CCLM / MMLM with the line that minimizes the distortion of the samples in the neighboring template. For example, after deriving the model parameters of CCLM / MMLM through a certain reference line, the model is applied to the luma samples of the neighboring template to obtain predicted chroma samples, and the cost is calculated based on the difference between the predicted chroma samples and the reconstructed chroma samples in the neighboring template. By using another reference line, the video codec can derive the model parameters of CCLM / MMLM, where the derived model is applied to the luma samples of the neighboring template to determine the cost. The costs of different models derived from different reference lines are then compared. By selecting and using the model / reference line with the smallest cost, the final chroma prediction of the current block is generated.

[0313] In addition, the use of multiple reference lines may depend on the current block size or the mode of CCLM / MMLM. In some embodiments, if the current block width is less than a threshold, multiple reference lines are used in CCLM_A or MMLM_A. Similarly, if the current block height is less than a threshold, multiple reference lines are used in CCLM_L or MMLM_L. If the (width+height) of the current block is less than a threshold, multiple reference lines are used in CCLM_LA or MMLM_LA. For another example, in some embodiments, if the area of ​​the current block is less than a threshold, more than two reference lines are used in CCLM or MMLM. In other embodiments, multiple reference lines are used in CCLM_A, CCLM_L, MMLM_A or MMLM_L. In yet other embodiments, the SPS, PPS, PH, SH, CTU, CU or PU level signaling syntax may be used to indicate whether multiple reference lines are allowed for the current block.

[0314] The LM mode described herein may refer to one or more CCLM modes and / or one or more MMLM modes. The LM mode may refer to any mode that uses cross-component information to predict the current component. The LM mode may also refer to any extension / variant from the CCLM mode and / or the MMLM mode.

[0315] The method proposed in the present invention can be enabled and / or disabled according to implicit rules (such as block width, height or area) or explicit rules (such as syntax on block, tile, slice, picture, SPS or PPS level). For example, when the block area is less than a threshold, reordering can be applied. The term "block" in the present invention may refer to TU / TB, CU / CB, PU / PB, predefined area or CTU / CTB.

[0316] For the template-related methods of the present invention, the size of the template can vary with the change of the block width, block height or block area. For larger blocks, the template size can be larger. For smaller blocks, the template size can be smaller. For example, for larger blocks, the template thickness is set to 4, and for smaller blocks, the template thickness is set to 2. In some embodiments, the reference line of the template prediction and / or the current block prediction is inferred as a line adjacent to the template. In some embodiments, the reference line of the template prediction and / or the current block prediction is indicated as a line adjacent to or not adjacent to the template or the current block.

[0317] Any combination of the methods proposed by the present invention may be used.

[0318] Any of the above-mentioned methods can be implemented in an encoder and / or a decoder. For example, any of the above-mentioned methods can be implemented in an inter-frame / intra-frame / prediction module of an encoder and / or an inter-frame / intra-frame / prediction module of a decoder. Alternatively, any of the above-mentioned methods can be implemented as a circuit coupled to an inter-frame / intra-frame / prediction module of an encoder and / or an inter-frame / intra-frame / prediction module of a decoder, thereby providing information required by the inter-frame / intra-frame / prediction module.

[0319] XI. Example Video Encoder

[0320] Fig.19 An exemplary video encoder 1900 for using a plurality of reference lines when encoding a pixel block is illustrated. As shown, the video encoder 1900 receives an input video signal from a video source 1905 and encodes the signal into a bitstream 1995. The video encoder 1900 has several components or modules for encoding the signal from the video source 1905, including at least some components selected from a transform module 1910, a quantization module 1911, an inverse quantization module 1914, an inverse transform module 1915, an intra-picture estimation module 1920, an intra-prediction module 1925, a motion compensation module 1930, a motion estimation module 1935, an in-loop filter 1945, a reconstructed picture buffer 1950, a motion vector (MV) buffer 1965, a motion vector prediction module 1975, and an entropy encoder 1990. The motion compensation module 1930 and the motion estimation module 1935 are part of the inter-prediction module 1940.

[0321] In some embodiments, modules 1910-1990 are modules of software instructions being executed by one or more processing units (e.g., processors) of a computing device or electronic device. In some embodiments, modules 1910-1990 are modules of hardware circuits implemented by one or more integrated circuits (ICs) of an electronic device. Although modules 1910-1990 are shown as separate modules, some of these modules can be combined into a single module.

[0322] The video source 1905 provides a raw video signal, which represents pixel data for each video frame without compression. The subtractor 1908 calculates the difference between the raw video pixel data of the video source 1905 and the predicted pixel data 1913 from the motion compensation module 1930 or the intra-picture prediction module 1925 as a prediction residual 1909. The transform module 1910 transforms the difference (or residual pixel data or residual signal 1908) into transform coefficients (e.g., by performing a discrete cosine transform or DCT). The quantization module 1911 quantizes the transform coefficients into quantized data (or quantized coefficients) 1912, which are encoded into the bitstream 1995 by the entropy encoder 1990.

[0323] The inverse quantization module 1914 dequantizes the quantized data (or quantized coefficients) 1912 to obtain transform coefficients, and the inverse transform module 1915 inversely transforms the transform coefficients to generate a reconstructed residual 1919. The reconstructed residual 1919 is added to the predicted pixel data 1913 to generate reconstructed pixel data 1917. In some embodiments, the reconstructed pixel data 1917 is temporarily stored in a line buffer (not shown) for intra picture prediction and spatial MV prediction. The reconstructed pixels are filtered by the in-loop filter 1945 and stored in a reconstructed picture buffer 1950. In some embodiments, the reconstructed picture buffer 1950 is a storage external to the video codec 1900. In some embodiments, the reconstructed picture buffer 1950 is a storage internal to the video encoder 1900.

[0324] The picture intra estimation module 1920 performs intra prediction based on the reconstructed pixel data 1917 to generate intra prediction data. The intra prediction data is provided to the entropy encoder 1990 to be encoded into a bitstream 1995. The intra prediction data is also used by the intra prediction module 1925 to generate predicted pixel data 1913.

[0325] The motion estimation module 1935 performs inter-frame prediction by generating motion vectors to reference pixel data of previously decoded frames stored in the reconstructed picture buffer 1950. These motion vectors are provided to the motion compensation module 1930 to generate predicted pixel data.

[0326] Instead of encoding the complete actual MV in the bitstream, the video codec 1900 uses MV prediction to generate a predicted MV, and the difference between the MV used for motion compensation and the predicted MV is encoded as residual motion data and stored in the bitstream 1995.

[0327] The motion vector prediction module 1975 generates a predicted motion vector, i.e., a motion compensated motion vector used to perform motion compensation, based on the reference motion vector generated for encoding the previous video information frame. The motion vector prediction module 1975 retrieves the reference motion vector from the previous video information frame from the motion vector buffer 1965. The video encoder 1900 stores these motion vectors generated for the current video information frame in the motion vector buffer 1965 as reference motion vectors for generating predicted motion vectors.

[0328] The motion vector prediction module 1975 uses the reference motion vector to create a predicted motion vector. The predicted motion vector can be calculated by spatial motion vector prediction or temporal motion vector prediction. The difference (residual motion data) between the predicted motion vector and the motion compensation motion vector (MC MV) of the current information frame is encoded into a bitstream 1995 by the entropy encoder 1990.

[0329] By using entropy coding techniques, such as context-adaptive binary arithmetic coding (CABAC) or Huffman coding, the entropy encoder 1990 encodes various parameters and data into a bitstream 1995. The entropy encoder 1990 encodes various header elements, flags, and quantized transform coefficients 1912 and residual motion data as syntax elements into the bitstream 1995. In turn, the bitstream 1995 is stored in a storage device or transmitted to a decoder via a communication medium such as a network.

[0330] The in-loop filter 1945 performs filtering or smoothing operations on the reconstructed pixel data 1917 to reduce encoding artifacts, especially artifacts located at the boundaries of pixel blocks. In some embodiments, the filtering operation or smoothing operation performed by the in-loop filter 1945 includes a deblocking filter (DBF), a sample adaptive offset (SAO), and / or an adaptive loop filter (ALF).

[0331] Fig. 20A -C illustrates a portion of the video encoder 1900 that implements prediction via a plurality of reference lines. Fig. 20A As shown, the reference line selection module 2010 selects one or more reference lines. The indication of the selected reference line is provided to the entropy encoder 1990, which can signal the index of the combination including the selected reference line or can signal the indices of the selected reference lines separately.

[0332] Based on the reference line selection, the corresponding samples are obtained from the reconstructed picture buffer 1950. The obtained samples are provided to the reference line blending module 2020, which uses the obtained samples to generate a fused reference line with blended or fused samples. The fused samples of the fused reference line are provided to the prediction generation module 2030. The prediction generation module 2030 uses the fused samples and other samples from the reconstructed picture buffer 1950 and / or the motion compensation module 1930 to generate the predicted pixel data 1913 of the current block.

[0333] In some embodiments, the prediction generation module 2030 uses samples of the fused reference line to perform DIMD intra prediction. Fig. 20B Components in the prediction generation module 2030 for performing DIMD are illustrated. As shown, the gradient accumulation module 2040 derives a gradient histogram (HoG) 2042 having data entries corresponding to different intra-frame prediction angles. Entries to the binary bit code of the gradient histogram 2042 are generated based on gradients calculated from blended samples of the fused reference line (provided by the reference line blending module 2020) and / or neighboring samples of the current block (provided by the reconstructed picture buffer 1950). The intra-frame mode selection module 2046 uses the gradient histogram to determine two or more DIMD intra-frame modes, and the intra-frame prediction generation module 2048 generates a prediction / predictor for the current block based on the two or more DIMD intra-frame modes.

[0334] In some embodiments, the prediction generation module 2030 uses the luma / chroma component samples of the fused reference line to derive a linear model and perform cross-component prediction. Fig. 20C Components in the prediction generation module 2030 for performing cross-component prediction are illustrated. As shown, the linear model generation module 2050 generates a linear model 2055 by performing data regression, etc. using component samples (luminance component samples or chrominance component samples) of the fused reference line and / or other reference lines. In some embodiments, the generated linear model 2055 can be applied to an initial predictor of the current block (e.g., inter-frame prediction performed through motion compensation) to generate a refined predictor of the current block. In some embodiments, the generated linear model can be applied to the luminance samples of the current block to generate predicted chrominance samples of the current block.

[0335] Fig.21 The process 2100 of generating predictions using a plurality of reference lines when encoding a pixel block is schematically illustrated. In some embodiments, the encoder 1900 is implemented to execute instructions stored in a computer-readable medium through one or more processing units (e.g., processors) of a computing device to perform the process 2100. In some embodiments, the electronic device implementing the encoder 1900 performs the process 2100.

[0336] The encoder receives (at block 2110) data for a pixel block of a current block of a current picture to be encoded as a video. The encoder signals (at block 2120) to select a first reference line and a second reference line from a plurality of reference lines adjacent to the current block. Each reference line includes a set of pixel samples that form an L shape near (e.g., above and to the left of) the current block. The plurality of reference lines may include one reference line adjacent to the current block and two or more reference lines that are not adjacent to the current block. For example, the first reference line may be adjacent to the current block, or neither the first reference line nor the second reference line is adjacent to the current block.

[0337] In some embodiments, the selection of the first reference line and the second reference line includes an index representing a combination including the first reference line and the second reference line, wherein different combinations of more than two reference lines are represented by different indices. Different indices representing different combinations of reference lines are determined based on different combinations (e.g., different combinations are sorted based on cost). In some embodiments, each combination further specifies an intra-prediction mode, and based on the fused reference line, a prediction of the current block is generated through the intra-prediction mode. In some embodiments, the selection of the first reference line and the second reference line includes a first index and a second index. The first index can determine the first reference line, and the second index is an offset to be added to the first index to determine the second reference line.

[0338] The encoder blends (at block 2130) the first reference line and the second reference line into a fused reference line. The encoder generates (at block 2140) a prediction of the current block using samples of the fused reference line. In some embodiments, the encoder may perform DIMD intra prediction based on the fused reference line. Specifically, the encoder derives a HoG having data entries corresponding to different intra prediction angles, wherein an entry is assigned to the data entry when a gradient calculated based on the fused reference line indicates a particular intra prediction angle corresponding to the data entry. The encoder may determine two or more intra prediction modes based on the HoG, and generate a prediction of the current block based on the determined two or more intra prediction modes.

[0339] In some embodiments, the encoder may perform cross-component prediction based on the fused reference line. For example, the encoder may derive a linear model based on a plurality of luma component samples and chroma component samples of the fused reference line, and the prediction of the current block is a chroma prediction generated by applying the derived linear model to a plurality of luma samples of the current block.

[0340] The encoder encodes the current block (at block 2150) by generating a prediction residual using the generated prediction.

[0341] XII. Example Video Decoder

[0342] In some embodiments, the encoder may signal (or generate) one or more syntax elements in a bitstream so that the decoder may parse the one or more syntax elements from the bitstream.

[0343] Fig. 22 An exemplary video decoder 2200 using a plurality of reference lines when decoding a pixel block is illustrated. As shown, the video decoder 2200 is a picture decoding or video decoding circuit that receives a bitstream 2295 and decodes the contents of the bitstream into pixel data of a video information frame for display. The video decoder 2200 has several components or modules for decoding the bitstream 2295, including some components selected from an inverse quantization module 2211, an inverse transform module 2210, an intra-frame prediction module 2225, a motion compensation module 2230, an in-loop filter 2245, a decoded picture buffer 2250, a motion vector buffer 2265, a motion vector prediction module 2275, and a parser 2290. The motion compensation module 2230 is a part of the inter-frame prediction module 2240.

[0344] In some embodiments, modules 2210-2290 are modules of software instructions executed by one or more processing units (e.g., processors) of a computing device. In some embodiments, modules 2210-2290 are hardware circuit modules implemented by one or more ICs of an electronic device. Although modules 2210-2290 are illustrated as independent modules, some of these modules may be combined into a single module.

[0345] The parser 2290 (or entropy decoder) receives the bitstream 2295 and performs initial parsing according to the syntax defined by the video codec or picture codec standard. The parsed syntax elements include various header elements, flags, and quantized data (or quantized coefficients) 2212. The parser 2290 parses the various syntax elements by using entropy coding techniques such as context-adaptive binary arithmetic coding (CABAC) or Huffman coding.

[0346] The inverse quantization module 2211 dequantizes the quantized data (or quantized coefficients) 2212 to obtain transform coefficients, and the inverse transform module 2210 inversely transforms the transform coefficients 2216 to generate reconstructed residuals 2219. The reconstructed residuals 2219 are added to the predicted pixel data 2213 from the intra prediction module 2225 or the motion compensation module 2230 to generate decoded pixel data 2217. The decoded pixel data is filtered by the in-loop filter 2245 and stored in the decoded picture buffer 2250. In some embodiments, the decoded picture buffer 2250 is a storage outside the video decoder 2200. In some embodiments, the decoded picture buffer 2250 is a storage inside the video decoder 2200.

[0347] The intra prediction module 2225 receives intra prediction data from the bitstream 2295 and generates predicted pixel data 2213 based on the intra prediction data from the decoded pixel data 2217 stored in the decoded picture buffer 2250. In some embodiments, the decoded pixel data 2217 is also stored in a line buffer (not shown) for intra picture prediction and spatial MV prediction.

[0348] In some embodiments, the contents of the decoded picture buffer 2250 are used for display. The display device 2255 retrieves the contents of the decoded picture buffer 2250 for direct display, or retrieves the contents of the decoded picture buffer to a display buffer. In some embodiments, the display device receives pixel values ​​from the decoded picture buffer 2250 via pixel transfer.

[0349] Based on motion compensation MV (MC MV), the motion compensation module 2230 generates predicted pixel data 2213 from the decoded pixel data 2217 stored in the decoded picture buffer 2250. These motion compensation MVs are decoded by adding the residual motion data received from the bitstream 2295 to the predicted MV received from the motion vector prediction module 2275.

[0350] The motion vector prediction module 2275 generates a predicted MV, for example, a motion compensated MV for performing motion compensation, based on a reference MV generated for decoding a previous video information frame. The motion vector prediction module 2275 retrieves the reference motion vector of the previous video information frame from the motion vector buffer 2265. The video decoder 2200 also stores the motion compensated motion vector generated for decoding the current video information frame in the motion vector buffer 2265 as a reference motion vector for generating a predicted motion vector.

[0351] The in-loop filter 2245 performs filtering or smoothing operations on the decoded pixel data 2217 to reduce encoding artifacts, especially artifacts located at the boundaries of pixel blocks. In some embodiments, the filtering or smoothing operations performed by the in-loop filter 2245 include a deblocking filter (DBF), a sample adaptive offset (SAO), and / or an adaptive loop filter (ALF).

[0352] Fig.23A -C illustrates a portion of the video decoder 2200 that implements prediction via a plurality of reference lines. Fig.23A As shown, the reference line selection module 2310 selects one or more reference lines. The indication of the selected reference line is provided by the entropy decoder 2290, which can receive an index indicating a combination including the selected reference line, or a plurality of indices indicating the selected reference lines respectively.

[0353] Based on the reference line selection, the corresponding samples are obtained from the decoded picture buffer 2250. The obtained samples are provided to the reference line blending module 2320, which uses the obtained samples to generate a fused reference line with blended or fused samples. The fused samples of the fused reference line are provided to the prediction generation module 2330. The prediction generation module 2330 uses the fused samples and other samples from the decoded picture buffer 2250 and / or the motion compensation module 2230 to generate the predicted pixel data 2213 of the current block.

[0354] In some embodiments, the prediction generation module 2330 uses samples of the fused reference line to perform DIMD intra prediction. Fig. 23B Components of the prediction generation module 2330 for performing DIMD are illustrated. As shown, the gradient accumulation module 2340 derives a gradient histogram (HoG) 2342 having data entries corresponding to different intra prediction angles. Entries to the data entry of the gradient histogram are generated based on gradients calculated from blended samples of the fused reference line (provided by the reference line blending module 2320) and / or neighboring samples of the current block (provided by the decoded picture buffer 2250). The intra mode selection module 2346 uses the gradient histogram to determine two or more DIMD intra modes, and the intra prediction generation module 2348 generates a prediction / predictor for the current block based on the two or more DIMD intra modes.

[0355] In some embodiments, the prediction generation module 2330 uses the luma / chroma component samples of the fused reference line to derive a linear model and perform cross-component prediction. Fig.23CComponents in the prediction generation module 2330 for performing cross-component prediction are illustrated. As shown, the linear model generation module 2350 generates a linear model 2355 by performing data regression, etc. using component samples (luminance component samples or chrominance component samples) of the fused reference line and / or other reference lines. In some embodiments, the generated linear model 2355 can be applied to an initial predictor of the current block (e.g., inter-frame prediction through motion compensation) to generate a refined predictor of the current block. In some embodiments, the generated linear model can be applied to the luminance samples of the current block to generate predicted chrominance samples of the current block.

[0356] Fig.24 The process 2400 of generating predictions using a plurality of reference lines when decoding a pixel block is schematically illustrated. In some embodiments, the decoder 2200 is implemented to execute instructions stored in a computer-readable medium through one or more processing units (e.g., processors) of a computing device to perform the process 2400. In some embodiments, the electronic device implements the decoder 2200 to perform the process 2400.

[0357] The decoder receives (at block 2410) data of a pixel block of a current block of a current picture to be decoded as a video. The decoder receives (at block 2420) a first reference line and a second reference line selected from a plurality of reference lines adjacent to the current block. Each reference line includes a set of pixel samples that form an L shape near (e.g., above and to the left of) the current block. The plurality of reference lines may include one reference line adjacent to the current block and two or more reference lines that are not adjacent to the current block. For example, the first reference line may be adjacent to the current block, or neither the first reference line nor the second reference line is adjacent to the current block.

[0358] In some embodiments, the selection of the first reference line and the second reference line includes an index representing a combination including the first reference line and the second reference line, wherein different combinations of more than two reference lines are represented by different indices. Different indices representing different combinations of reference lines are determined based on different combinations (e.g., different combinations are sorted based on cost). Each combination further specifies an intra-frame prediction mode, and based on the fused reference line, a prediction of the current block is generated through the intra-frame prediction mode. In some embodiments, the selection of the first reference line and the second reference line includes a first index and a second index. The first index can determine the first reference line, and the second index is an offset to be added to the first index to determine the second reference line.

[0359] The decoder blends (at block 2430) the first reference line and the second reference line into a fused reference line. The decoder generates (at block 2440) a prediction of the current block using samples of the fused reference line. In some embodiments, the decoder may perform DIMD intra prediction based on the fused reference line. Specifically, the decoder derives a HoG having data entries corresponding to different intra prediction angles, and assigns an entry to the data entry when a gradient calculated based on the fused reference line indicates a particular intra prediction angle corresponding to the data entry. The decoder may determine two or more intra prediction modes based on the HoG, and generate a prediction of the current block based on the determined two or more intra prediction modes.

[0360] In some embodiments, the decoder may perform cross-component prediction based on the fused reference line. For example, the decoder may derive a linear model based on a plurality of luma component samples and chroma component samples of the fused reference line, and the prediction of the current block is a chroma prediction generated by applying the derived linear model to a plurality of luma samples of the current block.

[0361] The decoder reconstructs (at block 2450) the current block by using the generated prediction. The decoder may then provide the reconstructed current block for display as part of a reconstructed current picture.

[0362] XIII. Example Electronic System

[0363] Many of the above features and applications can be implemented as software processes, which are specified as sets of instructions recorded on a computer readable storage medium (also referred to as a computer readable medium). When these instructions are executed by one or more computing units or processing units (e.g., one or more processors, processor cores, or other processing units), the instructions cause the processing units to perform the actions represented by these instructions. Examples of computer readable media include, but are not limited to, CD-ROMs, flash drives, random access memory (RAM) chips, hard disks, erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), etc. Computer readable media do not include carriers and telecommunications signals connected wirelessly or wired.

[0364] In this specification, the term "software" is meant to include firmware in a read-only memory or an application stored in a magnetic storage device, which can be read into the memory for processing by the processor. At the same time, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program, while retaining different software inventions. In some embodiments, multiple software inventions can be implemented as independent programs. Finally, any combination of independent programs that implement the software inventions described herein together is within the scope of the present invention. In some embodiments, when installed to operate on one or more electronic systems, the software program defines one or more specific machine implementations that execute and implement the operations of the software program.

[0365] Fig.25 An electronic system 2500 for implementing some embodiments of the present application is schematically shown, and some embodiments of the present invention are implemented through the electronic system. The electronic system 2500 can be a computer (e.g., a desktop computer, a personal computer, a tablet computer, etc.), a phone, a PDA, or other types of electronic devices. This electronic system includes various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 2500 includes a bus 2505, a processing unit 2510, a graphics processing unit (GPU) 2515, a system memory 2520, a network 2525, a read-only memory (ROM) 2530, a permanent storage device 2535, an input device 2540, and an output device 2545.

[0366] Bus 2505 collectively represents all system buses, peripheral buses, and chipset buses that are communicatively coupled to a multitude of devices within electronic system 2500. For example, bus 2505 is communicatively coupled to processing unit 2510 via GPU 2515, ROM 2530, system memory 2520, and permanent storage device 2535.

[0367] From these various memory units, the processing unit 2510 retrieves instructions to execute and data to process in order to perform the processes of the present invention. In different embodiments, the processing unit can be a single processor or a multi-core processor. Certain instructions are transmitted to and executed by the GPU 2515. The GPU 2515 can offload various calculations or supplement the image processing provided by the processing unit 2510.

[0368] ROM 2530 stores static data and instructions required by processing unit 2510 or other modules of the electronic system. On the other hand, permanent storage device 2535 is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when electronic system 2500 is turned off. Some embodiments of the present invention use a large-capacity storage device (such as a magnetic disk or optical disk and its corresponding magnetic disk drive) as permanent storage device 2535.

[0369] Other embodiments use removable storage devices (such as floppy disks, flash memory devices, etc., and their corresponding disk drives) as permanent storage devices. Like the permanent storage device 2535, the system memory 2520 is a read-write memory device. However, unlike the storage device 2535, the system memory 2520 is a volatile read-write memory, such as a random access memory. The system memory 2520 stores some instructions and data that the processor needs at runtime. In some embodiments, the processing according to the present invention is stored in the system memory 2520, the permanent storage device 2535 and / or the read-only memory 2530. For example, various memory units include instructions for processing multiple media clips according to some embodiments. For these various memory units, the processing unit 2510 retrieves the executed instructions and processed data in order to perform the processing of some embodiments.

[0370] The bus 2505 is also connected to input devices 2540 and output devices 2545. The input devices 2540 enable a user to communicate information and select commands to the electronic system. The input devices 2540 include an alphanumeric keyboard and pointing device (also known as a "cursor control device"), a camera (such as a webcam), a microphone or similar device for receiving voice commands, etc. The output device 2545 displays pictures generated by the electronic system or data output in other ways. The output device 2545 includes a printer and a display device, such as a cathode ray tube (CRT) or a liquid crystal display (LCD), as well as a speaker or similar audio output device. Some embodiments include devices such as a touch screen that serves as both an input device and an output device.

[0371] Finally, if Fig.25As shown, bus 2505 also couples electronic system 2500 to network 2525 via a network adapter (not shown). In this manner, the computer may be part of a computer network (e.g., a local area network (LAN), a wide area network (WAN), or an intranet) or a network of networks (e.g., the Internet). Any or all components of electronic system 2500 may be used in conjunction with the present invention.

[0372] Some embodiments include electronic components, such as a microprocessor, a storage device, and a memory that stores computer program instructions to a machine-readable medium or a computer-readable medium (optionally referred to as a computer-readable storage medium, a machine-readable medium, or a machine-readable storage medium). Some examples of computer-readable media include RAM, ROM, read-only compact discs (CD-ROMs), recordable compact discs (CD-Rs), rewritable compact discs (CD-RWs), read-only digital versatile discs (e.g., DVD-ROMs, dual-layer DVD-ROMs), various recordable / rewritable DVDs (e.g., DVD RAM, DVD-RWs, DVD+RWs, etc.), flash memory (e.g., SD cards, mini SD cards, micro SD cards, etc.), magnetic and / or solid-state hard drives, read-only and recordable Computer readable media may store computer programs executed by at least one processing unit and include sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as that produced by a compiler, and files containing high-level code executed by a computer, electronic component, or microprocessor using an interpreter.

[0373] While the above discussion refers primarily to microprocessors or multi-core processors that execute software, many of the above functions and applications are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuit itself. In addition, some embodiments execute software stored in programmable logic devices (PLDs), ROM, or RAM devices.

[0374] As used in the specification and any claims of the present invention, the terms "computer", "server", "processor" and "memory" refer to electronic devices or other technical devices. These terms do not include people or groups. For the purpose of this specification, the term display or display device refers to displaying on an electronic device. As used in the specification and any claims of the present invention, the terms "computer-readable medium", "computer-readable medium" and "machine-readable medium" are entirely limited to tangible, physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals and any other transient signals.

[0375] While the present invention has been described in conjunction with many specific details, those skilled in the art will recognize that the present invention may be implemented in other specific forms without departing from the spirit of the present invention. Fig.21 and Fig.24 ) conceptually illustrates the processes. The specific operations of these processes may not be performed in the exact order shown and described. These specific operations may not be performed in a continuous series of operations, and different specific operations may be performed in different embodiments. In addition, the processes are implemented by using several sub-processes, or as part of a larger macro process. Therefore, it will be understood by those skilled in the art that the present invention is not limited by the foregoing illustrative details, but is defined by the scope of the claims.

[0376] Additional Notes

[0377] The subject matter described herein sometimes represents different components, which are contained in or connected to other different components. It is understood that the described structure is only an example and can actually be implemented by many other structures to implement the same function. Conceptually, any arrangement of components that implement the same function is actually "associated" in order to implement the required function. Therefore, regardless of the structure or intermediate components, any two components combined to implement a specific function are considered to be "interrelated" to implement the required function. Similarly, any two associated components are considered to be "operably connected" or "operably coupled" to each other to implement a specific function. Any two components that can be associated with each other are also considered to be "operably coupled" to each other to implement a specific function. Specific examples of operable connections include, but are not limited to, physically pairable and / or physically interacting components, and / or wirelessly interactive and / or wirelessly interactive components, and / or logically interactive and / or logically interactive components.

[0378] Furthermore, with respect to the use of substantially any plural and / or singular terms, those of ordinary skill in the art can translate from the plural to the singular and / or from the singular to the plural as appropriate to the context and / or application. For clarity, the various singular / plural permutations are expressly provided herein.

[0379] In addition, it will be understood by those skilled in the art that, in general, the terms used in the present invention, especially in the claims, and the subject matter of the claims, are generally used as "open" terms, for example, "including" should be interpreted as "including but not limited to", "having" should be interpreted as "at least having", "including" should be interpreted as "including but not limited to", etc. It will be further understood by those skilled in the art that if a specific number of claim contents are intended to be introduced, it will be clearly indicated in the claim, and will not be displayed in the absence of such contents. For example, to aid understanding, the claims may contain the phrases "at least one" and "one or more" to introduce the contents of the claims. However, the use of these phrases should not be understood to imply the use of the indefinite article "a" to introduce the contents of the claims and limit any particular patent scope. Even when the same claim includes the introductory phrases "one or more" or "at least one", the indefinite article, such as "a", should be interpreted to mean at least one or more plural, and the same is true for the use of explicit descriptions used to introduce the claims. In addition, even if the introductory content explicitly refers to a specific number, a person of ordinary skill in the art will recognize that such content should be interpreted as indicating the number of references, for example, "two references" without other modifications means at least two references, or two or more references. In addition, when a statement similar to "at least one of A, B, and C" is used, it is generally stated so that a person of ordinary skill in the art can understand the statement, for example, "a system includes at least one of A, B, and C" will include but is not limited to a system with A alone, a system with B alone, a system with C alone, a system with A and B, a system with A and C, a system with B and C, and / or a system with A, B, and C, etc. A person of ordinary skill in the art will further understand that any separated words and / or phrases represented by two or more alternative terms, whether in the specification, claims, or drawings, should be understood to include the possibility of one of these terms, one of them, or both of these terms. For example, "A or B" should be understood as the possibility of "A", or "B", or "A and B".

[0380] As can be seen from the foregoing, various embodiments have been described herein for illustrative purposes, and various modifications may be made without departing from the scope and spirit of the invention. Therefore, the various embodiments disclosed herein are not intended to be limiting, and the scope of the claims represents the true scope and spirit.

Claims

1. A video encoding and decoding method, comprising: receiving data of a pixel block of a current block of a current picture to be encoded or decoded as a video; Receiving or signaling to select a first reference line and a second reference line from a plurality of reference lines adjacent to the current block; blending the first reference line and the second reference line into a fused reference line; Generate a prediction of the current block by using a plurality of samples of the fused reference line; as well as Using the generated prediction, the current block is encoded or decoded.

2. The video encoding and decoding method as claimed in claim 1, further comprising: deriving a histogram of gradients (HoG), the histogram of gradients comprising a plurality of data bins corresponding to different intra prediction angles, wherein an entry is assigned to a data bin when a gradient calculated based on the fused reference line indicates a specific intra prediction angle corresponding to the data bin; and Based on the gradient histogram, determining two or more intra prediction modes; The prediction of the current block is generated based on two or more determined intra-frame prediction modes.

3. The video encoding and decoding method as claimed in claim 1, further comprising: A linear model is derived based on a plurality of luma component samples and chroma component samples of the fusion reference line, wherein the prediction of the current block is a chroma prediction generated by applying the derived linear model to a plurality of luma samples of the current block.

4. The video encoding and decoding method as claimed in claim 1, wherein: Each reference line includes a set of pixel samples forming an L shape around the current block.

5. The video encoding and decoding method as claimed in claim 1, wherein: The plurality of reference lines include a reference line adjacent to the current block and two or more reference lines not adjacent to the current block.

6. The video encoding and decoding method as claimed in claim 5, wherein: The first reference line is adjacent to the current block.

7. The video encoding and decoding method as claimed in claim 5, wherein: The first reference line and the second reference line are not adjacent to the current block.

8. The video encoding and decoding method as claimed in claim 1, wherein: The selection of the first reference line and the second reference line includes: an index representing a combination including the first reference line and the second reference line, wherein different combinations of more than two reference lines are represented by different indexes.

9. The video encoding and decoding method as claimed in claim 8, wherein: Different indexes representing different reference line combinations are determined based on the different combinations.

10. The video encoding and decoding method as claimed in claim 8, wherein: Each combination further specifies an intra prediction mode, wherein a prediction of the current block is generated by the intra prediction mode based on the fused reference line.

11. The video encoding and decoding method as claimed in claim 1, wherein: The receiving or signaling selection of the first reference line and the second reference line includes a first index and a second index.

12. The video encoding and decoding method as claimed in claim 11, wherein: The first index determines a first reference line, and the second index is an offset to be added to the first index for determining the second reference line.

13. An electronic device comprising: The video encoder circuit is configured to perform the following operations: receiving data of a pixel block of a current block of a current picture to be encoded or decoded as a video; Receiving or signaling to select a first reference line and a second reference line from a plurality of reference lines adjacent to the current block; blending the first reference line and the second reference line into a fused reference line; Generate a prediction of the current block by using a plurality of samples of the fused reference line; as well as Using the generated prediction, the current block is encoded or decoded.

14. A video decoding method, comprising: receiving data of a pixel block of a current block of a current picture to be decoded as a video; Receiving or signaling to select a first reference line and a second reference line from a plurality of reference lines adjacent to the current block; blending the first reference line and the second reference line into a fused reference line; Generate a prediction of the current block by using a plurality of samples of the fused reference line; as well as Using the generated predictions, reconstruct the current block.

15. A video encoding method, comprising: receiving data of a pixel block of a current block of a current picture to be decoded as a video; signaling to select a first reference line and a second reference line from a plurality of reference lines adjacent to the current block; blending the first reference line and the second reference line into a fused reference line; Generate a prediction of the current block by using a plurality of samples of the fused reference line; as well as Using the generated prediction, the current block is encoded.

Images