Method and apparatus for using homogeneous syntax with coding tools

Abstract

Disclosed are flexible methods implemented using implementations of determining coding mode information that use combinations of coding modes while maintaining good performance tradeoffs. In one implementation, activation of a first coding mode is determined, and after determining whether to use a second coding mode, encoding or decoding is performed using the determined mode. Other implementations enable the use of multiple reference lines, intra subblock partitioning, matrix intra prediction, and low frequency non-separable transforms.

Description

Technical Field

[0001] At least one embodiment of this implementation relates in general to a method or apparatus for video encoding or decoding, compression or decompression. Background Technology

[0002] To achieve high compression efficiency, image and video coding schemes typically employ prediction (including motion vector prediction) and transform to utilize spatial and temporal redundancy in the video content. Generally, intra-frame or inter-frame prediction is used to leverage intra-frame or inter-frame correlations, and then the differences between the original and predicted images (usually represented as prediction error or prediction residuals) are transformed, quantized, and entropy-coded. To reconstruct the video, the compressed data is decoded through the inverse process corresponding to entropy coding, quantization, transform, and prediction. Summary of the Invention

[0003] At least one embodiment of this implementation relates in general to a method or apparatus for video encoding or decoding, and more specifically to a method or apparatus for using homogeneous coding syntax with a particular coding tool, such as in the VVC (Video Coding or H.266) standard.

[0004] According to a first aspect, a method is provided. The method includes the steps of: determining whether a first encoding tool is activated for encoding a portion of a video image; determining whether a second encoding tool is activated for encoding that portion of the video image, and the conditions for said activation; and conditionally executing at least one of the encoding tools.

[0005] According to a second aspect, a method is provided. The method includes the following steps: determining whether a first decoding tool is activated based on a video bitstream; determining whether a second decoding tool is activated, and the conditions for said activation; and conditionally executing at least one of the decoding tools.

[0006] According to another aspect, an apparatus is provided. The apparatus includes a processor. The processor can be configured to encode video blocks or decode bitstreams by performing any of the methods described above.

[0007] According to another general aspect of at least one embodiment, an apparatus is provided comprising means according to any embodiment of the decoding embodiment; and at least one of: (i) an antenna configured to receive a signal comprising a video block; (ii) a band limiter configured to limit the received signal to a band comprising the video block; and (iii) a display configured to display an output representing the video block.

[0008] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided, which contains data content generated according to any one of the said encoding embodiments or variations.

[0009] According to another general aspect of at least one embodiment, a signal is provided that includes video data generated according to any one of the encoding embodiments or variations.

[0010] According to another general aspect of at least one implementation, the bitstream is formatted to include data content generated according to any one of the implementations or variations of the encoding implementation.

[0011] According to another general aspect of at least one embodiment, a computer program product is provided, the computer program product including instructions that, when executed by a computer, cause the computer to perform any of the described coded embodiments or variations.

[0012] These and other aspects, features, and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments read in conjunction with the accompanying drawings. Attached Figure Description

[0013] Figure 1 An example of four reference rows adjacent to the prediction block is shown.

[0014] Figure 2 The sub-partitions are shown, depending on the block size.

[0015] Figure 3 The matrix-weighted intra-frame prediction process is illustrated.

[0016] Figure 4 The diagram shows the division of the CU into two triangular prediction units.

[0017] Figure 5 The weighting process on the diagonal edge between two triangular prediction units is shown.

[0018] Figure 6 The shape for 4-frame intra-hypothesis prediction is shown.

[0019] Figure 7 The low-frequency non-separable transform (LFNST) process is illustrated.

[0020] Figure 8 The location, type, and transformation type of the SBT are shown.

[0021] Figure 9 An example of the proposed interpolation filter is shown.

[0022] Figure 10 The values ​​for "IF index" and "MV" are shown, allowing the export of motion compensation filters.

[0023] Figure 11 The encoding unit tree syntax is shown.

[0024] Figure 12 A standard, general-purpose video compression scheme (encoder) is shown.

[0025] Figure 13 This illustrates a standard, general-purpose video decompression scheme (decoder).

[0026] Figure 14 The pattern exclusivity diagram is shown.

[0027] Figure 15 A diagram illustrating the non-exclusivity model is shown.

[0028] Figure 16 An overview of the value inference process is shown.

[0029] Figure 17 An overview of the parameters for sending signal notifications is shown.

[0030] Figure 18 The entropy encoding of the CHP tag is shown.

[0031] Figure 19 A processor-based system for encoding / decoding is shown, based on aspects described in general.

[0032] Figure 20 A method for encoding using the described aspects is shown.

[0033] Figure 21 A method for decoding using the described aspects is shown.

[0034] Figure 22 An apparatus for encoding or decoding using the described aspects is shown. Detailed Implementation

[0035] The described implementation is situated within the context of video compression and addresses the specific problem of combining encoding modes in the codec. More specifically, in VVC, certain mode combinations are canonically removed from the specification because they do not offer a good performance / complexity tradeoff at the encoder. In other words, restrictions are imposed on the use of certain modes that are not feasible depending on the context.

[0036] The described implementation schemes illustrate a flexible approach to using combinations of these patterns under normal circumstances while maintaining a good performance trade-off.

[0037] In the following text, several acronyms for encoding modes discussed in this invention are used. For the purposes of this explanation, the underlying processes associated with each mode are not particularly relevant, but rather to the limitations applied to activating those modes. Therefore, only brief explanations of those modes are provided below. More details can be found in the VVC documentation.

[0038] Multi-reference line (MRL) intra-frame prediction

[0039] Intra-frame prediction of a block is based on its already reconstructed neighboring samples. These samples are called reference samples. In normal intra-frame prediction, only the nearest row / column of the coded block is used as reference samples. In multi-reference row (MRL) intra-frame prediction, more reference rows are used for intra-frame prediction. Figure 1 The example depicts four reference rows where samples for segments A and F are not obtained from the reconstructed neighboring samples, but are instead filled with the closest samples from segments B and E, respectively.

[0040] Interactive MRL-MPM

[0041] The index (mrl_idx) of the selected reference row is signaled and used to generate the intra-prediction factor. MRL is applied only to one mode within the MPM (Maximum Probability Mode) framework, and not to any of the remaining modes. When using MRL, planar and DC modes are excluded from intra-prediction mode. This is one case of mode usage limitations addressed in this specification.

[0042] Intra-frame sub-partition (ISP)

[0043] The Intra-Frame Sub-Partition (ISP) tool divides the luma intra-frame prediction block vertically or horizontally into 2 or 4 sub-partitions. Figure 2 Two possible examples are shown. All sub-partitions satisfy the condition of having at least 16 samples.

[0044] In the ISP, the reconstructed sample values ​​of each sub-partition are used to generate predictions for the next sub-partition, and each sub-partition is processed repeatedly. The first sub-partition to be processed is the one containing the top-left sample of the CU. The next sub-partition is processed iteratively downwards (horizontal segmentation) or to the right (vertical segmentation). All sub-partitions share the same intra-frame mode.

[0045] Interaction between ISP and other encoding tools.

[0046] - Multiple Reference Line (MRL): If a block has an MRL index other than 0, the ISP encoding mode will be inferred to be 0, and therefore the ISP mode information will not be sent to the decoder.

[0047] -MPM usage: MPM tags will be inferred as a tag in a block encoded by ISP mode, and the MPM list is modified to exclude DC mode and prioritize ISP horizontally split intra-frame mode and vertically split intra-frame mode.

[0048] - Transform size limitation: All ISP transforms with a length greater than 16 points use DCT-II.

[0049] -MTS Flag: If the CU uses ISP encoding mode, the MTS CU flag will be set to 0 and will not be sent to the decoder. Therefore, the encoder will not perform RD tests for the different available transforms for each resulting sub-partition. The transform selection for ISP mode will instead be fixed and selected based on the intra-frame mode used, processing order, and block size. Therefore, no signaling is required.

[0050] Matrix-weighted intra-frame prediction (MIP)

[0051] In matrix-weighted intra-frame prediction (MIP), the prediction of a rectangular block of width W and height H uses H reconstructed neighboring boundary samples from the left row and W reconstructed neighboring boundary samples from the top row as input. The generation of the predicted signal is based on the following three steps: averaging the reference samples, matrix-vector multiplication of the averaged reference samples, and... Figure 3 The linear interpolation is shown. This average corresponds to filtering and downsampling of the reference sample. This produces values ​​set in a 1D vector. This vector is multiplied by a matrix selected by an index encoded in the stream, which produces a subsampled version of the prediction block. The final interpolation is performed to generate a prediction block at full resolution.

[0052] For each coding unit (CU) in intra-frame mode, an indication is sent as to whether the MIP mode should be applied to the marker on the corresponding prediction unit (PU). If the MIP mode is to be applied, it is encoded directly.

[0053] The number of supported MIP modes depends on the block size.

[0054] Interactive MIP-LFNST

[0055] For large MIP blocks, enable LFNST (Low Frequency Inseparable Transform). Here, the LFNST transform corresponding to the planar mode is used.

[0056] Triangle Prediction Model (TPM)

[0057] Triangular prediction is an inter-frame prediction that splits the prediction into two triangular partitions. For example... Figure 4As shown, the CU is divided into two triangular prediction units (PUs) along the diagonal or anti-diagonal direction. Each triangular prediction unit in the CU uses its own motion vector derived from the merged candidate list and a reference frame index to perform inter-frame prediction.

[0058] An adaptive weighting process is applied to the predicted samples at the diagonal or antidiagonal edges between two triangular prediction units to derive the final predicted sample values ​​for the entire CU, such as... Figure 5 As shown.

[0059] The triangular prediction unit mode is only applied to CUs in skip or merge modes. When the triangular prediction unit mode is applied to a CU, a signal is sent to notify the index and the motion vectors of the two triangular prediction units, which indicates the direction in which the CU is divided into two triangular prediction units (here labeled triangle_idx).

[0060] Interactive TMP-SBT

[0061] The triangular partitioning mode is not used in combination with SBT (subblock transform). That is, when the triangular mode for sending a signal is equal to 1, cu_sbt_flag (also marked as "sbt" here) is inferred to be 0 if no signal is sent.

[0062] Intra-frame and inter-frame joint prediction (CIIP)

[0063] The general concept of intra- and inter-frame joint prediction is to combine inter-frame prediction, performed in merge mode (sending a signal to the merge index to derive motion information for motion compensation prediction), with an intra-frame prediction mode or another inter-frame mode (e.g., one-way prediction AMVP, skip, or merge). The final prediction is a weighted average of the merge index prediction and the prediction generated by the intra-frame prediction mode, where different weights are applied based on the intra-frame direction and distance between the current sample and the intra-frame reference sample. Figure 6 The shape of the 4-frame intra-prediction used in multiple hypotheses is shown. (Example) Figure 6As shown, the current block is divided into four equal-area regions. The weight gradually decreases as the region moves away from the intra-frame reference sample. Each weight set is applied to the corresponding region, and each weight set is represented as (w_intrai, w_interi), where i is between 1 and 4, and (w_intra1, w_inter1) = (6, 2), (w_intra2, w_inter2) = (5, 3), (w_intra3, w_inter3) = (3, 5), and (w_intra4, w_inter4) = (2, 6). When DC or planar mode is selected, or when the CU width or height is less than 4, equal weights are applied to all samples. In intra-frame prediction in multi-hypothesis CUs, the chroma component uses the direct mode (the same frame direction as the luminance).

[0064] Interactive CIIP-Skip

[0065] CIIP cannot be used in skip mode.

[0066] Multiple Transform Selection (MTS)

[0067] Multiple Transform Selection (MTS) is used for residual coding of both inter-frame and intra-frame coded blocks. In addition to DCT-II already used in HEVC, a set of transforms is employed. The newly introduced transform matrices are DST-VII and DCT-VIII.

[0068] To control the MTS scheme, separate enabling flags are specified at the SPS level for intra-frame and inter-frame operations. When MTS is enabled at the SPS, a signal is sent to the CU-level flag to indicate whether MTS has been applied. Here, MTS applies only to lumen.

[0069] MTS restrictions

[0070] When both the block width and height are less than or equal to 32, and when the CBF flag (coded block flag, which indicates whether the block contains residual data or not) is equal to 1, a signal is sent to notify the MTS CU level flag.

[0071] If the MTS CU flag is equal to zero, DCT2 is applied in both directions. However, if the MTS CU flag is equal to 1, two additional signals are sent to notify two other flags to indicate the transformation type in the horizontal and vertical directions, respectively.

[0072] Transform selection for ISP and implicit MTS is used by removing intra-frame mode and block shape dependencies. If the current block is in ISP mode, or if the current block is an intra-frame block and both intra-frame and inter-frame explicit MTS are enabled, DST7 is used only for both the horizontal and vertical transform kernels.

[0073] Low-frequency non-separable transform (LFNST)

[0074] LFNST (Low Frequency Non-Separable Transform), also known as a reduced quadratic transform, is applied between the forward primary transform and quantization (at the encoder) and between dequantization and the inverse primary transform (at the decoder). Figure 7 As shown. In LFNST, either a 4x4 or 8x8 inseparable transformation is applied depending on the block size. For example, a 4x4 LFNST is applied to small blocks (i.e., min(width, height) < 8) and an 8x8 LFNST is applied to larger blocks (i.e., min(width, height) > 4).

[0075] The application of the inseparable transform used in LFNST is described below using the input as an example. To apply 4×4 LFNST, a 4×4 input block X...

[0076]

[0077] First, it is represented as a vector.

[0078]

[0079] The inseparable transformation is calculated as in This indicates the transformation coefficient vector, and T is a 16x16 transformation matrix. The 16x1 coefficient vector is then transformed using the scan order of this block (horizontal, vertical, or diagonal). Reorganize into 4x4 blocks. Coefficients with smaller indices will be placed in 4x4 coefficient blocks with smaller scan indices.

[0080] There are a total of 4 transform sets, and 2 inseparable transform matrices (kernels) are used for each transform set in LFNST. A predefined mapping from intra-prediction modes to transform sets is used. If one of the three Cross-Component Linear Model (CCLM) modes is used for the current block, transform set 0 is selected for the current chroma block. For each transform set, the selected inseparable quadratic transform candidate is further specified by an explicitly signaled LFNST index. Following the transform coefficients, the signaled index is sent in the bitstream once per intra-CU.

[0081] Interaction between LFNST and other tools

[0082] The LFNST index is context-coded but does not depend on the intra-prediction mode, and only the first bin is context-coded. Furthermore, LFNST is applied to intra-CUs in both intra- and inter-frame slices, and to both luma and chroma.

[0083] When ISP mode is selected, LFNST is disabled. When MIP mode is selected, LFNST is also disabled and no signal is sent to the index.

[0084] The maximum allowed size of LFNST is limited to 64x64. LFNST is only enabled with DCT2.

[0085] Subblock Transformation (SBT)

[0086] A sub-block transform is introduced for inter-frame prediction CUs. In this transform mode, only a sub-part of the residual block is encoded for each CU. When the inter-frame prediction CU with CBF equals 1, a signal can be sent to notify cu_sbt_flag (also labeled "sbt" in the described implementation) to indicate whether the entire residual block or a sub-part of the residual block was encoded. In the former case, MTS information is further parsed to determine the transform type of the CU. In the latter case, a portion of the residual block is encoded with an inferred adaptive transform, and the other portion of the residual block is zeroed out.

[0087] When SBT is used for inter-frame coding CU, signals are sent in the bitstream to notify the SBT type and SBT location information. For example... Figure 8 As indicated, there are two SBT types and two SBT locations. For SBT-V (or SBT-H), the TU width (or height) can be equal to half the CU width (or height) or 1 / 4 of the CU width (or height), resulting in a 2:2 split or a 1:3 / 3:1 split. A 2:2 split is similar to a binary tree (BT) split, while a 1:3 / 3:1 split is similar to an asymmetric binary tree (ABT) split. In an ABT split, only small regions contain non-zero residuals. If a CU's size is 8 in the luminance sample, a 1:3 / 3:1 split along that size is not allowed. A CU can have a maximum of 8 SBT modes.

[0088] Interactive SBT-TMP

[0089] SBT should not be applied to CUs encoded using intra-inter joint mode or TPM (triangle partitioning mode).

[0090] Switchable Filter

[0091] The principle of switchable interpolation filters (IF) is to improve motion compensation prediction by selecting the IF(ifIdx) used for prediction of each block.

[0092] IF can often differ from smoothing characteristics. For example, in the Universal Video Coding Proposal, the IF index (hpelIfIdx) can be selected based on the coding unit (CU) and is derived from the “amvr_precision_idx” index of the code that indicates the resolution of the motion vector difference (MVD) of the code: if IMV = HALF_PEL, then IF-idx = 1 is selected, otherwise IF-idx = 0 is selected.

[0093] In the case of merging, the IF index is not explicitly encoded but derived from the merge candidates. In VVC, the IF index value can be one of two filters (IF-0 or IF-1), but IF-1 can only be used for HALF_PEL motion vector values. Then, if the IF index is not equal to zero and the MV horizontal (or vertical) component is not HALF_PEL, IF-0 is used. Figure 10 ).

[0094] Encoding Unit Syntax Tree

[0095] Figure 11 The syntax tree of the coding unit is shown below. The tree design is based on the values ​​of different parameters that control the coding mode. A brief explanation of these parameters is given below.

[0096] Use three main Boolean parameters:

[0097] ●“skip”—A Boolean parameter indicating whether the block is encoded in split mode—In skip mode, the residual prediction signal is not encoded.

[0098] ● "merge" — A Boolean parameter indicating whether the block is encoded in inter-frame prediction with merging mode — In merging mode, motion vectors are derived from a list of candidate motion vectors, and motion vector differences are not encoded.

[0099] ●“intra” — A Boolean parameter indicating whether the block is coded in intra-prediction mode.

[0100] Based on the values ​​of these parameters, the encoded block may have four encoding options, which correspond to the four main branches of the decision tree.

[0101] Option 1—Inter-frame prediction with skip mode

[0102] When skipping is enabled, skip mode is used. Skip mode is an inter-frame prediction mode in which the residual prediction signal and the motion vector prediction difference are not encoded (except in mmVD mode where corrections can be added). Motion data (motion vectors, reference index) is predicted from a list of motion vector candidates, and motion data predictors are identified by an index (labeled here as mergeidx).

[0103] The following parameters can be used.

[0104] ● "subblock" — A Boolean parameter indicating whether to derive the block's motion vector for each subblock (typically a 4x4 block).

[0105] ●“regular merge”—A Boolean parameter indicating whether motion vector derivation will be based on the merging mode.

[0106] ● "subblock idx" — The parameter corresponding to the index of the subblock candidate (any ATMVP of the affine candidate).

[0107] ●“triangle idx” — Parameter corresponding to the type of triangle prediction pattern and the index of the candidates.

[0108] ● "mmvd" — A Boolean parameter indicating whether a merging mode with motion vector difference (MMVD) is selected.

[0109] ●“merge idx” — The parameter corresponding to the index of the predicted motion vector in the motion vector candidate list of the merge mode.

[0110] ●“mmvd idx” — The parameter corresponding to the index of the MMVD motion vector prediction.

[0111] Option 2—Intra-frame Prediction Case

[0112] Intra-frame prediction is activated when skipping is disabled and the "intra" parameter is available. The following parameters can be used when intra-frame prediction is applied:

[0113] ●“mip” — A Boolean parameter indicating whether the block is encoded in Matrix Weighted Intra Prediction (MIP) mode.

[0114] ●“mrl” — A Boolean parameter indicating whether the block is coded in Multi-Reference Line (MRL) intra-prediction mode.

[0115] ● "isp" — A Boolean parameter indicating whether the block is encoded in Intra-Segment Prediction (ISP) mode.

[0116] ●“mip_mode” — A parameter that provides the value of the MIP intra-prediction submode.

[0117] ● "mrl_idx" — A parameter indicating the line to be used for intra-prediction in MRL intra-prediction mode.

[0118] ● "isp_mode" — A parameter indicating the type of ISP prediction.

[0119] ●“mpm,mpm1 to mpm4” — Tags related to intra-frame pattern derivation of Maximum Probability Mode (MPM).

[0120] ●“planar” — A marker indicating whether the intra-frame mode is planar mode.

[0121] ●“intradir” — Specifies the parameter for the intra-frame prediction direction.

[0122] Option 3—Inter-frame prediction with merging mode

[0123] When "Skip" and "Intra-frame" are disabled and the "Merge" parameter is available, inter-frame prediction with merge mode is activated. Merge mode is an inter-frame prediction mode where motion vectors are derived from a list of candidate motion vectors and motion vector differences are not encoded (except in MMVD mode where corrections can be added). The index is encoded to indicate the candidate motion vectors used as predictors (here denoted as merge idx).

[0124] The following parameters can be used:

[0125] ● Use the same parameters as in the skip case: "subblock", "regularmerge", "subblockidx", "triangle idx", "mmvd", "mergeidx", "mmvd idx"

[0126] ● Additionally, one parameter is limited to merged cases:

[0127] ○ "ciip" — A Boolean parameter indicating whether the Inter-Frame / Intra-Frame Joint Prediction (CIIP) mode is selected.

[0128] Option 4—Inter-frame prediction with AMVP mode

[0129] When "Skip," "Intra-Frame," and "Merge" are disabled, inter-frame prediction with Advanced Motion Vector Prediction (AMVP) is used. AMVP is a general concept for motion prediction that involves performing motion vector prediction based on a list of motion vector candidates. To identify prediction factors, a signal is sent to notify the index (represented here as mvp_idx). In this case, residual prediction data and motion vector difference information are encoded.

[0130] When intra-frame prediction is applied, the following parameters can be used:

[0131] ●“dir”—A parameter indicating the type of inter-frame prediction (one-way or two-way) prediction.

[0132] ●“affine” — A parameter indicating whether the motion data in the block belongs to the affine or translation type.

[0133] ● "ref_idx0 / 1" — A parameter indicating the index of the reference image used for time prediction from the list of reference images (L0 / L1).

[0134] ●“mvd0 / 1” — A parameter related to the motion vector difference to be added to the motion vector prediction.

[0135] ● "mvp_idx0 / 1" — A parameter indicating the index of the motion vector predictor in the motion vector candidate list.

[0136] ● "amvr" is a Boolean parameter indicating whether adaptive motion vector resolution (AMVR) is used for motion compensation.

[0137] ● "gbi" (aka bcw) is a Boolean parameter indicating whether the Generalized Bidirectional Prediction (GBI) mode is used.

[0138] grammar

[0139] The table below shows the encoding units and merging syntax.

[0140] It can be noted that some modes are exclusive in the codec, and some tool combinations are disabled.

[0141] Examples can be given as follows:

[0142] -MRL (associated syntax element intra_luma_ref_idx) and ISP (associated syntax element intra_subpartitions_mode_flag)

[0143] -CIIP (associated syntax element ciip_flag) and skip mode (associated syntax element cu_skip_flag)

[0144] -TPM (because it is a leaf of the syntax tree, it has no associated syntax elements; TPM can be activated if all previously parsed merge patterns are false) and SBT (associated syntax element cu_sbt_flag)

[0145] -CIIP and SBT

[0146] -(the associated syntax element lfnst_idx) and ISP

[0147] - When the block height or width is less than 16, LFNST applies only to MIPs (the associated syntax element intra_mip_flag).

[0148] -LFNST and MTS (associated syntax element tu_mts_idx)

[0149] -TPM and AMVP (the associated syntax element general_merge_flag is false)

[0150] -CIIP and AMVP

[0151] Encoding unit syntax

[0152] According to VVC:

[0153]

[0154]

[0155]

[0156]

[0157]

[0158]

[0159] Merge data syntax According to VVC section 7.3.8.7:

[0160]

[0161] Inference pattern

[0162] In addition to these pattern exclusions, some patterns or values ​​are also automatically inferred in the encoder and decoder.

[0163] For example:

[0164] - In IBC mode, the half-sample interpolation filter index hpelIfIdx is set to 0. Otherwise, hpelIfIdx is derived from AMVR (Adaptive Motion Vector Resolution, amvr_resolution_idx). If amvr_resolution_idx equals half-pixel resolution, hpelIfIdx is set to 1; otherwise, it is set to 0. If the final (reconstructed) motion vector is a half-pixel position and hpelIfIdx equals 1, the interpolation filter IF-1 is used for motion compensation; otherwise, the interpolation filter (IF-0, the default interpolation filter) is used.

[0165] hpelIfIdx=(amvr_resolution_idx==HalfPelResolution)? 1:0

[0166] - MRL and non-MPM (maximum probability mode) modes: If the MRL index is non-zero, the inferred mode is in the MPM list (arbitrary intra-frame mode cannot be set).

[0167] -CIIP Intra-Frame Mode: Intra-frame portions always use planar mode (arbitrary intra-frame mode cannot be set).

[0168] - When using ISP or SBT, MTS (Multi-Transform Set) is reduced.

[0169] - Merge index in the case of MMVD (Merge Motion Vector Difference): Only index 0 or 1 can be set, corresponding to the first merge candidate or the second merge candidate. Other merge candidates cannot be set.

[0170] Entropy coding using CABAC

[0171] In HEVC and VVC, the context-based binary arithmetic encoder (CABAC) can be used for entropy coding. Another coding mode exists, known as "bypass" coding, which is not discussed here. Bypass modes for encoding bits involve simply encoding the bit values ​​in the bitstream.

[0172] CABAC performs the following operations for encoding / decoding syntax elements:

[0173] ● First, the syntax elements are binarized into a string of binaries;

[0174] ●Then for each bin

[0175] ○ Select context CTX; it is associated with a context model of the context probability of bin stored as "1" or "0"; this context probability corresponds to the estimated conditional probability P(b / CTX) that bin adopts value b given the context value CTX.

[0176] ■ This selection is based on contextual information, such as the encoded symbols (syntax elements) in the neighborhood of the current symbol (syntax element) to be encoded, the size or shape of the CU / PU / TU, and the slice type.

[0177] ○ Use an arithmetic encoder to encode / decode the bin based on the contextual probability P(b / CTX);

[0178] Once encoded, update the context probability P0 = P(0 / CTX) or P1 = P(1 / CTX).

[0179] In an arithmetic encoder, the value of probability P0 / P1 (and its initial value) has a significant impact on the number of bits generated. For example, if bin 1 is encoded with a low probability P1, more bits will be required compared to a high probability P1.

[0180] In the above scenario, although the encoder complexity is reduced by decreasing the number of combinations between modes (when two modes can only be used exclusively, the encoder only needs to test the use of each mode individually; there is no testing of the two combined modes), the operation may affect the original performance of the encoder and decoder by removing some flexibility.

[0181] The described implementation scheme suggests using a combination of syntax changes, ad-hoc entropy context, entropy initial probability parameters, and encoding selection to transform the aforementioned hard constraints into softer constraints.

[0182] As mentioned above, undesirable complexity / gain trade-offs or mode choices are canonically removed from encoding choices.

[0183] The described aspects offer a way to increase encoder-decoder flexibility by maintaining the same performance (i.e., the same complexity / gain tradeoff) when using low-complexity encoders, or by implementing these patterns for higher-complexity encoders using adaptive grammars, ad-hoc contexts, CABAC parameters, and encoding methods. In this case, the encoder is more complex, but the expected gain is higher. This could be useful, for example, for non-real-time encoders, such as in video-on-demand applications where the video stream is encoded once without the constraint of real-time delivery.

[0184] It is recommended to use the following combination to transform the above-mentioned hard constraints in the specification into softer constraints.

[0185] ●Syntax changes (e.g., removing or changing syntax elements based on another parameter)

[0186] ●ad-hoc entropy context,

[0187] ● Initial probability parameter of entropy

[0188] ● Encoding selection.

[0189] Exclusive pattern

[0190] exist Figure 14 and Figure 15 The diagram illustrates the overall change in the process of converting from hard exclusivity to soft exclusivity using two encoding tools M and N. Encoding tool N becomes available even when encoding tool M is activated with the associated probability model P1 (see below for details).

[0191] Figure 14 The example of "pattern exclusivity" is illustrated. This corresponds to a prior art where both patterns M and N are exclusive. Step 300 corresponds to the start of the entire process. In step 301, the activation of tool M is checked (e.g., by a test marker).

[0192] If tool M is activated, tool M is executed in step 302. The process then proceeds to step 305, which is a continuation of the entire process.

[0193] If tool M is not activated, then check the activation of tool N (step 303). Define the probability P0 of activating tool N. If tool N is activated, then execute tool N in step 304. Then, the process proceeds to step 305, which is a continuation of the entire process. If tool N is not activated, the process proceeds to step 305, which is a continuation of the entire process.

[0194] In this case, tool N can only be activated if tool M is not activated. The use of modes M and N is exclusive.

[0195] Figure 15 An example of a "non-exclusive mode" is shown. This corresponds to one aspect of the described implementation scheme, where modes M and N can be used non-exclusively. Step 400 corresponds to the start of the entire process. In step 401, the activation of tool M is checked.

[0196] If tool M is activated, tool M is executed in step 402. Then, in step 403, the activation of tool N is checked. The probability P1 of activating tool N is defined. If tool N is activated, tool N is executed in step 404. Then, the process proceeds to step 406, which is a continuation of the entire process. If tool N is not activated, the process proceeds to step 406, which is a continuation of the entire process.

[0197] If tool M is not activated, then check the activation of tool N (step 405). Define the probability P0 of activating tool N. If tool N is activated, then execute tool N in step 403. Then, the process proceeds to step 406, which is a continuation of the entire process. If tool N is not activated, the process proceeds to step 406, which is a continuation of the entire process.

[0198] In this scenario, tool N can be activated in two ways: tool M is activated, or tool M is not activated. The use of modes M and N is not exclusive.

[0199] Pattern Inference

[0200] exist Figure 16 and Figure 17The diagram illustrates the overall transformation from the inferred value using associated probabilities to the value sent as a signal notification (details below). Two tools, M and N, are considered here, with tool N controlled by parameters.

[0201] Figure 16 The example of "value inference" is shown. It corresponds to the prior art in which the parameters of tool N (uncoded) are inferred. Step 500 corresponds to the beginning of the entire process. In step 501, the activation of tool M is checked.

[0202] If tool M is activated, the activation condition of tool N is checked in step 502. If the activation condition of tool N is true, tool N is executed with parameter value V1 in step 503. Then, the process proceeds to step 505, which is a continuation of the entire process. If the activation condition of tool N is false, tool N is executed with parameter value V0 in step 504. Then, the process proceeds to step 505, which is a continuation of the entire process. The tool M process is not shown in the figure and is executed before or after tool N.

[0203] In this case, the parameters associated with tool N are continuously inferred.

[0204] Figure 17 The example of "sending signal notification parameters" is shown. It corresponds to one aspect of the described implementation, where the parameters of tool N are explicitly sent as a signal notification. Step 600 corresponds to the start of the entire process. In step 601, the activation of tool M is checked.

[0205] If tool M is activated, the activation condition of tool N is checked in step 602. If the activation condition N is true, the parameter value of tool N is checked (and possibly decoded from the stream). If the parameter is v0, tool N is activated with the parameter value v0 in step 605. The process then proceeds to step 606, which is a continuation of the entire process. Otherwise, tool N is activated with the parameter value v1 in step 604. The process then proceeds to step 606, which is a continuation of the entire process.

[0206] If tool M is not activated, tool N is activated with parameter value v0 in step 605. The process then proceeds to step 606, which is a continuation of the entire process. The tool M process is not shown in the figure and is executed before or after tool N.

[0207] In this case, the parameters associated with tool N are not inferred, but the parameters are decoded or derived from the specific conditions applied to tool N.

[0208] Syntax changes

[0209] The first change is a change in syntax covering pattern exclusivity and / or parameter inference.

[0210] Regarding pattern exclusivity, pattern exclusivity is removed. The activation of pattern N no longer depends on the activation of pattern M. The activation condition of tool N is transformed into a new adhoc CABAC context.

[0211] Regarding the inferred value, a new syntax element is introduced to explicitly encode the value. This new syntax element replaces parameter inference with one or more new adhoc CABAC contexts.

[0212] In the variant, the mode exclusivity aspect (removing mode exclusivity) and / or the inferred value aspect (new syntax elements are used to explicitly encode the value) are controlled / activated by conditions of encoded parameters (which are preferably encoded in the high-level syntax, such as in the slice header, picture parameter set (PPS), sequence parameter set (SPS), adaptive parameter set (APS), or decode parameter set).

[0213] CABAC context change

[0214] Exclusive pattern

[0215] Since the entropy encoder (in this case, CABAC) is used to encode the use of tool N, a different bin is used when tool M is activated. A new CABAC context is needed to separate the cases where pattern M is activated or not activated. A condition for pattern M activation is added to the CABAC context derivation for the syntax elements associated with pattern N. The encoding of the first bin (when pattern M is not activated) remains unchanged and uses the same probability model P0 (initialized to the same value as CABACCABAC).

[0216] Value inference

[0217] Since the entropy encoder (in this case, CABAC) is used to encode the value V0 or V1 associated with the tool N, the probability model P1 (equal to V0 or V1) associated with the encoding of the value V is set such that it has a high probability of being V1 (the previously inferred value).

[0218] CABAC initial value adaptive

[0219] Exclusive pattern

[0220] The following describes how to set up the new probabilistic model P1. This bin is initialized with low probabilities so as not to affect the encoder's encoding performance, and the encoder always excludes tool N when tool M is in use.

[0221] In practice, if a mode is rarely used but has a high probability of associated use in a CABAC model, many bits will be used to signal that the mode is not being used. On the other hand, if the probability is low, not using the mode will have no or minimal impact on the rate compared to a codec where the mode is unavailable.

[0222] The following text explains the principle behind the changes in the CABAC initialization process.

[0223] In VVC, the initial probability values ​​of the CABAC model are given by the linear model according to the QP (quantization parameter), which is controlled by two parameters, a (product factor or slope) and b (offset). Both the slope a and the offset b are hard-coded in the specification for a given bin corresponding to the syntax element. Since CABAC updates the probability at each bin read, two "window sizes" are also used to update the symbol probability. Depending on the window size, the probability update will be fast or slow.

[0224] For example, the CIIP labeling parameters are as follows: a = 0.024 and b = 0.29 (for clarity, the coefficients are expressed as floating-point values). Then, the initial probabilities of CABAC given by QP are used:

[0225] proba(QP)=(0.024 / 2)*(QP-16)+0.29

[0226] In fact, for a QP of 34, the probability is approximately 0.5.

[0227] To avoid penalizing low-complexity encoders that won't test mode N when mode M is activated, it's recommended to initialize the CABAC probabilities with low-probability models where a and b are small. Therefore, the encoding of bins indicating unused mode N (e.g., bin0) will require very few bits, since the probability of bin1 is low, and conversely, the probability of bin0 is high. For example: a = 0 and b = 0.008. In the current VTM-6.0, the current slope and offset encoding is used, corresponding to an initialization parameter of 32 for CABAC. Other values ​​that maintain low initial probabilities can be used. For example, maintaining the initial probability function of QP: a = 0.0117, b = 0.008. In the current VTM-6.0, the current slope and offset encoding is used, corresponding to an initialization parameter of 40 for CABAC.

[0228] By using these initial probabilistic models, a low-complexity encoder that does not execute mode N when N is activated (such as the low-complexity encoder described in VVC) does not suffer performance loss (performance remains roughly the same). However, more complex encoders can utilize new available modes if needed. In fact, such an encoder can test more combinations of modes, which is beneficial in terms of compression gain in most cases.

[0229] In the variant, the window size associated with the new syntax element is shortened to adapt to the probabilistic model more quickly (see below for details).

[0230] Value inference

[0231] The principle of probability initialization is the same as above: low probabilities are set to values ​​that were not previously inferred.

[0232] Variant

[0233] When the logic used for newly introduced patterns or non-inferred values ​​is inverted, the initial probability value is set to a value close to 1 (instead of close to 0).

[0234] Encoding method

[0235] To improve the performance of highly complex encoders, the probability of using newly introduced mode combinations / values ​​needs to be adjusted to its correct expected value. However, since the probability model is fixed for the encoder / decoder (see parameters a, b, and window size above), the initial cost of selecting a new mode may be too high for the new mode to be selected. When evaluating the cost of activating mode N, its cost is high due to its low probability, thus mode N is strongly penalized, and therefore the probability of mode N being selected is very small. In this case, since mode N is rarely selected, its probability of being used remains low, and the likelihood of it being used at all is very small.

[0236] To change the probability associated with a mode without altering its initial value, possible encoding methods include: calculating the bias RD (rate distortion) cost of mode N when mode M is activated so that mode N is used in the first few blocks.

[0237] C = D + λ(R - α)

[0238] In typical RD cost estimation, D is distortion, R is rate, and λ is the rate-distortion parameter. α is the fractional bit cost (positive) calculated to favor mode N when mode M is activated. For example, the factor α can be precisely calculated, such as (R-α) equaling 1 bit, which corresponds to a probability of 0.5 for the current encoded bin. Therefore, even if the probability of mode N is low, and the encoder can choose mode N without penalty as long as it provides a better distortion value than when not using the mode. When processing consecutive blocks, the probabilistic model gradually converges to the actual probability of using mode N.

[0239] After a sufficient number of blocks have been processed, i.e., when the probabilistic model converges to a good estimate of the probability of the true mode using the data, the factor α is reduced to 0 so that the true rate cost R can be used in the RD cost equation. The stability of the probabilistic model can be estimated by evaluating the change in probability values ​​P0 or P1 while bins are being encoded. If the change is below a given threshold (e.g., 0.01), the model is considered stable and α can begin to decrease. Alternatively, the reduction of α can be determined once a given number of blocks (e.g., 20) have been processed.

[0240] The number of blocks that adapt depends on the fixed window size in the encoder / decoder. The shorter the window size, the faster the adaptation.

[0241] Examples of hard-to-soft mode change

[0242] CIIP mode in skip mode

[0243] Here, mode M is skip (i.e., merge without residuals), and N is CIIP mode.

[0244] The first change involves the syntax in the coding unit data: the cu_skip_flag for enabling CIIP no longer depends on cu_skip_flag (which indicates skip mode activation). Furthermore, the conditions for reading or not reading the regular merge mode flags have also been adapted to avoid useless reads in the case of inactive CIIP and triangles.

[0245]

[0246]

[0247] CABAC Adaptive

[0248] One implementation adds a new context based on the skip flag (cu_skip_flag). If the skip flag is 0, CABAC encoding is performed as previously described. If the skip flag is 1, a separate bin is used to encode "ciip_flag" (see [link to implementation]). Figure 18 ).

[0249] In step 700, check cu_skip_flag. If cu_skip_flag equals 0, then bin1 is used with the initial probability of a low value (step 701). Otherwise, bin 0 is used with a CABAC model (including the initial probability) similar to cases that do not have the aspects described herein (step 702). CABAC decoding is then applied in step 703.

[0250] MRL and ISP

[0251] Here, M represents the MRL tool and N represents the ISP tool.

[0252]

[0253] ISP mode is allowed even when the MRL index is not 0.

[0254] CABAC Adaptive

[0255] A new context, dependent on the MRL index (intra_luma_ref_idx), is introduced to encode the intra_subpartitions_mode_flag. Similarly, when the MRL index is 0, the same CABAC process described earlier is performed; when it is not 0, a new bin with a low initial probability is used to encode the flag. The low-probability setting process is similar to that described previously.

[0256] TPM and SBT

[0257] Here, M represents the TPM tool and N represents the SBT tool. SBT mode is also allowed even when TPM mode is activated.

[0258]

[0259] CABAC Adaptive

[0260] A new context, dependent on the TPM mode (MergeTriangleFlag), is introduced to encode cu_sbt_flag. Similarly, when the MergeTriangleFlag is 0, the same CABAC process described earlier is performed; when it is not 0, a new bin with a low initial probability is used to encode that flag. The low-probability setting process is similar to the process described above.

[0261] CIIP and SBT

[0262] Here, M represents the CIIP tool, and N represents the SBT tool. SBT mode is also allowed even when CIIP mode is activated.

[0263]

[0264] CABAC Adaptive

[0265] A new context, dependent on the CIIP mode (ciip_flag), is introduced to encode cu_sbt_flag. When ciip_flag is 0, the same CABAC process as described above is performed; when it is not 0, a new bin with a low initial probability is used to encode the flag. The low-probability setting process is similar to that described earlier.

[0266] LFNST (Low-Frequency Non-Separable Transformer) and ISP

[0267] Here, M represents the ISP tool, and N represents the LFNST tool. LFNST mode is also allowed even when ISP mode is activated.

[0268]

[0269] CABAC Adaptive

[0270] A new context, dependent on the ISP mode (IntraSubPartitionsSplitType), is introduced to encode lfnst_idx. When IntraSubPartitionsSplitType is ISP_NO_SPLIT, the same CABAC process as described above is performed; when it is not ISP_NO_SPLIT, a new bin with a low initial probability is used to encode the tag. The low-probability setup process is similar to the process described earlier.

[0271] When the block height or width is less than 16, LFNST is only applicable to MIP (Matrix Intra Prediction).

[0272] Here, M represents the MIP tool and N represents the LFNST tool. LFNST mode is allowed even when MIP mode is activated and the block size (height or width) is less than 16.

[0273]

[0274] CABAC Adaptive

[0275] A new context, dependent on the MIP mode (intra_mip_flag) and block size, is introduced to encode lfnst_idx. When intra_mip_flag is 0 or the width and height are greater than 16, the same CABAC process as described earlier is performed; otherwise, a new bin with a low initial probability is used to encode the flag. The low-probability setting process is similar to that described previously.

[0276] Switchable Filter

[0277] For inferred patterns or indexes, add additional syntax elements to explicitly encode the pattern or index.

[0278] Derive the half-sample interpolation filter index hpelIfIdx from AMVR (Adaptive Motion Vector Resolution) mode (which is half-pixel): In this case, the index 0 or 1 is encoded to indicate which interpolation filter to use.

[0279]

[0280] In Intra-Block Copy (IBC) mode, the value of hpelIfIdx is set to 0.

[0281] During the decoding process, if the syntax element hpelIfIdx does not exist (e.g., in merge mode), the value of hpelIfIdx is inferred as: (amvr_precision_idx[x0][y0]====HalfPelIdx)? IF-1:IF-0.

[0282] In the variant, the encoding condition for hpelIfIdx is the high-level parameter "enabled_interpolation_filter_flag". If enabled_interpolation_filter_flag = 0, the syntax element does not exist and is inferred (e.g., inferred as the default such as IF-0, or inferred as "amvr_precision_idx[x0][y0] == HalfPelIdx").

[0283]

[0284] In another variant, the encoding condition for hpelIfIdx is the high-level parameter "enabled_interpolation_filter_flag", but not the value of amvr_precision_idx.

[0285]

[0286] CABAC Adaptive

[0287] The new syntax element hpelIfIdx uses CABAC encoding with a high initial probability. The high probability setting process is similar to that described earlier.

[0288] In the variant, the initial probability depends on the value of "enabled_interpolation_filter_flag".

[0289] Non-MPM with MRL

[0290] When the MRL index is not 0, the intra-frame mode is allowed to be non-mpm, so the intra_luma_mpm_flag is read in all cases.

[0291]

[0292] CABAC Adaptive

[0293] A new context, dependent on the MRL index, is introduced to encode the intra_luma_mpm_flag. When the MRL index is 0, the same CABAC process as described above is performed; when it is not 0, a new bin with a low initial probability is used to encode the flag.

[0294] Non-planar modes in CIIP

[0295] When CIIP is activated, all intra-frame modes are allowed. Therefore, when the ciip_flag is set to true, the same syntax as in intra-mode parsing is executed: parsing MIP modes, MRL, MPM, etc.

[0296]

[0297] CABAC Adaptive

[0298] When ciip_flag is true, all of the following flags with the following adaptations obtain the new context bin:

[0299] -intra_mip_flag: The initial probability is set to a low probability.

[0300] -intra_luma_ref_idx: The initial probability is set to high probability with index 0.

[0301] -intra_luma_mpm_flag: The initial probability is set to high probability.

[0302] -intra_luma_not_planar_flag: The initial probability is set to low.

[0303] Merged Index of MMVD

[0304]

[0305] Encode mmvd_merge_idx to allow setting any merge candidate in the list.

[0306] In the variant, the encoding condition for “mmvd_merge_idx” is the high-level parameter “enabled_mmvd_merge_idx_flag”. If enabled_mmvd_merge_idx_flag = 0, then “mmvd_merge_idx_minus1” does not exist and is inferred as (mmvd_cancer_flag[X0][Y0]?1:0).

[0307]

[0308] CABAC Adaptive

[0309] This adaptive approach uses the same method as merge_idx, except that the first two bits are CABAC encoded.

[0310] The first bit is CABAC encoded with an initial probability similar to that of the original mmvd_cand_flag. The second bit is CABAC encoded with a low initial probability (meaning that if the index is not 0, then the index has a higher probability of being 1). The remaining bits are bypassed and encoded in the same way as during the index merging process.

[0311] This document describes multiple aspects, including tools, features, implementation schemes, models, methods, etc. Many of these aspects are described in detail, and often in a way that may sound limited, at least to illustrate individual characteristics. However, this is for clarity and does not limit the application or scope of these aspects. In fact, all the different aspects can be combined and interchanged to provide further aspects. Furthermore, these aspects can also be combined and interchanged with those described in previous submissions.

[0312] The aspects described and envisioned in this document can be implemented in many different forms. The following... Figure 12 , Figure 13 and Figure 19 Some implementation schemes are provided, but other implementation schemes are envisioned, and Figure 2 , Figure 3 and Figure 9The discussion does not limit the breadth of specific implementations. At least one of these aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a bitstream generated or encoded. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions stored thereon for encoding or decoding video data according to any of the methods, and / or computer-readable storage media having a bitstream generated according to any of the methods stored thereon.

[0313] In this application, the terms "reconstruction" and "decoding" are used interchangeably, as are the terms "pixel" and "sample," and the terms "image," "picture," and "frame." Generally, but not necessarily, the term "reconstruction" is used at the encoding end, while "decoding" is used at the decoding end.

[0314] This document describes various methods, and each method includes one or more steps or actions for implementing the method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or purpose of a particular step and / or action may be modified or combined.

[0315] The various methods and other aspects described in this document can be used to modify modules of the video encoder 100 and decoder 200 (e.g., intra-frame prediction, entropy coding, and / or decoding modules (160, 360, 145, 330)). Figure 12 and Figure 13 As shown. Furthermore, aspects of the invention are not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations (whether pre-existing or developed in the future) and any extensions of such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this document may be used alone or in combination.

[0316] Various numerical values ​​are used in this document, such as {{1,0},{3,1},{1,1}}. Specific values ​​are for illustrative purposes only, and the aspects described are not limited to these specific values.

[0317] Figure 12 Encoder 100 is shown. Variations of this encoder 100 are envisioned, but for clarity, encoder 100 is described below without describing all anticipated variations.

[0318] Before encoding, the video sequence may undergo pre-coding (101), for example, by applying color transformations to the input color image (e.g., a conversion from RGB 4:4:4 to YCbCr 4:2:0), or by performing remapping of the input image components to obtain a more resilient signal distribution to compression (e.g., histogram equalization using one of the color components). Metadata may be associated with the pre-processing and attached to the bitstream.

[0319] In encoder 100, the image is encoded by encoder elements as described below. The image to be encoded is partitioned (102) and processed in units such as CUs. For example, each unit is encoded using either an intra-frame mode or an inter-frame mode. When a unit is encoded in intra-frame mode, it performs intra-frame prediction (160). In inter-frame mode, motion estimation (175) and compensation (170) are performed. The encoder determines (105) which of the intra-frame mode or inter-frame mode is used to encode the unit and indicates the intra-frame / inter-frame decision by, for example, a prediction mode flag. For example, the prediction residual is calculated by subtracting (110) the prediction block from the original image block.

[0320] The predicted residual is then transformed (125) and quantized (130). The quantized transform coefficients, motion vectors, and other syntax elements are entropy encoded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., encode the residual directly without applying the transform or quantization process.

[0321] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. A loop filter (165) is applied to the reconstructed image to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coded artifacts. The filtered image is stored in a reference image buffer (180).

[0322] Figure 13 A block diagram of a video decoder 200 is shown. In decoder 200, the bitstream is decoded by decoder elements, as described below. Video decoder 200 generally performs operations similar to... Figure 12 The encoding process is the opposite of the decoding process. Encoder 100 typically also performs video decoding as part of the video data encoding.

[0323] The decoder's input includes a video bitstream, which can be generated by the video encoder 100. First, entropy decoding (230) is performed on the bitstream to obtain transform coefficients, motion vectors, and other encoded information. Image partitioning information indicates how the image should be partitioned. Therefore, the decoder can partition (235) the image based on the decoded image partitioning information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residuals. The decoded prediction residuals and prediction blocks are combined (255) to reconstruct the image blocks. Prediction blocks (270) can be obtained from intra-frame prediction (260) or motion-compensated prediction (i.e., inter-frame prediction) (275). A loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference image buffer (280).

[0324] The decoded image can also undergo post-decoding processing (285), such as inverse color transformation (e.g., a transformation from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping of the remapping process performed in the pre-encoding process (101). Post-decoding processing can utilize metadata derived in the pre-encoding process and signaled in the bitstream.

[0325] Figure 19 A block diagram illustrating examples of systems implementing various aspects and embodiments therein is provided. System 1000 may be embodied as a device including the various components described below and configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 1000 may be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to other similar systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects described in this document.

[0326] System 1000 includes at least one processor 1010 configured to execute instructions loaded thereon for implementing various aspects, such as those described in this document. Processor 1010 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). System 1000 includes a storage device 1040 that may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 1040 may include internal storage devices, attached storage devices, and / or network-accessible storage devices.

[0327] System 1000 includes an encoder / decoder module 1030 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module that can be included in a device to perform encoding and / or decoding functions. It is well known that a device may include one or both of an encoding module and a decoding module. Furthermore, the encoder / decoder module 1030 may be implemented as a standalone element of system 1000, or may be incorporated within processor 1010 as a combination of hardware and software known to those skilled in the art.

[0328] Program code to be loaded onto processor 1010 or encoder / decoder 1030 to execute the various aspects described in this document may be stored in storage device 1040 and subsequently loaded onto memory 1020 for execution by processor 1010. According to various embodiments, one or more of processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 may store one or more items from various categories during the execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded or partially decoded video, bitstreams, matrices, variables, and intermediate or final results of processing equations, formulas, operations, and operational logic.

[0329] In several embodiments, the memory within processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be processor 1010 or encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, fast external volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as for MPEG-2, HEVC, or VVC (Variety Video Coding).

[0330] Inputs to the components of system 1000 may be provided via various input devices as shown in box 1130. Such input devices include, but are not limited to: (i) receiving, for example, inputs from a broadcaster via the air.

[0331] The RF section of the transmitted RF signal, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0332] In various embodiments, the input device of block 1130 has associated corresponding input processing elements as known in the art. For example, the RF section may be associated with elements required for: (i) selecting a desired frequency (also known as selecting a signal, or limiting a signal band to a band), (ii) down-converting the selected signal, (iii) further band-limiting to a narrower band to select (e.g.,) a signal band that may be referred to as a channel in some embodiments), (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements for performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include tuners that perform various of these functions, including, for example, down-converting received signals to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or to baseband. In one set-top box implementation, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to the desired frequency band. Various implementations rearrange the order of the aforementioned (and other) components, remove some of these components, and / or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as inserting amplifiers and analog-to-digital converters. In various implementations, the RF section includes an antenna.

[0333] Furthermore, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented as needed, for example, within a separate input processing IC or within the processor 1010. Similarly, various aspects of USB or HDMI interface processing may be implemented as needed, either within a separate interface IC or within the processor 1010. Demodulation, error correction, and demultiplexing streams are provided to various processing elements, including, for example, the processor 1010, and an encoder / decoder 1030 that operates in conjunction with memory and storage elements to process the data streams as needed for presentation on the output device.

[0334] Various components of system 1000 can be housed within an integrated housing. Within the integrated housing, various components can be interconnected and transmit data therebetween using a suitable connection arrangement 1140 (e.g., internal buses as known in the art, including inter-IC (I2C) buses, wiring, and printed circuit boards).

[0335] System 1000 includes a communication interface 1050 capable of communicating with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 1060 may be implemented, for example, in a wired and / or wireless medium.

[0336] In various implementations, data is streamed to system 1000 using a wireless network such as IEEE 802.11. For example, the wireless signals in these implementations are received via a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 in these implementations is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other cloud-based communications. Other implementations use a set-top box to provide streaming data to system 1000, delivering data via an HDMI connection to input block 1130. Still other implementations use an RF connection to input block 1130 to provide streaming data to system 1000.

[0337] System 1000 can provide output signals to various output devices, including display 1100, speaker 1110, and other peripheral devices 1120. In various embodiments, other peripheral devices 1120 include one or more of a standalone DVR, disk player, stereo system, lighting system, and other devices that provide output functionality based on system 1000. In various embodiments, control signals are transmitted between system 1000 and display 1100, speaker 1110, or other peripheral devices 1120 using signaling such as AV.Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. Output devices can be communicatively coupled to system 1000 via dedicated connections through corresponding interfaces 1070, 1080, and 1090. Alternatively, output devices can be connected to system 1000 via communication interface 1050 using communication channel 1060. Display 1100 and speaker 1110 can be integrated into a single unit with other components of system 1000 in electronic devices (e.g., television sets). In various implementations, the display interface 1070 includes a display driver, such as a timing controller (T Con) chip.

[0338] Alternatively, if the RF portion of input 1130 is part of a standalone set-top box, the display 1100 and speaker 1110 may be separate from one or more other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0339] These implementations may be executed by computer software implemented by processor 1010, or by hardware, or a combination of hardware and software. As a non-limiting example, these implementations may be implemented by one or more integrated circuits. As a non-limiting example, memory 1020 may be of any type suitable for the technical environment and may be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. Processor 1010 may be of any type suitable for the technical environment and, as a non-limiting example, may encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0340] Various specific implementations involve decoding. As used in this application, "decoding" may include all or part of a process, for example, performing on a received coded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also or alternatively includes processes performed by a decoder of various embodiments described in this application, such as extracting weight indices to be used for various intra-frame prediction reference arrays.

[0341] As a further example, in one embodiment, "decoding" refers only to entropy decoding; in another embodiment, "decoding" refers only to differential decoding; and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" specifically refers to a subset of operations or broadly refers to a wider decoding process will be clear based on the specific context of the description and is believed to be well understood by those skilled in the art.

[0342] Various specific implementations involve encoding. In a manner similar to the discussion above regarding “decoding,” the term “encoding,” as used herein, can encompass all or part of a process performed, for example, on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential coding, transform, quantization, and entropy coding. In various embodiments, such processes also or alternatively include processes performed by an encoder of the various embodiments described herein, such as weighting an intra-frame prediction reference array.

[0343] As a further example, in one implementation, "encoding" refers only to entropy encoding; in another implementation, "encoding" refers only to differential encoding; and in yet another implementation, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" specifically refers to a subset of operations or broadly refers to a wider encoding process will be clear based on the specific context of the description and is believed to be well understood by those skilled in the art.

[0344] Note that the grammatical elements used in this article are descriptive terms. Therefore, the use of other grammatical element names is not excluded.

[0345] When the accompanying drawings are presented as flowcharts, it should be understood that block diagrams of the corresponding devices are also provided. Similarly, when the accompanying drawings are presented as block diagrams, it should be understood that flowcharts of the corresponding methods / processes are also provided.

[0346] Various implementation schemes refer to rate-distortion computation or rate-distortion optimization. During the encoding process, a balance or trade-off between rate and distortion is typically considered, often taking into account computational complexity constraints. Rate-distortion optimization is generally formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. Different approaches exist to solve the rate-distortion optimization problem. For example, these methods may be based on extensive testing of all encoding options (including all considered modes or encoding parameter values) and a complete evaluation of their encoding costs and the associated distortion of the reconstructed signal after encoding and decoding. Faster methods can also be used to reduce encoding complexity, particularly for the computation of approximate distortion based on prediction or prediction of the residual signal rather than the reconstructed residual signal. A hybrid of these two approaches can also be used, such as by using approximate distortion for only some of the possible encoding options and full distortion for others. Other methods evaluate only a subset of the possible encoding options. More generally, many methods employ any of a variety of techniques to perform optimization, but optimization is not necessarily a complete evaluation of both encoding costs and associated distortion.

[0347] The specific embodiments and aspects described herein may be implemented, for example, in methods or processes, apparatus, software programs, data streams, or signals. Even if discussed only in the context of a single form of specific embodiment (e.g., discussed only as a method), specific embodiments of the discussed features may be implemented in other forms (e.g., apparatus or program). Apparatus may be implemented, for example, in suitable hardware, software, and firmware. Methods may be implemented, for example, in a processor, which generally refers to a processing device.

[0348] The processing device includes, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication devices, such as computers, mobile phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0349] The reference to “an implementation plan” or “implementation plan” or “a specific implementation” or “specific implementation”, and their other variations, means that the specific features, structures, characteristics, etc., described in connection with the implementation plan are included in at least one implementation plan. Therefore, the appearance of the phrase “in an implementation plan” or “in an implementation plan” or “in a specific implementation” or “in a specific implementation”, and any other variations appearing throughout this document, do not necessarily refer to the same implementation plan.

[0350] In addition, this document may refer to the various pieces of information marked "determined". Determined information may include one or more of the following: estimated information, calculated information, predicted information, or information retrieved from memory.

[0351] Furthermore, this document may refer to "accessing" each piece of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or more of these.

[0352] Furthermore, this document may refer to "receiving" individual pieces of information. Like "access," "receiving" is intended as a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory). Moreover, "receiving" typically participates in one or more of the following operations: such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0353] It should be understood that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one” is intended to cover selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such phrases are intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or selecting all three options (A, B, and C). As will be apparent to those skilled in the art and related fields, this can be extended to as many items as possible listed.

[0354] Moreover, as used herein, the term "signaling" refers to (among other things) instructing the corresponding decoder to do something. For example, in some implementations, the encoder signals a specific weight among several weights to be used in the intra-prediction reference array. Thus, in one implementation, the same parameters are used on both the encoder and decoder sides. Therefore, for example, the encoder can transmit (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. Conversely, if the decoder already has specific parameters and others, signaling can be used without transmission (implicit signaling) to simply allow the decoder to know and select specific parameters. Bit savings are achieved in various implementations by avoiding the transmission of any actual functionality. It should be understood that signaling can be implemented in various ways. For example, in various implementations, information is signaled to the corresponding decoder using one or more syntax elements, flags, etc. Although the verb form of the term "signal" has been used above, the term "signal" can also be used as a noun herein.

[0355] It will be apparent to those skilled in the art that embodiments can generate various signals formatted to carry, for example, storable or transmissible information. The information may include, for example, instructions for performing a method or data generated by one of the embodiments. For example, a signal may be formatted to carry a bitstream of the embodiment. Such signals may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of a spectrum) or baseband signals. Formatting may include, for example, encoding a data stream and modulating the carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is known that signals can be transmitted via various wired or wireless links. Signals may be stored on a processor-readable medium.

[0356] Implementations may include one or more of the following features or entities, individually or in combination, across a variety of different claim classes and types:

[0357] ●Relax the hard constraints in video coding standards to improve coding patterns

[0358] ●Relaxing the hard constraints mentioned above can be achieved through grammatical changes.

[0359] ● The relaxation of the hard constraints mentioned above can be achieved through ad-hoc entropy context.

[0360] ●The relaxation of the hard constraints mentioned above can be achieved by changing the initial probability parameter of the entropy.

[0361] ● Encoding / decoding scheme, which adds a CABAC context indicating whether the encoding mode is active.

[0362] ● Pattern inference mode, in which pattern exclusivity is removed and / or conditions of encoded parameters are used to control / activate the inferred value (new syntax elements are used to explicitly encode this value).

[0363] ● Switchable filter mode, in which the encoding condition of hpelIfIdx is the advanced parameter "enabled_interpolation_filter_flag".

[0364] ●mmvd_merge_idx is encoded to allow setting the pattern for any merge candidate in the list.

[0365] ● In the variant, the encoding condition for "mmvd_merge_idx" is a high-level parameter.

[0366] "enabled_mmvd_merge_idx_flag".

[0367] ● When mode M is activated so that the initial set of blocks uses mode N, calculate the bias rate distortion cost of mode N.

[0368] ● Add a new context based on the skip flag.

[0369] ● Includes a bitstream or signal of one or more syntax elements or their variants from the described syntax elements.

[0370] ● Create and / or transmit and / or receive and / or decode bit streams or signals including one or more syntax elements or variations thereof from the described syntax elements.

[0371] ● A television, set-top box, mobile phone, tablet computer, or other electronic device that performs in-loop filtering according to any of the embodiments described in the implementation plan.

[0372] ● A television, set-top box, mobile phone, tablet computer, or other electronic device that performs in-loop filtering and displays the resulting image (e.g., using a monitor, screen, or other type of display) according to any of the described implementation schemes.

[0373] ● A television, set-top box, mobile phone, tablet computer, or other electronic device that tunes (e.g., uses a tuner) a channel to receive signals including encoded images and performs in-loop filtering according to any of the embodiments described.

[0374] ● A television, set-top box, mobile phone, tablet computer, or other electronic device that receives (e.g., using an antenna) an air signal including an encoded image and performs in-loop filtering according to any of the described embodiments.

[0375] Figure 20 An embodiment of a method 2000 according to the general aspects described herein is illustrated. The method begins at a start box 2001, and control advances to box 2010 to determine whether a first encoding tool is activated for encoding a portion of a video image. Control advances from box 2010 to box 2020 to determine whether a second encoding tool is activated for encoding that portion of the video image, and the conditions for said activation. Control advances from box 2020 to box 2030 to conditionally execute at least one of the encoding tools.

[0376] Figure 21 The diagram illustrates one embodiment of method 2100 according to the general aspects described herein. The method begins at a start box 2101, and control advances to box 2110 to determine whether a first decoding tool is activated based on the video bitstream. Control advances from box 2110 to box 2120 to determine whether a second decoding tool is activated, and the conditions for said activation. Control advances from box 2120 to box 2130 to conditionally execute at least one of the decoding tools.

[0377] Figure 22 An embodiment of an apparatus 2200 for encoding, decoding, compressing, or decompressing video data using a simplified encoding mode based on a neighbor-sample dependency parameter model is shown. The apparatus includes a processor 2210 and is interconnected to a memory 2220 via at least one port. Both the processor 2210 and the memory 2220 may also have one or more additional interconnects for external connection.

[0378] The processor 2210 is also configured to insert or receive information in the bitstream and to use any of the described aspects to perform compression, encoding, or decoding.

[0379] Various other general and specific inventions and claims are also supported and envisioned throughout this specification.

Claims

1. A method in a video encoder, the method comprising: Determine whether to activate the first encoding tool for encoding a portion of the video image; The method is characterized in that it includes: Determine whether the second encoding tool is activated for encoding the portion of the video image; At least one of the first encoding tool and the second encoding tool shall be executed based on the result of the determination steps; Encode the first syntax element that indicates whether the first encoding tool is activated; and The second syntax element, which indicates whether the second encoding tool is activated, is encoded. If the first syntax element indicates that the first encoding tool is activated, the second syntax element is encoded using a first context bin, and otherwise the second syntax element is encoded using a second context bin, which is different from the first context bin.

2. The method of claim 1, wherein the context bin used when the first encoding tool is activated is initialized with low probability.

3. The method according to claim 1, wherein the first coding tool is a matrix intra-frame prediction with a block size less than a value, and the second coding tool is a low-frequency inseparable transform.

4. The method of claim 1, wherein the first encoding tool is a skip mode and the second encoding tool is an intra-frame-inter-frame joint prediction.

5. The method of claim 1, wherein the first encoding tool is a multi-reference line and the second encoding tool is an intra-frame sub-block partitioning.

6. An apparatus for video encoding, the apparatus comprising: Processor, the processor being configured to: Determine whether to activate the first encoding tool for encoding a portion of the video image; The device is characterized in that the processor is configured to: Determine whether the second encoding tool is activated for encoding the portion of the video image; At least one of the first encoding tool and the second encoding tool shall be executed based on the result of the determination steps; Encode the first syntax element that indicates whether the first encoding tool is activated; as well as The second syntax element, which indicates whether the second encoding tool is activated, is encoded. If the first syntax element indicates that the first encoding tool is activated, the second syntax element is encoded using a first context bin, and otherwise the second syntax element is encoded using a second context bin, which is different from the first context bin.

7. The apparatus of claim 6, wherein the context bin used when the first encoding tool is activated is initialized with low probability.

8. The apparatus of claim 6, wherein the first coding tool is a matrix intra-frame prediction with a block size less than a value, and the second coding tool is a low-frequency inseparable transform.

9. The apparatus of claim 6, wherein the first coding tool is a skip mode and the second coding tool is an intra-frame-inter-frame joint prediction.

10. The apparatus of claim 6, wherein the first encoding tool is a multi-reference line and the second encoding tool is an intra-frame sub-block partitioning.

11. A method in a video decoder, the method comprising: Decode the first syntax element that indicates whether the first decoding tool is activated; The method is characterized by: Decoding a second syntax element that indicates whether a second decoding tool is activated; decoding the second syntax element using a first context bin if the first syntax element indicates that a first decoding tool is activated, and otherwise decoding the second syntax element using a second context bin, wherein the second context bin is different from the first context bin; and At least one of the first decoding tool and the second decoding tool is executed based on the decoded first and second syntax elements.

12. The method of claim 11, wherein the context bin used when the first decoding tool is activated is initialized with low probability.

13. The method of claim 11, wherein the first decoding tool is a matrix intra-frame prediction with a block size less than a value, and the second decoding tool is a low-frequency inseparable transform.

14. The method of claim 11, wherein the first decoding tool is a skip mode and the second decoding tool is an intra-frame-inter-frame joint prediction.

15. The method of claim 11, wherein the first decoding tool is a multi-reference line and the second decoding tool is an intra-frame sub-block partitioning.

16. An apparatus for video decoding, the apparatus comprising: Processor, the processor being configured to: Decode the first syntax element that indicates whether the first decoding tool is activated; The device is characterized in that the processor is configured to: The second syntax element that indicates whether the second decoding tool is activated is decoded. If the first syntax element indicates that the first decoding tool is activated, the second syntax element is decoded using the first context bin, and otherwise the second syntax element is decoded using the second context bin, which is different from the first context bin. as well as At least one of the first decoding tool and the second decoding tool is executed based on the decoded first and second syntax elements.

17. The apparatus of claim 16, wherein the context bin used when the first decoding tool is activated is initialized with low probability.

18. The apparatus of claim 16, wherein the first decoding tool is a matrix intra-frame prediction with a block size less than a value, and the second decoding tool is a low-frequency inseparable transform.

19. The apparatus of claim 16, wherein the first decoding tool is a skip mode and the second decoding tool is an intra-frame-inter-frame joint prediction.

20. The apparatus of claim 16, wherein the first decoding tool is a multi-reference line and the second decoding tool is an intra-frame sub-block partitioning.

Images