Template matching prediction for versatile video coding

Abstract

Template matching prediction is combined with additional coding tools to provide a robust set of encoding / decoding tools by advantageously using syntax elements. Some of these additional coding tools include matrix intra prediction, intra sub-partition, low frequency non-separable transform, multiple transform selection.

Description

TECHNICAL FIELD

[0001] At least one of the embodiments in this disclosure relates generally to a method or apparatus for video encoding or decoding, compression or decompression. BACKGROUND

[0002] To achieve high compression efficiency, image and video coding schemes usually employ prediction, including motion vector prediction, and transform to exploit the spatial and temporal redundancy in the video content. Generally, intra- or inter-frame prediction is used to exploit the intra- or inter-frame correlation, and then the difference between the original image and the predicted image, usually denoted as prediction error or prediction residual, is transformed, quantized and entropy coded. To reconstruct the video, the compressed data is decoded by the inverse processes corresponding to the entropy coding, quantization, transform and prediction. SUMMARY

[0003] At least one of the embodiments in this disclosure relates generally to a method or apparatus for video encoding or decoding, and more particularly, to a method or apparatus using template matching prediction in combination with other coding tools as in the VVC (Versatile Video Coding or H.266) standard.

[0004] According to a first aspect, a method is provided. The method comprises a step for signaling use of at least a first coding tool and a second coding tool on a video block; and encoding the video block using the first coding tool and the second coding tool.

[0005] According to a second aspect, another method is provided. The method comprises a step for parsing a video bitstream to determine at least a first decoding tool and a second decoding tool for a video block; and decoding the video block using at least the first coding tool and the second coding tool to be used on the video block.

[0006] According to another aspect, an apparatus is provided. The apparatus comprises a processor. The processor can be configured to encode a video block or decode a bitstream by performing any of the preceding methods.

[0007] According to another general aspect of at least one embodiment, there is provided a device comprising: an apparatus according to any of the decoding embodiments; and at least one of: (i) an antenna configured to receive a signal, the signal comprising a video block; (ii) a frequency band limiter configured to limit the received signal to a frequency band comprising the video block; and (iii) a display configured to display an output representing the video block.

[0008] According to another general aspect in accordance with at least one embodiment, there is provided a non-transitory computer readable medium comprising data content generated according to any of the described encoding embodiments or variants.

[0009] According to another general aspect in accordance with at least one embodiment, there is provided a signal comprising video data generated according to any of the described encoding embodiments or variants.

[0010] According to another general aspect in accordance with at least one embodiment, a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variants.

[0011] According to another general aspect in accordance with at least one embodiment, there is provided a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out any of the described decoding embodiments or variants.

[0012] These and other aspects, features, and advantages of the general aspects will become apparent to those of ordinary skill in the art from the following detailed description of exemplary embodiments, read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 Intra prediction modes in Versatile Video Coding (VVC) are shown.

[0014] Figure 2 An example of template matching prediction (TMP) is shown.

[0015] Figure 3 A standard versatile video compression scheme is shown.

[0016] Figure 4 A standard versatile video decompression scheme is shown.

[0017] Figure 5 An embodiment of a method in the context of the general description aspect is shown.

[0018] Figure 6 Another embodiment of a method in the context of the general description aspect is shown.

[0019] Figure 7 An exemplary apparatus in the context of the described aspect is shown.

[0020] Figure 8 A processor-based system for encoding / decoding in the context of the general description aspect is shown.

[0021] Figure 9 An example of a template matched with a single template is shown.

[0022] Figure 10 An example of template matching prediction with a small search range is shown.

[0023] Figure 11 An example of mutually exclusive search ranges for IBC and TMP is shown.

[0024] Figure 12 Search ranges allowed within the current CTU are shown.

[0025] Figure 13 SIMD optimization is shown where the line width of R1 and R2 is a multiple of 4.

[0026] Figure 14 SIMD optimization is shown where the top-left part of the template is removed to have the same size as the current block. DETAILED DESCRIPTION

[0027] The embodiments described herein are in the field of video compression and relate generally to video compression and video encoding and decoding, and more specifically aim to improve the intra prediction part of video compression by employing template matching prediction. This mode generates a prediction signal from blocks having a similar template within the reconstructed part of the image.

[0028] To achieve high compression efficiency, image and video coding schemes typically employ prediction, including motion vector prediction, and transforms to exploit spatial and temporal redundancy in the video content. Generally, intra- or inter-frame prediction is used to exploit intra- or inter-frame correlation, followed by transform, quantization, and entropy coding of the difference between the original image and the predicted image, often denoted as prediction error or prediction residual. To reconstruct the video, the compressed data is decoded by the inverse processes corresponding to the entropy coding, quantization, transform, and prediction.

[0029] Intra-picture prediction is a fundamental part of image and video compression. Traditionally, the prediction signal is generated from L-shaped reconstructed pixels (reference samples), assuming they are along different angles. This mechanism is called angular prediction. VVC uses 65 intra prediction modes, among which 63 angles are defined, as well as DC and Planar prediction (see Figure 1 ).

[0030] Various tools are used to enhance the traditional intra prediction in VVC:

[0031] - Cross-component linear model (CCLM): color prediction blocks are generated by a linear model of the luminance reconstructed samples.

[0032] - Multi-reference line prediction (MRL): more reference samples are used to generate the prediction block.

[0033] Intra sub-partition (ISP): The prediction block is divided into 4 sub-blocks sharing the same prediction mode.

[0034] Matrix weighted intra prediction (MIP): The prediction block is generated by multiplying the reference samples with some offline optimized prediction matrix.

[0035] Intra block copy (IBC): The prediction block is generated by copying another block from already reconstructed image part with a displacement vector signaled in the bitstream.

[0036] Another mode (called multi-transform selection (MTS)) uses the core transform DCT-II or DST-VII combined with DCT-VIII to transform the residual block. The transformed block can be further transformed with a secondary non-separable transform to further compress the residual block. This process is called low-frequency non-separable transform (LFNST).

[0037] Template matching prediction (TMP) is yet another powerful intra prediction mode not included in VVC. TMP is performed by searching for a similar L-shaped neighborhood (called patch) to the target block. This is illustrated in the following figure. If TMP is used, the current template consists of the reconstructed L-shaped neighboring parts. Similar templates are found with small difference to the current template. The blocks belonging to these templates (the target block) are used to generate the prediction signal by averaging them or considering only the one with the smallest template difference.

[0038] Integrating TMP into VVC requires proper interoperability with existing intra tools. Namely:

[0039] Interaction with ISP, MIP and MRL

[0040] Interaction with transform tools (MTS and LFNST, implicit MTS)

[0041] Interaction with combined inter and intra prediction (CIIP)

[0042] The general aspects described herein relate to these interactions and propose embodiments implementing this mode for VVC.

[0043] Template matching prediction is an optional intra prediction mode in the Joint Exploration Test Model (JEM) of the Joint Video Exploration Team (JVET). It comes with an online process to derive a transform matrix using Karhunen Loueve Transform (KLT) using the same template as the one of TMP. The prediction block is generated by averaging up to 8 candidate blocks.

[0044] In the context of VVC development, region-based template matching prediction is proposed in existing works. Compared to the JEM version, these existing works define smaller search regions, where an index is signaled to guide the decoder to limit the search to a given region. This reduces the latency on the decoder side.

[0045] The general aspects described herein focus on implementing TMP in the context of VVC, where interactions with other tools as well as signaling are proposed.

[0046] The basic idea of the general aspects described herein is to integrate TMP into VVC, where interactions with these are defined as:

[0047] - Interactions with ISP, MIP and MRL

[0048] - Interactions with the transform tools (MTS and LFNST, implicit MTS)

[0049] - Interactions with combined inter and intra prediction (CIIP)

[0050] The affected encoder modules are Figure 3 the transform module (125) and the intra prediction module (160).

[0051] The affected decoder modules are Figure 4 the inverse transform module (250) and the intra prediction module (260).

[0052] Signaling of TMP flag

[0053] A CU flag is signaled to indicate the use of TMP. This flag can be signaled at different levels in the codec design. However, CU level signaling is consistent with other intra tools. The current signaling in VTM is as follows:

[0054] - The MIP flag (intra_mip_flag) is signaled first to indicate the use of MIP

[0055] - If MIP is not used, the MRL flag (intra_mip_flag) is signaled to indicate the use of MRL

[0056] - If MRL is not used, the ISP flag (intra_subpartitions_mode_flag) is signaled to indicate the use of ISP

[0057] - If MRL is not used (MIP or ISP can be used), the prediction mode is further signaled.

[0058] The corresponding syntax is shown as follows:

[0059]

[0060] For TMP, the following considerations are taken into account:

[0061] - Since MIP cannot be used with TMP, it should not be signaled to MIP.

[0062] - Since MRL cannot be used with TMP, it should not be signaled to MRL.

[0063] -Since TMP does not have a mode for sending a signal, it should not send a signal to notify the intra-prediction mode.

[0064] -ISP can be used with TMP, where each subpartition uses a different matching template.

[0065] Therefore, the TMP flag can be signaled before the MIP flag, so that if all other flags are used, it is not signaled and inferred to be 0. The corresponding changes are as follows (the added parts are shaded):

[0066]

[0067] Where `sps_TMP_enabled_flag` is the SPS-level flag used to activate TMP, and `intra_TMP_flag` is the CU-level flag used to signal the use of TMP. `MaxTMPSize` is the maximum allowed size of TMP. It is fixed at the SPS level or signaled.

[0068] In this method, ISP and TMP cannot be used together. This can be changed to allow their combination. This is done by signaling the TMP flag after or before the ISP flag. Signaling should only be done when MRL and MIP are not used. Make the following modifications:

[0069]

[0070] It should be noted that for both methods, in the case of region-based template matching prediction, the additional syntax can be signaled. The additional syntax is signaled immediately after the signal `intra_TMP_flag`.

[0071] In addition to the SPS flag, general constraint flags can be defined for TMP. This is similar to most coding tools in VVC, which have more advanced flags to indicate whether they have been deactivated. The following flags can be added to the specification:

[0072]

[0073] When gci_no_tmp_constraint_flag is equal to 1, it indicates that sps_TMP_enabled_flag is equal to 0.

[0074] Interaction with transform tools :

[0075] TMP can be used with MTS and / or LFNST. However, LFNST depends on the intra-prediction mode used to select the transform kernel. The same MIP method can be used. That is, for transform matrix selection, MIP is considered as planar intra-prediction. Therefore, TMP can also be considered in this way. That is, TMP is used when using an LFNST transform kernel corresponding to a planar mode.

[0076] Furthermore, since LFNST is used for the chromaticity component, a prediction mode is needed to select the transformation matrix. Specifically, for CCLM, if MIP is used for the luminance component, the prediction mode is considered a planar mode. The same operation is performed if TMP is used for the luminance component.

[0077] For MTS, there exists a special mode known as implicit MTS, where the MTS flag is not signaled; instead, the transform selection is derived from the block size. However, the transform selection used for MIP mode or LFNST is DCT-II. The same transform selection is used for TMP.

[0078] The corresponding changes to the draft working paper are as follows:

[0079]

[0080]

[0081]

[0082]

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094] To reduce encoder runtime, the MTS can be disallowed to have a TMP. That is, when the TMP is used as the predictive mode, the encoder does not attempt all possible MTS modes (DCT-II, DST-VII, and DCT-VIII) and does not signal any information to the decoder. Instead, a fixed transform selection can be used. One option is to use DCT-II for both vertical and horizontal directions. Another option is to use implicit transform selection for ISP modes. That is, selecting the vertical and horizontal transforms as follows:

[0095] TrHor = width <= 16? DST7 : DCT2

[0096] TrVer = Height <= 16? DST7 : DCT2

[0097] Interaction with MPMs

[0098] The Most Probable Mode (MPM) is an efficient way to signal the intra-frame prediction mode. MPM list generation requires knowledge of the current prediction mode. If TMP is used, it can be considered a planar mode for MPM list generation.

[0099] TMP for luma and chroma components

[0100] Generally, TMP is used only for the luminance component. However, it can be extended to use both the luminance and chrominance components. To allow this, use the following options:

[0101] - A single flag is used to signal the use of TMP for both the luminance and chrominance components.

[0102] - Template matching is performed only on the luma component, and the same positions of the matched blocks are used for the chroma component (with appropriate scaling based on the chroma formation, e.g., 4:2:0). This is to reduce complexity by avoiding duplicate searches for all components.

[0103] Interaction with CIIP

[0104] CIIP is a prediction mode that combines intra-frame prediction and inter-frame prediction. The intra-frame portion is a planar mode, and the inter-frame portion is a regular merged mode. TMP can be used to replace the intra-frame prediction portion (planar mode).

[0105] The weighting of the two predicted signals can also be modified. Currently, the weights are defined based on adjacent coding units as follows:

[0106]

[0107] Instead, an equal weighted average of the inter-frame and intra-frame portions can be used:

[0108] Interaction with IBC

[0109] In many cases, using both TMP and IBC simultaneously is useless because both perform intra-frame prediction by searching within the reconstructed portion of the image. Therefore, it is proposed that IBC and TMP should not be activated together. This can be done by adjusting their SPS flags relative to each other. That is, if the SPS flag of IBC is 1, then the SPS flag of TMP should be inferred as 0. Alternatively, if the SPS flag of TMP is 1, then the SPS flag of IBC will be inferred as 0.

[0110] Examples of such methods are given in the syntax table below (tuning TMP on IBC):

[0111]

[0112] Another example (adjusting IBC on TMP):

[0113]

[0114] Figure 5 The diagram illustrates one embodiment of method 500 under the general aspects described herein. The method begins at a start box 501, and control proceeds to box 510 to signal the use of at least a first encoding tool and a second encoding tool on a video block. Control proceeds from box 510 to box 520 for encoding the video block using the first encoding tool and the second encoding tool.

[0115] Figure 6 An embodiment of method 600 under the general aspects described herein is illustrated. The method begins at a start box 601, and control proceeds to box 610 for parsing the video bitstream to determine at least a first decoding tool and a second decoding tool for a video block. Control proceeds from box 610 to box 620 for decoding the video block using at least the first encoding tool and the second encoding tool to be used on the video block.

[0116] Figure 7An embodiment of an apparatus 700 for encoding, decoding, compressing, or decompressing video data using a simplified encoding mode based on a neighbor-sample dependency parameter model is shown. The apparatus includes a processor 710 and can be interconnected to a memory 720 via at least one port. Both the processor 710 and the memory 720 may also have one or more additional interconnects for external connection.

[0117] The processor 710 is also configured to insert or receive information in the bitstream and to use any of the aspects described to perform compression, encoding, or decoding.

[0118] Single template prediction

[0119] It was observed that a single matching block provides the best compression efficiency. That is, instead of searching for N similar templates, a single similar template is found and used as the prediction signal. The process is as follows: Figure 9 As shown in the image.

[0120] Variable search range

[0121] The search range used to find the best matching template (multiple templates) can be set to variable or fixed. The maximum coding gain associated with the maximum complexity occurs when using the full range, that is, finding the template within the entire reconstructed portion of the current frame. However, to reduce complexity, a smaller search range can be used. Figure 10 An example is shown in the image.

[0122] It was found that the maximum coding gain can be achieved with a search range of 1024. However, for video sequences with low resolution, a smaller search range can be used. It was also found that a search range of 64 offers a reasonable trade-off between complexity and gain. Therefore, in this implementation, the search range is defined as: 64, 128, 256, 1024, or all of them. High-level syntax elements can be used to signal the value of the search range.

[0123] Search range within CTU

[0124] The search range is allowed within the current CTU. That is, when the current block (PU / CU) is inside a larger CTU, predictive candidates can be searched both inside and outside the CTU. However, not all parts within the CTU are decoded and therefore cannot be used as predictive candidates.

[0125] To allow searching within the current CTU, only candidates located in the upper left corner of the current block are used. This is in Figure 12As shown in the diagram, the darker search range is within the current CTU, where all pixels are decoded and reconstructed. A search range outside the CTU is also used, and the best candidate with the smallest template difference is used as the prediction block.

[0126] Template matching CABAC context derivation

[0127] To signal a template match, an appropriate context for CABAC must be derived. In this implementation, the same approach as MIP is followed. This is consistent with the design of VVC. Four contexts are used:

[0128] 1- Initialize the context with 0

[0129] 2- If the left-hand CU uses template matching prediction, then add one.

[0130] 3- If the CU above uses template matching prediction, then add one.

[0131] 4. If the width is greater than twice the height, or the height is greater than twice the width, then set the context to 3.

[0132] That is, the four categories are:

[0133] 1- Adjacent CUs do not use TMP

[0134] 2- An adjacent CU is using TMP

[0135] 3- Two adjacent CUs are using TMP

[0136] The 4-CU shape is elongated in one dimension.

[0137] A similar approach can be used without using a fourth context.

[0138] Variable maximum size

[0139] The maximum CU size using TMP can be variable. It has been found that up to 16×16 CUs provide the maximum coding gain while reducing complexity. The maximum size value can be signaled using high-level syntax elements, or it can be kept constant at 16.

[0140] Removed redundant signaling

[0141] The signaling of the TMP flag can be redundant because in some cases, it is known that TMP cannot be used. An example of this is when the CU size is larger than the maximum permissible size of TMP, when considering previous implementations (variable maximum size), or when there are not enough reconstructed portions to search for a similar template. For example, for a CU with a size of 64×64, and the reconstructed portion is smaller than 64×64 plus the template size, TMP cannot be used and therefore its flag should not be signaled but inferred as 0.

[0142] Search range outside current CTU

[0143] The search range can be simplified to exclude the current CTU. It was observed that including the current CTU in the search range did not result in any improved coding gain. Therefore, as a simplification step, it is proposed to remove this region from the search range.

[0144] SIMD optimization

[0145] By employing SIMD to compute template differences, runtime can be reduced. That is, the differences between pixels of two templates can be processed in parallel through SIMD optimization.

[0146] For SIMD operations, each comparison line must be a multiple of 4. Figure 13 In this context, it's assumed that the block size (b_width and b_height) is always a multiple of 4, because in VVC, the block size is a multiple of 4. Therefore, for a comparison to have multiples of 4, the line widths of R1 and R2 (b_width and b_height) Figure 13 It must be a multiple of 4:

[0147] 1-R1: b_width + t_width must be a multiple of 4. Therefore, t_width must be a multiple of 4.

[0148] 2-R2: T_width must be a multiple of 4. Therefore, T_width must be a multiple of 4.

[0149] Otherwise, the top left portion of the template is removed to have the same size as the current block. This is in Figure 14 As shown in the image.

[0150] By doing this, the width of R1 is equal to b_width (always a multiple of 4), and R2 is transposed before calculating this difference, and will ultimately have a width of b_height (always a multiple of 4). This method provides SIMD optimization regardless of the template size.

[0151] Mutually exclusive search range for IBC and TMP

[0152] In this implementation, IBC and TMP use different reference regions. Typically, IBC uses a CTU region as a reference. Therefore, TMP should use the remaining portion of the reconstructed image as a reference.

[0153] In the case of a CTU of size 32×32 or 64×64, IBC can use more than one CTU region as a reference, but always on the same CTU line as the current coding unit, and in this case, TMP should use all reconstructed images except the current CTU line as references.

[0154] Figure 11 An example of mutually exclusive search ranges for IBC and TMP is shown in the figure.

[0155] The implementation described herein encompasses various aspects, including tools, features, implementation methods, models, methods, etc. Many of these aspects are specifically described, and at least individual characteristics are shown, often in a manner that may sound restrictive. However, this is for clarity and does not limit the application or scope of these aspects. In fact, all the different aspects can be combined and interchanged to provide further aspects. Furthermore, these aspects can also be combined and interchanged with those described in previous submissions.

[0156] The aspects described and envisioned in this patent application may be implemented in many different forms. Figure 3 , Figure 4 and Figure 8 Some implementation schemes are provided, but other implementation schemes are envisioned, and Figure 3 , Figure 4 and Figure 8 The discussion does not limit the breadth of specific implementations. At least one of these aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a bitstream generated or encoded. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions stored thereon for encoding or decoding video data according to any of the methods, and / or computer-readable storage media having a bitstream generated according to any of the methods stored thereon.

[0157] In this application, the terms "reconstruction" and "decoding" are used interchangeably, as are the terms "pixel" and "sample," and the terms "image," "picture," and "frame." Generally, but not necessarily, the term "reconstruction" is used at the encoding end, while "decoding" is used at the decoding end.

[0158] This document describes various methods, and each method includes one or more steps or actions for implementing the method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or purpose of a particular step and / or action may be modified or combined.

[0159] The various methods and other aspects described in this patent application can be used to modify modules of the video encoder 100 and decoder 200 (e.g., intra-frame prediction, entropy coding, and / or decoding modules (160, 260, 145, 230)), such as Figure 10 and Figure 11 As shown. Furthermore, aspects of the invention are not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations (whether pre-existing or developed in the future) and any extensions of such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this application may be used alone or in combination.

[0160] Various numerical values ​​are used in this application. Specific values ​​are for illustrative purposes only, and the aspects described are not limited to these specific values.

[0161] Figure 3 Encoder 100 is shown. Variations of this encoder 100 are envisioned, but for clarity, encoder 100 is described below without describing all anticipated variations.

[0162] Before encoding, the video sequence may undergo pre-coding (101), for example, by applying color transformations to the input color image (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0), or by performing remapping of the input image components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with pre-processing and attached to the bitstream.

[0163] As described below, in encoder 100, the image is encoded by encoder elements. The image to be encoded is partitioned (102) and processed in a unit, such as a CU. For example, each unit is encoded using either an intra-frame mode or an inter-frame mode. When a unit is encoded in intra-frame mode, intra-frame prediction (160) is performed. In inter-frame mode, motion estimation (175) and compensation (170) are performed. The encoder determines (105) which of the intra-frame or inter-frame modes to use for encoding the unit, and indicates the intra / inter-frame decision via, for example, a prediction mode flag. For example, the prediction residual is calculated by subtracting (110) the prediction block from the initial image block.

[0164] Then, the predicted residual is transformed (125) and quantized (130). The quantized transform coefficients, motion vectors, and other syntax elements are entropy encoded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, that is, encode the residual directly without applying the transform or quantization process.

[0165] The encoder decodes the coded blocks to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residuals. The decoded prediction residuals and prediction blocks are combined (155) to reconstruct the image blocks. A loop filter (165) is applied to the reconstructed image to perform, for example, deblocking / sample adaptive offset (SAO) filtering, thereby reducing coding artifacts. The filtered image is stored in a reference image buffer (180).

[0166] Figure 4 A block diagram of a video decoder 200 is shown. In decoder 200, the bitstream is decoded by decoder elements, as described below. Video decoder 200 typically performs operations similar to... Figure 3 The encoding phase is the opposite of the decoding phase. Encoder 100 typically also performs video decoding as part of the encoding of video data.

[0167] Specifically, the decoder's input includes a video bitstream, which can be generated by the video encoder 100. The bitstream is first entropy-decoded (230) to obtain transform coefficients, motion vectors, and other encoded information. Image partitioning information indicates how the image should be partitioned. Therefore, the decoder can partition (235) the image based on the decoded image partitioning information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residuals. The decoded prediction residuals and prediction blocks are combined (255) to reconstruct the image blocks. The prediction blocks (270) can be obtained through intra-frame prediction (260) or motion-compensated prediction (i.e., inter-frame prediction) (275). A loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference image buffer (280).

[0168] The decoded image can also undergo post-decoding processing (285), such as inverse color transformation (e.g., a transformation from YcbCr 4:2:0 to RGB 4:4:4) or inverse remapping of the remapping process performed in the pre-encoding process (101). Post-decoding processing can utilize metadata derived in the pre-encoding process and signaled in the bitstream.

[0169] Figure 8A block diagram illustrating examples of systems implementing various aspects and embodiments is shown. System 1000 may be embodied as a device including the various components described below and configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 1000 may be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects described in this document.

[0170] System 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing various aspects, such as those described in this document. Processor 1010 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). System 1000 includes a storage device 1040 that may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 1040 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0171] System 1000 includes an encoder / decoder module 1030 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module that can be included in a device to perform encoding and / or decoding functions. It is well known that a device may include one or both of an encoding module and a decoding module. Alternatively, the encoder / decoder module 1030 may be implemented as a standalone component of system 1000, or it may be incorporated within processor 1010 as a combination of hardware and software known to those skilled in the art.

[0172] Program code to be loaded onto processor 1010 or encoder / decoder 1030 to execute the various aspects described in this document may be stored in storage device 1040 and subsequently loaded onto memory 1020 for execution by processor 1010. According to various embodiments, one or more of processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 may store one or more items from various projects during the execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded or partially decoded video, bitstreams, matrices, variables, and intermediate or final results of processing equations, formulas, operations, and operational logic.

[0173] In some embodiments, the memory within processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be processor 1010 or encoder / decoder module 1030) is used for one or more of these functions. External memory may be memory 1020 and / or storage device 1040, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one implementation, a fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations, such as MPEG-2 (MPEG stands for Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2) or VVC (Universal Video Coding, a new standard developed by the Joint Video Experts Group (JVET)).

[0174] Inputs to the components of system 1000 can be provided through various input devices as shown in box 1130. Such input devices include, but are not limited to: (i) a radio frequency (RF) section that receives, for example, RF signals transmitted over the air by a broadcaster; (ii) component (COMP) input terminals (or a set of COMP input terminals); (iii) universal serial bus (USB) input terminals; and / or (iv) high-definition multimedia interface (HDMI) input terminals. Figure 8 Other examples not shown include composite video.

[0175] In various embodiments, the input device of block 1130 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting a signal band to a band), (ii) down-converting the selected signal, (iii) further band-limiting to a narrower band to select (e.g.,) a signal band that may be referred to as a channel in some embodiments), (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired data packet stream. The RF section of various embodiments includes one or more elements for performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include tuners that perform various functions among these functions, including, for example, down-converting received signals to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or to baseband. In one set-top box implementation, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to the desired frequency band. Various implementations rearrange the order of the aforementioned (and other) components, remove some of these components, and / or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as inserting amplifiers and analog-to-digital converters. In various implementations, the RF section includes an antenna.

[0176] Additionally, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented as needed, for example, within a separate input processing IC or within the processor 1010. Similarly, various aspects of USB or HDMI interface processing may be implemented as needed, either within a separate interface IC or within the processor 1010. The demodulated, error-corrected, and demultiplexed stream is provided to various processing elements, including, for example, the processor 1010 and an encoder / decoder 1030, which operates in conjunction with memory and storage elements to process the data stream as needed for presentation on an output device.

[0177] Various components of the system 1000 can be housed within an integrated housing, in which the various components can be interconnected and transmit data therebetween using a suitable connection arrangement (e.g., internal buses as known in the art, including inter-IC (I2C) buses, wiring and printed circuit boards).

[0178] System 1000 includes a communication interface 1050 capable of communicating with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0179] In various implementations, wireless networks such as Wi-Fi networks, such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers), are used to stream or otherwise provide data to system 1000. In these implementations, the Wi-Fi signal is received via a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 in these implementations is typically connected to an access point or router that provides access to external networks, including the Internet, for use in allowing streaming applications and other communications across carriers. Other implementations use a set-top box to provide streaming data to system 1000, delivering data via an HDMI connection in input box 1130. Other implementations use an RF connection in input box 1130 to provide streaming data to system 1000. As described above, various implementations provide data in a non-streaming manner. Additionally, various implementations use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0180] System 1000 can provide output signals to various output devices, including display 1100, speaker 1110, and other peripheral devices 1120. Display 1100 in various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 1100 can be used in televisions, tablets, laptops, cellular phones (mobile phones), or other devices. Display 1100 can also be integrated with other components (e.g., as in a smartphone) or with separate components (e.g., an external monitor for a laptop computer). In various examples of embodiments, other peripheral devices 1120 include one or more of a standalone digital video disc (or digital versatile disc) (DVR, which may refer to both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide functionality based on the output of system 1000. For example, a disc player performs the function of playing the output of system 1000.

[0181] In various embodiments, signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols enabling device-to-device control with or without user intervention are used to transmit control signals between system 1000 and display 1100, speaker 1110, or other peripheral devices 1120. Output devices may be communicatively coupled to system 1000 via dedicated connections through corresponding interfaces 1070, 1080, and 1090. Alternatively, output devices may be connected to system 1000 via communication interface 1050 using communication channel 1060. Display 1100 and speaker 1110 may be integrated into a single unit with other components of system 1000 in electronic devices such as televisions. In various embodiments, display interface 1070 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0182] If the RF section of input 1130 is part of a standalone set-top box, then display 1100 and speaker 1110 may optionally be separated from one or more other components. In various embodiments where display 1100 and speaker 1110 are external components, output signals may be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0183] These implementations may be executed by computer software implemented by processor 1010, or by hardware, or by a combination of hardware and software. As a non-limiting example, these implementations may be implemented by one or more integrated circuits. Memory 1020 may be of any type suitable for the technical environment and may be implemented using any suitable data storage technology, such as, as a non-limiting example, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. Processor 1010 may be of any type suitable for the technical environment and, as a non-limiting example, may encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0184] Various specific implementations involve decoding. As used in this application, "decoding" may include all or part of a process, for example, performing on a received encoded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also includes, or alternatively includes, processes performed by a decoder of the various specific implementations described in this application.

[0185] As a further example, in one implementation, "decoding" refers only to entropy decoding; in another implementation, "decoding" refers only to differential decoding; and in yet another implementation, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" specifically refers to a subset of operations or broadly refers to a wider decoding process will be clear based on the specific context of the description and is believed to be well understood by those skilled in the art.

[0186] Various specific implementations involve encoding. In a manner similar to the discussion above regarding “decoding,” the term “encoding,” as used herein, can encompass all or part of a process performed, for example, on an input video sequence to produce an encoded bitstream. In various implementations, such processes include one or more processes typically performed by an encoder, such as partitioning, differential coding, transform, quantization, and entropy coding. In various implementations, such processes also include, or alternatively include, processes performed by the encoders of the various specific implementations described in this application.

[0187] As a further example, in one implementation, "encoding" refers only to entropy encoding; in another implementation, "encoding" refers only to differential encoding; and in yet another implementation, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" specifically refers to a subset of operations or broadly refers to a wider encoding process will be clear based on the specific context of the description and is believed to be well understood by those skilled in the art.

[0188] Note that the grammatical elements used in this article are descriptive terms. Therefore, the use of other grammatical element names is not excluded.

[0189] When the accompanying drawings are presented as flowcharts, it should be understood that block diagrams of the corresponding devices are also provided. Similarly, when the accompanying drawings are presented as block diagrams, it should be understood that flowcharts of the corresponding methods / processes are also provided.

[0190] Various implementations can refer to parametric models or rate distortion optimization. Specifically, during the encoding process, a balance or trade-off between rate and distortion is typically considered, often taking into account computational complexity constraints. This can be measured by rate distortion optimization (RDO) or by least mean square (LMS), average absolute error (MAE), or other such measures. Rate distortion optimization is generally formulated as minimizing a rate distortion function, which is a weighted sum of rate and distortion. Different approaches exist to address the rate distortion optimization problem. For example, these approaches may be based on extensive testing of all encoding options, including all considered modes or encoding parameter values, and a complete evaluation of their encoding costs and the associated distortion of the reconstructed signal after encoding and decoding. Faster methods can also be used to reduce encoding complexity, particularly for calculating approximate distortion based on prediction or prediction of the residual signal rather than the reconstructed residual signal. A hybrid of these two approaches can also be used, such as by using approximate distortion for only some of the possible encoding options and full distortion for others. Other methods evaluate only a subset of the possible encoding options. More generally, many methods employ any of a variety of techniques to perform optimization, but optimization is not necessarily a complete assessment of both coding cost and associated distortion.

[0191] The specific embodiments and aspects described herein may be implemented, for example, in methods or processes, apparatus, software programs, data streams, or signals. Even if discussed only in the context of a single form of specific embodiment (e.g., discussed only as a method), specific embodiments of the discussed features may be implemented in other forms (e.g., apparatus or program). Apparatus may be implemented, for example, in suitable hardware, software, and firmware. Methods may be implemented, for example, in a processor that generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end users.

[0192] The reference to "an implementation scheme" or "implementation scheme" or "a specific implementation" or "specific implementation," and other variations thereof, means that the specific features, structures, characteristics, etc., described in connection with the implementation scheme are included in at least one implementation scheme. Therefore, the appearance of the phrase "in an implementation scheme" or "in an implementation scheme" or "in a specific implementation" or "in a specific implementation," and any other variations appearing throughout this application, do not necessarily refer to the same implementation scheme.

[0193] Additionally, this application may involve "determining" various types of information. Determining information may include, for example, one or more of the following: estimation information, calculation information, prediction information, or information retrieved from memory.

[0194] Furthermore, this application may relate to "accessing" various types of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or more of these.

[0195] Furthermore, this application may relate to "receiving" various types of information. Like "access," "receiving" is intended to be a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory) or more. Moreover, "receiving" typically involves one or more of the following during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0196] It should be understood that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one” is intended to cover selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such phrases are intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or selecting all three options (A, B, and C). As will be apparent to those skilled in the art and related fields, this can be extended to as many items as possible listed.

[0197] Moreover, as used herein, the term "signaling" refers to (among other things) instructing the corresponding decoder to do something. For example, in some implementations, the encoder signals a specific one of a plurality of transforms, encoding modes, or flags. Thus, in one implementation, the same transform, parameter, or mode is used on both the encoder and decoder sides. Therefore, for example, the encoder may transmit (explicit signaling) a specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter and others, signaling may be used without transmission (implicit signaling) to simply allow the decoder to know and select the specific parameter. Bit savings are achieved in various implementations by avoiding the transmission of any actual function. It should be understood that signaling can be implemented in various ways. For example, in various implementations, information is signaled to the corresponding decoder using one or more syntax elements, flags, etc. Although the verb form of the term "signal" has been used above, the term "signal" may also be used as a noun herein.

[0198] It will be apparent to those skilled in the art that embodiments can generate various signals formatted to carry, for example, storable or transmissible information. The information may include, for example, instructions for performing a method or data generated by one of the embodiments. For example, a signal may be formatted to carry a bitstream of the embodiment. Such signals may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of a spectrum) or baseband signals. Formatting may include, for example, encoding a data stream and modulating the carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is known that signals can be transmitted via various wired or wireless links. Signals may be stored on a processor-readable medium.

[0199] The preceding section describes multiple embodiments across various claim classes and types. Features of these embodiments may be provided individually or in any combination. Furthermore, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination, across various claim classes and types:

[0200] ●Signaling a signal may involve using at least a first encoding tool and a second encoding tool on a video block; and using the first encoding tool and the second encoding tool to encode the video block.

[0201] ● Parse the video bitstream to determine at least a first decoding tool and a second decoding tool for the video block; and decode the video block using at least a first encoding tool and a second encoding tool that will be used on the video block.

[0202] ●Any of the above-described embodiments having a first encoding / decoding tool including template matching prediction and a second encoding / decoding tool including one of matrix intra-frame prediction, intra-frame sub-partitioning, low-frequency inseparable transform, multiple transform selection, and intra-frame sub-partitioning.

[0203] ● Any of the above implementation schemes that use a single matching block as the prediction signal.

[0204] ● Any of the above implementation schemes using a variable or fixed search range.

[0205] ● Each of the above implementation schemes has a CABAC context export that signals a template match flag.

[0206] ● Any of the above embodiments having an encoding unit with a size using TMP can be variable.

[0207] ● Any of the above implementation schemes for removing redundant signaling.

[0208] ● The search scope extends to any of the above implementation schemes outside the current coding tree unit.

[0209] ● Any of the above implementation schemes that have IBC and TMP using different reference areas.

[0210] ● Use the general constraint flags of TMP to indicate deactivation.

[0211] ● In the case of using the LFNST transform kernel corresponding to the planar mode, TMP is used.

[0212] ● Template matching prediction is not allowed for multiple transformation selection.

[0213] ● TMP is used for both the luminance and chrominance components, where the syntax is used to signal the use of TMP for both the luminance and chrominance components.

[0214] ● Use template matching for the luminance component, where the matching block is positioned in the same location as the matching block used for the chrominance component.

[0215] ● Allows prediction of candidates both inside and outside the coding tree unit.

[0216] ● Use SIMD to calculate template differences.

[0217] ● Includes a bitstream or signal of one or more syntax elements or their variants from the described syntax elements.

[0218] ●Includes a bitstream or signal that transmits a syntax for information generated according to any of the embodiments described.

[0219] ●Creation and / or transmission and / or reception and / or decoding according to any one of the embodiments described.

[0220] ●A method, process, apparatus, medium for storing instructions, medium for storing data, or signal according to any one of the embodiments described above.

[0221] ● Insert a syntax element into the signaling that enables the decoder to determine the decoding information in a manner corresponding to that used by the encoder.

[0222] ● To create and / or send and / or receive and / or decode bit streams or signals that include one or more of the described syntax elements or their variants.

[0223] ● A television, set-top box, cellular phone, tablet computer, or other electronic device that performs the transformation method according to any of the embodiments described in the implementation plan.

[0224] ● A television, set-top box, cellular phone, tablet computer, or other electronic device that performs the transformation method to determine and display the resulting image (e.g., using a monitor, screen, or other type of display) according to any of the described implementation schemes.

[0225] ● A television, set-top box, cellular phone, tablet computer, or other electronic device that selects, band-limits, or tunes (e.g., uses a tuner) a channel according to any of the described implementation schemes to receive signals including encoded images and perform transformation methods.

[0226] ● A television, set-top box, cellular phone, tablet computer, or other electronic device that receives signals including encoded images over the air (e.g., using an antenna) and performs transformation methods.

Claims

1. An encoding method, the method comprising: Signal to notify the use of at least the first and second encoding tools on the video block; as well as The video block is encoded using the first encoding tool and the second encoding tool. The first encoding tool includes template matching prediction, and the second encoding tool includes a transform selection tool. Corresponding to the use of planar mode, or when using template matching prediction, no signal is sent to the multi-transformation selection flag, and the transformation selection to be applied to the residual of the block is implicitly determined based on the block size.

2. An encoding device, the device comprising: Processor, the processor being configured to: Signal to notify the use of at least the first and second encoding tools on the video block; as well as The video block is encoded using the first encoding tool and the second encoding tool. The first encoding tool includes template matching prediction, and the second encoding tool includes a transform selection tool. Corresponding to the use of planar mode, or when using template matching prediction, no signal is sent to the multi-transformation selection flag, and the transformation selection to be applied to the residual of the block is implicitly determined based on the block size.

3. A decoding method, the method comprising: Analyzing the video bitstream to determine at least a first decoding tool and a second decoding tool for video blocks, wherein the first decoding tool includes a template matching prediction (TMP) tool and the second decoding tool includes a transform selection tool; and The video block will be decoded using at least the first and second decoding tools to be used on the video block. When the template matching prediction tool is used on the video block, the multi-transform selection flag is not signaled in the bitstream, and the transformation to be applied to the residual of the video block is implicitly determined based on the size of the video block.

4. A decoding device, the device comprising: Processor, the processor being configured to: Analyzing the video bitstream to determine at least a first decoding tool and a second decoding tool for video blocks, wherein the first decoding tool includes a template matching prediction (TMP) tool and the second decoding tool includes a transform selection tool; and The video block will be decoded using at least the first and second decoding tools to be used on the video block. When the template matching prediction tool is used on the video block, the multi-transform selection flag is not signaled in the bitstream, and the transformation to be applied to the residual of the video block is implicitly determined based on the size of the video block.

5. The method of claim 1 or 3, or the apparatus of claim 2 or 4, wherein a universal constraint flag is used for TMP to indicate deactivation.

6. The method according to claim 1 or 3, or the apparatus according to claim 2 or 4, does not allow for multiple transformation selection using template matching prediction.

7. The method of claim 1 or 3, or the apparatus of claim 2 or 4, wherein the TMP is used for both the luminance and chrominance components, and a syntax is used to signal that the TMP is used for both the luminance and chrominance components.

8. The method according to claim 1 or 3, or the apparatus according to claim 2 or 4, wherein template matching is used for the luminance component, and the same position of the matching blocks is used for the chrominance component.

9. The method of claim 1 or 3, or the apparatus of claim 2 or 4, wherein prediction candidates are allowed from both inside and outside the coding tree unit.

10. The method of claim 1 or 3, or the apparatus of claim 2 or 4, wherein SIMD is used for calculating template differences.

Images