Template Matching Prediction for Versatile Video Coding

Abstract

Template matching prediction is combined with additional coding tools to provide a robust set of coding / decoding tools through advantageous use of syntax elements, some of which include matrix intra prediction, intra subdivision, low frequency non-separable transforms, and multiple transform selection.

Description

Technical Field

[0001] At least one of the embodiments generally relates to a method or apparatus for encoding or decoding video, compressing or decompressing.

Background Art

[0002] To achieve high compression efficiency, video and image coding schemes typically employ prediction including motion vector prediction, and transforms for exploiting spatial and temporal redundancy in video content. Generally, intra prediction or inter prediction is used to take advantage of correlations within or between frames, whereby the difference between the original image and the predicted image, often referred to as the prediction error or prediction residual, is transformed, quantized, and entropy coded. To reconstruct the video, the compressed data is decoded by inverse processes corresponding to entropy coding, quantization, transform, and prediction.

Summary of the Invention

[0003] At least one of the embodiments generally relates to a method or apparatus for encoding or decoding video, and more specifically, to a method or apparatus for using template matching prediction in combination with other coding tools such as the VVC (Versatile Video Coding) (versatile video coding or H.266) standard.

[0004] According to a first aspect, a method is provided. The method includes steps for signaling that at least a first and a second coding tool are used on a video block, and steps for encoding the video block using the first and second coding tools.

[0005] According to a second embodiment, another method is provided, which includes the steps of: analyzing a video bitstream to determine at least first and second decoding tools for a video block; and decoding the video block using the at least first and second coding tools to be used on the video block.

[0006] In another embodiment, the apparatus is provided, comprising a processor. The processor may be configured to encode blocks of video or decode a bitstream by performing one of the methods described above.

[0007] According to another general aspect of at least one embodiment, a device is provided comprising: an apparatus according to any of the decoding embodiments; and at least one of: (i) an antenna configured to receive a signal, the signal including a video block; (ii) a band limiter configured to restrict the received signal to a frequency band including a video block; or (iii) a display configured to display an output representing a video block.

[0008] Another generalized aspect of at least one embodiment provides a non-temporary computer-readable medium containing data content generated according to any of the decrypting embodiments or variations described.

[0009] According to another generalized aspect of at least one embodiment, a signal is provided that includes video data generated according to any of the described encoding embodiments or variations.

[0010] According to another generic aspect of at least one embodiment, the bitstream is formatted to include data content generated according to any of the decrypting embodiments or variations described.

[0011] According to another generalized form of at least one embodiment, a computer program product is provided which, when executed by a computer, includes instructions causing a computer to perform one of the decrypting embodiments or variations described.

[0012] The above and other embodiments, features, and advantages of the general-purpose embodiment will become apparent as you read the following detailed description of the exemplary embodiment with reference to the accompanying drawings. [Brief explanation of the drawing]

[0013] [Figure 1] This example illustrates the intra-predictive mode in multi-purpose video coding (VVC). [Figure 2] An example of template matching prediction (TMP) is provided. [Figure 3] This shows a standard, common video compression scheme. [Figure 4] This shows a standard, common video decompression scheme. [Figure 5] An embodiment of the method described in general terms is shown. [Figure 6] Another embodiment of the method under the generally described aspect is shown. [Figure 7] An exemplary apparatus under the described embodiment is shown. [Figure 8] This describes a processor-based system for encoding / decoding under the generally described manner. [Figure 9] This example demonstrates template matching using a single template. [Figure 10] An example of template matching prediction with a smaller search range is shown. [Figure 11] Examples of mutually exclusive search ranges for IBC and TMP are shown. [Figure 12] This indicates the currently permitted search range within the CTU. [Figure 13]This demonstrates SIMD optimization where the line widths of R1 and R2 are multiples of 4. [Figure 14] This shows a SIMD optimization where the top-left portion of the template is removed to have the same dimensions as the current block. [Modes for carrying out the invention]

[0014] The embodiments described herein are in the field of video compression and, in general, relating to video compression, video coding and decoding, more specifically, aim to improve the intra-predictive portion of video compression by employing template matching prediction. This mode generates a prediction signal from patches having similar templates within the reconstructed portion of an image.

[0015] To achieve high compression efficiency, image and video coding schemes typically employ predictions, including motion vector predictions, as well as transformations to leverage the spatial and temporal redundancy of video content. Generally, intra-prediction or inter-prediction is used to take advantage of intra-frame or inter-frame correlations, where the difference between the original and predicted images—often representing prediction error or prediction residual—is transformed, quantized, and entropicoded. To reconstruct the video, the compressed data is decoded through the reverse processes corresponding to entropicoding, quantization, transformation, and prediction.

[0016] Intra-picture prediction is a fundamental part of image and video compression. Traditionally, the prediction signal is generated from L-shaped reconstructed pixels (reference samples) by assuming them along different angles. This mechanism is known as angle prediction. VVC uses 65 intra-prediction modes, with 63 angle predictions, as well as DC and plane predictions defined (see Figure 1).

[0017] Traditional intranet prediction in VVC is enhanced with multiple tools. - Cross-component linear model (CCLM): The chroma prediction block is generated by a linear model of the remapped sample. - Multi-reference line prediction (MRL): More reference samples are used to generate the prediction block. - Intra sub-partitioning (ISP): The prediction block is divided into four sub-blocks that share the same prediction mode. - Matrix weighted intra prediction (MIP): The prediction block is generated by multiplying the reference samples by some offline optimized prediction matrices. - Intra block copy (IBC): The prediction block is generated by copying another block from a previously reconstructed image part, and the displacement vector is signaled in the bitstream.

[0018] The residual block is transformed in another mode which is known as the core transform DCT-II, or a combination of DST-VII and DCT-VIII known as multiple transform selection (MTS). The transformed block can be further transformed using a secondary non-separable transform to further compress the residual block. This process is called the low-frequency non-separable transform (LFNST).

[0019] Template Matching Prediction (TMP) is another powerful intra-prediction mode not included in VVC. It is performed by searching for similar L-shaped neighborhoods (called patches) to find the target block. This is illustrated in the following diagram. When TMP is used, the current template is formed by the reconstructed L-shaped neighborhoods. Similar templates with small differences from the current template are found. The blocks belonging to these templates (target blocks) are used to generate a prediction signal by averaging them or by considering only those with the smallest template difference.

[0020] Integrating TMP into VVC requires proper interoperability with existing intranet tools. That is, - Interaction with ISP, MIP, and MRL - Interaction with conversion tools (MTS and LFNST, implicit MTS) - Interaction with combined inter and intra prediction (CIIP)

[0021] The general embodiments described herein propose embodiments that enable this mode for VVCs with respect to these interactions.

[0022] Template matching prediction was an optional intra-prediction mode in the Joint Video Exploration Team's (JVET) Joint Exploration Test Mode (JEM). This involved an online process to derive the transformation matrix using the Karhunen-Loueve transform (KLT) and the same template from TMP. Prediction blocks were generated by averaging up to eight candidate blocks.

[0023] In relation to VVC development, region-based template matching prediction has been proposed in previous studies. Compared to the JEM version, these previous studies define smaller search areas, where an index is signaled to guide the decoder to restrict the search to a given area. This reduces latency on the decoder side.

[0024] The general-purpose embodiments described herein focus on implementing TMP in the context of VVC, and propose interactions with other tools and signals.

[0025] The basic idea of ​​the general-purpose embodiments described herein is to integrate the TMP into the VVC, and the interaction between them is defined as follows: - Interaction with ISP, MIP, and MRL - Interaction with conversion tools (MTS and LFNST, implicit MTS) - Interaction with combined inter- and intra-prediction (CIIP)

[0026] The affected encoder modules are the conversion module (125) and the intra prediction module (160) shown in Figure 3.

[0027] The affected decoder modules are the inverse transform module (250) and the intra prediction module (260) shown in Figure 4.

[0028] TMP Flag Signaling The CU flag is signaled to indicate the use of TMP. This flag can be signaled at different levels in the codec design. However, CU-level signaling is consistent with other intra-tools. The current signaling in VTM is as follows: - The MIP flag (intra_mip_flag) is the first to be signaled to indicate the use of MIP. -If MIP is not used, the MRL flag (intra_mip_flag) is signaled to indicate the use of MRL. -If MRL is not used, the ISP flag (intra_subpartitions_mode_flag) is signaled to indicate the use of an ISP. -If MRL is not used (MIP or ISP may be used), the prediction mode is further signaled.

[0029] The corresponding syntax is shown below.

[0030] [Table 1]

[0031] The following considerations are made regarding TMP. -MIP cannot be used with TMP, therefore MIP should not be signaled. - MRL cannot be used with TMP, therefore MRL should not be signaled. -TMP does not have a mode that should be signaled, therefore the intra-predictive mode should not be signaled. -ISP can be used with TMP, where each sub-partition uses a different matching template.

[0032] Therefore, the TMP flag may be signaled before the MIP flag, so that it is inferred to be 0 if all other flags used are not signaled. The corresponding changes are as follows (added parts are shaded):

[0033] [Table 2]

[0034] Here, sps_TMP_enabled_flag is the SPS level flag for activating the TMP, and intra_TMP_flag is the CU level flag for signaling the use of the TMP. MaxTMPSize is the maximum allowable size of the TMP. It is fixed or signaled at the SPS level.

[0035] This method does not allow the use of ISP and TMP together. This can be modified to enable the combination. This is done by signaling the TMP flag adjacent to the ISP flag, either before or after it. It should only be signaled when MRL and MIP are not used. The following modifications are made:

[0036] [Table 3]

[0037] For both methods, note that additional syntax may be signaled in the case of region-based template matching prediction. This is signaled immediately after the signal intra_TMP_flag.

[0038] In addition to the SPS flag, general constraint flags may be defined for TMP. This is similar to most VVC coding tools, which have higher-level flags to indicate whether something is deactivated or not. The following flags may be added to the specification:

[0039] [Table 4]

[0040] When gci_no_tmp_constraint_flag is equal to 1, it indicates that sps_TMP_enabled_flag is equal to 0.

[0041] Interaction with conversion tools: TMP can be used with MTS and / or LFNST. However, LFNST relies on an intra-predictive mode for the selection of the transformation kernel. The same approach as MIP can be used; that is, in the case of transformation matrix selection, MIP is considered a planar intra-predictive. Therefore, TMP can be thought of in this way as well; that is, when TMP is used, an LFNST transformation kernel corresponding to the planar mode is used.

[0042] Furthermore, since LFNST is used for the chroma component, the prediction mode is required to select the transformation matrix. Specifically, in the case of CCLM, the prediction mode is considered to be the planar mode when MIP is used for the lumana component. The same thing happens here when TMP is used for the lumana component.

[0043] In the case of MTS, there is a special mode known as implicit MTS, in which the MTS flag is not signaled, but the conversion selection is inferred from the block dimensions. However, the conversion selection for MIP mode or LFNST is DCT-II. The same applies to TMP.

[0044] The corresponding changes to the working draft are as follows:

[0045] [Table 5]

[0046] [Table 6]

[0047] To reduce encoder execution time, MTS may be disabled along with TMP. That is, when TMP is used as the predictive mode, the encoder does not attempt all possible MTS modes (DCT-II, DST-VII, and DCT-VIII) and does not signal any information to the decoder. Instead, a fixed conversion selection can be employed. The first selection is to use DCT-II for the vertical and horizontal directions. Another selection is to employ the implicit conversion selection used for ISP mode, namely the vertical and horizontal conversions are selected as follows: TrHor=width≦16?DST7:DCT2 TrVer=height≦16?DST7:DCT2

[0048] Interaction with MPM The most probable mode (MPM) is an effective way to signal the intra-predictive mode. MPM list generation requires knowledge of the current predictive mode. When TMP is used, it can be considered a planar mode for MPM list generation.

[0049] TMP for lunar and chromatic components Generally, TMP is used only for the lumern component. However, it can be extended to use both the lumern and chromatic components. To enable this, the following options are used. - A single flag is used to signal the use of TMP for both the lumens and chromats. - Template matching is performed only on the rumor components, and the same position of the matching block is used for the chroma components (using appropriate scaling according to chroma formation, e.g., 4:2:0). This reduces complexity by avoiding repeated searches for all components.

[0050] Interaction with CIIP CIIP is a prediction mode that combines both intra-prediction and inter-prediction. The intra-prediction portion is planar mode, and the inter-prediction portion is regular merge mode. TMP can be used to replace the intra-prediction portion (planar mode).

[0051] The weighting of the two prediction signals can also be modified. Currently, the weights are defined as follows, depending on the neighboring coding units:

[0052] [Table 7]

[0053] Alternatively, equal weighting of the inter and intra portions can be used.

[0054] Interaction with IBC In many cases, using both TMP and IBC is not useful because both perform intra-prediction by exploring within the reconstructed portion of the image. Therefore, it is proposed that IBC and TMP should not be activated together. This can be done by conditioning their SPS flags on each other; that is, if the SPS flag of IBC is 1, then the SPS flag of TMP is estimated to be 0. The reverse is also possible; if the SPS flag of TMP is 1, then the SPS flag of IBC is estimated to be 0.

[0055] An example of such a method is given in the following syntax table (conditioning TMP in IBC).

[0056] [Table 8]

[0057] Another example (conditioning IBC in TMP):

[0058] [Table 9]

[0059] One embodiment of method 500 under the general-purpose configuration described herein is shown in Figure 5. This method begins with a start block 501, and control proceeds to block 510, where it signals that at least the first and second coding tools will be used on the video block. Control proceeds from block 510 to block 520, where it encodes the video block using the first and second coding tools.

[0060] One embodiment of method 600 under the general-purpose aspects described herein is shown in Figure 6. This method begins with a start block 601, and control proceeds to block 610, where the video bitstream is analyzed to determine at least first and second decoding tools for a video block. Control proceeds from block 610 to block 620, where the video block is decoded using the at least first and second coding tools to be used on that video block.

[0061] Figure 7 shows one embodiment of a device 700 for encoding, decoding, compressing, or decompressing video data using a simplified coding mode based on a neighboring sample-dependent parametric model. The device comprises a processor 710 which can be interconnected to memory 720 through at least one port. Both the processor 710 and memory 720 may also have one or more additional interconnections to external connections.

[0062] Furthermore, the processor 710 is configured to insert or receive information in the bitstream and compress, encode, or decode it using any of the embodiments described.

[0063] Single-template prediction It is observed that a single matching block provides the best compression efficiency. That is, instead of searching for N similar templates, one similar template is found and used as the prediction signal. This process is illustrated in Figure 9.

[0064] Variable search range The search range for finding the optimal matching template(s) can be set to be variable or fixed. The maximum coding gain, related to the greatest complexity, occurs when the entire range is used; that is, the template is found within the entire reconfigured portion of the current frame. However, a smaller search range can be used to reduce complexity. An example is shown in Figure 10.

[0065] It is known that the maximum coding gain can be achieved with a search range equal to 1024. However, for video sequences with low resolution, a smaller search range can be used. It is also known that a search range of 64 has a reasonable trade-off between complexity and gain. Therefore, in this embodiment, the search range is defined as either 64, 128, 256, 1024, or full. High-level syntax elements may be used to signal the value of the search range.

[0066] Search range within CTU The search range can be limited to within the current CTU. That is, when the current block (PU / CU) is within a larger CTU, prediction candidates can be searched not only within the CTU but also outside of it. However, not all parts within the CTU can be decoded and therefore cannot be used as prediction candidates.

[0067] To allow searching within the current CTU, only candidates located in the upper left of the current block are used. This is shown in Figure 12. The darker search range is the interior of the current CTU where all pixels are decoded and reconstructed. The search range outside the CTU is also used, and the best candidate with the smallest template difference is used as the predicted block.

[0068] Template matching CABAC context derivation To signal the template matching flags, an appropriate context derivation for CABAC must be used. In this embodiment, the same procedure as for MIP is followed. This should be consistent with the VVC design. The following four contexts are used: 1. Initialize the context with 0. 2. Add 1 if the left CU uses template matching prediction. 3. Add 1 if the above CU uses template matching prediction. 4. If the width is greater than twice the height, or if the height is greater than twice the width, set the context to 3.

[0069] In other words, the four categories are as follows: 1-The nearest CU is not using TMP. 2-1 neighboring CUs are using TMP. 3-2 neighboring CUs are using TMP. The 4-CU shape is elongated in a single dimension.

[0070] If the fourth context is not used, a similar method can be used.

[0071] Variable maximum size The maximum CU size using TMP can be variable. It has been found that a maximum of 16 × 16 CUs provides the greatest coding gain with reduced complexity. The maximum dimension value can be signaled using high-level syntax elements or can be kept as 16.

[0072] Removed redundant signaling Signaling the TMP flag can be redundant, as it is known that TMP may be unavailable in some cases. Examples include cases where the CU size is larger than the maximum allowable size of TMP, when considering the previous embodiment (variable maximum size), or when there is not enough reconfiguration space to search for a similar template. For example, in the case of a CU of size 64x64, the reconfiguration space is less than 64x64 + the size of the template, so TMP is unavailable, and therefore its flag is not signaled and is presumed to be 0.

[0073] Current search range outside of CTU The search range can be simplified to exclude the current CTU. It is observed that there is no improved coding gain by including the current CTU in the search range. Therefore, as a simplification step, it is proposed to remove this region from the search range.

[0074] SIMD optimization By employing SIMD to calculate template differences, execution time can be reduced. In other words, the difference between pixels in two templates can be calculated in parallel via SIMD optimization.

[0075] In SIMD operations, each line being compared must be a multiple of 4. In Figure 13, since the block size is a multiple of 4 in VVC, the block dimensions (b_width and b_height) are always assumed to be multiples of 4. Therefore, in order to perform a comparison of multiples of 4, the line widths of R1 and R2 (Figure 13) must be multiples of 4. 1-R1: b_width+t_width must be a multiple of 4. Therefore, t_width must be a multiple of 4. 2-R2: T_width must be a multiple of 4. Therefore, T_width must be a multiple of 4.

[0076] Otherwise, the upper left portion of the template is removed to have the same dimensions as the current block. This is shown in Figure 14.

[0077] By doing so, the R1 width is equal to b_width (always a multiple of 4), and R2 is transposed before calculating the difference, ultimately having a width of b_heigh (always a multiple of 4). This method provides SIMD optimization regardless of the template size.

[0078] Mutually exclusive search range for IBC and TMP In this embodiment, the IBC and TMP use different reference areas. Typically, the IBC uses one CTU area as a reference. Therefore, the TMP should use the remainder of the already reconstructed picture as a reference.

[0079] For CTUs of size 32x32 or 64x64, the IBC can use two or more CTU areas as references, but it can always be used on the same CTU line as the current coding unit. In this case, the TMP should use all reconfigured pictures except for the current CTU line as references.

[0080] An example of mutually exclusive search ranges for IBC and TMP is shown in Figure 11.

[0081] The embodiments described herein include a variety of aspects, including tools, features, examples, models, approaches, and so on. Many of these aspects are described specifically and, in many cases, in a manner that may sound restrictive, at least to illustrate their individual characteristics. However, this is for the purpose of clarifying the description and not to limit the application or scope of those aspects. In practice, all of the different aspects can be combined and substituted to provide further embodiments. Furthermore, these embodiments can also be combined and substituted with embodiments described in previous applications.

[0082] The embodiments described and intended in this application can be implemented in many different forms. Figures 3, 4, and 8 provide some embodiments, but other embodiments are also intended, and the description of Figures 3, 4, and 8 does not limit the scope of implementation forms. At least one of the embodiments generally relates to video encoding and decoding, and at least one other embodiment generally relates to transmitting a generated or encoded bitstream. These and other embodiments can be implemented as a computer-readable storage medium storing in itself instructions for encoding or decoding video data according to any of the described methods, apparatus, and / or a computer-readable storage medium storing in itself a bitstream generated according to any of the described methods.

[0083] In this application, the terms “reconstructed” and “decoded” may be used interchangeably, the terms “pixel” and “sample” may be used interchangeably, and the terms “image,” “picture,” and “frame” may be used interchangeably. Typically, though not always, the term “reconstructed” is used on the encoder side, while the term “decoded” is used on the decoder side.

[0084] Various methods are described herein, each of which includes one or more steps or actions to achieve the described method. Unless a particular order of steps or actions is required for the proper operation of the method, the order and / or use of any particular steps and / or actions may be modified or combined.

[0085] Modules of the video encoder 100 and video decoder 200, such as those shown in Figures 10 and 11, including an intra-prediction module, an entropic coding module, and / or decoding modules (160, 360, 145, 330), can be modified using the various methods and other embodiments described herein. Furthermore, embodiments of this disclosure are not limited to VVC or HEVC and can be applied to other standards and recommendations, whether existing or future, and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, embodiments described herein can be used individually or in combination.

[0086] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes only, and the embodiments described are not limited to these specific values.

[0087] Figure 3 illustrates encoder 100. While variations of encoder 100 are also considered, for the sake of clarity, encoder 100 will be described below without explaining all anticipated variations.

[0088] Before encoding, the video sequence may undergo preprocessing (101), such as applying a color conversion to the input color picture (e.g., converting from RGB4:4:4 to YCbCr4:2:0), or performing a remapping of the input picture components to obtain a signal distribution more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the preprocessing and appended to the bitstream.

[0089] In encoder 100, the picture is encoded by encoder elements as described below. The picture to be encoded is divided into units, for example, CUs (102), and processed. Each unit is encoded using either intra-mode or inter-mode, for example. When a unit is encoded in intra-mode, it performs intra-prediction (160). In inter-mode, motion estimation (175) and motion compensation (170) are performed. The encoder determines whether to use intra-mode or inter-mode to encode a unit (105), and indicates the intra / inter decision, for example, by a prediction mode flag. The prediction residual is calculated, for example, by subtracting the predicted blocks from the original image blocks (110).

[0090] The predicted residual is then transformed (125) and quantized (130). The quantized transformation coefficients, as well as the motion vector and other syntax elements, are entropicoded to output a bitstream (145). The encoder can skip the transformation and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transformation and quantization, i.e., the residual is coded directly without applying either the transformation or quantization process.

[0091] The encoder decodes the encoded blocks to provide a reference for further prediction. The quantized transformation coefficients are inversely quantized (140) and inversely transformed (150) to decode the prediction residuals. The decoded prediction residuals and the predicted blocks are combined (155) to reconstruct the image blocks. An in-loop filter (165) is applied to the reconstructed picture to perform unblocking / sample adaptive offset (SAO)() filtering, for example, to reduce encoding artifacts. The filtered image is stored in a reference picture buffer (180).

[0092] Figure 4 shows a block diagram of the video decoder 200. In the decoder 200, the bitstream is decoded by the decoder elements, as described below. The video decoder 200 generally performs a decoding path that is the reverse of the encoding path described in Figure 3. The encoder 100 also generally performs video decoding as part of encoding the video data.

[0093] In particular, the input to the decoder includes a video bitstream, which may be generated by the video encoder 100. The bitstream is first entropy-decoded to obtain transformation coefficients, motion vectors, and other coded information (230). Picture segmentation information indicates how the picture is segmented. The decoder can therefore segment the picture according to the decoded picture segmentation information (235). The transformation coefficients are inversely quantized (240) and inversely transformed (250) to decode the prediction residuals. The image blocks are reconstructed by combining the decoded prediction residuals and the predicted blocks (255). Predicted blocks can be obtained from intra-prediction (260) or motion-compensated prediction (i.e., inter-prediction) (275) (270). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0094] The decoded picture may undergo further post-decoded processing (285), such as inverse color conversion (e.g., conversion from YcbCr4:2:0 to RGB4:4:4), or reverse remapping, which performs the reverse of the remapping process performed in pre-encoding processing (101). The post-decoded processing may use metadata derived in pre-encoding processing and signaled in the bitstream.

[0095] Figure 8 shows a block diagram of an example of a system in which various embodiments and models are implemented. System 1000 can be embodied as a device comprising various components described below and configured to perform one or more of the embodiments described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of System 1000 can be embodied individually or in combination in one integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of System 1000 are distributed across multiple ICs and / or separate components. In various embodiments, System 1000 is communicably coupled to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, System 1000 is configured to implement one or more of the embodiments described herein.

[0096] System 1000 includes, for example, at least one processor 1010 configured to execute instructions loaded internally in order to implement various embodiments described herein. The processor 1010 may include embedded memory, input / output interfaces, and various other circuits known in the art. System 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). System 1000 includes a storage device 1040, which may include non-volatile memory and / or volatile memory, such as electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, magnetic disk drives, and / or optical disk drives. The storage device 1040 may, in non-limiting examples, include an internal storage device, an attached storage device (including removable and non-removable storage devices), and / or a network-accessible storage device.

[0097] System 1000 includes, for example, an encoder / decoder module 1030 configured to process data to provide encoded or decoded video, the encoder / decoder module 1030 of which may include its own processor and memory. The encoder / decoder module 1030 corresponds to a module that can be included in a device that performs encoding and / or decoding functions. As is known, the device may include one or both of the encoding and decoding modules. Furthermore, the encoder / decoder module 1030 may be implemented as a separate element of System 1000, or it may be incorporated into the processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0098] Program code loaded into the processor 1010 or encoder / decoder 1030 to perform the various embodiments described herein can be stored in the storage device 1040 and subsequently loaded into memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 can store one or more of various items during the execution of the storage process described herein. Such stored items may include, but are not limited to, input video, decoded video, or a portion of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0099] In some embodiments, the internal memory of the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory outside the processing device (for example, the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, for example, dynamic volatile memory and / or non-volatile flash memory. In some embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, a high-speed external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations such as MPEG-2 (MPEG stands for Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Variable-Purpose Video Coding, a new standard under development by JVET).

[0100] Inputs to the elements of system 1000 can be provided through various input devices shown in block 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) section for receiving RF signals transmitted over the entire broadcast by a broadcaster, (ii) a component (COMP) input terminal (or set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Although not shown in Figure 8, another example is composite video.

[0101] In various embodiments, the input device of block 1130 has associated input processing elements as known in the Art. For example, the RF portion may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency in a certain band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band in order to select a signal frequency band that may be referred to as a channel in a particular embodiment, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a stream of desired data packets. The RF portion of various embodiments includes one or more elements that perform these functions, e.g., frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion may include tuners that perform various of these functions, e.g., down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or to baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements receive an RF signal transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering it to a desired frequency band. Various embodiments may involve rearranging the order of the elements described above (and others), removing some of these elements, and / or adding other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting an amplifier and an analog-to-digital converter. In various embodiments, the RF section includes an antenna.

[0102] In addition, the USB terminal and / or HDMI terminal may include their respective interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, can be implemented, for example, in a separate input processing IC or within the processor 1010, as needed. Similarly, aspects of USB or HDMI interface processing can be implemented, for example, in a separate interface IC or within the processor 1010, as needed. The demodulated, error-corrected, and demultiplexed stream is provided to various processing elements, for example, the processor 1010 and an encoder / decoder 1030 that operates in conjunction with memory and storage elements to process the data stream as needed for presentation on an output device.

[0103] Various elements of system 1000 may be provided within an integrated housing, where the various elements are interconnected using internal buses known in the art, such as Inter-IC (I2C) buses, wiring, and printed circuit boards, and can transmit data between them.

[0104] System 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented, for example, in a wired and / or wireless medium.

[0105] In various embodiments, the data is streamed to system 1000 or otherwise provided using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (Institute of Electrical and Electronics Engineers (IEEE) refers to the American Institute of Electrical and Electronics Engineers). The Wi-Fi signals in these embodiments are received on a communication channel 1060 and a communication interface 1050 adapted for Wi-Fi communication. In these embodiments, communication channel 1060 is typically connected to an access point or router that provides access to an external network, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, streaming data is provided to system 1000 using a set-top box that distributes data via an HDMI connection on input block 1130. In yet another embodiment, streaming data is provided to system 1000 using an RF connection on input block 1130. As shown above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, e.g., cellular networks or Bluetooth networks.

[0106] System 1000 can provide output signals to various output devices, including a display 1100, a speaker 1110, and other peripheral devices 1120. In various embodiments, the display 1100 includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 1100 may be for a television, tablet, laptop, mobile phone, or other device. The display 1100 may also be integrated with other components (e.g., as in a smartphone) or separate (e.g., an external monitor for a laptop). Other peripheral devices 1120, in various embodiments of the embodiment, include one or more of a standalone digital video disc (or digital versatile disc, DVR) (for both terms, digital versatile disc, DVR), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide functionality based on the output of System 1000. For example, the disc player performs the function of playing the output of system 1000.

[0107] In various embodiments, control signals are communicated between the system 1000 and the display 1100, speaker 1110, or other peripheral devices 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable control between devices with or without user intervention. Output devices can be communicably coupled to the system 1000 via dedicated connections through their respective interfaces 1070, 1080, and 1090. Alternatively, output devices can be connected to the system 1000 using communication channel 1060 via communication interface 1050. The display 1100 and speaker 1110 can be integrated into a single unit with other components of the system 1000 in an electronic device such as a television. In various embodiments, the display interface 1070 includes a display driver, such as a timing controller (TCon) chip.

[0108] For example, if the RF section of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 can, alternatively, be separate from one or more of the other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal can be provided via a dedicated output connection, such as an HDMI port, a USB port, or a COMP output.

[0109] The embodiments can be implemented by a processor 1010, by hardware, or by a combination of hardware and software, or by computer software. In a non-limiting example, the embodiments can be implemented by one or more integrated circuits. The memory 1020 can be of any type appropriate to the technical environment and, in a non-limiting example, can be implemented using any suitable data storage technology such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and live-bubble memory. The processor 1010 can be of any type appropriate to the technical environment and, in a non-limiting example, can include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0110] Various implementations include decoding. As used in this application, “decoding” can encompass all or part of the processing performed on the received encoded sequence to produce, for example, a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes may also include, or alternatively, processes performed by the decoders of the various implementations described in this application.

[0111] As further examples, in one embodiment, “decoding” refers only to entropy decoding; in another embodiment, “decoding” refers only to differential decoding; and in yet another embodiment, “decoding” refers to a combination of entropy decoding and differential decoding. Whether the phrase “encoding process” is intended to refer specifically to a working subset or to refer to a broader encoding process as a whole will become clear from the context of the specific explanation and will be well understood by those skilled in the art.

[0112] Various implementations involve encoding. As used in this application, “encoding” can encompass, for example, all or part of the processing performed on an input video sequence to generate an encoded bitstream, similar to the above description of “decoding.” In various embodiments, such processing includes one or more processes typically performed by an encoder, such as, for example, splitting, differential coding, transformation, quantization, and entropy coding. In various embodiments, such processing may further or alternatively include processes performed by the encoders of the various implementations described in this application.

[0113] As further examples, in one embodiment, “encoding” refers only to entropy coding; in another embodiment, “encoding” refers only to differential coding; and in yet another embodiment, “encoding” refers to a combination of differential coding and entropy coding. Whether the phrase “encoding process” is intended to refer specifically to a working subset or to refer to a broader encoding process as a whole will become clear from the context of the specific explanation and will be well understood by those skilled in the art.

[0114] Please note that the syntax elements used herein are descriptive terms; therefore, they do not preclude the use of other syntax element names.

[0115] If a diagram is presented as a flowchart, it should be understood that the diagram also provides a block diagram of the corresponding device. Similarly, if a diagram is presented as a block diagram, it should be understood that the diagram also provides a flowchart of the corresponding method / process.

[0116] Various embodiments may relate to parametric models or rate distortion optimization. Specifically, during the coding process, the balance or trade-off between rate and distortion is usually considered to impose computational complexity constraints. This can be measured through the Rate Distortion Optimization (RDO) metric, or through Least Mean Square (LMS), Mean of Absolute Errors (MAE), or other such measurements. Rate distortion optimization is usually formulated to minimize a rate distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving rate distortion optimization problems. For example, these approaches are obtained based on extensive testing of all coding options, including all considered mode or coding parameter values, and involve a complete evaluation of their coding costs, as well as the associated distortions of the reconstructed signals after coding and decoding. To reduce coding complexity, faster methods may also be used, particularly with the calculation of approximate distortions based on the predicted or predicted residual signal rather than the reconstructed signal. A mixture of these two methods can also be used, for example, by using approximate distortion for only a subset of the possible coding options and full distortion for the others. Other methods evaluate only a subset of the possible coding options. More generally, many approaches employ one of various techniques to perform optimization, but the optimization is not necessarily a full evaluation of both the coding cost and the associated distortion.

[0117] The implementations and embodiments described herein may be implemented, for example, in methods or processes, apparatus, software programs, data streams, or signals. Even if considered only in the context of a single implementation (for example, considered only as a method), the implementations of the considered features may also be implemented in other forms (for example, apparatus or programs). For example, apparatus may be implemented in appropriate hardware, software, and firmware. A method may be implemented, for example, in a processor, where a processor refers to a general processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. A processor may also include communication devices, such as, for example, computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate the communication of information between end users.

[0118] References to “one embodiment” or “a certain embodiment,” or “one implementation” or “a certain implementation,” or to other variations thereof, mean that the specific features, structures, characteristics, etc. described in relation to that embodiment are included in at least one embodiment. Therefore, when the phrases “in one embodiment” or “in a certain embodiment,” or “in one implementation” or “in a certain implementation,” or other variations appear in various places throughout this application, they do not necessarily all refer to the same embodiment.

[0119] In addition, this application may refer to "determining" various types of information. Determining information may include, for example, one or more of the following: estimating information, calculating information, predicting information, or retrieving information from memory.

[0120] Furthermore, this application may refer to "accessing" various types of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0121] In addition, this application may refer to "receiving" various types of information. Receiving is intended to be a broad term, similar to "accessing." Receiving information may include, for example, accessing information or retrieving information (for example, from memory). Furthermore, "receiving" generally involves in some way operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0122] For example, in the cases of "A / B", "A and / or B", and "at least one of A and B", it should be understood that the use of any of the following " / ", "and / or", and "at least one of" is intended to encompass the selection of only the first listed option (A), only the second listed option (B), or both options (A and B). In further embodiments, in the cases of "A, B, and / or C" and "at least one of A, B, and C," such expressions are intended to encompass the selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or the selection of all three options (A, B, and C). This can be extended to the number of listed items, as will be apparent to those skilled in the art in this and related fields.

[0123] Furthermore, as used herein, the term “signaling” specifically means indicating something to a corresponding decoder. For example, in certain embodiments, an encoder signals one particular one of several transformations, encoding modes, or flags. Thus, in certain embodiments, the same transformation, parameter, or mode is used on both the encoder and decoder sides. Therefore, for example, an encoder can transmit a particular parameter to a decoder so that the decoder can use the same particular parameter (explicit signaling). In contrast, if the decoder already has other parameters along with that particular parameter, it can use non-transmitting signaling (implicit signaling) so that the decoder simply knows and can select that particular parameter. Bit saving is achieved in various embodiments by avoiding the transmission of any actual function. It will be understood that signaling can be achieved in various ways. For example, one or more syntax elements, flags, etc., are used in various embodiments to signal information to a corresponding decoder. The above relates to the verb form of the word “signal,” which may also be used as a noun herein.

[0124] As will be obvious to those skilled in the art, the implementation can result in a variety of signals formatted to carry information that can be stored or transmitted. The information may include, for example, instructions for performing a method, or data generated by one of the implementations described. For example, a signal can be formatted to carry a bitstream of the embodiment described. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, signals can be transmitted over a variety of different wired or wireless links. The signals can be stored in a processor-readable medium.

[0125] The preceding sections describe several embodiments across various claim categories and types. Features of these embodiments can be provided individually or in any combination. Furthermore, embodiments may include, individually or in any combination, one or more of the following features, devices, or aspects across various claim categories and types: - Signal that at least the first and second coding tools may be used on the video block, and encode the video block using the first and second coding tools. The video bitstream is analyzed to determine at least first and second decoding tools for a video block, and the video block is decoded using the at least first and second coding tools to be used on that video block. Any of the above embodiments, comprising a first coding / decoding tool including template matching prediction, and a second coding / decoding tool including one of matrix intra prediction, intra subdivision, low-frequency inseparable transformation, multiple transformation selection, and intra subdivision. Any of the above embodiments that use a single matching block as a predictive signal. • Any of the above embodiments that use a variable or fixed search range. Any of the above embodiments, which involve CABAC context derivation that signals a template matching flag. Any of the above embodiments having a coding unit having a size that uses a TMP may be variable. • Any of the above embodiments in which redundant signaling has been removed. Any of the above embodiments having a search scope outside the current coding tree unit. • Any of the above embodiments having IBC and TMP using different reference regions • Use general constraint flags for TMP to indicate deactivation. • Uses a TMP that employs an LFNST conversion kernel compatible with planar mode. • Do not allow multiple transformation selections based on template matching prediction. • TMP is used for both the lumens and chromens components, and the syntax is used to signal the use of TMP for both the lumens and chromens components. • Use template matching for lumen components that have the same position in the matching block used for chroma components. • Allows prediction candidates both inside and outside the coding tree unit. • Use SIMD to calculate template differences. A bitstream or signal containing one or more of the described syntax elements, or variations thereof. A bitstream or signal containing a syntax that carries information generated in accordance with any of the embodiments described. • Create and / or transmit and / or receive and / or decode by any of the embodiments described. • A method, process, apparatus, medium for storing instructions, medium for storing data, or signal, according to any of the embodiments described. - Insert syntax elements into the signaling that allow the decoder to determine the decoded information in a manner corresponding to its use by the encoder. • Create and / or transmit and / or receive and / or decode a bitstream or signal containing one or more of the described syntax elements or their variations. • An electronic device such as a television, set-top box, mobile phone, or tablet that performs a conversion method according to any of the embodiments described. An electronic device such as a television, set-top box, mobile phone, or tablet that determines a conversion method according to any of the embodiments described and displays the resulting image (for example, using a display of a type such as a monitor or screen). A television, set-top box, mobile phone, tablet, or other electronic device that selects, band-limits, or tunes (e.g., using a tuner) a channel to receive a signal containing an encoded image, and performs a conversion method according to any of the embodiments described. A TV, set-top box, mobile phone, tablet, or other electronic device that receives and converts signals containing encoded images throughout the entire broadcast.

Claims

1. It is a method, A step of signaling that at least the first and second coding tools are to be used on the video block, The steps include encoding the video block using the first and second coding tools, Equipped with, A method that uses template matching prediction and employs an LFNST transform kernel corresponding to the planar mode.

2. It is a device, Signal that at least the first and second coding tools are used on the video block, and A processor configured to encode the video block using the first and second coding tools, A device that uses template matching prediction and employs an LFNST transform kernel corresponding to planar mode.

3. It is a method, A step of analyzing a video bitstream to determine at least first and second decoding tools for a video block, The steps include: decoding the video block using the at least first and second decoding tools which will be used for the video block; Equipped with, A method that uses template matching prediction and employs an LFNST transform kernel corresponding to the planar mode.

4. It is a device, The video bitstream is analyzed to determine at least first and second decoding tools for the video block, and A processor configured to decode the video block using at least the first and second decoding tools to be used for the video block, A device that uses template matching prediction and employs an LFNST transform kernel corresponding to planar mode.

5. The method according to claim 1 or 3, wherein a general constraint flag is used in the template matching prediction (TMP) to indicate deactivation.

6. The method according to claim 1 or 3, wherein template matching prediction (TMP) is used together with LFNST, and an LFNST transform kernel corresponding to the planar mode is used for the template matching prediction.

7. Steps to prevent multiple transformation selections during template matching prediction. The method according to claim 1 or 3, further comprising:

8. The method according to claim 1 or 3, wherein template matching prediction (TMP) is used for the lumern and chroma components, and syntax is used to signal the use of TMP for both the lumern and chroma components.

9. The method according to claim 1 or 3, wherein template matching is used for the rumor component, and the same position in the matching block is used for the chroma component.

10. The method according to claim 1 or 3, wherein prediction candidates from inside and outside the coding tree unit are permitted.

11. The apparatus according to claim 2 or 4, wherein a general constraint flag is used in template matching prediction (TMP) to indicate deactivation.

12. The apparatus according to claim 2 or 4, wherein template matching prediction (TMP) is used together with LFNST, and an LFNST transformation kernel corresponding to the planar mode is used for template matching prediction.

13. The apparatus according to claim 2 or 4, wherein the processor is further configured not to allow multiple transformation selections during template matching prediction.

14. The apparatus according to claim 2 or 4, wherein template matching prediction (TMP) is used for the lumern and chroma components, and syntax is used to signal the use of TMP for both the lumern and chroma components.

15. The apparatus according to claim 2 or 4, wherein template matching is used for the rumor component, and the same position of the matching block is used for the chroma component.

16. The apparatus according to claim 2 or 4, wherein prediction candidates from inside and outside the coding tree unit are permitted.

17. The apparatus according to claim 2 or 4, wherein SIMD is used to calculate the template difference.

Images