Template type selection for video coding and decoding

Abstract

A video coding system may use different template types when using a template-based prediction mode, such as an intra-template matching prediction mode. The "L-shaped" template traditionally used by template-based coding tools may be divided into a horizontal "top" (or "upper") template and a vertical "left" template. This allows the encoder to select the appropriate portion of the template that better captures the statistical properties of the current block, resulting in improved coding performance. When using a template-based prediction mode, such as an intra-template matching prediction mode, the use of the template type is signaled in the encoded data so that the decoder can use the appropriate template for reconstruction.

Description

[Technical Field]

[0001] This application claims priority to European Application No. 22306519.4, filed October 10, 2022, which is incorporated herein by reference in its entirety.

[0002] FIELD OF THE INVENTION The present disclosure is in the field of video compression, and at least one embodiment relates more particularly to signaling the type of template used in a template-based coding tool. [Background technology]

[0003] To achieve high compression efficiency, image and video coding schemes typically use prediction and transform to exploit spatial and temporal redundancy in the video content. Generally, intra- or inter-prediction is used to exploit intra- or inter-frame correlation, and then the difference between the original and predicted picture block, often denoted as prediction error or prediction residual, is transformed, quantized, and entropy coded. To reconstruct the video, the compressed data is decoded by the inverse processes corresponding to entropy coding, quantization, transformation, and prediction. Summary of the Invention

[0004] At least one embodiment provides the encoder with the possibility to use different types of templates for template-based coding tools. The "L-shaped" templates traditionally used by template-based coding tools can be split into horizontal "top" (or "upper") templates and vertical "left" templates. This allows the encoder to select the appropriate part of the template that better captures the statistical properties of the current block, resulting in improved coding performance. The use of template types is signaled in the encoded data so that the decoder uses the appropriate template for reconstruction.

[0005] A first aspect is directed to a method comprising providing information indicative of a type of template to be used by a template-based prediction mode, such as an intra-template matching prediction mode, for a current block of a picture, wherein the type of template is selected from among a first template including a set of neighboring pixels above the current block, a second template including a set of neighboring pixels to the left of the current block, or a third template that is a combination of the first and second templates.

[0006] A second aspect is directed to a method including: selecting a prediction mode and a type of template for a current block of a picture based on a coding cost; predicting the block based on the selected prediction mode and type of template; encoding the current block based on the predicted block; and providing coding information for the current block including at least information representing use of a template-based prediction mode according to the first aspect and information representing the type of template.

[0007] A third aspect is directed to a device comprising a processor configured to: select a prediction mode and a type of template for a current block of a picture based on a coding cost; predict the block based on the selected prediction mode and type of template; encode the current block based on the predicted block; and provide coding information for the current block including at least information representing use of a template-based prediction mode according to the first aspect and information representing the type of template.

[0008] A fourth aspect is directed to a method including: obtaining, for a current block of a picture, coding information for the current block, the coding information including at least information representing the use of a template-based prediction mode according to the first aspect and information representing a type of template; predicting the block based on the prediction mode and the type of template; and decoding the current block based on the predicted block.

[0009] A fifth aspect is directed to a device comprising a processor configured to obtain, for a current block of a picture, coding information for the current block, the current block including at least information representing the use of a template-based prediction mode according to the first aspect and information representing a type of template; predicting the block based on the prediction mode and the type of template; and decoding the current block based on the predicted block.

[0010] A sixth aspect is directed to a computer program product comprising instructions that, when executed by a computer, cause the computer to perform a method according to the first, second or fourth aspect.

[0011] A seventh aspect is a non-transitory computer-readable medium storing executable program instructions, the instructions causing a computer executing the instructions to perform a method of the first aspect, the second aspect, or the fourth aspect.

[0012] An eighth aspect is directed to a bitstream representing a coded picture formatted to include a syntax element indicating a template type of a current block according to the first aspect.

[0013] A ninth aspect is directed to a non-transitory computer-readable medium that stores information representing an encoded picture, the information being formatted to include a syntax element that indicates a template type for a current block according to the method of the first aspect.

[0014] The foregoing presents a simplified summary of the subject matter in order to provide a basic understanding of some aspects of the disclosure. This summary is not an extensive overview of the subject matter. It is not intended to identify key / critical elements of embodiments or to delineate the scope of the subject matter. Its sole purpose is to present some concepts of the subject matter in a simplified form as a prelude to the more detailed description provided below. [Brief explanation of the drawings]

[0015] The present disclosure may be better understood upon consideration of the following detailed description in conjunction with the accompanying figures. [Figure 1] 1 shows a block diagram of a video encoder according to one embodiment. [Figure 2] 1 shows a block diagram of a video decoder according to one embodiment. [Figure 3] 1 illustrates a block diagram of an example system in which various aspects and embodiments may be implemented. [Figure 4A] We present the principle of template-based intra-mode derivation. [Figure 4B] We present the principle of template-based intra-mode derivation. [Figure 4C] We present the principle of template-based intra-mode derivation. [Figure 5] The principle of intra mode derivation on the decoder side is shown. [Figure 6] The principle of intra-template matching is shown. [Figure 7] The principle of template matching is shown. [Figure 8] 10 shows the reference region restriction for intra block copy using template matching mode. [Figure 9A]1 illustrates a flowchart of an example of an encoding process for a template-based prediction mode, according to at least one embodiment. [Figure 9B] 1 illustrates a flowchart of an example of a decoding process for a template-based prediction mode, according to at least one embodiment. [Figure 10] Indicates the type of template. [Figure 11] 10 illustrates an example of CTU partitioning with CU decoding order. [Figure 12] 1 shows a flowchart of an example of an encoding process using intra template matching prediction mode, according to at least one embodiment. [Figure 13] 1 illustrates a flowchart of an example of a decoding process using an intra-template matching prediction mode, according to at least one embodiment.

[0016] It should be understood that the drawings are intended to illustrate examples of various aspects, features, and embodiments according to the present disclosure, and are not necessarily the only possible configuration. Like reference designators throughout the various views refer to the same or similar features. DETAILED DESCRIPTION OF THE INVENTION

[0017] As described in more detail below, the video codec can involve determining a predictive block for a current block based on samples of the selected block, the block being selected within the area of ​​decoded picture information based on a template matching process involving a comparison of a template associated with the current block to at least one other template associated with at least one other block within the area of ​​decoded picture information. Encoding methods, decoding methods, encoding devices, and decoding devices according to present principles are described.

[0018] Additionally, although the present aspects describe principles related to particular drafts of the VVC (Versatile Video Coding) or HEVC (High Efficiency Video Coding) specifications, they are not limited to VVC or HEVC and may be applied, for example, to other standards and recommendations, whether existing or developed in the future, and to extensions of any such standards and recommendations (including VVC and HEVC).Unless otherwise indicated or technically excluded, the aspects described in this application may be used individually or in any combination.

[0019] 1 shows a block diagram of a video encoder according to one embodiment. While variations of this encoder 100 are contemplated, the encoder 100 is described below for clarity without describing all possible variations. Before being encoded, a video sequence may undergo a pre-encoding process (101), such as applying a color transformation to an input color picture (e.g., from RGB 4:4:4 to YCbCr 4:2:0) or performing a remapping of the input picture components to obtain a signal distribution more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata associated with the pre-processing can be attached to the bitstream.

[0020] In encoder 100, a picture is coded by encoder elements as described below. The picture to be coded is partitioned (102) and processed, for example, in units of CUs. Each unit is coded, for example, using either intra mode or inter mode. If the unit is coded in intra mode, intra prediction is performed (160). In inter mode, motion estimation (175) and motion compensation (170) are performed. The encoder decides (105) whether to use intra mode or inter mode to code the unit, and indicates the intra / inter decision, for example, by a prediction mode flag. A prediction residual is calculated, for example, by subtracting (110) the predicted block from the original image block.

[0021] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, plus motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., the residual is coded directly without applying a transform or quantization process.

[0022] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150), and the prediction residual is decoded. The decoded prediction residual is combined (155) with the predicted block to reconstruct an image block. An in-loop filter (165) is applied to the reconstructed picture, performing, for example, deblocking / sample adaptive offset (SAO) and adaptive loop-filter (ALF) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0023] FIG. 2 shows a block diagram of a video decoder according to one embodiment. In the decoder 200, the bitstream is decoded by elements of the decoder, as described below. The video decoder 200 generally performs a decoding pass that is the inverse of the encoding pass. The encoder 100 also generally performs video decoding as part of encoding the video data. Specifically, the decoder's input includes a video bitstream, which may be generated by the video encoder 100. First, the bitstream is entropy decoded (230) to obtain transform coefficients, motion vectors, and other coded information. Picture partition information indicates how the picture is partitioned. Thus, the decoder may divide the picture according to the decoded picture partition information (235). The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual is combined with the prediction block (255) to reconstruct an image block. The predicted block (270) can be obtained from intra prediction (260) or motion compensated prediction (i.e., inter prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0024] The decoded picture may further undergo a post-decoding process (285), such as an inverse color transform (e.g., YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping that performs the inverse of the remapping process performed in the pre-encoding process (101). The post-decoding process may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0025] FIG. 3 shows a block diagram of an example system in which various aspects and embodiments can be implemented. System 1000 can be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of system 1000, either singly or in combination, can be embodied in a single integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or separate components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more of the aspects described herein.

[0026] The system 1000 includes at least one processor 1010 configured to execute loaded instructions, for example, to implement various aspects described herein. The processor 1010 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 1010 may include embedded memory, input / output interfaces, and various other circuits as known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include non-volatile memory and / or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drives, and / or optical disk drives.Storage devices 1040 may include, by way of non-limiting example, internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0027] System 1000 includes an encoder / decoder module 1030 configured to process data to provide, for example, encoded video or decoded video, which may include its own processor and memory. Encoder / decoder module 1030 represents a module that may be included in a device that performs encoding and / or decoding functions. As is known, a device may include one or both of an encoding and a decoding module. Additionally, encoder / decoder module 1030 may be implemented as a separate element of system 1000 or may be incorporated within processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0028] Program code loaded into the processor 1010 or the encoder / decoder 1030 to perform various aspects described herein may be stored in the storage device 1040 and subsequently loaded into the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items during execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, expressions, operations, and computational logic.

[0029] In some embodiments, memory internal to the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be either the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be the memory 1020 and / or the storage device 1040, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, for example, external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, a high-speed external dynamic volatile memory such as RAM is used as working memory for video coding and decoding operations such as MPEG-2 (MPEG stands for Moving Picture Experts Group, MPEG-2 is also referred to as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, an emerging standard developed by JVET, i.e., the Joint Video Experts Team).

[0030] Inputs to the elements of system 1000 may be provided through various input devices, as shown in block 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives, for example, RF signals transmitted over the air by a broadcast station, (ii) a component (COMP) input terminal (or set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and / or (iv) a High-Definition Multimedia Interface (HDMI) input terminal. Other examples not shown in FIG. 3 include composite video.

[0031] In various embodiments, the input devices of block 1130 have associated respective input processing elements as known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or bandlimiting a signal to a frequency band), (ii) downconverting the selected signal, (iii) bandlimiting again to a narrower frequency band to select a signal frequency band, which in certain embodiments may be referred to as a channel (for example), (iv) demodulating the downconverted, bandlimited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements that perform these functions, e.g., a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF section may include, for example, a tuner that performs various of these functions, including downconverting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency near baseband) or to baseband. In one set-top box embodiment, the RF section and its associated input processing elements perform frequency selection by receiving, filtering, downconverting, and filtering again to the desired frequency band an RF signal transmitted over a wired (e.g., cable) medium. In various embodiments, the order of these (and other) elements is rearranged, some of these elements are removed, and / or other elements that perform similar or different functions are added. Adding elements can include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0032] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or within processor 1010, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented, as desired, within a separate interface IC or within processor 1010. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 1010 and an encoder / decoder 1030, which operates in combination with memory and storage elements to process the data stream as desired for display on an output device.

[0033] The various elements of system 1000 may be provided within a unitary housing in which the various elements are interconnected and capable of transmitting data therebetween using a suitable connection arrangement 1140, e.g., an internal bus as known in the art, including an Inter-IC (I2C) bus, wiring, and printed circuit boards.

[0034] System 1000 includes a communication interface 1050 that enables communication with other devices over a communication channel 1060. Communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 1060. Communication interface 1050 may include, but is not limited to, a modem or a network card, and communication channel 1060 may be implemented in a wired and / or wireless medium, for example.

[0035] In various embodiments, data is streamed or otherwise provided to system 1000 using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in these embodiments is received via communication channel 1060 and communication interface 1050, which are adapted for Wi-Fi communication. Communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, enabling streaming applications and other over-the-top communications. Other embodiments provide streamed data to system 1000 using a set-top box that delivers data via an HDMI connection in input block 1130. Still other embodiments provide streamed data to system 1000 using an RF connection in input block 1130. As noted above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0036] System 1000 can provide output signals to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. Display 1100 in various embodiments includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 1100 may be for a television, a tablet, a laptop, a mobile phone, or other device. Display 1100 may also be integrated with other components (e.g., as in a smartphone) or may be separate (e.g., an external monitor for a laptop). Other peripheral devices 1120, in various example embodiments, include one or more of a standalone digital video disc (or digital versatile disc) (both terms DVR), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 to provide functionality based on the output of system 1000. For example, a disc player performs the function of playing the output of system 1000.

[0037] In various embodiments, control signals are communicated between system 1000 and display 1100, speakers 1110, or other peripheral devices 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable inter-device control with or without user intervention. Output devices may be communicatively coupled to system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, output devices may be connected to system 1000 using communication channel 1060 via communication interface 1050. Display 1100 and speakers 1110 may be integrated into a single unit with other components of system 1000 in an electronic device such as a television. In various embodiments, display interface 1070 includes a display driver, such as a timing controller (TCon) chip.

[0038] Display 1100 and speakers 1110 may alternatively be separate from one or more of the other components, for example, if the RF portion of input 1130 is part of a separate set-top box. In various embodiments in which display 1100 and speakers 1110 are external components, the output signal may be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0039] The embodiments may be implemented by the processor 1010, or by computer software implemented by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type appropriate to the technical environment and may be implemented using any suitable data storage technology, such as, by way of non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 1010 may be of any type appropriate to the technical environment and may include, by way of non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0040] The technical field of the embodiments relates to the intra prediction stage of a video compression scheme, and more particularly to a template matching tool. A template matching tool is based on the assumption that if neighboring pixels of a block to be reconstructed can be found within an already reconstructed area, then an already reconstructed block corresponding to a matching template is likely to be similar to the block to be reconstructed. When a template matching tool is used, a device first determines, for a current block, a set of pixels that form the neighborhood of the current luma block, i.e., pixels adjacent to the block. This set of pixels, which takes the form of a traditional L-shape, is called a template. Various sizes (e.g., 1, 2, or 4 pixels wide) can be used as templates, but they are still based on the same principle. This process is similar on both sides. In other words, it is implemented by both an encoder device and a decoder device. Only a few signaling elements are required to determine which template-based tool should be used. Various template matching-based tools may be used to code and decode video. Some of them are described below.

[0041] 4A, 4B, and 4C illustrate the principle of template-based intra mode derivation. Template-based intra mode derivation (TIMD) is the first tool based on template matching. For a given luma coding block (CB) 400, the following mode derivation via TIMD is applied in the same way on the encoder and decoder sides. For each intra prediction mode in the most probable mode (MPM) list of this luma coding block (CB), supplemented if necessary with a default mode, this mode calculates a prediction 401, 402 of this luma coding block's template from the template's decoded reference samples 403, and calculates the sum of absolute transformed difference (SATD) between this prediction and the luma coding block's template. The two intra prediction modes with the smallest SATD are selected as TIMD modes. Note that for TIMD, the set of directional intra prediction modes is expanded from 65 to 129. This means that the set of possible intra-prediction modes derived via the TIMD includes 131 modes. After retaining two intra-prediction modes from the first pass of testing, which includes the MPM list supplemented with default modes, for each of these two modes, if this mode is neither PLANAR nor DC, the TIMD also tests its two closest extended directional intra-prediction modes for predicted SATD. Note that in the above description, it is assumed that the luma CB template does not fall outside the boundaries of the current frame. If at least a portion of the luma CB template falls outside the boundaries of the current frame, the template area on which the prediction and SATD are calculated is modified as shown in Figures 4B and 4C if reconstructed left or top samples, respectively, are not available.

[0042] To predict the current luminance CB via TIMD, the two predictions of luminance CB via the two TIMD modes resulting from the two passes of testing are fused with weights after applying PDPC. The weights used depend on the predicted SATD of the two TIMD modes.

[0043] In Figure 4A, the current W × H luminance CB(100) is the w t ×H part (402) and W×h above it t The TIMD derivation step is performed using the 1+2w of the template. t +2W+2h t Predict the template for the current luminance CB from a set of +2H decoded reference samples 403. In at least one implementation, w t is equal to 2 (if W≦8), and w t is equal to 4 (otherwise). t is equal to 2 (if H≦8), and h t is equal to 4 (otherwise).

[0044] In FIG. 4B, the current W×H luminance CB(400) is calculated based on the available W×h t Only part (401) is surrounded by the template. During the TIMD derivation step, the intra prediction modes tested are 1+2W+2h of the template. t A template for the current luminance CB is predicted from a set of +2H decoded reference samples (403).

[0045] In Figure 4C, the current W x H luminance CB(400) is calculated by dividing the available W x H luminance by the t Only the ×H portion (402) is surrounded by the template. During the TIMD derivation step, the intra prediction modes tested are the 1+2w t A template for the current luminance CB is predicted from a set of +2W+2H decoded reference samples (403).

[0046] The principle of decoder-side intra mode derivation is shown in Figure 5. Decoder-side intra mode derivation (DIMD) is another template-based tool.

[0047] When DIMD is applied, two intra-modes are derived from the reconstructed L-shaped template. Essentially, a histogram of the gradients of the template pixels is constructed, and two peak points are selected as angles of prediction, and the corresponding intra-modes are selected. These two predictors are combined with a planar mode predictor with weights derived from their histogram values. The division operation in the weight derivation is performed using the same lookup table (LUT)-based integerization scheme used by cross-component linear model (CCLM) prediction tools. For example, the orientation calculation Orient = G y / G x The division operation in is calculated by the following LUT-based scheme: x=Floor(Log2(G x )) normDiff=((G x <<4)>>>x)&15 x+=(3+(normDiff!=0)?1:0) Orient=(G y * (DivSigTable[normDiff]|8)+(1<<(x-1)))>>x where DivSigTable

[16] ={0, 7, 6, 5, 5, 4, 4, 3, 3, 2, 2, 1, 1, 1, 1, 0}.

[0048] The derived intra modes are included in the primary list of intra most probable modes (MPMs), and therefore the DIMD process is performed before the MPM list is constructed. The primary derived intra mode of a DIMD block is stored with the block and is used for constructing the MPM lists of neighboring blocks.

[0049] The DIMD chroma mode uses the DIMD derivation method to derive the chroma intra prediction mode of the current block based on neighboring reconstructed Y, Cb, and Cr samples in the second neighboring row and column, as shown in Figure 5. Specifically, horizontal and vertical gradients are calculated for each co-located reconstructed luma sample of the current chroma block, as well as the reconstructed Cb and Cr samples, to construct a histogram of oriented gradients (HoG). The intra prediction mode with the largest histogram amplitude value is then used to perform chroma intra prediction for the current chroma block.

[0050] If the intra prediction mode derived from the DIMD chroma mode is the same as the intra prediction mode derived from the direct mode, the intra prediction mode with the second largest histogram amplitude value is used as the DIMD chroma mode. A coding unit level flag is signaled to indicate whether the proposed DIMD chroma mode is applied.

[0051] The Multi-model Based Cross-component Linear Model (MMLM) is another tool based on template analysis that extends the cross-component linear model (CCLM) prediction by adding three MMLM modes. In each MMLM mode, reconstructed template samples are classified into two classes using a threshold that is the average of neighboring luma-reconstructed samples. A linear model for each class is derived using the least-mean-square (LMS) method. For the CCLM mode, which uses a single class, the LMS method is also used to derive the linear model. A slope adjustment is applied to the CCLM and MMLM predictions. The adjustment tilts the linear function that maps luma values ​​to chroma values ​​relative to a center point determined by the average luma value of the reference sample.

[0052] Figure 6 illustrates the principle of intra-template matching. Intra-template matching (intra-TMP) is another tool based on template matching. Intra-TMP is a special intra-prediction mode that copies the best predicted block from the reconstructed part of the current frame, whose L-shaped template matches the current template. In the range, the encoder searches for the template that is most similar to the current template in the reconstructed part of the current frame and uses the corresponding block as the predicted block. The encoder signals the use of this mode so that the same prediction operation is performed at the decoder side.

[0053] The prediction signal is generated by matching the L-shaped causal neighborhood of the current block with another block within a predefined search area in Figure 6, which includes four regions: R1 (current CTU), R2 (upper-left CTU), R3 (upper CTU), and R4 (left CTU). The sum of absolute differences (SAD) is used as the cost function. Within each region, the decoder searches for the template with the smallest SAD relative to the current template and uses its corresponding block as the prediction block. The region dimensions (SearchRange_w, SearchRange_h) are set proportional to the block dimensions (BlkW, BlkH) and have a fixed number of SAD comparisons per pixel. That is, SearchRange_w=a * BlkW SearchRange_h=a * BlkH where "a" is a constant that controls the gain / complexity tradeoff. "a" is, for example, equal to 5.

[0054] Intra TMP prediction mode is enabled for CUs with width and height sizes less than or equal to 64. This maximum CU size for intra template matching is configurable.

[0055] The intra-TMP prediction mode is signaled at the CU level through a dedicated flag when DIMD is not currently used for the CU.

[0056] FIG. 7 illustrates the principle of template matching. Template matching (TM) is another tool based on template matching. It is a decoder-side motion vector (MV) derivation method for refining the motion information of a current CU by finding the closest match between a template of a neighboring sample in the current picture and a block (i.e., the same size as the template) in a reference picture. A better MV is searched for around the initial motion of the current CU within a [-8, +8] pel search range. In at least one implementation of TM, the search step size is determined based on an Adaptive Motion Vector Resolution (AMVR) mode, and TM can be cascaded with a bidirectional matching process in merge mode.

[0057] In Adaptive Motion Vector Prediction (AMVP) mode, an MVP candidate is determined based on the template matching error to select the one that achieves the minimum difference between the current block template and the reference block template, and then TM is performed only on this specific MVP candidate for MV refinement. TM refines this MVP candidate by using an iterative diamond search, starting with full-pel MVD accuracy (or 4-pel for 4-pel AMVR mode) within the [-8, +8]-pel search range. The AMVP candidate can be further refined by using a cross search with full-pel MVD accuracy (or 4-pel for 4-pel AMVR mode), followed by half-pel and quarter-pel accuracy, depending on the AMVR mode, as specified in Table 1. This search process ensures that the MVP candidate still maintains the same MV accuracy as indicated by AMVR mode after the TM process. In the search process, if the difference between the previous minimum cost and the current minimum cost in an iteration is less than a threshold equal to the area of ​​the block, the search process ends.

[0058] Table 1 shows the search patterns for AMVR and merge mode using AMVR.

[0059] [Table 1]

[0060] In merge mode, a similar search method is applied to the merge candidates indicated by the merge index. As shown in Table 1, TM may run all the way up to 1 / 8-pel MVD accuracy or skip beyond half-pel MVD accuracy, depending on whether an alternative interpolation filter (used when AMVR is in half-pel mode) is used according to the merged motion information. Furthermore, when TM mode is enabled, template matching may act as an independent process or an additional MV refinement process between the block-based bilateral matching (BM) method and the sub-block-based BM method, depending on whether BM can be enabled according to its enablement condition check.

[0061] FIG. 8 shows reference region restriction for intra block copy using template matching mode. Intra block copy using template matching mode (IBC-TM) is another tool based on template matching. Template matching is used in IBC for both IBC merge mode and IBC AMVP mode. Compared to that used by the normal IBC merge mode, the IBC-TM merge list is modified so that candidates are selected according to a pruning method using motion distance between candidates, as in the normal TM merge mode. The end 0 motion fulfillment is replaced by motion vectors to the left (-W, 0), up (0, -H), and up-left (-W, -H), where W is the width of the current CU and H is the height of the current CU.

[0062] In IBC-TM merge mode, the selected candidate is refined using a template matching method before the RDO or decoding process. The IBC-TM merge mode is in competition with the regular IBC merge mode, and the TM merge flag is signaled.

[0063] In IBC-TM AMVP mode, up to three candidates are selected from the IBC-TM merge list. Each of the three selected candidates is refined using a template matching method and sorted according to their resulting template matching costs. Then, only the first two are considered in the motion estimation process as usual.

[0064] Template matching refinement for both IBC-TM merge and AMVP modes is fairly simple because IBC motion vectors are constrained (i) to be integers and (ii) to be within a reference region. Figure 8 shows the reference region constraints for four example current CU positions, where regions marked with an "X" symbol are not considered for refinement for the current CU position.

[0065] In IBC-TM merge mode, all refinements are performed with integer precision, while in IBC-TM AMVP mode, they are performed with either integer or 4-pel precision depending on the AMVR value. Such refinements access only samples without interpolation. In both cases, the refined motion vectors and the templates used in each refinement step must obey the reference region constraints.

[0066] The embodiments described below are designed with the above in mind. Traditional template-based tools are based on the assumption that an L-shaped template captures the statistics of the current block. This assumption is utilized to estimate the best prediction mode (TIMD, DIMD), the best copy block (intra template matching), or to refine the merge motion vector (inter template matching and IBC template matching). However, in some scenarios, the left template or the top template may be distorted, for example, by edges that result in different statistics. This typically occurs when coding content that is not captured by a camera (game content, screen content).

[0067] It is therefore proposed to rely on partial templates (above or left) rather than the traditional L-shaped templates, in other words to consider a "above" template that includes pixels adjacent to and located above a block, or a "left" template that includes pixels adjacent to and located to the left of the block. Multiple lines or rows of pixels can also be used for the templates.

[0068] At least one embodiment proposes that an encoder be able to select the template type for a template-based coding tool from among a "top" template that includes pixels adjacent to and located above the block, a "left" template that includes pixels adjacent to and located to the left of the block, or a combination of "top" and "left" templates. In the latter case, in at least one embodiment, the template may also include a top-left element, as in the case of a conventional L-shaped template. The template type is signaled in the encoded data and used by the decoder to select a template of the appropriate type to predictably perform prediction using the selected template-based tool.

[0069] FIG. 9A shows a flowchart of an example of an encoding process for a template-based prediction mode, according to at least one embodiment. This encoding process 900 may be implemented, for example, by the encoder 100 of FIG. 1 in the device 1000 of FIG. 3. For a current block, in step 911, the device selects a template from neighboring samples according to the template type. In step 912, the reconstructed area is analyzed to find a template matching the selected template. In step 913, a block corresponding to the matching template is selected, and a coding cost for this block is determined according to the template-based prediction mode. Steps 911, 912, and 913 are repeated in step 910 for different template-based prediction modes (with different parameter sets, if available) and different template types. In step 920, a prediction mode (using the selected template type) is selected based on the coding cost. Note that these steps 910-920 may be performed within RDO optimization, which is conventionally part of the encoder. Also, the circumstances under which the template-based prediction mode is selected are described here. If another mode that does not use a template is selected as the prediction mode for the block, steps 925-935 are replaced with conventional encoding steps according to the selected prediction mode. In step 925, the current block is predicted according to the selected prediction and associated parameters or related data (e.g., values ​​of samples of the block selected for prediction). In step 930, the current block is then coded based on the predicted block, and in step 935, the use of a template type is signaled for the current block according to the selected template-based prediction mode.

[0070] FIG. 9B shows a flowchart of an example of a decoding process for template-based prediction mode, according to at least one embodiment. This decoding process 950 is implemented, for example, by the encoder 100 of FIG. 1 in the device 1000 of FIG. 3. In step 960, the device obtains information signaling the use of template-based prediction mode for the current block and the type of template selected as described above. In step 965, the device determines a template for neighboring samples of the current block. In step 970, the device finds a matching (e.g., best-matching) template in the reconstructed area of ​​the image, and in step 975, selects parameters corresponding to the matching template based on the prediction mode. In at least one embodiment, the set of parameters is the set of samples of the block corresponding to the matching template. Then, in step 980, these parameters are used to predict the current block. In step 985, the current block is conventionally decoded (reconstructed) based on the predicted block.

[0071] 10 illustrates template types. Element 1004 represents the current coding unit. It is surrounded by a top template 1001 and a left template 1002 that are adjacent to the CU. A traditional "L-shape" may be obtained by combining top template 1001 and left template 1002 with a top-left region 1003. The template depicted in this figure is multiple pixels wide (or high), but templates described herein may also be single-pixel wide (or high) templates.

[0072] In at least one embodiment, in addition to at least a first element signaling the use of a template-based tool, an additional syntax element is signaled to indicate the type of template selected from among a top template, a left template, or both templates used by the template-based tool. This signaling may be done, for example, in a high-level SPS or slice header, or at the CU level. It is generated by the encoder device for a block of an image of video and provided to the decoding device through the coded stream to enable correct reconstruction of the coded block of an image of video. The binary format of this signaling is shown in Table 2.

[0073] [Table 2]

[0074] In at least one embodiment, the syntax element representing the template type is optimized to reduce signaling overhead. For non-square blocks, it is unlikely to use the left template for wide blocks (width greater than height) because more elements are available in the top template. Similarly, it is unlikely to use the top template for thin blocks. Therefore, in at least one embodiment, it is proposed to use single-bit signaling to indicate whether the top and left templates are used, or whether only one of them is used. The selection of the top or left template is made based on the block shape, with the top template being selected for wide blocks and the left template for tall / narrow blocks. For square blocks, the top template is used by default. The binary format of this signaling is shown in Table 3.

[0075] [Table 3]

[0076] In an alternative embodiment, the signaling format of Table 3 is used only if the block proportions comply with certain conditions, for example, if the ratio of width divided by height when width is greater than height, or the ratio of height divided by width when height is greater than width, is better than a predetermined threshold.

[0077] In another variation shown in FIG. 10, when a top template 1001 and a left template 1002 are used, the template also includes a top left region 1003 so that the template is "L-shaped."

[0078] In another variation, the templates are not limited to a single line but use multiple lines, for example the top template uses multiple rows of pixels and the left template uses multiple columns of pixels.

[0079] 11 shows an example of CTU partitioning with CU decoding order, which illustrates the latency issue for template matching (TM) prediction mode.

[0080] At least one embodiment proposes to reduce the latency caused by the TM prediction mode by signaling the template to be used. In fact, TM causes some latency in the decoding of a CU because the surrounding reconstructions must be completed before the current template for a particular CU can be calculated. For example, the current template for the eighth CU must wait for the complete reconstruction of the fifth, sixth, and seventh CUs. Therefore, the process for the eighth CU cannot start until the seventh CU is completely reconstructed.

[0081] Generally, the "top" CU is available, but the "left" CU may be missing. In this embodiment, if both (left and top) templates or only the top template can be used for TM, it is proposed to signal at the SPS, picture header, or slice header level. In the latter case, the TM latency problem can be overcome.

[0082] In an alternative embodiment, since some CUs do not suffer from such latency issues (e.g., the seventh CU in the figure requires the reconfiguration of the first and second CUs), it is proposed to signal the use of the top template, the left template, or both templates at the CU level: for example, 2, 3, 4, 6, 8, and 11 use the top, 10 uses the left, and 1, 5, 7, and 9 use both.

[0083] Another variation proposes to signal for each CU whether it can use the top template, the left template, both templates, or no template, in order to keep the CTUs independent. For example, 1, 2, 3, 4, and 10 use none; 6, 8, 9, and 11 use the top; and 5 and 7 use both.

[0084] Figure 12 shows a flowchart of an example of an encoding process using intra-template matching prediction mode, according to at least one embodiment. This encoding process 1200 may be implemented, for example, by the encoder 100 of Figure 1 in the device 1000 of Figure 3. The process operates on a current block of an image or video. In step 1210, the encoder selects an intra-template matching prediction mode and a template type for the current block. In step 1220, the encoder predicts the block using the intra-template matching prediction mode based on the template type. In step 1230, the encoder encodes the predicted block. In step 1240, the encoder provides coding information for the current block, including at least the use of the template intra-matching prediction mode and the template type.

[0085] Figure 13 shows a flowchart of an example of a decoding process using intra-template matching prediction mode, according to at least one embodiment. This decoding process 1300 may be implemented, for example, by the decoder 200 of Figure 2 in the device 1000 of Figure 3. The process operates on a current block of an image or video. In step 1310, the decoder obtains coding information for the current block, including at least the use of the template intra-matching prediction mode and the type of template. In step 1320, the decoder predicts the block using the intra-template matching prediction mode based on the type of template. In step 1330, the decoder decodes the predicted block.

[0086] At least one example embodiment may involve an apparatus described herein and a device including at least one of: (i) an antenna configured to receive a signal, the signal including data representing image information; (ii) a band limiter configured to limit the received signal to a frequency band including the data representing the image information; and (iii) a display configured to display an image from the image information.

[0087] At least one example embodiment may involve a device described herein, the device including one of a television, a television signal receiver, a set-top box, a gateway device, a mobile device, a mobile phone, a tablet, a computer, a laptop, or other electronic device.

[0088] In general, another example embodiment may involve a bitstream or signal formatted to include syntax elements and picture information, where the syntax elements are generated and the picture information is encoded by a process based on any one or more of the example embodiments of the method according to the present disclosure.

[0089] Generally, one or more other example embodiments may also provide a computer-readable storage medium, e.g., a non-volatile computer-readable storage medium, having stored thereon instructions for encoding or decoding picture information, such as video data, according to the methods or apparatus described herein. One or more embodiments may also provide a computer-readable storage medium having stored thereon a bitstream generated according to the methods or apparatus described herein. One or more embodiments may also provide methods and apparatus for transmitting or receiving a bitstream or signal generated according to the methods or apparatus described herein.

[0090] Many of the example embodiments described herein are described with specificity, and often in a manner that may be considered restrictive, at least to illustrate individual characteristics. However, this is for clarity of description and does not limit the application or scope of these aspects. In fact, all of the different aspects can be combined and interchanged to provide further aspects. Furthermore, embodiments, features, and the like can be combined and interchanged with those described in previous applications.

[0091] Various implementations involve decoding. As used herein, "decoding" can encompass, for example, all or some of the processes performed on a received encoded sequence to generate a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also, or alternatively, include processes performed by decoders in various implementations described herein.

[0092] As a further example, in one embodiment, "decoding" refers to entropy decoding only, in another embodiment, "decoding" refers to differential decoding only, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to the broader decoding process generally will be clear based on the context of a particular description and will be well understood by one of ordinary skill in the art.

[0093] Various implementations involve encoding. Similar to the above discussion of "decoding," "encoding," as used herein, can encompass, for example, all or some of the processes performed on an input video sequence to generate an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential encoding, transforming, quantizing, and entropy encoding.

[0094] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to the broader encoding process in general will be clear based on the context of the particular description and will be well understood by one of ordinary skill in the art.

[0095] Note that the syntax elements used herein are descriptive terms, and therefore they do not preclude the use of other syntax element names.

[0096] When a figure is presented as a flow diagram, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow diagram of the corresponding method / process.

[0097] Generally, the embodiments, implementations, features, etc. described herein may be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed in the context of only one type of implementation (e.g., discussed only as a method), the implementation of the discussed feature may also be implemented in other forms (e.g., an apparatus or a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. One or more examples of a method may be implemented in, for example, a processor, which refers generally to processing devices including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users. Furthermore, use of the term "processor" herein is intended to broadly encompass various configurations of one processor or more than one processor.

[0098] References to "one embodiment" or "one embodiment," or "one implementation" or "one implementation," as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with an embodiment is included in at least one embodiment. Thus, appearances of the phrases "in one embodiment" or "an embodiment," or "in one implementation," or "in an implementation," as well as any other variations, in various places throughout this application are not necessarily all referring to the same embodiment.

[0099] Additionally, the application may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0100] Additionally, the application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0101] Additionally, the present application may refer to "receiving" various information. Receiving, like "accessing," is intended to be a broad term. Receiving information can include, for example, one or more of accessing information or retrieving information (e.g., from memory). Furthermore, "receiving" typically involves in some manner, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0102] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass the selection of only the first listed option (A), or the selection of only the second listed option (B), or the selection of both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed option (A), or the selection of only the second listed option (B), or the selection of only the third listed option (C), or the selection of only the first and second listed options (A and B), or the selection of only the first and third listed options (A and C), or the selection of only the second and third listed options (B and C), or the selection of all three options (A, B, and C). This may be expanded as many times as the number of listed items, as would be apparent to one of ordinary skill in the art.

[0103] As will be apparent to those skilled in the art, implementations can generate a variety of signals formatted to carry information that can be stored or transmitted, for example. The information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal can be formatted to carry a bitstream of the described embodiments. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0104] Various embodiments are described herein, the features of which may be provided singly or in any combination across various claim categories and types.

Claims

1. 1. A method comprising: providing, for a current block of a picture, information indicative of a type of template to be used in an intra template matching prediction mode, wherein the type of template is selected from among a first template comprising a set of neighboring pixels above the current block, a second template comprising a set of neighboring pixels to the left of the current block, or a third template that is a combination of the first template and the second template.

2. 2. The method of claim 1, wherein the information representing the type of the template is coded as two-bit information, a first bit indicating whether a combination of the first template and the second template is used or a single template is used, and a second bit indicating whether the first template or the second template is used.

3. 2. The method of claim 1, wherein the first and second types of templates are represented by the same element, the selection is made according to the size of the current block, and the information representing the type of the template is coded as a single bit.

4. The method of any one of claims 1 to 3, wherein the third template also includes the set of neighboring pixels to the top left of the current block.

5. The method according to any one of claims 1 to 4, wherein the height of the first template and the width of the second template are greater than one pixel.

6. For the current block of the picture, selecting an intra template matching prediction mode and a template type based on a coding cost; predicting a block using the intra template matching prediction mode based on the type of the template; encoding the current block based on the predicted block; providing coding information for the current block, the information including at least information representing use of an intra template matching prediction mode and information representing a type of template selected from a first template including a set of neighboring pixels above the current block, a second template including a set of neighboring pixels to the left of the current block, or a third template that is a combination of the first template and the second template.

7. For the current block of the picture, selecting an intra-template matching prediction mode and a template type based on a coding cost; predicting a block using the intra template matching prediction mode based on the type of the template; encoding the current block based on the predicted block; 1. A device comprising: a processor configured to provide coding information for a current block, the coding information including at least information representing use of an intra template matching prediction mode; and information representing a type of template selected from among a first template including a set of neighboring pixels above the current block, a second template including a set of neighboring pixels to the left of the current block, or a third template that is a combination of the first template and the second template.

8. For the current block of the picture, obtaining coding information for the current block, the coding information including at least information indicating use of an intra template matching prediction mode and information indicating a type of template selected from a first template including a set of neighboring pixels above the current block, a second template including a set of neighboring pixels to the left of the current block, or a third template being a combination of the first template and the second template; predicting a block using the intra template matching prediction mode based on the type of the template; and decoding a current block based on the predicted block.

9. For the current block of the picture, obtain coding information for the current block, the coding information including at least information indicating use of an intra template matching prediction mode and information indicating a type of template selected from a first template including a set of neighboring pixels above the current block, a second template including a set of neighboring pixels to the left of the current block, or a third template being a combination of the first template and the second template; predicting a block using the intra template matching prediction mode based on the type of the template; A device comprising: a processor configured to decode a current block based on the predicted block.

10. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 7.

11. 8. A non-transitory computer readable medium storing executable program instructions, the instructions causing a computer executing the instructions to perform the method of any one of claims 1 to 7.

12. A bitstream representing a coded picture formatted to include a syntax element indicating the type of the template for a current block according to the method of any one of claims 1 to 5.

13. A non-transitory computer-readable medium storing information representing an encoded picture formatted to include a syntax element indicating the type of the template for a current block according to the method of any one of claims 1 to 5.

Images