Encoding and decoding methods using template-based tools and corresponding devices

Abstract

A decoding method is disclosed. Information is obtained for identifying which pixels (e.g., decoded pixels) within a template of a current block of a picture are unavailable. A template-based tool is further applied using the obtained information to determine information to be used for decoding the current block. Finally, the current block is decoded using the determined information.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of European application number 22306323.1 filed on September 7, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] At least one of the present embodiments generally relates to a method and apparatus for encoding and decoding picture blocks using output of a template-based tool. Background Art

[0004] In order to achieve high compression efficiency, image and video coding schemes usually use prediction and transformation to fully exploit the spatial and temporal redundancy in video content. In general, intra-frame or inter-frame prediction is used to exploit intra-picture or inter-picture correlation, and then the difference between the original block and the predicted block (often expressed as a prediction error or prediction residual) is transformed, quantized, and entropy encoded. In order to reconstruct the video, the compressed data is decoded by the inverse process corresponding to entropy coding, quantization, transformation, and prediction. Summary of the invention

[0005] In one embodiment, a decoding method is disclosed, the decoding method comprising:

[0006] Obtaining information about which pixels (eg, decoded pixels) within a template for identifying a current block of a picture are unavailable;

[0007] applying a template-based tool using the obtained information to determine information to be used for decoding the current block; and

[0008] The current block is decoded using the determined information.

[0009] A decoding device is disclosed, the decoding device comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to execute the method disclosed above.

[0010] In another embodiment, a coding method is disclosed, the coding method comprising:

[0011] Obtaining information about which pixels (eg, decoded pixels) within a template for identifying a current block of a picture are unavailable;

[0012] applying a template-based tool using the obtained information to determine information to be used to encode the current block; and

[0013] The current block is encoded using the determined information.

[0014] A coding apparatus is disclosed, the coding apparatus comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to perform the method just disclosed above.

[0015] Further embodiments are described herein which may be used alone or in combination.

[0016] One or more embodiments also provide a computer program including instructions that, when executed by one or more processors, cause the one or more processors to perform a method for predicting chrominance samples or encoding / decoding image or video data according to any one of the embodiments described herein. One or more of the present embodiments also provide a non-transitory computer-readable medium and / or a computer-readable storage medium having stored thereon instructions for predicting chrominance samples or encoding / decoding image or video data according to the methods described herein.

[0017] One or more embodiments further provide a computer-readable storage medium on which a bitstream generated according to the method described herein is stored. One or more embodiments further provide a method and apparatus for transmitting or receiving a bitstream generated according to the method described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 illustrates a block diagram of a system in which aspects of the present embodiment may be implemented;

[0019] Figure 2 A block diagram of an embodiment of a video encoder is illustrated;

[0020] Figure 3 illustrates a block diagram of an embodiment of a video decoder;

[0021] Figure 4A , Figure 4B and Figure 4C The horizontal gradient G calculated at a given decoded reference sample in the middle row or middle column of the template of the current W×H luminance CB to be encoded / decoded is shown. HOR and the vertical gradient G VER The increase in the HOG bin caused by this;

[0022] Figure 5 illustrates the input of the spatial 5-tap components of the filter when the CCCM predicts the current chroma CB to be encoded / decoded based on a potentially downsampled version of the reconstructed luma CB concatenated with the current chroma CB;

[0023] Fig. 6A , 6B6C illustrates a current W×H chroma CB (201) to be encoded / decoded, its chroma reference region (203), a downsampled reconstructed luma CB (200) juxtaposed with this chroma CB, and a luma reference region (202) in the case of chroma format 4:2:0;

[0024] Figure 7 illustrates the search for a candidate reconstructed W×H luminance block (302) belonging to the search range of a TMP whose template (303) is closest in terms of SAD to the template (301) of the current W×H luminance CB to be encoded / decoded (300);

[0025] Fig. 8A illustrates the identification of reference (in terms of motion) samples of the template T of the current block to be encoded / decoded in case of a bidirectional merging candidate;

[0026] Figure 8B illustrates identification of reference (in terms of motion) samples of each sub-template of the template T of the current block to be encoded / decoded in case of sub-block based merge candidates;

[0027] FIG.9A illustrates the MIP mode via indexing The prediction of the current 4×4 brightness CB;

[0028] Fig. 9B The MIP mode via index is shown The prediction of the current 8×4 brightness CB;

[0029] Fig. 9C and Fig.9D illustrates the mapping from the index of the MIP mode to the index of the MIP matrix for the current 4×4 luma CB and the current 8×4 luma CB, respectively;

[0030] Fig. 10A and Fig. 10B illustrates two templates (311)(314) of a given W×H block (310) fed into a typical template-based neural network without translation equivariance;

[0031] Fig.11 illustrates a method of encoding a block using a template-based tool according to one embodiment;

[0032] Fig.12 illustrates a method of decoding a block using a template-based tool according to one embodiment; and

[0033] Figures 13A-26D Various embodiments of encoding / decoding methods are illustrated. DETAILED DESCRIPTION

[0034] The present application describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are specifically described, and at least in order to illustrate individual characteristics, are usually described in a manner that may sound restrictive. However, this is to describe clearly, without limiting the application or scope of those aspects. In fact, all different aspects can be combined and interchanged to provide further aspects. In addition, this aspect can also be combined and interchanged with the aspects described in the earlier applications.

[0035] The aspects described and contemplated in this application may be implemented in many different forms. Figure 1 , Figure 2 and Figure 3 Some embodiments are provided, but other embodiments are contemplated and Figure 1 , Figure 2 and Figure 3 The discussion of does not limit the breadth of the embodiments. At least one of the aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the methods.

[0036] In this application, the terms "reconstructed" and "decoded" are used interchangeably, the terms "encoded" or "coded" are used interchangeably, the terms "pixel" and "sample" are used interchangeably, and the terms "image", "picture" and "frame" are used interchangeably. Typically, but not necessarily, the term "reconstructed" is used on the encoder side, and "decoded" is used on the decoder side.

[0037] Various methods are described herein, and each method includes one or more steps or actions for realizing the method. Unless the correct operation method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined. In addition, terms such as "first", "second", etc. can be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding". Unless specifically required, the use of such terms does not mean the sequencing of the operation to modification. Therefore, in this example, the first decoding does not need to be performed before the second decoding, and can occur in, for example, before, during, or in a time period overlapping with the second decoding.

[0038] The present aspects are not limited to VVC or HEVC, and may be applied, for example, to other standards and recommendations (whether pre-existing or developed in the future), and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise stated, or technically excluded, the aspects described in this application may be used alone or in combination.

[0039] Figure 1 A block diagram of an example of a system in which various aspects and embodiments may be implemented is illustrated. System 100 may be embodied as a device including various components described below, and is configured to perform one or more of the aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances and servers. The elements of system 100 may be embodied in a single integrated circuit, multiple ICs and / or discrete components singly or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed over multiple ICs and / or discrete components. In various embodiments, system 100 is coupled to other systems or other electronic devices via, for example, a communication bus or by dedicated input and / or output ports. In various embodiments, system 100 is configured to implement one or more of the aspects described in this application.

[0040] The system 100 includes at least one processor 110 configured to execute instructions loaded therein to implement, for example, various aspects described in the present application. The processor 110 may include embedded memory, input and output interfaces, and various other circuits as known in the art. The system 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). The system 100 includes a storage device 140, which may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, magnetic disk drive, and / or optical disk drive. As a non-limiting example, the storage device 140 may include an internal storage device, an attached storage device, and / or a network accessible storage device.

[0041] The system 100 includes an encoder / decoder module 130, which is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents a module (or modules) that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. In addition, as known to those skilled in the art, the encoder / decoder module 130 may be implemented as a separate element of the system 100, or may be incorporated into the processor 110 as a combination of hardware and software.

[0042] Program code to be loaded onto the processor 110 or the encoder / decoder 130 to perform various aspects described herein may be stored in the storage device 140 and subsequently loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder / decoder module 130 may store one or more of various items during execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from processing of equations, formulas, operations, and operational logic.

[0043] In some embodiments, memory internal to the processor 110 and / or encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be memory 120 and / or storage device 140, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations, such as working memory for MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG H Part 2), or VVC (Versatile Video Coding, a new standard being developed by the Joint Video Experts Group JVET).

[0044] Inputs to the elements of system 100 may be provided through various input devices, as indicated in block 105. Such input devices include, but are not limited to: (i) an RF section that receives a radio frequency (RF) signal transmitted over the air, for example, by a broadcast device, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 1 Other examples not shown include composite video.

[0045] In various embodiments, the input devices of block 105 have associated respective input processing elements, as known in the art. For example, the RF portion may be associated with elements adapted to: (i) select a desired frequency (also referred to as selecting a signal, or band limiting a signal to a frequency band), (ii) down-convert the selected signal, (iii) again band-limit to a narrower frequency band to select a signal frequency band that may be referred to as a channel in some embodiments, (iv) demodulate the down-converted and band-limited signal, (v) perform error correction, and (vi) demultiplex to select a desired data packet stream. The RF portion of various embodiments includes one or more elements to perform these functions, such as a frequency selector, a signal selector, a frequency band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF portion may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near baseband frequency) or baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted by wired (for example, cable) medium, and filter, down-convert and filter to the desired frequency band again to perform frequency selection. Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements, and / or add other elements of similar or different functions. Adding element can include inserting element between existing elements, for example inserting amplifier and analog to digital converter. In various embodiments, the RF part includes antenna.

[0046] In addition, the USB and / or HDMI terminals may include respective interface processors for connecting the system 100 to other electronic devices across the USB and / or HDMI connections. It is to be understood that various aspects of input processing (e.g., Reed Solomon error correction) may be implemented, for example, within a separate input processing IC or within the processor 1010 as desired. Similarly, various aspects of USB or HDMI interface processing may be implemented, for example, within a separate interface IC or within the processor 1010 as desired. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 110 and the encoder / decoder 130, to operate in conjunction with memory and storage elements to process the data streams as desired for presentation on an output device.

[0047] The various components of system 100 may be disposed within an integrated housing. Within the integrated housing, the various components may be interconnected and transmit data between them using a suitable connection arrangement 1140 (eg, an internal bus as known in the art, including an I2C bus, wiring, and printed circuit boards).

[0048] The system 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data through the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or a network card, and the communication channel 190 may be implemented, for example, in a wired and / or wireless medium.

[0049] In various embodiments, data is streamed to the system 100 using a WiFi network such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signals of these embodiments are received through a communication channel 190 and a communication interface 150 suitable for Wi-Fi communication. The communication channel 190 of these embodiments is typically connected to an access point or router that provides access to an external network including the Internet to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box that delivers data through an HDMI connection of an input block 105 to provide streaming data to the system 100. Still other embodiments use an RF connection of an input block 105 to provide streaming data to the system 100. As indicated above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use a wireless network instead of WiFi, such as a cellular network or a Bluetooth network.

[0050] The system 100 can provide output signals to various output devices, including a display 165, a speaker 175, and other peripheral devices 185. The display 165 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 165 can be used for a television, a tablet computer, a laptop computer, a cell phone (mobile phone), or other devices. The display 165 can also be integrated with other components (e.g., as in a smart phone), or separated (e.g., an external monitor for a laptop computer). In various examples of embodiments, other peripheral devices 185 include one or more of a stand-alone digital video disk (or digital versatile disk) (DVR for both terms), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 185 that provide functions based on the output of the system 100. For example, a disk player performs the function of playing the output of the system 100.

[0051] In various embodiments, control signals are communicated between the system 100 and the display 165, speaker 175, or other peripherals 185 using signaling such as AV.Link, CEC, or other communication protocols that enable inter-device control with or without user intervention. The output devices may be communicatively coupled to the system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, the output devices may be connected to the system 100 via the communication interface 150 using the communication channel 190. The display 165 and speaker 175 may be integrated into a single unit with other components of the system 100 in an electronic device such as a television. In various embodiments, the display interface 1070 includes a display driver, such as a timing controller (T Con) chip.

[0052] For example, if the RF portion of input 105 is part of a stand-alone set-top box, the display 165 and speaker 175 may alternatively be separate from one or more of the other components. In various embodiments where the display 165 and speaker 175 are external components, the output signals may be provided via dedicated output connections, including, for example, an HDMI port, a USB port, or a COMP output.

[0053] Embodiments may be implemented by computer software implemented by processor 110 or by hardware or by a combination of hardware and software. As a non-limiting example, embodiments may be implemented by one or more integrated circuits. As a non-limiting example, memory 120 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, processor 110 may be of any type suitable for the technical environment and may encompass one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0054] Figure 2 An example video encoder 200 is illustrated, such as a VVC (Versatile Video Coding) encoder. Figure 2 It is also possible to illustrate an encoder that has improved the VVC standard or an encoder that uses a technique similar to VVC.

[0055] Before being encoded, the data sequence may undergo a pre-encoding process (201), for example, applying a color transform to an input color picture (e.g., conversion from RGB 4:4:4 to YCbCr4:2:0), or performing a remapping of input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and attached to the bitstream.

[0056] In encoder 200, as described below, a picture is encoded by encoder elements. The picture to be encoded is segmented (202) and processed in units of, for example, CUs (coding units). Each unit is encoded using, for example, intramode or intermode. When a unit is encoded in intramode, it performs intraprediction (260). In intermode, motion estimation (275) and compensation (270) are performed. The encoder decides (205) which of intramode or intermode is used to encode the unit and indicates the intra / inter decision by, for example, a prediction mode flag. For example, a prediction residual is calculated by subtracting (210) the predicted block from the original image block.

[0057] The prediction residual is then transformed (225) and quantized (230). The quantized transform coefficients, along with motion vectors and other syntax elements (such as picture segmentation information), are entropy encoded (245) to output a bitstream. The encoder may skip the transform and apply quantization directly to the untransformed residual signal. The encoder may bypass both the transform and quantization, i.e., encode the residual directly without applying the transform or quantization process.

[0058] The encoder decodes the encoded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The image block is reconstructed by combining (255) the decoded prediction residual and the prediction block. An in-loop filter (265) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) / ALF (adaptive in-loop filter) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (280).

[0059] Figure 3 A block diagram of an example video decoder 300 is shown. In the decoder 300, a bitstream is decoded by decoder elements as described below. The video decoder 300 generally performs the same operations as described below. Figure 2 The encoding pass described in is the inverse of the decoding pass, such as Figure 2 Encoder 200 also typically performs video decoding as part of encoding the video data.

[0060] In particular, the input to the decoder includes a video bitstream, which may be generated by the video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, prediction modes, motion vectors, and other encoding information. Picture segmentation information indicates how the picture is segmented. Thus, the decoder can divide (335) the picture according to the decoded picture segmentation information. The transform coefficients are dequantized (340) and inverse transformed (350) to decode the prediction residual. By combining (355) the decoded prediction residual and the prediction block, the image block is reconstructed. The prediction block may be obtained (370) from an intra-frame prediction (360) or a motion compensated prediction (i.e., inter-frame prediction) (375). An in-loop filter (365) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (380). Note that for a given picture, the contents of the reference picture buffer 380 on the decoder 300 side are the same as the contents of the reference picture buffer 280 on the encoder 200 side for the same picture.

[0061] The decoded picture may further undergo post-decoding processing (385), such as an inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or performing an inverse remapping that is the inverse of the remapping process performed in the pre-encoding process (201). The post-decoding process may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0062] In the following, some template-based tools in ECM (Enhanced Compression Model) are described in detail. The template-based tools are configured to output information to be used for encoding blocks of pictures. This information can be of various types, such as predictions of blocks to be encoded, one or more prediction modes of blocks to be encoded, one or more transforms to be used with blocks to be encoded, a reordered list of merge candidates to be used for blocks to be encoded, etc. This list is not exhaustive, and the present embodiment is neither limited to a specific template-based tool nor to a specific type of output information.

[0063] Decoder-side intra mode derivation (DIMD)

[0064] In ECM-4.0, DIMD derives indices of two intra prediction modes from the gradients in the template of the decoded reference samples of the current luma coding block (CB) to be encoded / decoded, which may be the two best intra prediction modes for predicting the current luma CB in terms of rate distortion. Later, the current luma CB is predicted by blending the two prediction blocks obtained by applying the two derived intra prediction modes with the prediction block obtained by applying PLANAR, which is one of the intra prediction modes defined in VVC. The weights involved in the blending are derived from the gradients in this template.

[0065] More specifically, for the current brightness CB, the indexes of the two intra prediction modes are as follows: Figure 4A , Figure 4B and Figure 4C The gradients in this template depicted in are derived. First, a Histogram of Oriented Gradients (HOG) with 65 bins corresponding to the 65 directional intra prediction modes is initialized to 0. Then, for each decoded reference sample in the middle row or middle column of the template for the three rows of decoded reference samples above the current luminance CB and the three columns of decoded reference samples to its left, the following procedure is applied.

[0066] like Figure 4A As shown, the 3×3 horizontal Sobel filter and the 3×3 vertical Sobel filter centered on this decoded reference sample generate horizontal gradients G HOR and the vertical gradient G VER .

[0067] like Figure 4C As shown in the figure, G HOR and G VER The sign of indicates in which of the four directions the perpendicular to the horizontal component G is found. HOR and the vertical component G VER The "target" direction of the gradient G. If |G VER |>|G HOR|, the anchoring direction corresponds to the horizontal direction. If |G HOR |≥|G VER |, then the anchor direction corresponds to the vertical direction. The “target” direction forms an angle θ with respect to the anchor direction, such as Figure 4B shown.

[0068] By discretizing a scaled version of tan(θ), find the index i of the ECM directional intra prediction mode whose direction is closest to the “target” direction.

[0069] The HOG interval of index i is represented by |G HOR |+|G VER |Increment, such as Figure 4C shown.

[0070] Finally, the indices of the two largest HOG intervals are the indices of the two derived intra prediction modes.

[0071] Note that in the above procedure, the fact that the “target” direction is perpendicular to the gradient G is justified by the fact that when the directional intra prediction mode moves the reference sample along the direction When extrapolated to a given region, the dominant gradient in this region will most likely be perpendicular to

[0072] Convolutional Cross-Component Model (CCCM)

[0073] In the Exploration Experiments (EE) on top of ECM-4.0, the Convolutional Cross Component Model (CCCM) predicts the current chroma CB to be encoded / decoded by applying a convolution filter to a potentially downsampled version of the reconstructed luma CB collocated with the current chroma CB. When chroma subsampling is used, this downsampling is performed so that the resolution of the downsampled collocated reconstructed luma CB matches the resolution of the chroma grid.

[0074] The CCCM convolution 7-tap filter consists of a 5-tap plus-shaped spatial component, a nonlinear term, and a bias term. Figure 5 As shown, the input to the spatial 5-tap component of the filter consists of the center (C) luma sample concatenated with the current chroma sample to be predicted, and its above / north (N), below / south (S), left / west (W), and right / east (E) neighbors.

[0075] The nonlinear term P is expressed as a power of 2 of the center luminance sample C and scaled to the sample value range of the content.

[0076] P=(C*C+midVal)>>bitDepth

[0077] where bitDepth represents the pixel bit depth, and midVal represents the middle value of the bit depth range.

[0078] For example, for 10-bit content it is calculated as

[0079] P=(C*C+512)>>10

[0080] The bias term B represents a scalar offset between the input and output. B is set to the middle chroma value, e.g., 512 for 10-bit content.

[0081] Let c0, c1, c2, c3, c4, c5 and c6 be called the seven coefficients of the 7-tap filter, and the current predicted chrominance sample "predChromaVal" is expressed as:

[0082] predChromaVal=clip(c0C+c1N+c2S+c3E+c4W+c5P+c6B)

[0083] where "clip" is clipped to the range of valid chroma sample values.

[0084] The filter coefficients c0, c1, c2, c3, c4, c5 and c6 are calculated by minimizing the mean square error (MSE) between predicted chroma samples generated by applying a convolutional 7-tap filter to reconstructed luma samples in the luma reference region (202) and to potentially downsampled versions of reconstructed chroma samples in the chroma reference region (203), as shown in Fig. 6A , Figure 6B and Figure 6C shown.

[0085] Fig. 6A A reconstructed luma CB is shown concatenated with a current WxH chroma CB (201) to be encoded / decoded. Figure 6B The diagram shows a downsampled reconstructed luma CB (200) juxtaposed with this chroma CB and a luma reference area (202) in the case where the chroma format is 4:2:0, ie the resolution of each chroma channel is divided by 2 via subsampling before encoding. Figure 6C Its chromaticity reference region (203) is illustrated.

[0086] The luma reference region (202) consists of six rows / columns of potentially downsampled reconstructed luma samples above and to the left of a potentially downsampled version of the reconstructed luma CB (200) juxtaposed with the current chroma CB. The chroma reference region (203) consists of six rows / columns of reconstructed chroma samples above and to the left of the current chroma CB (201) to be encoded / decoded. Each reference region is extended to the right by a CB width and below a CB boundary by a CB height. Each reference region is adjusted to include only available decoded reference samples. Figure 6B and 6CThe extension of the black filled areas in necessitates edge sampling of the supporting plus-shaped spatial filter and is filled in when in unavailable areas.

[0087] MSE minimization is performed by calculating the autocorrelation matrix for the luma input and the cross-correlation vector between the luma input and the chroma output. The autocorrelation matrix is ​​decomposed by LDL, and back substitution is used to calculate the final filter coefficients. This process follows the calculation of the adaptive linear filtering (ALF) filter coefficients in ECM, except that LDL decomposition is chosen instead of Cholesky decomposition to avoid the use of square root operations. This calculation uses only integer operations.

[0088] Note that a single model or a multi-model variant of CCCM can be used. The multi-model variant uses two models, one derived for samples above the average luminance reference value and another derived for the remaining samples. The multi-model CCCM mode can be selected for coding units (CUs) containing at least 128 available decoded reference samples.

[0089] Note also that the term "reference region" has been chosen to match the standard nomenclature of CCCMs. However, the reference region of a given CB is equivalent to the template of this CB.

[0090] Template Matching Prediction (TMP)

[0091] In ECM-4.0, Template Matching Prediction (TMP) is an intra prediction mode that predicts the current W×H luma CB to be encoded / decoded.

[0092] To this end, the template (301) of the current brightness CB (300) consists of 4 rows of decoded reference samples above the current brightness CB and 4 columns of decoded reference samples to its left, as shown in Figure 7 As shown. In the search step, for each allowed position in a given search range of the TMP in the current luma channel, a candidate reconstructed W×H luma block (302) whose upper left pixel is located at this allowed position is considered, and the sum of absolute differences (SAD) between its template (303) of the 4 rows of decoded reference samples above it and the 4 columns of decoded reference samples to its left and the template (301) of the current luma CB (300) is calculated. The selected reconstructed luma block (best candidate) is the one with the smallest template matching the SAD.

[0093] The selected candidate reconstructed luma block is then used to predict the current luma CB.

[0094] Adaptive Re-ranking of Merge Candidates Using Template Matching (ARMC-TM)

[0095] In ECM-4.0, merge candidates are adaptively reordered via TM. For a given CU of inter prediction, merge mode derives all motion information from spatially and temporally adjacent CUs, which are called merge candidates. The reordering method is applied to normal merge mode, TM merge mode, and affine merge mode (excluding SbTMVP candidates). For TM merge mode, merge candidates are reordered before the refinement process. Basically, when ARMC is used, merge candidates with a smaller template distance than the current block template are placed at the top of the list.

[0096] After establishing the merge candidate list, the merge candidates are divided into several subgroups. For the normal merge mode and the TM merge mode, the subgroup size is set to 5. For the affine merge mode, the subgroup size is set to 3. The merge candidates in each subgroup are reordered in ascending order according to the cost value based on template matching. For simplicity, the merge candidates in the last subgroup instead of the first subgroup are not reordered.

[0097] The template matching cost of the merge candidate is measured by the SAD between the reconstructed samples of the template of the current block and its corresponding reference (in terms of motion, not intra sense) samples of the template associated with the reference (in terms of motion, not intra sense) block. The template includes a set of reconstructed samples surrounding the current block to be encoded / decoded. The reference (in terms of motion, not intra sense) samples of the template of the reference block are located by the motion information of the merge candidate.

[0098] When the merge candidate uses bidirectional prediction, the reference (in terms of motion, not in the intra sense) samples of the merge candidate's template are also generated by bidirectional prediction, such as Fig. 8A Depicted in. Fig. 8A Therefore, the identification of reference (in terms of motion) samples of template T of the current block to be encoded / decoded in the case of a bidirectional merge candidate is illustrated: (1) using the motion vector of the merge candidate in reference list 0, the identification of reference (in terms of motion) samples in the reference picture of reference list 0 surrounding the template T of the current block to be encoded / decoded; and (2) using the motion vector of the merge candidate in reference list 1, the identification of reference (in terms of motion) samples in the reference picture of reference list 1 surrounding the template T of the current block to be encoded / decoded.

[0099] For sub-block based merge candidates with sub-block size equal to Wsub×Hsub, the template above includes several sub-templates of size Wsub×1, and the template on the left includes several sub-templates of size 1×Hsub. Figure 8B As illustrated in , the motion information of the sub-blocks in the first row and first column of the current block to be encoded / decoded is used to derive reference (in terms of motion) samples for each sub-template.

[0100] Matrix-based Intra Prediction (MIP)

[0101] MIP includes a linear intra prediction mode in which the learning matrix is ​​fixed on both the encoder and decoder sides. The prediction of the current W×H luma CB via the MIP mode includes the following three steps. First, the W decoded reference samples above the current luma CB and the H decoded reference samples to its left are downsampled. Then, the downsampled result is linearly transformed into a reduced prediction. Finally, if necessary, the reduced prediction is linearly interpolated so that the interpolated prediction has the same size as the current W×H luma CB.

[0102] More precisely, if W=4 and H=4, the downsampling factor is 2. In addition, the size of the MIP matrix in the linear transformation is 16×4 (4 input samples and 16 output samples), as Fig.9A As shown. If W = 4 and H = 8 or W = 8 and H = 4 or W = 8 and H = 8, the downsampling factor of the W decoded reference samples is W / 4, and the downsampling factor of the H decoded reference samples is H / 4. In addition, the size of the MIP matrix in the linear transformation is 16×8 (8 input samples and 16 output samples), as shown Fig. 9B As shown. For all other block sizes, the downsampling factor of the W decoded reference samples is W / 4, and the downsampling factor of the H decoded reference samples is H / 4. In addition, the size of the MIP matrix in the linear transform is 64×8 (8 input samples and 64 output samples). Note that for the interpolation step, the horizontal interpolation of the downscaled prediction uses some of the H decoded reference samples instead of their downsampled versions. The vertical interpolation of the downscaled prediction uses some of the W decoded reference samples instead of their downsampled versions.

[0103] If W=4 and H=4, there are 32 MIP modes. These modes are split into pairs, each pair uses the same MIP matrix, but for the second mode of each pair, the downsampled reference samples above the current brightness CB and the downsampled reference samples to its left are swapped. The mapping from MIP mode index to MIP matrix index is as follows Fig. 9C When the exchange of downsampled reference samples is applied, the downsampled prediction is transposed before being interpolated. If W=4 and H=8 or W=8 and H=4 or W=8 and H=8, there are 16 MIP modes and the mode pairing still applies, as Fig.9D For all other block sizes, 12 MIP modes are used and the mode pairing still applies.

[0104] Template-based neural networks without translation equivariance

[0105] In parallel with standardization, new template-based tools based on neural networks have been developed. Considering a two-dimensional translation, the translation equivariance of neural networks means that if the input is shifted by Then the neural network output is also shifted (s0, s1). Assume that the input of the neural network has four dimensions, such as YC b C r The translation equivariance of a neural network along the first two dimensions and the last dimension (e.g., two spatial dimensions and a temporal dimension) means that if the input is shifted by and shifted along the last dimension Then the neural network output is also shifted along its first two dimensions by (s0,s1) and along its last dimension by t.

[0106] In addition to template-based tools with translational isovariance, there are also template-based tools without translational isovariance. Fig. 10A and Fig. 10B Depicted is an example of a template fed into a typical template-based neural network without translation equivariance. Fig. 10A In the first example, for a given W×H block (310), the template (311) inserted into the neural network consists of n a The row decoded pixel and its left n l The template is extended by W on the right side of this block and by H below this block. In the extended part of the template, the unavailable decoded pixels are replaced / filled following the procedure for replacing unavailable decoded reference samples specified by the HEVC and VVC standards. Therefore, the bottom n pixels in the template are b ∈[|0,H|] row (313) unavailable decoded pixels and rightmost n r ∈[|0,W|] columns (312) where unavailable decoded pixels are replaced, n b and n r Depends on the encoding / decoding split history. Fig. 10B In the second example, for a given W×H block (310), the template (314) fed into the neural network is not expanded this time. In both examples, the sequence of computations within the neural network fed with the W×H block as template does not change with the availability of decoded pixels in the input template. This reduces the trade-off between the quality of the neural network output and its inference complexity.

[0107] For a given block to be encoded (respectively decoded), in its general design, the template includes no decoded pixels on the upper right side of this block and no decoded pixels on its lower left side. In the case where the template includes pixels on the upper right side of this block and pixels on its lower left side, the unavailable pixels in these two extended parts are usually replaced / filled following the procedure of replacement of unavailable decoded reference samples specified by the HEVC and VVC standards. Either the template is restricted to its general design, or it is extended while replacing / filling the unavailable pixels, which reduces the relevance of the output of the template-based tool and thus reduces the coding efficiency of the block encoded from this output.

[0108] However, depending on the size of the current block, its position in its current coding tree unit (CTU) and its position in the current frame, decoded pixels to the upper right and / or lower left of this block may be available. If most of the relevant luma texture is located to the upper right and / or lower left of this block, the fact that these decoded pixels are not included in the template may be seen as a critical loss of available information.

[0109] Therefore, it may be advantageous to extend the template towards the upper right side of the block and towards its lower left side. In a first embodiment, the extension towards the upper right side of the block may cover as many available decoded pixels as possible, subject to the restriction of W additional columns of decoded pixels. The extension towards the lower left side of the block may cover as many available decoded pixels as possible, subject to the restriction of H additional rows of decoded pixels. In other embodiments, there is no restriction on the extension. Finally, various forms of extended templates are proposed, such as a template that completely surrounds the current block.

[0110] In the template of the block to be encoded (respectively decoded), a pixel (e.g., a decoded pixel) may be unavailable because it has not been reconstructed / decoded due to the encoding / decoding segmentation history, or because it is inaccessible. Due to specific coding constraints, a pixel may be inaccessible even if it is reconstructed, for example, because it belongs to a tile different from the tile to which the block to be encoded belongs, or because it is located outside the frame boundary. In the following, "unavailable pixel" and "unavailable decoded pixel" are used interchangeably.

[0111] In this embodiment, the operation (e.g., vector-matrix product) implemented in the template-based tool and involving this template is further fed with information for identifying which decoded pixels within the template are unavailable, so as to skip at least a portion of this operation involving unavailable decoded pixels. In some embodiments, a complete module of calculations involving unavailable decoded pixels can be skipped. As an example, in the case where the template-based tool is fed with a template surrounding the current block to be encoded / decoded, this template is extended towards the upper right side of the block and its lower left side, if the template-based tool contains filters dedicated to two extended template parts and the decoded pixels in both extended parts are unavailable, this filtering can be skipped. This partial operation can be part of an element-by-element multiplication between two tensors, part of a vector-matrix product, part of a vector reduction via downsampling.

[0112] Fig.11 A method 1100 for encoding a block using a template-based tool according to one embodiment is illustrated. At 1110, information is obtained, wherein the information is used to identify which decoded pixels within the template of the block are unavailable. This information is obtained, for example, in response to a segmentation history. For example, in the case of a given frame encoded via VVC (with a CTU scan order from top left to bottom right and a hierarchical Z scan order for CUs), if the current CTU is split via a quadtree split and the current CU is the bottom right CU resulting from this split, then the segmentation history (i.e., the split depth (1), the type of split "quadtree", and the index of the current CU resulting from this split (3)) directly informs that all decoded reference samples to the top right side of the current CB in the current CU are unavailable. If a template of decoded reference samples around the current CB is to be extracted, the information about the unavailability of adjacent decoded reference samples can be the index of the column of unavailable decoded pixels in the template.

[0113] At 1120, a template-based tool is applied using the information obtained at 1110 to determine information to be used to encode the block. More specifically, the information obtained at 1110 is used to skip at least a portion of operations implemented in the template-based tool and involving unavailable decoded pixels.

[0114] At 1130, the block is encoded using the information determined by the template-based tool (e.g., prediction block, prediction mode index(es), transform type, merge candidate ordering, etc.) Encoding the block is accomplished by determining the residual between the pixels of the block and the prediction and encoding the residual.

[0115] In one example, an encoding device is disclosed, which includes one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to execute the above encoding method.

[0116] Fig.12 A method 1101 of decoding a block using a template-based tool according to one embodiment is illustrated. At 1111, information is obtained, wherein the information is used to identify which decoding pixels within the template of the block are unavailable. This information is obtained, for example, in response to a segmentation history.

[0117] At 1121, the template-based tool is applied using the information obtained at 1111 to determine the information to be used to decode the block. The information to be used to decode the block is the same as the information to be used to encode the block. More precisely, the information obtained at 1111 is used to skip at least a portion of the operations implemented in the template-based tool and involving unavailable decoded pixels. Steps 1111 and 1121 are the same as steps 1110 and 1120 on the encoder side.

[0118] At 1131, the index of the block is decoded using information determined by the template-based tool (e.g., prediction block, prediction mode index(es), etc.) Decoding the block is accomplished by decoding the residual and adding the residual to the prediction to reconstruct the pixels of the block.

[0119] The following examples apply to both the encoding method and the decoding method.

[0120] In one example, using the obtained information to apply the template-based tool includes skipping a calculation if the calculation involves pixels identified as unusable by the obtained information.

[0121] In one example, the method further includes flattening the template prior to applying the template-based tool.

[0122] In one example, obtaining information for identifying which decoded pixels within the template of the current block are unavailable includes obtaining indices of all unavailable decoded pixels.

[0123] In one example, obtaining information for identifying which decoded pixels within the template of the current block are unavailable includes obtaining indices of all available decoded pixels.

[0124] In one example, obtaining information for identifying which decoded pixels in the template of the current block are unavailable includes obtaining flags, each flag indicating, for a pixel in the template, whether the pixel is available.

[0125] In one example, obtaining information for identifying which decoded pixels in the template of the current block are unavailable includes: for a group of adjacent unavailable pixels, obtaining an index of a first unavailable pixel and an index of a last unavailable pixel in the group.

[0126] In one example, obtaining information for identifying which decoded pixels in the template of the current block are unavailable includes: for a group of adjacent available pixels, obtaining an index of a first available pixel and an index of a last unavailable pixel in the group.

[0127] In one example, the method further includes spatially reorganizing pixels within the template so as to increase the number of available decoded pixels that are contiguous to the memory.

[0128] In one example, the unavailable decoding pixel is one of a pixel that has not been reconstructed, a pixel that belongs to a tile different from the tile to which the current block belongs, or a pixel outside a picture boundary.

[0129] In one example, the template-based tool belongs to a group of template-based tools, which include:

[0130] - Intra mode derivation at decoder side;

[0131] -Convolutional cross-component model;

[0132] - Adaptive re-ranking of merge candidates using template matching;

[0133] - Intra prediction based on template matching prediction matrix; and

[0134] - Template-based neural network prediction without translation equivariance.

[0135] In one example, the template includes all pixels located around the current block.

[0136] In one example, the template includes a line of pixels located above the current block and columns of pixels located to the right and left of the current block.

[0137] In one example, a decoding device is disclosed, which includes one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to execute the above decoding method.

[0138] In one example, a computer program comprises program code instructions for implementing the steps of the above encoding (respectively decoding) method.

[0139] In one example, a computer-readable storage medium having stored thereon instructions for encoding or decoding a block of a picture according to the above encoding (respectively decoding) method.

[0140] Below about Figures 13A-26D Additional embodiments are described.

[0141] Example 1

[0142] The information for identifying which decoded pixels within the template of the block are unavailable may be a set of indices of unavailable decoded pixels in the template or a transformed version of the template. This transformation can be any transformation, such as filtering, reshaping, rotation, flipping or splitting.

[0143] For example, Fig.13A and 13B Illustration of the case where the operation in the template-based tool is a vector-matrix product and the input template is first flattened Fig.11 and Fig.12 The template (401) of a given W×H block (400) contains n b ∈[|0,H|]| row (403) has unavailable decoded pixels and the n pixels to its right r ∈[|0,W|] columns (402) are unavailable decodable pixels. The notation [|a,b|] represents all integers included in the range [a;b]. The template is first flattened, resulting in (404). The product (407) between the flattened template (404) and the weight matrix (406) is then multiplied using the index set of unavailable decodable pixels (405) within the flattened template. to skip the computation, i.e., multiplication and addition. In this figure, the light grey squares indicate unavailable decoded pixels. In the weight matrix (406), the light grey areas contain weights that are not used due to skipped computation. In other words, the output coefficient of index j is represented as:

[0144]

[0145] T i : the coefficient of index i in the flattened template,

[0146] w i,j : The weight of index (i,j),

[0147] s: The size of the flattened template.

[0148] In a first variation of Embodiment 1, the index of any group of adjacent unavailable decoded pixels may be defined as the index of the first unavailable decoded pixel and the last unavailable decoded pixel. Each pair of indices in becomes the set of all indices between the two indices of the pair, and the index pair is included in the set. The above (Equation 1) remains unchanged, except that The definition is different:

[0149] In a second variation of embodiment 1, the information for identifying which decoded pixels within the template of the block are unavailable may be a flag set Each flag indicates whether the associated decoded pixel in the template or in a transformed version of the template is unavailable. In this embodiment, a flag equal to true indicates that the associated pixel is unavailable, and a flag equal to false indicates that the associated pixel is available. Fig.13A and 13B In the example shown in the figure, can be defined as follows:

[0150]

[0151] In fact, the template includes n a lines, each line includes (n l +2W-n r ) available pixels (so flag equals false), followed by n r unavailable pixels (so flag equals true), then n l (2H-n b ) available pixel lines (so flag equals false), and finally n l n b unavailable pixels (hence flag equals false).

[0152] Then, the output coefficient of index j is calculated as follows:

[0153]

[0154] Example 2

[0155] Instead of specifying which pixels in the template are unavailable, the information used to identify which decoded pixels within the template of the block are unavailable may be a set of indices of available decoded pixels in the template or a transformed version of the template. This transformation can be any transformation, such as filtering or reshaping or rotation or flipping or splitting.

[0156] For example, Fig.14A and 14B Illustration of the case where the operation in the template-based tool is a vector-matrix product and the input template is first flattened Figure 11 to Figure 12 Given a W×H block (500), the template (501) contains n b∈[|0, H|]| rows (503) have unavailable decodable pixels. The template is first flattened, resulting in (504). The product (507) between the flattened template (504) and the weight matrix (506) is then multiplied using the index set of available decodable pixels (505) within the flattened template. To skip multiplication and addition. In this figure, the light gray square indicates an unavailable decoded pixel. In the weight matrix (506), the light gray area contains the weights that are not used because the calculation is skipped. In other words, the output coefficient of index j is expressed by the following (Equation 2):

[0157]

[0158] In a first variation of Embodiment 2, the index of any set of adjacent available decoded pixels may be defined as the index of the first available decoded pixel and the last available decoded pixel. Each pair of indices in becomes the set of all indices between the two indices of the pair, and the index pair is included in the set. The above (Equation 2) remains unchanged, except that The definition is different:

[0159] In a second variation of Embodiment 2, the information for identifying which decoded pixels within the template of the block are unavailable may be a flag set: Each flag indicates whether the associated decoded pixel in the template or in a transformed version of the template is available. In this embodiment, a flag equal to true indicates that the associated pixel is available, and a flag equal to false indicates that the associated pixel is not available. Fig.14A and 14B In the example shown in the figure, can be defined as follows:

[0160]

[0161] Then, the output coefficient of index j is calculated as:

[0162]

[0163] Example 3

[0164] In this embodiment, the template can be transformed so that the number of available decoded pixels that are continuous in memory is increased in the transformed version of the template in the operation of interest fed into the template-based tool. The transformation is, for example, a spatial reorganization of pixels within the template in order to increase the number of available decoded pixels that are continuous in memory. In fact, the benefit of having as many decoded pixels of the same type of availability close to each other as possible is that when calculations as explained in the previous embodiment are skipped, several acceleration methods can be better utilized for the calculations that are not skipped. For example, it is assumed that AVX-512 is used. It is also assumed that in the potential transformed version of the template fed into the operation of interest, each decoded pixel is stored as a 16-bit integer. The higher the number of packets of 32 memory-contiguous available decoded pixels, the better the AXV-512 acceleration.

[0165] This embodiment can be combined with any of the previous embodiments 1 or 2 and any of their variations. Fig.13A and Fig. 13B An example of a combination of Embodiment 1 disclosed.

[0166] Figures 15A-15B An exemplary combination of embodiment 3 and embodiment 1 is provided, ie the information corresponds to the index of the unavailable decoded pixel. Furthermore, the operation of interest in the template-based video coding tool is the vector-matrix product, and the flattening occurs before the vector-matrix product.

[0167] A template (601) of a given W×H block (600) contains n b ∈[|0,H|]| row (603) has unavailable decoded pixels and the n pixels to its right r ∈[|0,W|] columns (602) are unavailable decodable pixels. The template is first split into two parts at line (604), and the part above line (604) is transposed, resulting in (605) and (606). Then, (605) and (606) are flattened into a single vector (607). Finally, the product (610) between the flattened template (607) and the weight matrix (609) is used to index the set of unavailable decodable pixels (608) within the flattened template. To skip multiplication and addition. In this example, (607) contains only two different groups of memory-contiguous available decoded pixels.

[0168] Figures 16A-16B Another exemplary combination of Embodiment 3 and Embodiment 1 is provided, namely the information corresponds to the index of the unavailable decoded pixel. In addition, the operation of interest in the template-based video coding tool is the vector-matrix product, and the flattening occurs before the vector-matrix product. In this figure, the crosses, circles and diamonds just help to visualize the cascade of flips and transpositions about the vertical axis.

[0169] The template (701) of a given W×H block (700) contains n b ∈[|0,H|]| row (703) has unavailable decoded pixels and the n pixels to its right r ∈[|0,W|] columns (702) are unavailable decodable pixels. The template is first split into two parts at line (704), and the part above line 704 is flipped about the vertical axis and then transposed, resulting in (705) and (706). Then, (705) and (706) are flattened into a single vector (707). Finally, the product (710) between the flattened template (707) and the weight matrix (709) is used to index the set of unavailable decodable pixels (708) within the flattened template. To skip multiplication and addition. In this example, (707) contains a single set of memory-contiguous available decoded pixels.

[0170] In this embodiment, The index n of the last unavailable decoded pixel in (707) belonging to the first memory continuous unavailable decoded pixel group may be included a n r -1, and the index n of the first unavailable decoded pixel belonging to the unavailable decoded pixel group in the second memory in (707) b n l -1, the index starts from the end of the vector this time. This makes it possible to have a compact representation of the information for identifying which decoded pixels within the template are not available.

[0171] Figures 17A-17B Another exemplary combination of Embodiment 3 and Embodiment 1 is provided, namely the information corresponds to the index of the unavailable decoded pixel. In addition, the operation of interest in the template-based video coding tool is the vector-matrix product, and the column-by-column flattening occurs before the vector-matrix product. In this figure, the crosses, circles and diamonds are just to help visualize the cascade of flips and transpositions about the horizontal axis.

[0172] The template (801) of a given W×H block (800) contains n b ∈[|0,H|]| row (803) has unavailable decoded pixels and the n pixels to its right r∈[|0,W|] columns (802) are unavailable for decoding. The template is first split into two parts at line (804), and the part to the left of line 804 is flipped about the horizontal axis and then transposed, resulting in (805) and (806). Then, (805) and (806) are flattened column by column into a single vector (807). Column by column flattening means scanning (806) and (805) column by column from left to right, scanning each column in turn from top to bottom to produce (807). Finally, the product (810) between the flattened template (807) and the weight matrix (809) uses the index set of the unavailable decoding pixels (808) within the flattened template. To skip multiplication and addition. In this example, (807) contains a single set of memory-contiguous available decoded pixels.

[0173] In this embodiment, may contain the index n of the last unavailable decoded pixel belonging to the first memory contiguous unavailable decoded pixel group in (807) b n l -1, and the index n of the first unavailable decoded pixel belonging to the second memory continuous unavailable decoded pixel group in (807) a n r -1, the index starts from the end of the vector this time. This makes it possible to have a compact representation of the information for identifying which decoded pixels within the template are not available.

[0174] Example 4

[0175] In this embodiment, due to specific constraints, such as independence of tiles during encoding / decoding, some of the decoded pixels may not be accessible while they have been reconstructed.

[0176] For example, Fig.18A and 18B Illustration of the case where the operation in the template-based tool is a vector-matrix product, the current frame is split into multiple tiles, and the input template is flattened first Figure 11 to Figure 12 The template (901) of a given W×H block (900) contains n r ∈[|0,W|] column (902) is not available for decoding pixels. In addition, the leftmost p∈[|0,n l|] The decoded pixel (903) belongs to the tile to the left of the tile that includes block (900). The boundary between these two tiles is represented as (904). The template is first flattened, resulting in (905). Then, because the input template overlaps two tiles (906) within the flattened template, the product (908) between the flattened template (905) and the weight matrix (907) uses the index set of decoded pixels that are not available or accessible. to skip multiplication and addition. In this figure, light gray squares indicate decoded pixels that are unavailable due to the encoding / decoding segmentation history, or decoded pixels that are inaccessible because the input template overlaps two tiles. In the weight matrix (907), the light gray area contains weights that are not used due to computational skipping. As in Example 1, the output coefficient of index j is represented as:

[0177]

[0178] All embodiments 1-3 and their variants can be combined with embodiment 4. In particular, an index set of available or accessible decoded pixels can be indicated. Instead of the above collection

[0179] Example 5

[0180] In this embodiment, some of the decoded pixels may not be accessible while already being reconstructed due to specific constraints (eg independence of tiles during encoding / decoding) or due to being outside a frame boundary.

[0181] For example, Fig.19A and 19B Illustration of the case where the operation in the template-based tool is a vector-matrix product, the current frame is split into multiple tiles, the template of the tile exceeds the boundary of the frame containing the tile, and the input template is first flattened Figure 11 to Figure 12 Given a W×H block (1000), the template (1001) contains n r ∈[|0,W|] column (1002) is not available for decoding pixels. In addition, the top p∈[|0,n a |] row decoded pixel (1003) belongs to the tile above the tile including block (1000). The boundary between these two tiles is represented as (1005). In addition, the leftmost q∈[|0,n l|] The column decoded pixels (1004) are beyond the left boundary (1006) of the frame that includes the block (1000). The template is first flattened, resulting in (1007). Then, assuming that the position of the template with respect to the tile boundary and the frame boundary (1008) is within the flattened template, the product (1010) between the flattened template (1007) and the weight matrix (1009) is calculated using the index set of the available or accessible decoded pixels. to skip multiplication and addition. In this figure, light gray squares indicate decoded pixels that are unavailable due to the encoding / decoding segmentation history, or decoded pixels that are inaccessible because the input template overlaps two tiles or exceeds the frame boundary. In the weight matrix (1009), the light gray area contains weights that are not used due to calculation skipping. As in Example 2, the output coefficient of index j is expressed as:

[0182]

[0183] All embodiments 1-3 and their variants can be combined with embodiment 5. In particular, an index set of unavailable or inaccessible decoded pixels can be indicated. Instead of the above collection

[0184] Example 6

[0185] The previously described embodiments can be extended so that the template-based tool handles templates of various sizes. In this case, the template-based tool can be used with a block size set associated, where you can dispose Template-based tools can then be set up to be fed a “maximum template”, i.e. one with each dimension being The template with the largest corresponding dimension over the block size in .

[0186] Fig. 20A Provides a set of block sizes and for disposal Example definition of the “maximum template” for a template-based tool for each block size template in . i ×H i The number of decoded pixel columns n to the left of this block in the template of the block l,i and the number of decoded pixel rows above this block n a,i It can be H i and / or W i Function, n l,i =f(H i ,W i ) and n a,i =g(H i ,W i). f and g can be H i and / or W i As an example, n l,i =min(H i ,W i )=n a,i . According to another example:

[0187] <![CDATA[H i ]]> <![CDATA[W i ]]> <![CDATA[n l,i ]]> 2 4 8 4 2 4 8 2 12 … 16 16 16

[0188] This simplifies the definition of the “maximum template”, since the shape of the “maximum template” can be directly obtained from Derived.

[0189] For any given W×H block, The template of this block can be put into the "maximum template". The coefficients in the "maximum template" that are outside the template of this block can be considered unusable. Therefore, all the embodiments disclosed previously can be applied.

[0190] For example, Figures 20B-20C Illustration of the case where the operation in the template-based tool is a vector-matrix product and the input template is first flattened Figure 11 to Figure 12 method.

[0191] The "maximum template" (1100) is first Assuming the size of the block of interest is W×H and deriving from it (n r ,n b ), the template (1101) of this block is placed in the "maximum template". Two vertical thick dashed lines separate (1101) from the coefficients of the "maximum template" that do not belong to (1101). The coefficients of the "maximum template" that do not belong to (1101) (denoted as (1102)) are considered unusable. (1101) is contained in the n b ∈[|0,H|] row (1104) has unavailable decoded pixels and the n pixels to its right r ∈[|0,W|] columns (1103) are unavailable for decoding. The padded "max template" is flattened, resulting in (1105). Finally, the product (1108) between the flattened padded "max template" (1105) and the weight matrix (1107) is used to index the set of unavailable decoded pixels (1106) within (1105). to skip multiplications and additions. In the template (1100) and its flattened version (1105), light grey squares indicate decoded pixels that are unavailable due to segmentation history, or decoded pixels that are inaccessible due to other constraints (e.g., tile, frame boundaries). In the weight matrix (1107), the light grey areas contain weights that are not used due to computational skipping. In this embodiment, the set L is therefore used to specify pixels in the "maximum template" (1105) that are not part of the template (1101) of the block and are therefore considered unavailable.

[0192] For example, Figures 20D-20E Illustration of the case where the operation in the template-based tool is an element-wise vector-matrix product and the input template is first flattened Figure 11 to Figure 12 method.

[0193] First use to establish a "maximum template" (1200). Assuming the size W×H of the block of interest, the encoding / decoding segmentation history, and other coding constraints, the template (1201) of this block is inserted into the "maximum template". The two thick dashed lines separate (1201) from the coefficients of the "maximum template" that do not belong to (1201). The coefficients of the "maximum template" that do not belong to (1201) (denoted as (1202)) are considered unavailable. The rightmost p∈[|0,W|] column (1203) of the template (1201) belongs to the block located to the right of the block that includes the block of interest. In Fig.20D The boundary between these two tiles is a thick vertical line. The padded "maximum template" is flattened, resulting in (1204). Finally, the element-wise product (1207) between the flattened padded "maximum template" (1204) and the weight vector (1206) is performed using the index set of the available decoded pixels (1205) within (1204). to skip the multiplication. In the template (1200) and its flattened version (1204), the light grey squares indicate decoded pixels that are not available due to the segmentation history, or decoded pixels that are inaccessible due to other constraints. In the weight vector (1206), the light grey areas contain weights that are not used due to computation skipping. The coefficient of index j obtained from the vector-vector element-wise multiplication can be expressed as: T i w i , for others, it is the default value.

[0194] If the index of a given decoded pixel in the potentially transformed version of the template fed into the vector-vector element-wise multiplication does not belong to The output coefficient of this index can use the default value, such as 0.

[0195] All embodiments 1-5 and their variations can be combined with embodiment 6. Template-based tools can be constructed to feed a "larger maximum template", i.e., a template with each dimension larger than The template with the largest corresponding dimension in the block size is not constructed to feed the "maximum template"

[0196] The embodiments disclosed for the operation of vector-vector element-wise product may also be applied to other types of operations, such as vector-matrix product.

[0197] Example 7

[0198] Within a template-based video coding tool, an operation of interest that is fed with information for identifying which decoded pixels are unavailable may need to reinterpret this information. This reinterpretation may depend on how the template was processed before being fed to the operation.

[0199] To illustrate this point, FIG. 21A to FIG. 21B It is shown that in the case where the template-based tool is a template-based neural network consisting of two convolutional layers and one fully connected layer Fig.13A and Fig. 13B The stride of the convolutions of both layers is equal to 1, and the type of both convolutions is the same. In this embodiment, the operation under consideration in the template-based tool is the vector-matrix product within a fully connected layer. The template (3001) of a given W×H block (3000) contains n b ∈[|0,H|]| row (3003) has unavailable decoded pixels and the n pixels to its right r ∈[|0,W|] columns (3002) are not available for decoding pixels. (3002) and (3003) are removed from the template (3001), and (3001) is split into two parts (3004) and (3005). The first convolutional layer with stride 1, same type and n0 kernels uses (3004) to generate a three-dimensional stack (3006). The second convolutional layer with stride 1, same type and n1 kernels uses (3005) to produce a three-dimensional stack (3007). The same type means that the input of the convolutional layer is padded so that each of the two spatial dimensions of the output is equal to its corresponding dimension in the input divided by the stride. Then, (3006) and (3007) are flattened into a single vector (3008). For this flattening, priority is given to the third dimension, followed by the second dimension. (3008) is fed into the fully connected layer of the neural network. As Fig. 21B As shown, in the set L (3009), each index of an unusable decoded pixel within the flattened template must be reinterpreted to incorporate the fact that the template (3001) has already been processed (removal of unusable parts and application of convolution). For example, the index nl +2W-n r becomes n0(n l +2W-n r As another example, index n l +2W-1 becomes n0(n l +2W)-1. FIG. 21A to FIG. 21B In , the calculation skipping is equivalent to removing the weight matrix (3010) with The row with the index in .

[0200] As another example, FIG. 22A to FIG. 22B Shows the FIG. 21A to FIG. 21B Same situation, but using convolution stride 2 instead of convolution stride 1. FIG. 22A to FIG. 22B middle, No changes. However, The reinterpretation of each index in is adaptively modified because the processing of the template is changed before being fed into the vector-matrix product in the neural network.

[0201] In templates (3001) and (4001), light grey squares indicate unavailable decoded pixels. In weight matrices (3010) and (4010), light grey areas contain weights that are not used due to computation skipping.

[0202] In all previously disclosed embodiments, the template is extended by an additional W columns of decoded pixels towards the upper right side of the block, and by an additional H rows of decoded pixels towards the lower left side of the block. However, these embodiments are not limited to these extensions, and can also be generalized to different extensions, i.e. different sizes and different forms.

[0203] Example 8

[0204] As an example, FIG. 23A to FIG. 23B Shows Fig.13A and Fig. 13B An adaptation of the example of w =2W+4 and e H =2H+4. Given a W×H block (1300), the template (1301) contains n b ∈[|0,e H -H|] line (1303) is not available for decoding pixels and the n r ∈[|0,e w The template is first flattened, thereby producing (1304). Then, the product (1307) between the flattened template (1304) and the weight matrix (1306) is used to index the set of unavailable decoded pixels (1305) within the flattened template. to skip multiplication and addition.

[0205] In all previously disclosed embodiments, the template extended towards the upper right side of the block to be encoded may cover as many available decoded pixels as possible, within the limit of an additional W columns of decoded pixels. The extension towards the lower left side of this block may cover as many available decoded pixels as possible, within the limit of an additional H rows of decoded pixels. In other embodiments, there is no limit on the extension. Finally, various forms of extended templates are proposed, such as a template that completely surrounds the current block.

[0206] In all previously disclosed embodiments, the template extends toward the upper right side of the block and toward the lower left side of the block. However, the embodiments are not limited to these extensions.

[0207] Example 9

[0208] As an example, Figures 24A-24B and 25A-25B show Fig.13A and Fig. 13B An adaptation of the example in , in which the template is further extended to the upper left side of the block and towards the lower right side of the block. This form of extended template can be advantageously used with a coding order different from the classical coding / decoding order (i.e., from left to right and from top to bottom), such as the fixed coding / decoding order of VVC and ECM. Therefore, at a given macroblock level, the coding / decoding order can be switched horizontally from left to right or from right to left. However, it should be understood that this type of template can also be used with the classical left to right and top to bottom coding / decoding order.

[0209] exist FIG. 24A to FIG. 24B A first example of using such an extended template is depicted above. Fig.24A An extended template (1401) is shown, which includes a line of pixels above the current block W×H block (1400) and additional columns of pixels both to the left and to the right of the block (1400). More precisely, Fig.24A The extended template includes (2e at the top of the current block (1400) w -W)×n a block and two n on the right and left respectively l ×e H piece.

[0210] In one example, e w =2W+4 and e H =2H+4. However, different values ​​may be used. A template (1401) of a given W×H block (1400) contains n b ∈[|0,e H -H|] line (1403) is not available for decoding pixels and the nr ∈[|0,e w -W|] column (1402) is not available for decoding. In the case where the encoding / decoding order is from left to right, the e at (1404) H n l decoded pixels are always unavailable. The template is first flattened, resulting in (1405). Note that the height on the right side of block (1400) is e H and width n l The rectangle is finally flattened. Then, the product (1408) between the flattened template (1405) and the weight matrix (1407) uses the index set of unavailable decoded pixels (1406) within the flattened template to skip multiplication and addition.

[0211] exist FIG. 25A to FIG. 25B Another example of using such an extended template is depicted above. Fig.25A An extended template (1501) is shown, which includes a line of pixels above the current block W×H block (1500) and additional columns of pixels both to the left and to the right of the block (1500). More precisely, Fig.25A The extended template includes (2e at the top of the current block (1500) w -W)×n a block and two n on the right and left respectively l ×e H piece.

[0212] In this example, e w =2W+4 and e H =2H+4. Given a W×H block (1500), the template (1501) contains n b ∈[|0,e H -H|] line (1503) has unavailable decoded pixels and the n r ∈[|0,e w -W|] column (1502) is not available for decoding. In the case where the encoding / decoding order is from right to left, the pixel at (1504) H n l decoded pixels are always unavailable. The template is first flattened, resulting in (1505). Note that the height on the right side of block (1500) is e H and width n l The rectangle is finally flattened. Again, the product (1508) between the flattened template (1505) and the weight matrix (1507) uses the index set of the unavailable decoded pixels (1506) within the flattened template to skip multiplication and addition.

[0213] Example 10

[0214] The particularity of an inter slice is that it contains both intra-predicted CUs and inter-predicted CUs. The decoding of a given CTU in an inter slice is broken down into three steps. In the first step, called parsing, all bits of the syntax associated with this CTU are read from the bitstream. In the second step, called decoding, these bits are interpreted. For example, the bits associated with the prediction of a given CU are interpreted as intra / inter modes. In the third step, the pixels of the CTU are reconstructed. For an inter-predicted CU, prediction requires only decoded pixels from an already decoded reference frame. In contrast, for an intra-predicted CU, prediction requires only decoded pixels located above and to the left of the current CU. This is why, in a given CTU in an inter slice, the third step can be run in parallel for all CUs predicted in the inter. However, for any CU predicted in the intra frame, the third step must be run after reconstructing the pixels in the CU located above and to the left of it. It is known that in some decoders, for a given CTU in an inter slice, the third step is parallelized for all CUs predicted in the inter frame. Then, at a given time step during the third step of the inter-predicted CU, the third step of the intra-predicted CU can start. For simplicity, it is assumed that for a given CTU in an inter slice, the time step for starting the third step of the intra-predicted CU follows immediately after the third step of all CUs that have completed inter-prediction. Then, for an intra-predicted CU, the prediction can access more decoded pixels than those located above and to the left of the CU.

[0215] For example, on the decoder side, Fig.26A In the CTU in the inter slice shown in , for an intra-predicted CU (shown in hatched lines), prediction has access to all decoded pixels located around this CU. For the current CTU in the current inter slice, the time step for starting reconstruction of the pixels of the intra-predicted CU occurs after reconstruction of the pixels of all inter-predicted CUs is completed. The dotted line thus describes the decoded pixels that can be accessed by the prediction module for an intra-predicted CU. As another example, at the decoder side, in Fig.26B In the CTU in the inter slice shown, for the leftmost CU predicted by intra, the prediction can access the decoded pixels located above / below and to the left of it. For the leftmost CU predicted by intra, the dotted line describes the decoded pixels that can be accessed by its prediction module.

[0216] In inter slices, given the above mentioned decoder implementation, the previously disclosed embodiments can be advantageously used with template based tools specific to templates of intra predicted blocks.

[0217] For example, Fig.26C and Fig.26DDepicted Fig.13A and Fig. 13B In an adaptation of the example in , when the template-based tool is fed with a template of a block to be intra-predicted within an inter slice, the operation of interest in the template-based tool is the vector-matrix product, and the input template is first flattened. Furthermore, Fig.26C and Fig.26D An extended template is shown, which includes all pixels around the current block W×H block (1600). More precisely, in Fig.26C and Fig.26D The extended template includes (W+2*n) at the top of the current block (1600). l )×n a block, (W+2*n below the current block (1600) l )×n a block, and two n on the right and left sides respectively l ×H blocks. Different values ​​may be used for the height and width of the blocks surrounding the current block (1600).

[0218] In the template (1601) of the intra-predicted W×H block (1600), for example Fig.26B For the leftmost CU in intra prediction, all decoded pixels to the right of block (1602) are unavailable. The template is first flattened, resulting in (1603). Then, the product (1606) between the flattened template (1603) and the weight matrix (1605) is used to index the set of unavailable decoded pixels (1604) within the flattened template. To skip multiplication and addition. In the template (1601), the light grey squares indicate unavailable decoded pixels. In the weight matrix (1605), the light grey areas contain weights that are not used due to computation skipping.

[0219] In various embodiments, the operation for which some parts / computations are skipped is a vector-matrix product. This is just one example. In all various embodiments, other operations may be considered for computation skipping, such as, for example, an element-wise vector-vector product as implemented by "tf.matmul" in Tensorflow or "torch.matmul" in PyTorch, a matrix product of tensors, an outer product between two vectors as implemented by "numpy.outer" in Numpy, for example. TensorFlow, PyTorch, Numpy are all libraries.

[0220] The various methods and other aspects described in this application can be used to modify Figure 2 and Figure 3Modules of the video encoder 200 and decoder 300 shown, such as intra prediction modules (260, 360). In addition, the present aspects are not limited to ECM, VVC or HEVC, and can be applied to, for example, other standards and recommendations, and extensions of any such standards and recommendations. Unless otherwise specified or technically excluded, the aspects described in this application can be used alone or in combination.

[0221] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes, and the described aspects are not limited to these specific values.

[0222] Various embodiments relate to decoding. "Decoding" as used in this application may encompass, for example, all or part of a process performed on a received encoded sequence in order to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also includes or alternatively includes a process performed by a decoder of various embodiments described in this application, such as decoding resampling filter coefficients, resampling decoded pictures; or, for example, obtaining information which pixels within a template for identifying a current block of a picture are unavailable; using the obtained information to apply a template-based tool to determine the information to be used to decode the current block; using the determined information to decode the current block.

[0223] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding, and in another embodiment, "decoding" refers to the entire reconstructed picture process including entropy decoding. Based on the context of the specific description, whether the phrase "decoding process" is intended to specifically refer to a subset of operations or generally refer to a broader decoding process will be clear and is considered to be well understood by those skilled in the art.

[0224] Various embodiments relate to encoding. In a manner similar to the discussion above regarding "decoding", "encoding" as used in this application can cover, for example, all or part of the processes performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as segmentation, differential coding, transforms, quantization, and entropy coding. In various embodiments, such processes also include or alternatively include processes performed by encoders of various embodiments described in this application, such as determining resampling filter coefficients, resampling decoded pictures; or, for example, obtaining information which pixels within a template for identifying a current block of a picture are unavailable; using the obtained information to apply a template-based tool to determine information to be used to decode the current block; using the determined information to encode the current block.

[0225] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or generally to a broader encoding process will be clear based on the context of the particular description and is considered to be well understood by those skilled in the art.

[0226] This disclosure has described various pieces of information that may be transmitted or stored, such as, for example, syntax. This information may be packaged or arranged in various ways, including, for example, those common in video standards, such as placing the information in an SPS, PPS, NAL unit, header (e.g., a NAL unit header or a slice header), or SEI message. Other approaches are also available, including, for example, those common to system-level or application-level standards, such as placing the information in one or more of the following:

[0227] a. SDP (Session Description Protocol), a format for describing multimedia communication sessions, used for the purpose of session announcement and session invitation (eg described in RFC) and used in conjunction with RTP (Real-time Transport Protocol) transport.

[0228] b. DASH MPD (Media Presentation Description) descriptor, such as that used in DASH and delivered over HTTP, which is associated with a representation or a set of representations to provide additional characteristics to the content representation.

[0229] c. RTP header extension, eg as used during RTP streaming.

[0230] d. ISO Base Media File Format, such as the format used in OMAF, and uses boxes which are object-oriented building blocks defined by a unique type identifier and a length (also referred to as "atoms" in some specifications).

[0231] e. HLS (HTTP Live Streaming) manifest delivered over HTTP. A manifest may be associated with, for example, a version or set of versions of content to provide characteristics of the version or set of versions.

[0232] When a figure is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0233] Various embodiments may refer to rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between bit rate and distortion is usually considered, and constraints on computational complexity are usually considered. Rate-distortion optimization is usually formulated as minimizing a rate-distortion function, which is a weighted sum of bit rate and distortion. There are different methods to solve the rate-distortion optimization problem. For example, these methods can be based on extensive testing of all coding options, including all considered modes or coding parameter values, and a complete evaluation of their coding costs and the associated distortion of the reconstructed signal after encoding and decoding. Faster methods can also be used to save coding complexity, especially by calculating approximate distortion based on prediction or prediction residual signals rather than reconstructed signals. A mixture of these two methods can also be used, such as by using approximate distortion only for some possible coding options and full distortion for other coding options. Other methods only evaluate a subset of possible coding options. More generally, many methods use any of a variety of techniques to perform optimization, but optimization is not necessarily a complete evaluation of coding costs and associated distortions.

[0234] The embodiments and aspects described herein can be implemented in, for example, methods or processes, devices, software programs, data streams or signals. Even if only discussed in the context of a single embodiment form (e.g., discussed only as a method), the embodiments of the features discussed can also be implemented in other forms (e.g., devices or programs). The device can be implemented in, for example, suitable hardware, software and firmware. The method can be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit or a programmable logic device. The processor also includes a communication device, such as, for example, a computer, a cellular phone, a portable / personal digital assistant ("PDA") and other devices that facilitate the communication of information between terminal-users.

[0235] Reference to "one embodiment" or "an embodiment" or "one implementation" or "an implementation", and other variations thereof, means that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearance of the phrase "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation", and any other variations thereof throughout this application, are not necessarily all referring to the same embodiment.

[0236] Additionally, this application may refer to "determining" various pieces of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0237] Additionally, the present application may refer to "accessing" various pieces of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0238] Furthermore, the present application may refer to "receiving" various pieces of information. Like "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" is generally referred to in one way or another during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0239] It is to be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As another example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to encompass selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to as many of the listed items as are apparent to one of ordinary skill in this and related arts.

[0240] In addition, as used herein, the word "signaling" especially refers to indicating something to the corresponding decoder. For example, in some embodiments, the encoder signals a specific one of a plurality of resampled filter coefficients or coding blocks. In this way, in one embodiment, the same parameters are used at both the encoder side and the decoder side. Therefore, for example, the encoder can transmit (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters and other parameters, signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select specific parameters. By avoiding the transmission of any actual function, bit savings are achieved in various embodiments. It is to be understood that signaling can be done in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to the corresponding decoder. Although the verb form of the word "signaling" is mentioned above, the word "signal" can also be used as a noun in this article.

[0241] It will be apparent to one of ordinary skill in the art that embodiments may generate various signals that are formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing a method, or data generated by one of the described embodiments. For example, a signal may be formatted to carry a bitstream of the described embodiment. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, a signal may be transmitted over a variety of different wired or wireless links. A signal may be stored on a processor readable medium.

[0242] A number of embodiments have been described above. The features of these embodiments may be provided individually or in any combination across the various claim categories and types.

Claims

1. A decoding method, comprising: Obtain information about which pixels in a template for identifying a current block of the image are unavailable; applying a template-based tool using the obtained information to determine information to be used to decode the current block; as well as The current block is decoded using the determined information.

2. The method according to claim 1, wherein: Using the obtained information to apply the template-based tool includes skipping calculations if the calculations involve pixels identified as unusable by the obtained information.

3. The method according to claim 1 or 2, wherein: The method further includes flattening the template prior to applying the template-based tool.

4. The method according to any one of claims 1 to 3, wherein: Obtaining information for identifying which pixels within the template of the current block are unavailable includes obtaining indices of all unavailable pixels.

5. The method according to any one of claims 1 to 3, wherein: Obtaining information for identifying which pixels within the template of the current block are unavailable includes obtaining indices of all available pixels.

6. The method according to any one of claims 1 to 3, wherein: Obtaining information for identifying which pixels in the template of the current block are unavailable includes obtaining flags, each flag indicating, for a pixel in the template, whether the pixel is available.

7. The method according to any one of claims 1 to 3, wherein: Obtaining information for identifying which pixels in the template of the current block are unavailable includes: for a group of adjacent unavailable pixels, obtaining an index of a first unavailable pixel and an index of a last unavailable pixel in the group.

8. The method according to any one of claims 1 to 3, wherein: Obtaining information for identifying which pixels in the template of the current block are unavailable includes: for a group of adjacent available pixels, obtaining an index of a first available pixel and an index of a last unavailable pixel in the group.

9. The method according to any one of claims 1 to 8, further comprising spatially reorganizing pixels within the template so as to increase the number of available pixels that are contiguous in memory.

10. The method according to any one of claims 1 to 9, wherein: The unavailable pixel is one of a pixel that has not been reconstructed, a pixel that belongs to a tile different from the tile to which the current block belongs, or a pixel outside a picture boundary.

11. The method according to any one of claims 1 to 10, wherein: The template-based tool belongs to a group of template-based tools, which include: - Intra mode derivation at decoder side; -Convolutional cross-component model; - Adaptive re-ranking of merge candidates using template matching; - Intra prediction based on template matching prediction matrix; and - Template-based neural network prediction without translation equivariance.

12. The method according to any one of claims 1 to 11, wherein: The template includes all pixels located around the current block.

13. The method according to any one of claims 1 to 11, wherein: The template includes a pixel line located above the current block and a pixel column located on the right and left of the current block.

14. A coding method comprising: Obtain information about which pixels in a template for identifying a current block of the image are unavailable; applying a template-based tool using the obtained information to determine information to be used to encode the current block; as well as The current block is encoded using the determined information.

15. The method according to claim 14, wherein: Using the obtained information to apply the template-based tool includes skipping calculations if the calculations involve pixels identified as unusable by the obtained information.

16. The method according to claim 14 or 15, wherein: The method further includes flattening the template prior to applying the template-based tool.

17. The method according to any one of claims 14 to 16, wherein: Obtaining information for identifying which pixels within the template of the current block are unavailable includes obtaining indices of all unavailable pixels.

18. The method according to any one of claims 14 to 16, wherein: Obtaining information for identifying which pixels within the template of the current block are unavailable includes obtaining indices of all available pixels.

19. The method according to any one of claims 14 to 16, wherein: Obtaining information for identifying which pixels in the template of the current block are unavailable includes obtaining flags, each flag indicating, for a pixel in the template, whether the pixel is available.

20. The method according to any one of claims 14 to 16, wherein: Obtaining information for identifying which pixels in the template of the current block are unavailable includes: for a group of adjacent unavailable pixels, obtaining an index of a first unavailable pixel and an index of a last unavailable pixel in the group.

21. The method according to any one of claims 14 to 16, wherein: Obtaining information for identifying which pixels in the template of the current block are unavailable includes: for a group of adjacent available pixels, obtaining an index of a first available pixel and an index of a last unavailable pixel in the group.

22. A method according to any one of claims 14 to 21, further comprising spatially reorganizing pixels within the template so as to increase the number of available pixels that are contiguous in memory.

23. The method according to any one of claims 14 to 22, wherein: The unavailable pixel is one of a pixel that has not been reconstructed, a pixel that belongs to a tile different from the tile to which the current block belongs, or a pixel outside a picture boundary.

24. The method according to any one of claims 14 to 23, wherein: The template-based tool belongs to a group of template-based tools, which include: - Intra mode derivation at decoder side; -Convolutional cross-component model; - Adaptive re-ranking of merge candidates using template matching; - Intra prediction based on template matching prediction matrix; and - Template-based neural network prediction without translation equivariance.

25. The method according to any one of claims 14 to 24, wherein: The template includes all pixels located around the current block.

26. The method according to any one of claims 14 to 24, wherein: The template includes a pixel line located above the current block and a pixel column located on the right and left of the current block.

27. A decoding device comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to perform the method of any one of claims 1-13.

28. An encoding device comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to perform the method of any one of claims 14-26.

29. A computer program comprising program code instructions for implementing the method according to any one of claims 1 to 26 when executed by a processor.

30. A computer readable storage medium having stored thereon instructions for implementing the method of any one of claims 1-26.

Images