Deriving a coding mode from one or more template-based costs.

Abstract

A method and apparatus for encoding or decoding video are provided, wherein a block is encoded using a coding mode that uses one or more template-based costs. In one embodiment, the determination of the template-based costs is adapted according to the coding mode. In one embodiment, one or more template-based costs are determined using a first cost obtained with respect to a first component of the block, e.g., luminance, and at least one additional cost determined with respect to the block. In a variant, the additional cost is a template-based cost determined for another component of the block, e.g., a chrominance component. In another variant, the additional cost is the cost for signaling one or more syntax elements of the coding mode. In another embodiment, one or more template-based costs are determined using a template shape adapted according to a given criterion. Once the parameters of the coding mode are determined, predictions about the block are obtained based on the coding mode, and the block is encoded or decoded based on the predictions.

Description

Technical Field

[0001] This application claims priority to European Patent Application Publication No. 23306631.5 filed on September 28, 2023, which is hereby incorporated by reference in its entirety into the present specification.

[0002] Technical Field The present embodiments relate generally to video compression. The present embodiments relate to methods and apparatuses for encoding or decoding images or video. More specifically, the present embodiments relate to improving coding modes in video compression systems that use template-based costs.

Background Art

[0003] Background To achieve high compression efficiency, image and video coding schemes usually employ prediction and transform to leverage spatial and temporal redundancy in the video content. Generally, intra or inter prediction is used to exploit intra or inter picture correlation, then the differences between the original block and the predicted block, often referred to as prediction error or prediction residual, are transformed, quantized, and entropy coded. In inter prediction, motion vectors used in motion compensation are often predicted from motion vector predictors. To reconstruct the video, the compressed data is decoded by inverse processes corresponding to entropy coding, quantization, transform and prediction.

Summary of the Invention

[0004] Summary According to one embodiment, a method for encoding or decoding a block of video is provided. The method includes: obtaining a prediction about a block based on a coding mode using at least one template-based cost determined using at least one first cost obtained with respect to a first component of the block; and encoding or decoding the block based on the prediction, wherein at least one template-based cost is determined using at least one additional cost determined for the block.

[0005] In another embodiment, an apparatus for encoding or decoding blocks of video is provided. The apparatus comprises one or more processors, which are operable to obtain predictions about a block based on a coding mode using at least one template-based cost determined using at least one first cost obtained with respect to a first component of the block, and to encode or decode the block based on the predictions, wherein at least one template-based cost is determined using at least one additional cost determined for the block.

[0006] In some embodiments, the additional cost is the template-based cost acquired for another component of the block. In other embodiments, the additional cost is the cost of signaling one or more syntax elements associated with the coding mode.

[0007] In another embodiment, a method for encoding or decoding blocks of video is provided. This method includes obtaining a prediction about a block based on a coding mode using at least one template base cost, and encoding or decoding the block based on the prediction, wherein at least one template base cost is determined using a template shape determined based on at least one parameter of the coding mode.

[0008] In another embodiment, an apparatus for encoding or decoding blocks of video is provided. The apparatus comprises one or more processors, one or more of which are operable to obtain predictions about blocks based on a coding mode using at least one template base cost, and to encode or decode blocks based on the predictions, the at least one template base cost being determined using a template shape derived from at least one parameter of the coding mode.

[0009] Further embodiments that can be used individually or in combination are described herein.

[0010] One or more embodiments also provide a computer program that, when executed by one or more processors, causes one or more processors to perform one of the methods of encoding or decoding video according to any of the embodiments described herein. One or more embodiments also provide a non-temporary computer-readable medium and / or computer-readable storage medium storing instructions for encoding or decoding video according to the methods described herein.

[0011] One or more embodiments also provide a computer-readable storage medium storing a bitstream generated according to the method described herein. One or more embodiments also provide a method and apparatus for transmitting or receiving a bitstream generated according to the method described above. [Brief explanation of the drawing]

[0012] Brief explanation of the drawing [Figure 1] A block diagram of a system in which an embodiment of this model may be implemented is shown. [Figure 2] A block diagram of one embodiment of a video encoder in which an aspect of this embodiment may be implemented is shown. [Figure 3] A block diagram of one embodiment of a video decoder in which an aspect of this embodiment may be implemented is shown. [Figure 4] Here is an example of a template used to determine the template cost for sorting merge candidates in a list. [Figure 5] This section shows an example of a template for a block and a sample template reference, along with the movement of a subblock that uses the movement information of the current block's subblocks. [Figure 6] An example of a geometric partition block template is shown. [Figure 7] An example of a prediction direction for intra-prediction is shown. [Figure 8] An example flowchart for encoding or decoding at least one block of video according to one embodiment is shown. [Figure 9] An example flowchart for encoding or decoding at least one block of video according to another embodiment is shown. [Figure 10] An example flowchart for encoding or decoding at least one block of video according to another embodiment is shown. [Figure 11] An example flowchart for encoding or decoding at least one block of video according to another embodiment is shown. [Figure 12] A block diagram of a system in which an aspect of this embodiment may be implemented is shown according to another embodiment. [Figure 13] This example illustrates two remote devices communicating via a communication network, based on this principle. [Figure 14] The syntax of a signal based on this principle is shown as an example. [Figure 15] An example of a spatially adjacent block used to derive spatial merge candidates is shown. [Modes for carrying out the invention]

[0013] Detailed explanation This application describes various aspects including tools, features, embodiments, models, techniques and the like. Many of these aspects are specifically described, and in many cases, are described in a manner that may seem limiting, at least to illustrate individual characteristics. However, this is for the purpose of clarifying the description and is not intended to limit the application or scope of these aspects. In fact, all of the different aspects can be combined and interchanged to provide further aspects. Furthermore, these aspects can be combined and interchanged with the aspects described in prior applications.

[0014] The aspects described and contemplated in the present disclosure may be implemented in many different forms. The following Figure 1, Figure 2 and Figure 3 provide several embodiments, but other embodiments are contemplated, and the discussion of Figure 1, Figure 2 and Figure 3 does not limit the breadth of the implementations. At least one of the plurality of aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to the transmission of a generated or encoded bitstream. These and other aspects may be implemented as a method, an apparatus, a computer-readable storage medium storing instructions for encoding or decoding video data according to any of the methods described above, and / or a computer-readable storage medium storing a bitstream generated according to any of the methods described.

[0015] In this application, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image", "picture" and "frame" may be used interchangeably.

[0016] Various methods are described herein, each of which includes one or more steps or actions to implement the described method. Unless a particular order of steps or actions is required for the proper operation of the method, the order and / or use of any particular steps and / or actions may be modified or combined. In addition, terms such as “first,” “second,” etc., may be used in various embodiments to modify elements, components, steps, operations, etc., for example, “first decoding” and “second decoding.” The use of such terms does not imply a modified ordering of operations unless specifically required. Thus, in this embodiment, the first decoding does not necessarily have to be performed before the second decoding, and may be performed, for example, before, during, or during an overlapping period of the second decoding.

[0017] These embodiments are not limited to VVC or HEVC and may apply, for example, to other standards and recommendations, whether existing or to be developed in the future, and to extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically excluded, the embodiments described herein may be used individually or in combination.

[0018] Figure 1 shows a block diagram of an example of a system in which various embodiments and forms can be implemented. System 100 may be realized as a device including various components described later and configured to implement one or more of the embodiments described in this application. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected consumer electronics, and servers. The elements of System 100 may be embodied individually or in combination in a single integrated circuit, a plurality of ICs, and / or individual components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of System 100 are distributed across a plurality of ICs and / or individual components. In various embodiments, System 100 is communicably coupled to other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, System 100 is configured to implement one or more of the embodiments described in this application.

[0019] System 100 includes, for example, at least one processor 110 configured to execute loaded instructions for implementing various embodiments described in this application. The processor 110 may include embedded memory, input / output interfaces, and various other circuits known in the art. System 100 includes at least one memory 120 (e.g., volatile memory devices and / or non-volatile memory devices). System 100 includes a storage device 140 which may include, but is not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drives, and / or optical disk drives, as well as non-volatile memory and / or volatile memory. The storage device 140 may, in non-limiting examples, include internal storage devices, external storage devices, and / or network-accessible storage devices.

[0020] System 100 includes, for example, an encoder / decoder module 130 configured to process data and provide encoded or decoded video, the encoder / decoder module 130 of which may include its own processor and memory. The encoder / decoder module 130 represents a module that may be included in a device to perform encoding and / or decoding functions. As is known, a single device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 130 may be implemented as a separate element of System 100 or may be incorporated into the processor 110 as a combination of hardware and software, as is known to those skilled in the art.

[0021] Program code loaded into the processor 110 or encoder / decoder 130 to implement the various embodiments described in this application may be stored in the storage device 140 and subsequently loaded into the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, memory 120, storage device 140, and encoder / decoder module 130 may store one or more of various items during the implementation of the processes described in this application. Items such as input video, decoded video or a portion of decoded video, bitstreams, matrices, variables, and intermediate or final results obtained from the processing of equations, formulas, operations and operational logic may be, but are not limited to, input video, decoded video or a portion of decoded video, bitstreams, matrices, variables, and equations, formulas, operations and operational logic.

[0022] In some embodiments, internal memory of the processor 110 and / or encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory outside the processing device (for example, the processing device may be either the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be memory 120 and / or storage device 140, such as dynamic volatile memory and / or non-volatile flash memory. In some embodiments, external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, high-speed external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations such as MPEG-2, HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Variable-Purpose Video Coding, also known as H.266, a standard developed by JVET, The Joint Video Experts Team).

[0023] Inputs to the elements of system 100 may be provided through various input devices shown in block 105. Such input devices may include, but are not limited to, (i) a radio frequency (RF) portion that receives RF signals transmitted through the air, for example by a broadcasting station, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Other embodiments not shown in Figure 1 include composite video.

[0024] In various embodiments, the input device of block 105 has associated input processing elements, as known in the Art. For example, the RF section may be associated with suitable elements for (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band to select a signal frequency band that may be called a channel in a particular embodiment, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various functions, such as down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or baseband. In one embodiment of the set-top box, the RF unit and its associated input processing elements receive an RF signal transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, downconverting, and refiltering to a desired frequency band. Various embodiments may rearrange the order of the elements described above (and others), remove some of these elements, and / or add other elements that perform similar or different functions. Addition of elements may include inserting elements between existing elements, such as amplifiers and analog-to-digital converters. In various embodiments, the RF unit includes an antenna.

[0025] In addition, the USB and / or HDMI terminals may include their respective interface processors for connecting the system 100 to other electronic devices over USB and / or HDMI connections. It should be understood that various forms of input processing, such as Reed-Solomon error correction, may be implemented as needed, for example, within a separate input processing IC or within the processor 110. Similarly, forms of USB or HDMI interface processing may be implemented as needed, within a separate interface IC or within the processor 110. Demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 110 operating in conjunction with memory and storage elements, and an encoder / decoder 130, to process the data stream as needed for presentation to an output device.

[0026] Various elements of system 100 may be housed within an integrated housing. Within the integrated housing, the various elements can be interconnected and data can be transmitted between them using an internal bus well known in the art, such as an I2C bus, wiring, and a printed circuit board, which may be suitable connection configuration 115.

[0027] System 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 190. The communication interface 150 may also include, but is not limited to, a modem or a network card, and the communication channel 190 may be implemented, for example, in a wired and / or wireless medium.

[0028] In various embodiments, data is streamed to system 100 using a Wi-Fi network, such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). In these embodiments, the Wi-Fi signal is received via a communication channel 190 and a communication interface 150 adapted for Wi-Fi communication. In these embodiments, the communication channel 190 is typically connected to an access point or router that provides access to an external network, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, the streamed data is provided to system 100 using a set-top box that distributes data via an HDMI connection in input block 105. Still other embodiments provide the streamed data to system 100 using an RF connection in input block 105. As described above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0029] System 100 may provide output signals to various output devices, including a display 165, a speaker 175, and other peripheral devices 185. In various embodiments, the display 165 may include, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 165 may be for a television, tablet, laptop, mobile phone, or other device. The display 165 may also be integrated with other components (e.g., a smartphone) or separate (e.g., an external monitor for a laptop). In various examples of embodiments, the other peripheral devices 185 may include one or more of a standalone digital video disc (or digital multi-purpose disc) (both terms DVR), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 185 that provide functions based on the output of System 100. For example, a disc player performs the function of playing back the output of System 100.

[0030] In various embodiments, control signals are communicated between the system 100 and the display 165, speaker 175, or other peripheral devices 185 using signaling such as AV.Link, CEC, or other communication protocols that enable control between devices with or without user intervention. Output devices may be communicably coupled to the system 100 via dedicated connections through their respective interfaces 160, 170, and 180. Alternatively, output devices may be connected to the system 100 via the communication interface 150 using a communication channel 190. The display 165 and speaker 175 may be integrated into a single unit together with other components of the system 100 within the electronic device, such as a television. In various embodiments, the display interface 160 includes a display driver, such as a timing controller (TCon) chip.

[0031] The display 165 and speaker 175 may, alternatively, be separate from one or more of the other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments where the display 165 and speaker 175 are external components, the output signal may be provided via a dedicated output connection, for example, including an HDMI port, a USB port, or a COMP output.

[0032] These embodiments can be implemented by computer software implemented on the processor 110, by hardware, or by a combination of hardware and software. In non-limiting examples, these embodiments can be implemented by one or more integrated circuits. The memory 120 can be of any type appropriate to the technical environment and, in non-limiting examples, can be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 110 can be of any type appropriate to the technical environment and, in non-limiting examples, may include one or more microprocessors, general-purpose computers, dedicated computers, and processors based on multi-core architectures.

[0033] Figure 2 shows an example of a block-based hybrid video encoder 200. While variations of this encoder 200 are intended, for clarity, the encoder 200 is described below without describing all anticipated variations.

[0034] In some embodiments, Figure 2 also shows encoders that improve upon the HEVC or VVC standard (Versatile Video Coding, standard ITU-T H.266, ISO / IEC 23090-3, 2020), or encoders that use HEVC or VVC-like technologies, such as the ECM encoder currently under development by JVET (Joint Video Exploration Team).

[0035] Before encoding, the video sequence may undergo pre-encoding processing (201), such as applying a color conversion to the input color picture (e.g., from RGB4:4:4 to YCbCr4:2:0), or performing a remapping of the input picture components to obtain a more compression-resistant signal distribution (e.g., using histogram equalization of color components), or resizing (e.g., downscaling) the picture. Metadata may be associated and appended to the bitstream by the pre-processing.

[0036] In encoder 200, a picture is encoded by encoder elements as described below. The encoded picture is partitioned (202) and processed, for example, in CUs (coding units) or units of blocks. Different expressions may be used in this disclosure to refer to such units or blocks resulting from the partitioning of a picture. Such expressions may be coding units or CUs, coding blocks or CBs, luminance CBs, or blocks. A CTU (coding tree unit) refers to a group of blocks, a group of units, or a group of coding units (CU). In some embodiments, a CTU may be considered a block or unit in itself. An example of partitioning using CTUs and CUs is shown in Figure 15.

[0037] For example, in VVC, as with HEVC, pictures are partitioned into multiple non-overlapping CTUs. In VVC, the CTU size can be set to a maximum of 128x128 or 256x256 lumens, while in HEVC, it can be set to a maximum of 64x64. In HEVC, recursive quad-tree (QT) partitioning can be applied to each CTU, resulting in one or more CUs, all of which have a square shape. In VVC, rectangular CUs are supported along with square CUs. VVC also employs binary tree (BT) and terminally tree (TT) partitioning. Figure 16 shows some examples of CTU partitioning. Further partitioning of the acquired CUs is also possible.

[0038] Each unit is encoded using either intra-mode or inter-mode, for example. If a unit is encoded in intra-mode, it performs intra-prediction (260). In inter-mode, motion estimation (275) and compensation (270) are performed. The encoder decides whether to use intra-mode or inter-mode to encode a unit (205), and indicates the intra / inter decision by, for example, a prediction mode flag. The encoder may also blend the intra-prediction results and the inter-prediction results (263), or blend the results from different intra / inter-prediction methods. For example, the prediction residual is calculated by subtracting the predicted blocks from the original image blocks (210).

[0039] The motion refinement module (272) uses an already available reference picture to refine the motion field of a block without referencing the original block. The motion field of a region can be considered as the collection of motion vectors for all pixels in that region. If the motion vectors are subblock-based, the motion field can also be represented as the collection of motion vectors for all subblocks within that region (all pixels within a subblock have the same motion vector, although the motion vectors may differ for each subblock). If a single motion vector is used for the region, the motion field of that region can also be represented by a single motion vector (the same motion vector for all pixels within the region).

[0040] The predicted residual is then transformed (225) and quantized (230). The quantized transformation coefficients, as well as the motion vector and other syntax elements, are entropy coded (245) to output a bitstream. The encoder can skip the transformation and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transformation and quantization, i.e., the residual is coded directly without applying either the transformation or quantization process.

[0041] The encoder decodes (reconstructs) the encoded blocks to provide a reference for further prediction. The quantized transformation coefficients are inversely quantized (240) and inversely transformed (250) to decode the prediction residuals. The image blocks are reconstructed by combining the decoded prediction residuals and the predicted blocks (255). For example, an in-loop filter (265) is applied to the reconstructed picture to perform deblocking / SAO (sample adaptive offset) filtering to reduce encoding artifacts. The filtered image is stored in a reference picture buffer (280).

[0042] Figure 3 shows a block diagram of the video decoder 300. In the decoder 300, the bitstream is decoded by the decoder elements, as will be described later. The video decoder 300 generally performs a decoding path that is the reverse of the encoding path described in Figure 2. The encoder 200 also generally performs video decoding as part of the encoding of the video data.

[0043] In particular, the decoder input includes a video bitstream that may be generated by the video encoder 200. The bitstream is first entropy-decoded to obtain transformation coefficients, motion vectors, and other coded information (330). Picture partitioning information indicates how the picture is partitioned. Thus, the decoder may partition the picture according to the decoded picture partitioning information (335). The transformation coefficients are inversely quantized (340) and inversely transformed (350) to decode the predicted residuals. The decoded predicted residuals and predicted blocks are combined (355) to reconstruct the image blocks.

[0044] The predicted blocks can be obtained from intra-prediction (360) or motion-compensated prediction (i.e., inter-prediction) (375) (370). The decoder may blend the intra-prediction results and the inter-prediction results (373), or blend the results from multiple intra / inter-prediction methods. The motion field may be refined by using an already available reference picture before motion compensation (372). An in-loop filter (365) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (380).

[0045] The decoded picture may undergo further post-decoded processing (385), such as reverse color conversion (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4), reverse remapping (the reverse of the remapping process performed in pre-encoded processing (201)), or resizing of the reconstructed picture (e.g., upscaling). The post-decoded processing may use metadata derived in pre-encoded processing and signaled in the bitstream.

[0046] Some of the embodiments described herein relate to template-based coding modes and improvements to the determination of template-based costs used in these coding modes.

[0047] Any one of the embodiments described herein can be implemented, for example, in an intra-prediction module or inter-prediction module of a video encoder or video decoder. For example, the embodiments described herein can be implemented in an intra-prediction module 260, or motion estimation module 275, or motion refinement module 272, or motion compensation 270 of a video encoder 200, or in an intra-prediction module 360, or motion refinement module 372, or motion compensation 375 of a video decoder 300.

[0048] In the ECM algorithm description (Muhammed Coban, Ru-Ling Liao, Karam Naser, Jacob Stroem, Li Zhang, "Algorithm description of Enhanced Compression Model 9 (ECM 9)," 30th Meeting, Antalya, TR, 21-28 April 2023), several tools or coding modes use template-based costs to determine or derive one or more parameters of the coding mode. Some of these are described below.

[0049] Fusion for template-based intra-mode derivation (TIMD) In ECM, for each intra-prediction mode in the MPM (Most Probable Mode List), and for wide-angle modes if upper-right and / or lower-left reference samples are available, the SATD (Sum of Absolute Transform Differences, e.g., Hadamard transform) is calculated between the predicted sample and the reconstructed sample of the template for the current block to be encoded or decoded.

[0050] Next, two intra-prediction modes that provide the minimum SATD are selected as TIMD modes. These two TIMD modes are fused with weights after applying the PDPC process (position-dependent intra-prediction fusion), and such weighted intra-predictions are now used to encode CUs / blocks. Position-dependent intra-prediction fusion (PDPC) is included in the derivation of the TIMD modes.

[0051] The costs of the two selected modes (costMode1, costMode2) are compared to a threshold. In the comparison, a cost factor of 2 is applied as follows: costMode2 < 2 * costMode1.

[0052] If this condition is true, a fusion of the two selected modes is applied; otherwise, only mode1 (which provides CostMode1) is used.

[0053] When fusing two selected modes, the mode weights (weight1, weight2) are calculated from the SATD cost as follows: weight1=costMode2 / (costMode1+costMode2) weight2 = 1 - weight1

[0054] Adaptive reordering of merge candidates with template matching (ARMC-TM) In ECM, in merge mode, the motion information of previously coded blocks considered is read in a given order for the candidate motion list.

[0055] The merge list candidates, also called merge candidates, are adaptively sorted using template matching (TM). This sorting method applies to the following merge modes: normal merge mode, TM merge mode, and affine merge mode (except for SbTMVP candidates (subblock time-motion vector predictors)). In TM merge mode, the merge candidates are sorted before the refinement process is applied to the selected merge candidates.

[0056] An initial merge candidate list is first constructed according to a given checking order of candidates. For example, the following candidates may be considered: spatial candidates, time-motion vector predictors (TMVP), non-adjacent candidates, hysteresis-motion vector predictors (HMVP), pairwise candidates, and virtual merge candidates. The candidates in the initial list are then divided into several subgroups.

[0057] In template matching (TM) merge mode and adaptive DMVR (decoder-side motion vector refinement) mode, each merge candidate in the initial list is first refined by using TM or multipath DMVR.

[0058] The merge candidates within each subgroup are sorted to generate a sorted list of merge candidates. This sorting is based on the template-based cost determined for each candidate. The index of the selected merge candidate in the sorted list is signaled to the decoder.

[0059] For simplification, merge candidates within the last subgroup, rather than the first, are not sorted. All zero candidates from the ARMC sorting process are excluded during the construction of the merge motion vector candidate list. In normal merge mode and TM merge mode, the subgroup size is set to 5. In affine merge mode, the subgroup size is set to 3.

[0060] Template-based cost calculation: The template matching cost of merge candidates during the sorting process is measured by the SAD (Sum of Absolute Differences) between the current block's template samples and their corresponding reference samples. A template contains a set of reconstructed samples neighboring the current block. The template's reference (predicted) samples are positioned by the merge candidate's motion information (motion compensation). If the merge candidate utilizes bidirectional prediction, the merge candidate's template reference samples are also generated by bidirectional prediction, as shown in Figure 4.

[0061] Refining the initial merge candidate list: When a multi-pass DMVR is used to derive refined motion to the initial merge candidate list, only the first pass of the multi-pass DMVR (i.e., the PU level) is applied in the sorting. When template matching is used to derive refined motion, the template size is set to equal 1. If each block is flat with a block width greater than twice its height, or narrow with a height greater than twice its width, only the top or left template is used during TM motion refinement. TM is extended to implement 1 / 16 PEL MVD precision. In TM merge mode, the first four merge candidates are sorted by refined motion.

[0062] For subblock-based merge candidates with subblock sizes equal to Wsub × Hsub, the top template contains several subtemplates with a size of Wsub × 1, and the left template contains several subtemplates with a size of 1 × Hsub. As shown in Figure 5, the subblock movement information in the first row and first column of the current block is used to derive a reference sample for each subtemplate.

[0063] Sorting criteria: In the sorting process, a candidate is considered redundant if the cost difference between a candidate and its predecessor is less than the lambda value, for example, if |D1-D2|<λ, where D1 and D2 are the costs obtained during the first ARMC sort, and λ is the Lagrangian parameter used in the RD criterion on the encoder side.

[0064] The algorithm is defined as follows: The minimum cost difference is determined between a given candidate and its predecessors among all candidates in the list. If the minimum cost difference is greater than or equal to λ, the list is considered sufficiently diverse, and the sorting stops. Otherwise (if this minimum cost difference is less than λ), the candidate is considered redundant, and it is moved to a further position in the list. This further position is the first position in which the candidate is sufficiently diverse compared to its predecessors. The algorithm stops after a finite number of iterations (if the minimum cost difference is not less than λ).

[0065] The sorting algorithms described above are applicable to standard, TM, BM (bidirectional matching), and affine merge modes. Similar algorithms are also applicable to Merge MMVD (Merge with MVD) and sine MVD (Motion Vector Difference) prediction methods, which similarly use ARMC for sorting.

[0066] The value of λ is set to be equal to the rate-distortion criterion λ used by the encoder to select the best merge candidate for low-latency configurations, and equal to the value λ corresponding to another QP for random-access configurations. A set of λ values ​​corresponding to each signaled QP offset is provided in the SPS, or in the slice header for QP offsets that are not present in the SPS.

[0067] AMVP mode expansion: The ARMC design is also applicable to AMVP (Adaptive Motion Vector Prediction) mode, in which case AMVP candidates are sorted according to their TM cost. For template matching in Advanced Motion Vector Prediction (TM-AMVP) mode, an initial AMVP candidate list is constructed, and then refined using TM to construct a refined AMVP candidate list. In addition, MVP candidates with a TM cost greater than a threshold equal to five times the cost of the first MVP candidate are skipped.

[0068] Geometric partitioning mode (GPM) using template matching (GPM-TM) In ECM, template matching is applied to GPM (Geometry Partition Mode). When GPM mode is enabled for a CU / block, a CU-level flag (GPM-TM flag) is signaled to indicate whether TM is applied to both geometric partitions. Motion information for each geometric partition is refined using TM. If TM is selected, the template is constructed using adjacent samples to the left, top, or left and top, depending on the partition angle, as shown in Table 1 below. The motion is then refined by minimizing the difference between the current template and the template in the reference picture using the same search pattern in merge mode with the half-per interpolation filter disabled.

[0069] [Table 1]

[0070] In addition to GPM-TM, GPM-MMVD allows signaling MVD information to modify selected motion vector candidates. GPM-MMVD and GPM-TM are exclusively enabled for one GPM CU. This is done by first signaling the GPM-MMVD syntax. If both GPM-MMVD control flags are equal to false (i.e., GPM-MMVD is disabled for two GPM partitions), the GPM-TM flag is signaled to indicate whether template matching applies to the two GPM partitions. Otherwise (if at least one GPM-MMVD flag is equal to true), the value of the GPM-TM flag is presumed to be false.

[0071] Template matching base replacement for GPM splitting mode (GPM-SPLIT-TM) ECM can also use template matching to sort different geometric partitionings of blocks corresponding to different edge angles. These partitionings are also called GPM partitioning modes. GPM modes as 64 partitioning modes.

[0072] In template matching-based sorting for GPM partitioning modes, given the movement information of the current GPM block, the TM cost value for each GPM partitioning mode is calculated. All GPM partitioning modes are then sorted in ascending order based on their TM cost value (SAD). Instead of sending GPM partitioning modes, an index is signaled using a Golomb-Rice code indicating where a specific GPM partitioning mode exists within the sorted list.

[0073] The sorting method for GPM partitioning mode is a two-step process performed after the reference templates for each of the two GPM partitions within the coded unit are generated as follows: The GPM partition edge is extended to the reference templates of the two GPM partitions, resulting in the generation of 64 reference templates, for which the respective TM cost is calculated.

[0074] Next, the GPM partitioning modes are sorted in ascending order based on their TM cost values, and the 32 best partitioning modes are marked as available partitioning modes.

[0075] Edges on the template are extended from the edges of the current CU, as shown in Figure 6, but the GPM blending process is not used in the template area that crosses the edges. After sorting in ascending order using TM cost, the index of the selected partitioning mode is signaled.

[0076] Template matching-based BCW index derivation for merge mode (BCW-TM) In BCW for merge mode, weighted bidirectional prediction is used to predict blocks, given, for example, B = (W0 x P0 + W1 x P1) / 8, where B is the predicted block, P0 and P1 are the reference blocks, and W0 and W1 are the respective weights applied to the reference blocks. Movement information inherited from the candidates in the merge list leads to the retrieval of the reference blocks. In BCW, in merge mode, the weights W0 and W1 are selected from a given table using a BCW index. An example of such a table may be as follows:

[0077] [Table 2]

[0078] In BCW-TM mode, the BCW (Bidirectional Prediction with CU-Based Weighting) index for merged code CUs is derived based on template matching costs, rather than from the BCW index of adjacent blocks.

[0079] Given a selected merge candidate, the TM cost value is calculated using different bidirectional prediction weights, and then the merge CU is predicted using the bidirectional prediction weight with the minimum TM cost value.

[0080] When calculating the TM cost for bidirectionally predicted weights, the following rules apply: Since inherited bidirectionally predicted weights are likely to have higher accuracy than other weights, only inherited bidirectionally predicted weights and their two adjacent weights (i.e., ±1) are considered. For example, if the inherited bidirectionally predicted weight is 4, only the three weights {3, 4, 5} are involved in the TM cost calculation. The TM cost for inherited BCW indices is multiplied by 0.90625, meaning the cost is reduced by 3 / 32. Since bidirectionally predicted samples are beneficial to BDOF, and BDOF is applied only to CUs with equal weights, the TM cost for equal weights is multiplied by 0.90625.

[0081] BCW index derivation based on template matching is typically applied to merges, template matching, adaptive decoder-side motion vector refinement, and CUs coded in MMVD mode.

[0082] In addition, the bidirectional prediction weights in merge mode are expanded from {-2,3,4,5,10} to {1,2,3,4,5,6,7}. Furthermore, the negative bidirectional predicted weights {-2,10} in non-merge mode are replaced with positive weights {1,7}.

[0083] In the example above, it appears that the determination of template costs could be improved. It can be seen that the template cost, or template matching cost, is calculated using only a single component (e.g., the rumor component). Furthermore, while the template matching cost does not consider mode signaling costs, in some coded modes, signaling can affect the rate / distortion cost of the selected candidate. In addition, it can be seen that in the example above, the TM shape is fixed for all mode candidates, and the mode candidates do not consider other mode parameters. Therefore, the template-based coded modes need to be improved.

[0084] In this specification, the terms template-based cost and template-matching cost may be used interchangeably.

[0085] Embodiments for improving the template-based coding mode are provided herein, namely, a coding mode that uses a template-based cost, for any of the following purposes: determining coding parameters such as motion vectors or intra-prediction modes, refining derived coding parameters, or sorting candidate predictors for a block.

[0086] Some embodiments described herein may include at least one of the following: determining a template matching cost using at least two components (e.g., a lumen component and a chroma component); determining a template matching cost including a mode signaling cost (e.g., a candidate index); or determining a template matching cost such that the shape of the template is fitted according to a given criterion. For example, such a criterion may be at least one of a mode candidate or a coding parameter of a coded mode. The TM shape can be implicitly derived as a function of several mode parameters (e.g., a spatial candidate position relative to the current block). The TM cost can be appropriately renormalized when compared with template costs having different shape sizes.

[0087] According to a general embodiment, a method is provided for encoding or decoding blocks of video, where the blocks are encoded / decoded using an encoding mode that uses one or more template-based costs. Such a general embodiment is shown, for example, in Figure 8 illustrating a method 800 for encoding / decoding blocks of video.

[0088] In 810, for a given coding mode that uses template-based costs, coding mode parameters are determined based on one or more template-based costs. The determination of template-based costs is adapted according to the coding mode.

[0089] In one embodiment, one or more template base costs are determined using a first cost obtained with respect to a first component of the block, e.g., luminance, and at least one additional cost determined with respect to the block. In a variant, the additional cost is a template base cost determined with respect to another component of the block, e.g., chrominance component. In another variant, the additional cost is the cost of signaling one or more syntax elements of the coding mode.

[0090] In another embodiment, one or more template base costs are determined using template shapes adapted according to a given criterion, such as a coding mode or a candidate used in a coding mode.

[0091] In 820, once the encoding mode parameters are determined, a prediction about the block is obtained based on the encoding mode, and in 830, the block is encoded or decoded based on the prediction.

[0092] The embodiments described above in step 810, along with further variations of these embodiments, are described below. These embodiments / variations can be used individually or in combination to further enhance template-based cost determination or to reduce the complexity of template-based cost determination. The embodiments / variations described above and below are described in consideration of a given coding mode that utilizes template-based cost to determine the parameters of the coding mode. These parameters can be any of the following: candidate motion, selected candidate in the candidate list, intra-mode prediction, geometric partitioning mode, weights of weighted prediction, etc.

[0093] For example, the embodiments / modifications described above and below can be applied to any coding mode of the ECM algorithm described above, where applicable. For example, the coding mode is one of the following: an intra mode that uses template-based costs to derive an intra prediction mode; a merge mode that uses template-based costs to sort a list of candidate movements; an adaptive motion vector prediction mode that uses template-based costs to sort a list of candidate movements; a geometric partitioning mode that uses template-based costs to determine the movements of geometric partitions; a geometric partitioning mode that uses template-based costs to sort a list of candidate partition modes for blocks; or a merge mode that uses template-based costs to determine the weights for block-based weighted bidirectional predictions.

[0094] The embodiments / modifications can also be applied to any coding mode that inevitably involves the determination of template-based costs.

[0095] In ECM, several coding tools (e.g., TIMD, ARMC-TM, GPM-TM, GPM-SPLIT-TM, BCW-TM) estimate the cost of a set of candidate modes using template-based costs. The cost is measured as the difference between the reconstructed samples in the template and the predictions obtained for the template using the candidate modes under consideration. The difference scaling function can be any quadratic error function, such as SAD (Sum of Absolute Differences), SATD (Hadamar), MR-SAD (Mean-Removed SAD), MR-SATD (Mean-Removed SATD), or MSE (Sum of Squared Errors). Generally, the difference is measured using a single component, i.e., a luminance sample.

[0096] In the embodiments described herein, the difference is measured in several components, i.e., at least two components, where nc > 1, and nc is the number of components used in the template-based cost determination. The two components may be a luminance component and a chrominance component. Then the final measurement is the sum of the nc measurements.

[0097] In one variant, the final measurement may be a weighted sum of nc measurements, each associated with a predefined weight. In further variants, the weights may depend on the chroma format.

[0098] For example, in the case of 4:4:4 RGB, equal weights may be used for each RGB component. In another example, in the case of 4:4:4 YCbCr, unequal weights (e.g., 4,1,1) may be used to emphasize the distortion of the rumor component. In yet another example, in the case of 4:2:0 YCbCr, using equal weights implicitly emphasizes the rumor because the number of samples in the template for the Y component is greater than the number of samples in the template for the Cb or Cr component.

[0099] In the variant form, the use of one or more components to determine the template base cost can be signaled to the decoder.

[0100] Alternatively, in another variant, the use of one or more components to determine the template base cost can be implicitly derived from the coding mode. For example, in the case of an intra-mode where the prediction mode differs between the lumera and chroma, or in the case of a dual tree, a regular difference metric using a single component (nc=1) is used, while in the case of an intra-mode where the prediction mode is identical between the lumera and chroma, a combination of several difference metrics (nc>1) is used.

[0101] In other variants, some components (e.g., nc=3) are used only for the intermode, or for a subset of the intermode (e.g., BCW-TM).

[0102] Figure 9 shows a method 900 for encoding / decoding blocks of video according to this embodiment. In this embodiment, in 910, the encoding mode parameters for a block are determined based on one or more template base costs, the template base costs are determined using at least two components according to any of the variations described above. Once the encoding mode parameters for a block are determined, in 920, a prediction about the block is obtained based on the encoding mode, and in 930, the block is encoded or decoded based on the prediction.

[0103] In another variant, to reduce complexity, the cost is initially calculated for only one component for all candidates considered in the coding mode, and the costs of additional components are added only for the best N candidates (e.g., N=3, or N can have different values ​​for each tool / coding mode). For example, depending on the metric being considered, the best candidate is the tested candidate that yields the lowest template-based cost. This variant refines the template-based costs of these N best candidates, and therefore allows for improving that candidate among other candidates when template-based costs are used for candidate refinement, or when improving the sorting of candidates by template-based cost is for sorting the candidate list.

[0104] In some embodiments, the number of candidates for which two or more components are used in the template-based cost depends on a comparison between the cost of a candidate obtained using only the first component and the cost of the best candidate obtained for that component. For example, if a candidate's cost is less than t*BestCost (e.g., t=1.1), then this candidate's cost is refined using another component. The best candidate's cost is also refined using another component, thereby allowing for a further comparison of these two candidates. This variation makes it possible to refine the template-based costs of candidates that are too close, and thus further enhance the distinction between close candidates.

[0105] As previously mentioned, according to one embodiment, the additional cost may be the cost of signaling one or more syntax elements of the coding mode. In ECM, coding tools that use TMs for candidate sorting (e.g., ARMC-TM, GPM-TM, GPM-SPLI-TM), candidate selection (e.g., BCW-TM, TIMD), and / or deriving weighted fusions (e.g., TIMD) use a difference scale as the cost function.

[0106] In this embodiment, the estimated signaling cost (VLC, index number, or log2(index number)), or the actual coding cost (e.g., using CABAC entropy coding), is added to the calculation of the template base cost. For example, the signaling may include at least one of the following: the index of a candidate in a candidate list ordered according to the template base cost; the motion vector residual from a motion vector predictor determined using the template base cost; a flag indicating the use of the template base cost to determine the motion information of block partitions within a geometric block partition; an intra-prediction mode; or a geometric partition mode index.

[0107] For example, the template-based cost is determined by cost = diff + λ·bits, where diff is the difference scale determined in the template sample, bits is the estimated coded mode signaling cost, and λ can be the Lagrangian (a predetermined value that may be a function of the picture type and / or quantization parameters), or any other equivalent rescaling coefficient related to the difference scale function. In fact, formulas based on the normal Lagrangian use MSE instead of SAD, so the value of λ can be adjusted accordingly.

[0108] Figure 10 shows a method 1000 for encoding / decoding blocks of video according to one embodiment, where additional costs are used to refine the initially determined template base cost. In 1010, template matching costs are determined for all candidates considered in the encoding mode. In 1020, the candidate that provides the lowest template matching cost is determined as the best candidate. In 1030, a loop is performed on the other candidates or a subset of the other candidates to verify whether the template matching costs of the candidates are close to the template matching costs of the best candidate. For example, it is verified whether the difference between the template base cost of a candidate and the template base cost of the best candidate is less than a given value.

[0109] In this case, the template matching cost of the candidate and the template matching cost of the best candidate are refined using an additional cost. This additional cost may be a template base cost determined for another component, or the cost of signaling one or more syntax elements of the coding mode associated with the candidate. Once the template matching cost is refined, the parameters of the coding mode are determined based on the refined template matching cost, and the block is encoded / decoded as in the method shown in Figure 8 or Figure 9.

[0110] In ECM, the TM shape is either fixed (a predetermined set of samples above and / or to the left of the current block for encoding or decoding) or the TM shape is explicitly signaled (e.g., GPM-TM).

[0111] According to one embodiment, the TM shape is not explicitly signaled but is implicitly derived from the coding parameters of the current block.

[0112] In one variant form, the TM shape can be derived from the block shape. For example, if W > k * H, the template is created from the reconstructed sample of the block described above, where {W, H, k} are the current width and height of the block, and k is a predefined value (e.g., k = 4). If H > k * W, the template is created from the left reconstructed sample of the block, using only the left template.

[0113] For example, this variant can be used when the coding mode is intra-mode, or when the coding mode is inter-mode and at least one template base cost is used to sort the candidate list.

[0114] In another variant, the TM shape is defined based on the candidates considered for the template matching cost. That is, the TM shape is different for several candidates. For example, in the case of angular intra prediction, when intraPredAngle is negative, the TM shape is created from reconstructed samples located to the left and above the current block. In another example, when intraPredAngle is zero (or Abs(intraPredAngle) < TMthreshold) and the direction is vertical, the TM shape is created only from the reconstructed samples above the current block, whereas when intraPredAngle is zero (or Abs(intraPredAngle) < TMthreshold) and the direction is horizontal, the TM shape is created only from the reconstructed samples to the left of the current block, where TMthreshold is a given threshold. Figure 7 shows an example of intra prediction directions for encoding / decoding the current block.

[0115] In another variant, the template shape is determined based on the location of the candidate for the block to be encoded / decoded. Figure 15 shows an example of adjacent blocks used to derive spatial candidates.

[0116] For example, when a block is coded in merge mode, the merge candidate is a spatial candidate, and the spatial candidate is located above (or top-right of) the block (e.g., referenced by 2 and 3 in Figure 15), the TM shape is created only from the reconstructed samples above the block ({Above}), while when the merge candidate is a spatial candidate and the spatial candidate is located to the left (or bottom-left of) the block (e.g., referenced by 1 and 4 in Figure 15), the TM shape is created only from the reconstructed samples to the left of the block ({Left}). In other cases, the TM shape is created from the reconstructed samples above the block and the reconstructed samples to the left of the block ({Above+Left}). Similar conditions for defining the template shape can also be applied when the candidate is a non-spatially adjacent block, as shown in Figure 15 (blocks 6-7, 9-12, 14, 17, 19, 22).

[0117] If several candidates use template shapes of different sizes, the difference scale relative to the template may be renormalized. For example, if the reference TM shape (e.g., {Above+Left}) is N R If it includes individual samples, N C The difference (Diff) measured in the TM shape created from individual samples. C The renormalization applied to ) can be done as follows: Diff=( Diff C ×N C ) / N R In the formula, Diff is a renormalized scale.

[0118] Figure 11 shows a method 1100 for encoding / decoding blocks of video according to an embodiment, where the template shape is defined according to the encoding parameters of the block. In 1110, the encoding parameters and template shape are determined for the block based on the encoding mode. The encoding mode uses one or more template base costs. For each template base cost used in the encoding mode, the template shape is determined according to one of the variations described above. In 1120, once the parameters of the encoding mode are determined, a prediction is determined based on the encoding mode, and in 1130, the block is encoded or decoded based on the prediction.

[0119] Figure 12 shows a block diagram of a system in which an aspect of this embodiment may be implemented according to another embodiment. Figure 12 shows one embodiment of a device 1200 for encoding or decoding video according to any one of the embodiments described herein. The device comprises a processor 1210 which can be interconnected to a memory 1220 via at least one port. Both the processor 1210 and the memory 1220 may also have one or more additional interconnections to external connections.

[0120] The processor 1210 is also configured, according to any one of the embodiments described herein, to obtain a prediction about at least one block of video based on an encoding mode using at least one template base cost determined using at least one first cost obtained with respect to a first component of the block, and to encode or decode the block based on the prediction, wherein the at least one template base cost is determined using at least one additional cost determined for the block.

[0121] In another embodiment, the processor 1210 is also capable of obtaining predictions about a block based on a coding mode using at least one template base cost, and operating to encode or decode the block based on the prediction, the at least one template base cost being determined using a template shape derived from at least one parameter of the coding mode.

[0122] For example, the processor 1210 uses a computer program product that includes code instructions implementing any one of the embodiments described herein.

[0123] In one embodiment shown in Figure 13, in relation to transmission between two remote devices A and B over a communication network NET, device A includes a processor associated with RAM and ROM, which are memory configured to implement a method of encoding video as described in relation to Figures 1-12, and device A also includes a processor associated with RAM and ROM, which are memory configured to implement a method of decoding video as described in relation to Figures 1-12. For example, the network is a broadcast network adapted to broadcast / transmit encoded video from device A to decoding devices including device B.

[0124] Figure 14 shows an example of the syntax of a signal transmitted over a packet-based transmission protocol. Each transmitted packet P includes a header H and a payload. In some embodiments, the payload may include video data encoded according to any one of the embodiments described above. The payload may also include any of the signaling described above.

[0125] Various implementations involve decoding. As used in this application, “decoding” can encompass all or part of the processes performed on a received encoded sequence to produce, for example, a final output suitable for display. In various embodiments, such processes typically include one or more processes performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes may further or alternatively include processes performed by a decoder in one of the various implementations described in this application, such as the process of entropy decoding a set of binary symbols for reconstructing image or video data.

[0126] As further examples, in one embodiment, “decoding” refers only to entropy decoding; in another embodiment, “decoding” refers only to differential decoding; in yet another embodiment, “decoding” refers to a combination of entropy decoding and differential decoding; and in yet another embodiment, “decoding” refers to the entire process of reconstructing a picture, including entropy decoding. Whether the phrase “decoding process” is intended to refer specifically to a subset of operations or to a broader decoding process in general should become clear from the context of the specific description and should be easily understood by those skilled in the art.

[0127] Various implementations involve encoding. Similar to what is discussed above regarding "decoding," as used in this application, "encoding" can encompass, for example, all or part of the processes performed on an input video sequence to generate an encoded bitstream. In various embodiments, such processes typically include one or more processes performed by the encoder (e.g., partitioning, differential encoding, transformation, quantization, and entropy encoding). In various embodiments, such processes may further or alternatively include processes performed by the encoder in the various implementations described in this application, such as determining resampling filter coefficients and resampling the decoded picture.

[0128] As further examples, in one embodiment, “encoding” refers only to entropy coding; in another embodiment, “encoding” refers only to differential coding; and in yet another embodiment, “encoding” refers to a combination of differential coding and entropy coding. Whether the phrase “encoding process” is intended to refer specifically to a subset of operations or to a broader encoding process in general should become clear from the context of the specific description and should be easily understood by those skilled in the art.

[0129] Please note that the syntax elements used herein are descriptive terms; therefore, they do not preclude the use of other syntax element names.

[0130] This disclosure has described various types of information that can be transmitted or stored, such as syntax. This information can be packaged or placed in various ways, including methods common in video standards, such as including information in SPS, PPS, NAL units, headers (e.g., NAL unit headers, picture headers, or slice headers), or SEI messages. Other methods are also available, including methods common with system-level or application-level standards, such as transmitting information in one or more of the following: a. SDP (Session Description Protocol). This is a format for describing multimedia communication sessions for purposes such as session announcements and session invitations, as described in RFCs, for example, and is used in conjunction with RTP (Real-time Transport Protocol) transmission. b. DASH MPD (Media Presentation Description) descriptor. This is, for example, one used in DASH and transmitted over HTTP, and the descriptor is associated with a representation or collection of representations to provide additional characteristics to the content representation. c. RTP header extensions. For example, those used during RTP streaming. d. ISO Base Media File Format. For example, the one used in OMAF, which uses boxes (also called "atoms" in some specifications), which are object-oriented building blocks defined by a unique type identifier and length. e. An HLS (HTTP Live Streaming) manifest sent via HTTP. The manifest can, for example, be associated with a version or collection of versions of content and provide characteristics of the version or collection of versions.

[0131] When a drawing is presented as a flowchart, it will be understood that the same drawing also provides a block diagram of the corresponding device. Similarly, when a drawing is presented as a block diagram, it will be understood that a flowchart of the corresponding method / process also provides.

[0132] Several embodiments refer to rate-distortion optimization. In particular, during the coding process, a balance or trade-off between rate and distortion is generally considered, often under computational complexity constraints. Rate-distortion optimization is typically formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are various methods for solving rate-distortion optimization problems. For example, these methods may be based on a comprehensive examination of all coding options, including all considered mode or coding parameter values, along with a complete evaluation of the coding cost and associated distortion of the reconstructed signal after coding and decoding. Faster methods may also be used, in particular, that use approximate distortion calculations based on the predicted or predicted residual signal rather than the reconstructed signal, in order to avoid coding complexity. These two methods can also be used in combination, for example, by using approximate distortion for only some of the possible coding options and full distortion for others. Other methods evaluate only a subset of the possible coding options. More generally, many methods employ one of various techniques to perform optimization, but the optimization does not necessarily involve a complete evaluation of both the coding cost and associated distortion.

[0133] The implementation forms and embodiments described herein can be implemented, for example, in methods or processes, apparatus, software programs, data streams, or signals. Even when discussing only in relation to a single form of implementation (for example, only as a method), the implementation forms of the features discussed can also be implemented in other forms (for example, apparatus or programs). Apparatus can be implemented, for example, in appropriate hardware, software, and firmware. These methods can be implemented within processors, for example, which refer to processing devices in general, including computers, microprocessors, integrated circuits, or programmable logic devices. Processors also include communication devices, such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate the communication of information between end users.

[0134] References to "one embodiment" or "one implementation" or "one implementation," as well as other variations thereof, mean that the specific features, structures, characteristics, etc. described in relation to that embodiment are included in at least one embodiment. Therefore, the appearance of "in one embodiment" or "in one embodiment" or "in one implementation," as well as any other variations, in various places throughout this application, does not necessarily all refer to the same embodiment.

[0135] In addition, this application may refer to "determining" various types of information. Identifying information may include, for example, one or more of the following: estimating information, calculating information, predicting information, or retrieving information from memory.

[0136] Furthermore, this application may refer to "accessing" various types of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0137] In addition, this application may refer to "receiving" various types of information. Receiving is intended to be a broad term, similar to "accessing." Receiving information may include, for example, one or more of the following: accessing information or retrieving information (for example, from memory). Furthermore, "receiving" typically occurs in some way during operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0138] It should be understood that the use of any of the following " / ", "and / or", and "at least one of" is intended to encompass, for example, in the cases of "A / B", "A and / or B", and "at least one of A and B", the selection of only the first enumerated option (A), or only the second enumerated option (B), or the selection of both options (A and B). As a further example, in the cases of "A, B, and / or C", and "at least one of A, B, and C", such a clause is intended to encompass the selection of only the first enumerated option (A), or only the second enumerated option (B), or only the third enumerated option (C), or only the first and second enumerated options (A and B), or only the first and third enumerated options (A and C), or only the second and third enumerated options (B and C), or the selection of all three options (A, B, and C). This can be extended as many times as there are enumerated items, as will be apparent to those skilled in the art.

[0139] Furthermore, as used herein, the term "signal" refers, in particular, to indicating something to a corresponding decoder. In this way, in one embodiment, the same parameter is used on both the encoder and decoder sides. Therefore, for example, the encoder can transmit a specific parameter to the decoder so that the decoder can use the same specific parameter (explicit signaling). Conversely, if the decoder already has a specific parameter along with other parameters, signaling can be used without transmitting that parameter to allow the decoder to recognize and select the specific parameter (implicit signaling). Bit savings are achieved in various embodiments by avoiding the transmission of actual functions at all. It will be understood that signaling can be implemented in various ways. For example, in various embodiments, information is signaled to a corresponding decoder using one or more syntax elements, flags, etc. The above-mentioned "signaling" relates to the verb form of the word "signal," but in this specification, the word "signal" may also be used as the noun "signal."

[0140] As will be apparent to those skilled in the art, implementations can generate various signals formatted to carry information that can be stored or transmitted. This information may include, for example, instructions for carrying out the method or data generated by one of the described implementations. For example, a signal can be formatted to carry the bitstream of the described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (using, for example, the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. The signal may be transmitted over various different wired or wireless links of known type. The signal can be stored in a processor-readable medium.

[0141] Several embodiments have been described above. Features of these embodiments can be provided individually or in any combination across various categories and types of claims.

Claims

1. It is a method, Obtaining a prediction for at least one block of video based on a coding mode that uses at least one template-based cost determined using at least one first cost obtained with respect to the first component of the block, Encoding or decoding the block based on the prediction, Includes, A method in which the at least one template base cost is determined using at least one additional cost determined for the block.

2. A device comprising one or more processors, wherein the one or more processors Based on a coding mode that uses at least one template-based cost determined using at least one first cost obtained with respect to the first component of the block, a prediction is obtained for at least one block of the video. The system is configured to encode or decode the block based on the prediction, An apparatus in which the at least one template base cost is determined using at least one additional cost determined for the block.

3. The method according to claim 1 or the apparatus according to claim 2, wherein the additional cost is a template-based cost obtained for another component of the block.

4. The method or apparatus according to claim 3, wherein the first component is a luminance component and the other component is a chrominance component.

5. The method according to any one of claims 1 or 3 to 4, wherein the at least one template base cost is a weighted sum of the first cost and the at least one additional cost, or the apparatus according to any one of claims 2 to 4.

6. The method of the apparatus according to claim 5, wherein the weights depend on the chroma format of the chrominance components.

7. The method according to any one of claims 1 or 3 to 6, wherein the use of the at least one additional cost for the block depends on the coding mode, or the apparatus according to any one of claims 2 to 6.

8. The method according to any one of claims 1 or 3 to 7, or the apparatus according to any one of claims 2 to 7, wherein determining the at least one template base cost includes determining a first template base cost for a given number of candidates using the first cost, and determining a second template base cost for a subset of the candidates using the first cost and the additional cost.

9. The method or apparatus according to claim 8, wherein the subset of candidates from which the second template base cost is determined using the first cost and the additional cost is a candidate whose difference between the first template base cost and the best candidate first template base is less than a given value.

10. The method according to any one of claims 1 or 3 to 9, or the apparatus according to any one of claims 2 to 9, wherein the additional cost is the cost of signaling one or more syntax elements associated with the coding mode.

11. The one or more syntax elements associated with the coding mode are Candidate index in the candidate list ordered according to the aforementioned at least one template base cost, The motion vector residual from the motion vector predictor determined using the above at least one template base cost, A flag indicating the use of template-based costing to determine the movement information of block partitions within a geometric block partition. The method or apparatus according to claim 10, comprising at least one of the following.

12. A method, wherein for at least one block of video, Obtaining a prediction about the block based on a coding mode, wherein the coding mode uses at least one template-based cost, Encoding or decoding the block based on the prediction, Includes, A method in which the at least one template base cost is determined using a template shape determined based on at least one parameter of the coding mode.

13. For at least one block of video, Based on a coding mode that uses at least one template-based cost, a prediction for the block is obtained, One or more processors capable of encoding or decoding the block based on the prediction, A device equipped with, An apparatus in which the at least one template base cost is determined using a template shape derived from at least one parameter of the coding mode.

14. The method according to claim 12 or the apparatus according to claim 13, wherein the coding mode is an intra mode and the template shape is derived based on an angle intra prediction mode.

15. The method according to claim 12 or the apparatus according to claim 13, wherein the coding mode is a merge mode and the template shape depends on the location of a spatial merge candidate for the block.

16. The method according to claim 12 or 14, or the apparatus according to claim 13 or 14, wherein the coding mode is intra-mode or inter-mode, and the at least one template base cost is used to sort the candidate list, and the template shape is derived from the shape of the block.

17. The method or apparatus according to claim 16, wherein if the width of the block is greater than k times the height of the block, the template is formed by pixels above the block, and if the height of the block is greater than k times the width of the block, the template is formed by pixels to the left of the block.

18. The aforementioned coding mode is - Intra mode that uses the template base cost to derive the intra prediction mode, - A merge mode that uses the template base cost to sort the list of candidate movements. - Adaptive motion vector prediction mode that uses the template base cost to sort the list of candidate motions, - A geometric partitioning mode that uses the template base cost to determine the movement of the geometric partitions. - Geometric partitioning modes that use the template base cost to sort the list of candidate partitioning modes for the block, - A merge mode that uses the template-based cost to determine the weights for block-based weighted bidirectional predictions. One of these is the method according to any one of claims 1, 3 to 12 or 14 to 17, or the apparatus according to any one of claims 2 to 11 or 13 to 17.

19. A computer program product comprising instructions for causing one or more processors to perform the method according to any one of claims 1, 3-9, 11-12, or 14-18.

20. A non-temporary computer-readable medium for storing executable program instructions, wherein the instructions cause a computer executing the instructions to carry out the method according to any one of claims 1, 3 to 9, 11 to 12, or 14 to 18.

21. It is a device, The apparatus according to any one of claims 2 to 11 or 13 to 18, (i) an antenna configured to receive or transmit a signal containing data representing video, (ii) a band limiter configured to limit the signal to a bandwidth of frequencies containing data representing video, or (iii) a display configured to display the video, A device equipped with the following features.

22. The device according to claim 21, comprising at least one of a television, a mobile phone, a tablet, and a set-top box.

Images