Histogram normalization of blocks used for decoder-side intra-frame mode derivation

Abstract

A method and apparatus for encoding or decoding images or videos are provided. For at least one block of an image, at least one first-direction gradient histogram is obtained based on at least one second-direction gradient histogram obtained for neighboring blocks of the at least one block. The at least one second-direction gradient histogram is normalized based on distances determined between the at least one block and its neighboring blocks. A prediction is determined for the at least one block using one or more intra-frame prediction modes, said one or more intra-frame prediction modes being determined using the at least one first-direction gradient histogram, and the determined prediction is used to encode or decode the at least one block.

Description

[0001] This application claims priority to European Application No. 24305025.9, filed on 8 January 2024, which is incorporated herein by reference in its entirety. Technical Field

[0002] This embodiment generally relates to video compression. Specifically, it relates to a method and apparatus for encoding and decoding images or videos. More particularly, this embodiment relates to an improved DIMD (decoder-side intra-frame mode derivation) prediction mode. Background Technology

[0003] To achieve high compression efficiency, image and video coding schemes typically employ prediction and transform to fully utilize the spatial and temporal redundancy in video content. Generally, intra-frame or inter-frame prediction is used to leverage intra- or inter-frame correlations, followed by transform, quantization, and entropy coding of the differences between the original and predicted blocks (often represented as prediction error or prediction residual). In inter-frame prediction, the motion vectors used for motion compensation are typically predicted from a motion vector predictor. To reconstruct the video, the compressed data is decoded through the inverse processes corresponding to entropy coding, quantization, transform, and prediction. Summary of the Invention

[0004] According to one aspect, a method for encoding an image or video is provided. For at least one block of an image, at least one first directional gradient histogram is obtained based on at least one second directional gradient histogram obtained for neighboring blocks of the at least one block. The at least one second directional gradient histogram is normalized based on a distance determined between the at least one block and the neighboring blocks. A prediction is determined for the at least one block using one or more intra-frame prediction modes, the one or more intra-frame prediction modes being determined using the at least one first directional gradient histogram, and the at least one block is encoded using the determined prediction.

[0005] According to another aspect, an apparatus for encoding an image or video is provided. The apparatus includes one or more processors operable to obtain, for at least one block of an image, at least one first directional gradient histogram based on at least one second directional gradient histogram obtained for neighboring blocks of the at least one block, wherein the at least one second directional gradient histogram is normalized based on a distance determined between the at least one block and the neighboring blocks; determine a prediction for the at least one block using one or more intra-frame prediction modes, the one or more intra-frame prediction modes being determined using the at least one first directional gradient histogram; and encode the at least one block using the determined prediction.

[0006] According to one aspect, a method for decoding an image or video is provided. For at least one block of an image, at least one first-direction gradient histogram is obtained based on at least one second-direction gradient histogram obtained for neighboring blocks of the at least one block. The at least one second-direction gradient histogram is normalized based on a distance determined between the at least one block and the neighboring blocks. A prediction is determined for the at least one block using one or more intra-frame prediction modes, the one or more intra-frame prediction modes being determined using the at least one first-direction gradient histogram, and the at least one block is decoded using the determined prediction.

[0007] According to another aspect, an apparatus for decoding an image or video is provided. The apparatus includes one or more processors operable for, for at least one block of an image, obtaining at least one first directional gradient histogram based on at least one second directional gradient histogram obtained for neighboring blocks of the at least one block, wherein the at least one second directional gradient histogram is normalized based on a distance determined between the at least one block and the neighboring blocks; determining a prediction for the at least one block using one or more intra-frame prediction modes, the one or more intra-frame prediction modes being determined using the at least one first directional gradient histogram; and decoding the at least one block using the determined prediction.

[0008] This document describes additional embodiments that can be used alone or in combination.

[0009] One or more embodiments also provide a computer program including instructions that, when executed by one or more processors, cause the one or more processors to perform any of the methods for encoding or decoding images or videos according to any of the embodiments described herein. One or more of these embodiments also provide a non-transitory computer-readable medium and / or a computer-readable storage medium storing instructions thereon for encoding or decoding images or videos according to the methods described herein.

[0010] One or more embodiments also provide a computer-readable storage medium storing a bit stream generated according to the method described herein. One or more embodiments also provide a method and apparatus for transmitting or receiving a bit stream generated according to the described method. Attached Figure Description

[0011] Figure 1A The illustration shows a block diagram of a system in which aspects of this embodiment can be implemented, according to one embodiment.

[0012] Figure 1B A block diagram of a system according to another embodiment in which aspects of this embodiment can be implemented is illustrated.

[0013] Figure 1C A block diagram of a system according to another embodiment in which aspects of this embodiment can be implemented is illustrated.

[0014] Figure 2 A block diagram illustrating an embodiment of a video encoder in which various aspects of this embodiment can be implemented is shown.

[0015] Figure 3 A block diagram illustrating an embodiment of a video decoder in which various aspects of this embodiment can be implemented is shown.

[0016] Figure 4 The illustration shows an example of extracting gradients from the context of the WxH block to be predicted.

[0017] Figure 5 The diagram illustrates G according to ECM-6.0. VER and G HOR The absolute value of G and G VER and G HOR Examples of symbols used to identify the range of the target intra-frame prediction mode index.

[0018] Figure 6 The illustration shows that and In the case of, when Calculate the reference axis and the component perpendicular to G. VER and G HOR The angle between the directions of the gradient G Examples.

[0019] Figure 7 The illustration shows that and In the case of, when Calculate the reference axis and the component perpendicular to G. VER and G HOR The angle between the directions of the gradient G Examples.

[0020] Figure 8 The illustration shows that in and In the case of, Figure 6 Under the conditions (i.e., Example of calculating the index of the target intra-frame prediction mode index.

[0021] Figure 9 The illustration shows that and In the case of, Figure 7 Under the conditions (i.e., Example of calculating the index of the target intra-frame prediction mode index.

[0022] Figure 10 An example of a DIMD region used to infer the location correlation of a DIMD pattern is illustrated.

[0023] Figure 11 The illustration shows an example of selecting DIMD and merge DIMD modes for intra-frame prediction, and determining the MHOG of the current block based on the two HOGs of the two blocks that are neighbors of the current WxH block.

[0024] Figure 12 The diagram illustrates an example of the workflow for normalizing each HOG involved in the merging process before applying the merge.

[0025] Figure 13 An example of a non-adjacent candidate is illustrated for DIMD merging.

[0026] Figure 14 An example of a method for encoding an image or video according to one embodiment is illustrated.

[0027] Figure 15 An example of a method for decoding an image or video according to one embodiment is illustrated.

[0028] Figure 16 An example of a method for obtaining a merged HOG of the current block to be encoded or decoded, according to one embodiment, is illustrated.

[0029] Figure 17 An example of a non-adjacent block for obtaining a merged HOG of blocks is illustrated according to one embodiment.

[0030] Figure 18 The illustration shows an example of a method for determining the merged HOG (MHOG) of the current block based on two HOGs of two blocks that are not adjacent to the current WxH block, according to an embodiment using spatial distance normalization.

[0031] Figure 19 The illustration shows an example of a method for determining the merged HOG (MHOG) of the current block based on the two HOGs of two blocks that are not adjacent to the current WxH block, according to an embodiment using HOG distance (SAD) normalization.

[0032] Figure 20 An example of two remote devices communicating over a communication network is shown, based on this principle.

[0033] Figure 21 The syntax of a signal according to this principle is shown. Detailed Implementation

[0034] This application describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described in detail and, at least to illustrate individual characteristics, are generally described in a manner that may sound restrictive. However, this is for clarity and not to limit the application or scope of those aspects. In fact, all the different aspects can be combined and interchanged to provide other aspects. Furthermore, this aspect can also be combined and interchanged with aspects described in earlier applications.

[0035] The aspects described and envisioned in this application can be realized in many different forms. The following... Figure 1A , 1B 1C Figure 2 and Figure 3 Some embodiments have been provided, but other embodiments are contemplated and Figure 1A , 1B 1C Figure 2 and Figure 3 The discussion does not limit the breadth of implementation. At least one aspect typically relates to video encoding and decoding, and at least one other aspect typically relates to the transmission of generated or encoded bitstreams. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions stored thereon for encoding or decoding video data according to any of the methods, and / or computer-readable storage media having a bitstream generated according to any of the methods stored thereon.

[0036] In this application, the terms “reconstructed” and “decoded” are used interchangeably, the terms “pixel” and “sample” are used interchangeably, and the terms “image”, “picture” and “frame” are used interchangeably.

[0037] Various methods are described herein, and each method includes one or more steps or actions for implementing the method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as "first," "second," etc., may be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding." Unless specifically required, the use of such terms does not imply an ordering of the modified operations. Therefore, in this example, the first decoding does not need to be performed before the second decoding and may occur, for example, before, during, or in a time period overlapping with the second decoding.

[0038] This aspect is not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations (whether pre-existing or developed in the future), and any extensions of such standards and recommendations (including VVC and HEVC). Unless otherwise stated or technically excluded, the aspects described in this application may be used alone or in combination.

[0039] Figure 1A-1C The illustration shows a block diagram of an example system in which various aspects and embodiments may be implemented. Any of systems 100A, 100B, or 100C may be embodied as a device including the various components described below and configured to perform one or more of the aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. In various embodiments, systems 100A, 100B, or 100C are communicatively coupled to other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, systems 100A, 100B, or 100C are configured to implement one or more of the aspects described in this application.

[0040] Figure 1A The diagram illustrates an example of a system in which various aspects and embodiments can be implemented. System 100A includes at least one processor 110 configured to execute instructions loaded therein to implement various aspects, such as those described in this application. Processor 110 may include embedded memory, input / output interfaces, and various other circuitry as known in the art. System 100A includes at least one memory 120, such as a volatile memory device and / or a non-volatile memory device, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drive, and / or optical disk drive. As a non-limiting example, memory 120 may include an internal storage device, an attached storage device, and / or a network-accessible storage device. Processor 110 may be interconnected with memory 120 via interconnect bus 115.

[0041] The program code to be loaded onto processor 110 to execute the various aspects described in this application is then loaded onto memory 120 for execution by processor 110.

[0042] In some embodiments, the memory within processor 110 is used to store program code instructions and provide working memory for processing required during encoding or decoding. Input to the components of system 100A can be provided through various input devices (not shown). Both processor 110 and memory 120 may also have one or more additional interconnects to external connections.

[0043] Figure 1B The diagram illustrates an example block diagram of a system 100B in which various aspects and embodiments may be implemented. System 100B includes combinations of... Figure 1A The processor 110 and memory 120 are described. Inputs to the components of system 100B can be provided through various input devices, as indicated in block 105, which are described below. Figure 1C Further description. Such input devices include, but are not limited to: (i) an RF section that receives radio frequency (RF) signals, for example, transmitted over the air by a broadcasting device; (ii) a component (COMP) input terminal (or a set of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 1B Other examples not shown include composite videos.

[0044] Various components can be interconnected and data can be transferred between them using a suitable connection arrangement 115, such as an internal bus known in the art, including an I2C bus, wiring, and printed circuit board.

[0045] System 100B includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 190 may be implemented, for example, in a wired and / or wireless medium.

[0046] System 100B can provide output signals to various output devices, including displays, speakers, and other peripheral devices. Output devices can be communicatively coupled to system 100B via dedicated connections through their respective interfaces 160, 170, and 180. Alternatively, output devices can be connected to system 100B via communication interface 150 using communication channel 190.

[0047] Figure 1CThe illustration shows an example block diagram of a system 100C according to another embodiment, in which various aspects and embodiments may be implemented. The elements of the system 100C may be embodied individually or in combination in a single integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of the system 100C are distributed across multiple ICs and / or discrete components.

[0048] System 100C includes, for example, combinations Figure 1A or Figure 1B The processor 110 and memory 120 are described.

[0049] System 100C includes storage device 140, which may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drive, and / or optical disk drive. As a non-limiting example, storage device 140 may include internal storage device, attached storage device, and / or network-accessible storage device.

[0050] System 100C includes an encoder / decoder module 130 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents one or more modules that can be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Additionally, as is known to those skilled in the art, the encoder / decoder module 130 may be implemented as a separate element of system 100C, or it may be incorporated within processor 110 as a combination of hardware and software.

[0051] Program code to be loaded onto processor 110 or encoder / decoder 130 to execute the various aspects described in this application may be stored in storage device 140 and subsequently loaded onto memory 120 for execution by processor 110. According to various embodiments, one or more of processor 110, memory 120, storage device 140, and encoder / decoder module 130 may store one or more of various items during execution of the processes described in this application. Such stored items may include, but are not limited to, input data (images, videos, volumetric content), decoded data (images, videos, volumetric content) or portions of decoded data, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0052] In some embodiments, memory within processor 110 and / or encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be processor 110 or encoder / decoder module 130) is used for one or more of these functions. External memory may be memory 120 and / or storage device 140, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, fast external volatile memory such as RAM is used as working memory for data encoding and decoding operations, such as for MPEG-2, HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG H Part 2), or VVC (Multi-Functional Video Coding (also known as H.266, a new standard developed by the Joint Video Experts Group JVET)).

[0053] The inputs to the components of System 100C can be provided through various input devices, such as... Figure 1B As indicated in block 105 mentioned above. Such input devices of system 100B or 100C include, but are not limited to: (i) an RF section that receives radio frequency (RF) signals transmitted over the air, for example by a broadcasting device; (ii) a component (COMP) input terminal (or a set of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 1B or Figure 1C Other examples not shown include composite videos.

[0054] In various embodiments, as known in the art, the input device of block 105 in system 100B or 100C has associated respective input processing elements. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting the signal band to a band), (ii) down-converting the selected signal, (iii) band-limiting it again to a narrower band to select, for example, a signal band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section in various embodiments includes one or more elements to perform these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, down-converters, demodulators, error correctors, and demultiplexers. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0055] Additionally, the USB and / or HDMI terminals may include their respective interface processors for connecting system 100B or 100C to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed Solomon error correction) can be implemented as needed, for example, within a separate input processing IC or within processor 1010. Similarly, various aspects of USB or HDMI interface processing can be implemented as needed, either within a separate interface IC or within processor 1010. Demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 110 and encoder / decoder 130, to operate in conjunction with memory and storage elements to process the data streams as needed for presentation on the output device.

[0056] Various components of system 100A, 100B, or 100C can be housed within an integrated enclosure. Within the integrated enclosure, various components can be interconnected and transmit data between them using a suitable connection arrangement 1140 (e.g., internal buses as known in the art, including I2C buses, wiring, and printed circuit boards).

[0057] Regarding Figure 1BSimilar to system 100B, system 100C includes a communication interface 150 that enables communication with other devices via communication channel 190. Communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data via communication channel 190. Communication interface 150 may include, but is not limited to, a modem or network interface card (NIC), and communication channel 190 may be implemented, for example, in wired and / or wireless media.

[0058] In various embodiments, data is streamed to system 100B or 100C using a Wi-Fi network such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in these embodiments is received via a communication channel 190 and a communication interface 150 suitable for Wi-Fi communication. The communication channel 190 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box that delivers data via an HDMI connection to input block 105 to provide streaming data to system 100B or 100C. Still other embodiments use an RF connection to input block 105 to provide streaming data to system 100B or 100C. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0059] System 100C can provide output signals to various output devices, including display 165, speaker 175, and other peripheral devices 185. Display 165 in various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a flexible display, and / or a foldable display. Display 165 can be used in televisions, tablets, laptops, cellular phones (mobile phones), or other devices. Display 165 can also be integrated with other components (e.g., as in a smartphone) or separate (e.g., an external monitor for a laptop). In various examples of embodiments, other peripheral devices 185 include one or more of a standalone digital video disc (or digital multifunction disc) (DVR for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 185 that provide functionality based on the output of system 100C. For example, a disc player performs the function of playing the output of system 100C.

[0060] In various embodiments, signaling such as AV links, CEC, or other communication protocols enabling inter-device control with or without user intervention is used to communicate control signals between system 100C and display 165, speaker 175, or other peripheral devices 185. Output devices can be communicatively coupled to system 100C via dedicated connections through their respective interfaces 160, 170, and 180. Alternatively, output devices can be connected to system 100C via communication interface 150 using communication channel 190. Display 165 and speaker 175 can be integrated into a single unit with other components of system 100C in electronic devices such as televisions. In various embodiments, display interface 1070 includes a display driver, such as a timing controller (TCon) chip.

[0061] For example, if the RF portion of input 105 is part of a standalone set-top box, then display 165 and speaker 175 may alternatively be separate from one or more other components. In various embodiments where display 165 and speaker 175 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0062] In any of systems 100A, 100B, or 100C, embodiments may be implemented by a computer program product comprising code instructions that implement any of the embodiments described herein. This computer program product may be computer software implemented by processor 110, or by hardware, or by a combination of hardware and software. As a non-limiting example, embodiments may be implemented by one or more integrated circuits. As a non-limiting example, the memory 120 of any of systems 100A, 100B, or 100C may be of any type suitable for the technical environment and may be implemented using any suitable data storage technology, such as optical storage devices, magnetic storage devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, the processor 110 of any of systems 100A, 100B, or 100C may be of any type suitable for the technical environment and may encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0063] Figure 2 An example of a block-based hybrid video encoder 200 is illustrated. Variations of this encoder 200 are envisioned. However, for clarity, encoder 200 is described below without depicting all anticipated variations.

[0064] In some embodiments, Figure 2 The diagram also illustrates the HEVC or VVC standard ( Multifunctional video coding standard ITU-T H.266, ISO / IEC 23090-3, 2020 This refers to an improved encoder or an encoder that employs a technology similar to HEVC or VVC, such as the ECM (Enhanced Compression Model) encoder being developed by JVET (Joint Video Experts Group).

[0065] Before being encoded, the video sequence may undergo pre-coding (201), for example, applying color transformations to the input color images (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), performing remapping of the input image components to obtain a more compression-resistant signal distribution (e.g., histogram equalization using color components), or resizing the images (excluding, for example, scaling down). Metadata may be associated with pre-processing and attached to the bitstream.

[0066] In encoder 200, as described below, the image is encoded by encoder elements. The image to be encoded is segmented (202) and processed in units such as CUs (coding units) or blocks. In this disclosure, different terms may be used to refer to such units or blocks resulting from the segmentation of the image. Such terms may be coding unit or CU, coding block or CB, luminance CB or block. CTU (coding tree unit) refers to a set of blocks or a set of units or a set of coding units (CUs). In some embodiments, a CTU may be considered a block, or as a unit itself.

[0067] Each unit is encoded using, for example, an intra-frame mode or an inter-frame mode. When a unit is encoded in an intra-frame mode, it performs intra-frame prediction (260). In inter-frame mode, motion estimation (275) and compensation (270) are performed. Intra-frame and / or inter-frame modes may include several different sub-modes. For example, intra-frame modes may include directional intra-frame prediction, template-based intra-frame prediction, decoder-side intra-frame mode derivation prediction, intra-block copy prediction, or other modes that spatially predict sample values ​​of the unit. Inter-frame modes may include skip modes, merge modes (in which motion information is derived from a motion candidate list and no motion vector prediction residual is encoded), inter-frame modes (in which motion information is derived from a motion candidate list and further refined by encoding the motion vector prediction residual or by performing template matching at both the encoder and decoder), and other inter-frame modes are also possible. The encoder decides (205) which of the intra-frame or inter-frame modes to use for encoding the unit. When different intra-frame and / or inter-frame modes are possible, the encoder decides (205) which of the intra-frame or inter-frame modes to use. The encoder indicates the intra-frame / inter-frame decision by signaling the prediction mode, for example, through one or more syntax elements. The encoder can also mix (205) intra-frame and inter-frame prediction results, or mix results from different intra-frame / inter-frame prediction methods. For example, the prediction residual is calculated by subtracting (210) the prediction block from the original image block.

[0068] The motion refinement module (272) uses an existing reference image to refine the motion field of a block without referencing the original block. The motion field of a region can be considered as the set of motion vectors of all pixels within that region. If the motion vectors are based on sub-blocks, the motion field can also be represented as the set of motion vectors of all sub-blocks within that region (all pixels within a sub-block have the same motion vector, and the motion vector can vary from sub-block to sub-block). If a single motion vector is used for the region, the motion field of that region can also be represented by that single motion vector (the same motion vector for all pixels in that region).

[0069] The predicted residual is then transformed (225) and quantized (230). The quantized transform coefficients, along with the motion vector and other syntax elements, are entropy encoded (245) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., directly encode the residual without applying either the transform or quantization process.

[0070] The encoder decodes (reconstructs) the encoded blocks to provide a reference for further prediction. Quantized transform coefficients are dequantized (240) and inversely transformed (250) to decode the prediction residuals. The decoded prediction residuals and prediction blocks are combined (255) to reconstruct the image blocks. In-loop filters (265) are applied to the reconstructed image to perform one or more of, for example, deblocking filtering, SAO (Sampling Adaptive Shift) filtering, or ALF (Adaptive Loop Filter) filtering to reduce coding artifacts. The filtered image is stored at a reference image buffer (280). Such a filtered image is also referred to hereinafter as the reference image.

[0071] Figure 3 A block diagram of a video decoder 300 is shown. In decoder 300, as described below, the bitstream is decoded by decoder elements. The video decoder 300 typically performs functions similar to... Figure 2 The decoding rounds that are the inverse of the encoding rounds described in the text, such as Figure 2 As described in [the document]. Encoder 200 typically also performs video decoding as part of the encoding of video data.

[0072] Specifically, the decoder's input includes a video bitstream, which can be generated by the video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, motion vectors, and other encoded information. Image segmentation information indicates how the image is segmented. Therefore, the decoder can segment (335) the image based on the decoded image segmentation information. The transform coefficients are dequantized (340) and inverse transformed (350) to decode the prediction residuals. By combining (355) the decoded prediction residuals and prediction blocks, image patches are reconstructed.

[0073] The prediction block can be obtained from intra-frame prediction (360) or motion-compensated prediction (i.e., inter-frame prediction) (375). In a similar manner to that in the encoder, intra-frame prediction and / or inter-frame prediction can include several different sub-modes. The decoder obtains (370) a predictor block based on one or more syntax elements that signal the prediction mode among the available intra-frame and inter-frame modes. The decoder can mix (370) the intra-frame prediction results and the inter-frame prediction results, or mix the results from multiple intra-frame / inter-frame prediction methods. Before motion compensation, the motion field can be refined (372) by using an already available reference picture. An in-loop filter (365) is applied to the reconstructed image. The filtered image is stored at the reference picture buffer (380). Note that for a given picture, the contents of the reference picture buffer 380 on the decoder 300 side are the same as the contents of the reference picture buffer 280 on the encoder 200 side for the same picture.

[0074] The decoded image can undergo further post-decoding processing (385), such as inverse color transformation (e.g., conversion from YCbCr4:2:0 to RGB 4:4:4) or inverse remapping of the remapping process performed in pre-encoding processing (201), or resizing of the reconstructed image (excluding: proportional scaling). Post-decoding processing can utilize metadata derived in pre-encoding processing and signaled in the bitstream.

[0075] Some of the embodiments described herein relate to intra-frame prediction of blocks of an image or video to be encoded or decoded, and more specifically, to improved decoder-side intra-frame mode derivation (DIMD). Any of the embodiments described herein can be implemented, for example, in the intra-frame prediction module of a video encoder and the intra-frame prediction module of a video decoder. For example, the embodiments described herein can be implemented in… Figure 2 The intra-frame prediction module 260 of the video encoder 200 or Figure 3 It is implemented in the intra-frame prediction module 360 ​​of the video decoder 300.

[0076] Decoder-side intra-mode derivation (DIMD) is a block-based intra-prediction mode that relies on the assumption that the reconstructed pixels surrounding the block to be predicted carry information for inferring the texture orientation within that block, i.e., the intra-prediction mode most likely to generate the highest quality prediction. DIMD is a block-based prediction mode, and the determination of the block is performed in the same manner on both the encoder and decoder. In the following explanation, the same interpretation applies to both the encoder and decoder sides.

[0077] Inference in DIMD According to DIMD, the inference of the index of the intra-prediction mode most likely to generate the highest quality prediction is decomposed into three steps. First, gradients are extracted from the context of the reconstructed pixels around the given block to be predicted. Then, these gradients are used to fill the Histogram of Oriented Gradients (HOG). Finally, the index of the intra-prediction mode most likely to give the highest quality prediction according to the DIMD derivation process is derived from the HOG, and merging can be performed.

[0078] Extract gradients from context For a given block to be predicted, consider the area above that block. Row reconstruction pixels and the left side of the block The L-shaped context of the reconstructed pixels, such as Figure 4 As shown. In Figure 4 In the image, the block to be predicted is shown in white, and the context is shown in gray. The context includes the area above the block. Row reconstruction pixels and pixels located on the left side of the block Column reconstructed pixels. Gradient filter in Figure 4It is enclosed in black.

[0079] The L-shaped context is often referred to as a template for a given block. At each reconstructed pixel of interest within this context, local vertical and horizontal gradients are computed. Mohsen Abdoli,Thomas Guionnet,Mickael Raulet, Gosala Kulupana, Saverio Blasi's "Decoder-side intra mode derivation for next-generation video coding” , ICME, 2020 (hereinafter referred to as [1]) and in Mohsen "NonCE-3: decoder-side intra" by Abdoli, Elie Mora, Thomas Guionnet, Mickael Raulet mode derivation with prediction fusion” , Paper submitted to the 4th JVET Conference in Geneva, JVET-N0342. March 19 to March 29, 2019 (Hereinafter referred to as [2]) and in ECM-6.0, respectively via Vertical and horizontal Sobel filters compute local vertical and horizontal gradients. Furthermore, in [1], [2], and ECM-6.0, the reconstructed pixels of interest in this context refer to those reconstructed pixels where the gradient filter does not exceed the context boundaries. Therefore, in [1], [2], and ECM-6.0, the complete extraction of gradients can be achieved through… Vertical and horizontal Sobel filters are summarized by "effective" convolution with the context. Note that in ECM-6.0, and .

[0080] Filling the Histogram of Oriented Gradients (HOG) In HOG, each histogram is associated with an index of a different directional intra-prediction mode. During initialization, all HOG histograms are equal to 0. This is used for calculating local vertical gradients. and local horizontal gradient Each reconstructed pixel of interest, from and The direction is derived, and a histogram of the indices of the directional intra-prediction modes whose directions are closest to the derived direction is incremented. This index is called the "target intra-prediction mode index".

[0081] More precisely, for a given reconstructed pixel of interest, from and The derivation of the direction is based on the following observation: During block prediction via directional intra-prediction modes, the gradient with the largest absolute value typically follows a direction perpendicular to the mode direction. Therefore, from and The direction of the output must be perpendicular to the component. and The gradient. For example, in ECM-6.0 using 65 VVC directional intra-prediction modes, considering vertical and horizontal gradient filters with positive vertical gradients from top to bottom and positive horizontal gradients from right to left, in Figure 5 The text shows from and absolute value and and The mapping of symbols to the range of the target frame intra-prediction mode index.

[0082] From Figure 5 In the framework of ECM using VVC directional intra-prediction mode: In (1), the target intra-frame prediction mode index belongs to , In (2), the target intra-frame prediction mode index belongs to , In (3), the target intra-frame prediction mode index belongs to , In (4), the target intra-frame prediction mode index belongs to , if If the target intra-frame prediction mode is vertical, then its index is 50. if If the target intra-frame prediction mode is horizontal, then its index is 18.

[0083] Now, if If the reference axis is perpendicular to the component, then it is a horizontal axis. Otherwise, the reference axis is a vertical axis. and gradient G Angle between directions Given by the following formula: If ,but ,otherwise, ,like Figure 6 and Figure 7 As shown.

[0084] For calculating local vertical gradient and local horizontal gradient The currently interesting reconstructed pixel, for example Figure 5 The range of the intra-prediction mode index found now makes it possible to find the closest one with respect to the reference axis. The index of the intra-prediction mode is then used. Then, the histogram of indices associated with the found target intra-prediction mode is incremented. This means that by representing the histogram of indices associated with the found target intra-frame prediction patterns as... i , Note that for the currently interesting reconstructed pixel, if If so, then there are no increasing histograms in the HOG.

[0085] Angle Discretization For calculating the local vertical gradient and local horizontal gradient Given a reconstructed pixel, the range of the intra-predictive mode index of the found target frame (see [reference]). Figure 5 (The angle mentioned above) It is not directly compared with the angle of each intra-prediction mode relative to the reference axis within this range. In fact, the absolute angle of each intra-prediction mode relative to its reference axis is stored as a scaled integer. Therefore, with index from reference axis i ( The scaling integer form of the angle of the directional intra-prediction mode. In comparison, `floor` represents the floor operation—that is, for input `x`, the operation returns the largest integer less than or equal to `x` as the result of `floor(x)`. Thus, the absolute offset from the reference axis index to the index of the target intra-prediction mode. yes The target intra-frame prediction mode index is eventually equal to the offset of the reference axis. The index. In Figure 6 Under the conditions, Figure 8 The diagram illustrates the use of the method mentioned above. The calculation of the index of the discretized target intra-frame prediction mode. Figure 7 Under the conditions, Figure 9 The diagram illustrates the use of the method mentioned above. The calculation of the index of the discretized target intra-prediction mode.

[0086] Inference of (one or more) intra-prediction modes Once HOG padding is complete, the index of the directional intra-prediction mode most likely to generate the highest quality prediction is selected as the index associated with the histogram histogram bar with the largest amplitude. In [1] and ECM-6.0, the two histogram bars with the largest amplitude are identified to find the index of the directional intra-prediction mode most likely to generate the two highest quality predictions according to DIMD, and the two modes are linearly combined, optionally with PLANAR.

[0087] In JVET-AB0148, the number of selected histograms has been expanded to 5, meaning that 5 predictions provided by 5 selected indices of the directional intra-prediction mode are combined.

[0088] Notify DIMD with a signal In ECM-6.0, for a given luminance coded block (CB) to be predicted, the DIMD is signaled via a DIMD flag, which is first placed in the decision tree of the signaling of the intra-prediction mode selected for predicting the luminance CB, i.e. before the Template Match Prediction (TMP) flag and the Matrix-Based Intra-Prediction (MIP) flag.

[0089] DIMD patterns based on region location exist Saverio Blasi, Jani Lainema of "AHG12–Location-dependent Decoder-side "Intra Mode Derivation", submitted to the 28th JVET Conference in Mainz, JVET-AB0116, October 20-28, 2022. (Hereinafter referred to as [3]), a sample-based non-uniform weighting method is proposed to mix predictions obtained from selected indices of the HOG. The use of sample-based mixing and the specific weights used for a given prediction are inferred during the DIMD derivation process. When deriving a DIMD pattern (i.e., a directional intra-prediction pattern), it is determined whether the derivation of such a pattern is primarily influenced by the template region above or to the left of the current block. If the DIMD pattern is primarily derived from samples above the current block, then higher weights should be used for samples closer to the portion above the block when mixing the corresponding predictions. To determine whether a particular sample in the template contributes to the inference of a particular DIMD pattern, such as Figure 10 As shown, three separate regions are considered within the DIMD template. Gradient calculations are performed on samples in each region, resulting in three histograms. , and For directional patterns m , Indicates direction m The cumulative amplitude of all samples in the upper ABOVE region. It should be noted that, compared to traditional DIMD, the template region is expanded by one sample in the upper left corner and one sample in the lower right corner.

[0090] Then, the complete gradient histogram of the entire template can be calculated as the sum of three individual histograms. As in traditional DIMD, the two directional modes with the largest and second largest cumulative magnitudes in the histogram are selected as the primary DIMD modes. and sub-DIMD mode .

[0091] Additionally, histogram and It can be used to determine and / or Whether this depends on the specific template area ABOVE or LEFT. Specifically, Locational correlation (represented as) ) can be defined as: if: ,but: ,This is It depends on the region ABOVE.

[0092] Otherwise if: ,but: ,This is It depends on the LEFT region.

[0093] otherwise: ,This is It's not location-related.

[0094] Then a blending process is performed to merge the master DIMD predictions. and sub-DIMD prediction With planar prediction In the absence of a DIMD pattern being determined as location-dependent (meaning...), In the case of [missing information], uniform mixing is applied. Uniform weights are derived based on the relative magnitudes of the patterns in the histogram. , and And the final DIMD prediction is calculated as follows: (Equation 1).

[0095] Otherwise, if at least one DIMD pattern is inferred to be location-dependent, sample-based mixing is used. Different weights are used to mix each location. The prediction at that location. If Then the calculation is used for prediction Sample-based weights This makes the average weight used within the block approximately equal to the uniform weight. And it allows for the use of higher weights in block portions closer to the ABOVE or LEFT regions, depending on... .scope It is predefined, corresponding to from The maximum deviation. The higher the value, the greater the change in weights within the block. Specifically, for a block of size... The block, if ,but: Otherwise if ,but: .

[0096] If two Then according to The values ​​are used to calculate the weights of the two predictions according to one of the two equations above. .

[0097] Conversely, if and ,but The weights are determined as follows: .

[0098] Finally, the weights for planar prediction. It was identified as: .

[0099] Then, the final location-related DIMD prediction was determined as follows: (Equation 2).

[0100] DIMD merge exist Saverio Blasi, Ivan Zupancic, Jani Lainema of AHG12–Decoder Side Intra "Mode Derivation Merge", submitted to the 31st JVET Conference in Geneva, JVET-AE0071, July 11-19, 2023. In ECM-9.0, a new DIMD merging mode was introduced (hereinafter referred to as [4]). In this mode, for the current block, the merged gradient histogram (MHOG) is determined by combining the HOGs of neighboring blocks encoded using intra-frame prediction with either conventional DIMD or DIMD in the merging mode. Conventional DIMD is DIMD intra-frame prediction, in which the HOG of the current block is determined using reconstructed samples of the template of the current block, while DIMD in the merging mode is DIMD intra-frame prediction, in which the HOG (MHOG) of the current block is determined using the HOGs previously determined for neighboring blocks.

[0101] In DIMD in merge mode, the index of the intra-prediction mode used to predict the current block and the weights of the potential DIMD blend are derived from the MHOG.

[0102] More precisely, regarding the combination of HOGs, for the current block, if a single neighboring block chooses either the traditional DIMD or the DIMD in the merge mode, then the HOG of that neighboring block is equal to the MHOG of the current block. If at least two neighboring blocks choose either the traditional DIMD or the DIMD in the merge mode, then the two HOGs of those two neighboring blocks are averaged to form the MHOG of the current block.

[0103] For a given block, DIMD in merge mode is allowed only if the current block has at least one neighbor that selects either traditional DIMD or DIMD in merge mode. If allowed, the use of DIMD in merge mode is signaled using a CABAC-encoded CU-level flag.

[0104] Figure 11 Presented the method for determining the current An example of a block's MHOG. Current Block (1100) has three neighboring blocks (1101), (1102) and (1103). Blocks (1102) and (1103) select DIMD for intra-frame prediction, while (1101) selects neither conventional DIMD nor DIMD in merge mode for intra-frame prediction.

[0105] During the encoding / decoding of the current frame to which the block belongs, the HOG of block (1103) is filled from the template (1104) of the reconstructed sample of block (1103), and is denoted as HOG1. The HOG of block (1102) is filled from the template (1105) of the reconstructed sample of block (1102), and is denoted as HOG2. The MHOG of block (1100) is then determined by averaging the two HOGs (HOG1 and HOG2).

[0106] DIMD merging with histogram normalization In [4], in DIMD in merge mode, for a current block with at least two neighboring blocks that have chosen either traditional DIMD or DIMD in merge mode for intra-frame prediction, the merging of HOGs does not take into account the dynamics of the HOGs to be merged. For example, in Figure 11 In the MHOG, since block (1105) is larger than block (1104), HOG2 contains more gradient magnitude increments than HOG1. Therefore, during the merging of HOG1 and HOG2, HOG2 contributes more to the MHOG than HOG1. This makes the underlying principles of HOG merging less relevant. This can be addressed by incorporating histogram normalization before merging.

[0107] It is possible Figure 12 The summary describes the normalization of each HOG involved in the merging of combined HOGs before forming the MHOG of the current block.

[0108] At 1200, the current event can be triggered. The determination of the MHOG of the block. At 1201, neighboring blocks selected for intra-frame prediction using either conventional DIMD or DIMD in merge mode can be placed into a set of candidate blocks, blockSet.

[0109] At 1202, as an optional step, the candidate neighbor block set `blockSet` can be reduced by filtering according to a given criterion, resulting in a reduced candidate neighbor block set `blockSetFiltered`. As an example, the criterion for eliminating a given neighbor block from the candidate set could be based on the distance between the current block and the given neighbor block. As another example, the criterion for eliminating a given neighbor block could be based on the difference between the size of the current block and the size of the given neighbor block. At 1203, normalization information `normInfo` can be collected from the current block and the candidate set `blockSetFiltered`. At 1204, using `normInfo` and `blockSetFiltered`, the HOG of each neighbor block in `blockSetFiltered` can be normalized, resulting in... .

[0110] At 1205, the normalized HOG It can be merged into the MHOG of the current block.

[0111] The normalization information collected at point 1203 can be one or more of the following normalization factors. In one variant, HOG can be normalized by the number of reference samples within the template of the associated neighboring blocks. In this variant, Figure 12 The normInfo at (1203) can be the size of the template for each neighboring block involved in the merging. In another variation, HOG can be normalized by the number of pixels in the associated neighboring blocks. In this variation, Figure 12 The normInfo at (1203) can be the size of each neighboring block involved in the merge. In another variation, the HOG can be normalized by any value derived from the template or block size. As a first example, for each HOG of a neighboring block involved in the merge, each histogram can be divided by twice the number of pixels in the associated neighboring block. As a second example, for each HOG of a neighboring block involved in the merge, each histogram can be divided by the number of reference samples inside the template of the associated neighboring block plus 2. In another variation, the HOGs of the neighboring blocks of the current block involved in the merge that produce the MHOG can be normalized relative to the histogram magnitude. For example, the normalization of this HOG can be as follows: , n This indicates the number of histogram bars in the HOG. For example, in ECM-9.0, This is associated with 65 directional intra-frame prediction modes in brightness. As another example, the normalization of this HOG can be as follows: .

[0112] DIMD merge with non-adjacent blocks In [4] and as described above, when neighboring blocks are encoded using one of the DIMD modes, the DIMD histograms are combined to form a new DIMD merge histogram for the current block. The DIMD merge mode and weights are calculated based on this merged histogram.

[0113] exist Junyan Huo, Jiawei Fan, Zhenyao Zhang, Yanzhuo Ma, Fuzheng Yang, Ming Li of "AHG12 – Non-adjacent spatial candidates for DIMD merge", submission to the 32nd Hanover Conference AF0106, October 13-20, 2023 In [5] (hereinafter referred to as [5]), a novel DIMD merging mode with non-adjacent blocks is proposed. In this submission, non-adjacent spatial candidates are used to construct the DIMD merging candidate list to determine the intra-frame prediction of the current block.

[0114] As mentioned above, although DIMD in the merge mode uses DIMD information from neighboring (adjacent) blocks to predict the current block [4], however... Figure 13 As shown, new DIMD merging with non-adjacent spatial blocks uses candidates from non-adjacent spatial blocks. Figure 13 Examples of non-adjacent spatial blocks (1301, 1303, 1305, 1304) are illustrated, which can be used as candidates for DIMD in the merging mode of the current block (1302). As detailed in [5], the distance between the non-adjacent candidate blocks (e.g., (1303), (1304), (1305)) and the current block (1302) is defined based on the width and height of the current coded block.

[0115] In [5], the DIMD in the merge pattern includes a list of non-adjacent blocks. The HOG normalization described above can also be applied to HOGs obtained from non-adjacent blocks; however, this does not take into account the spatial distance between the current block and non-adjacent blocks, or the HOG distance between HOG candidates from non-adjacent blocks and the HOG of the current block. This makes the principle of HOG merging less relevant in the case of non-adjacent blocks.

[0116] Some embodiments described herein provide a method in which histogram normalization is applied to at least non-adjacent blocks, taking into account spatial distance and / or HOG distance.

[0117] In some embodiments, prior to merging, the HOGs of non-adjacent blocks are normalized by their spatial distances to the current block and / or by their HOG distances to the current block. It should be noted that the following description applies in the same manner on both the encoder and decoder sides.

[0118] In one variant, for each HOG involved in a non-adjacent block in the MHOG that is merged into the current block, the HOG is normalized by the HOG distance (SAD) to the current block.

[0119] In another variation, for each HOG involved in a non-adjacent block in the MHOG that is merged into the current block, the HOG is normalized by its spatial distance from the current block.

[0120] In yet another variation, the HOG of each non-adjacent block involved in the merge of the current block involved in the MHOG can be filtered by a filter before the merge. In yet another variation, after merging the HOGs of the non-adjacent blocks involved in the merge of the current block, an MHOG is generated, which can be filtered.

[0121] As described above, each histogram bar of the HOG (Histogram of Oriented Gradients) is associated with an index of a directional intra-prediction mode, allowing one or more intra-prediction modes to be derived from the HOG to predict the current block. In the following text, word HOG and gradient orientation histogram are used interchangeably. Furthermore, histograms of word intra-prediction orientations can also be used.

[0122] Figure 14 An example of a method 1400 for encoding an image or video according to one embodiment is illustrated. For example, method 1400 is composed of... Figure 2 The method described here is the execution of the encoder. Consider that at least one current block of an image or video to be encoded will be encoded using DIMD in a merged mode. At 1401, the merged HOG (MHOG) of the current block is obtained based on one or more HOGs previously obtained for one or more neighboring blocks. Neighboring blocks may or may not be adjacent to the current block. For example, neighboring blocks can be obtained from a set of candidate blocks encoded using the following DIMD prediction modes: conventional DIMD or DIMD in a merged mode. For each considered neighboring block, the HOG has been previously obtained either from the determination of reconstructed samples based on the template of the neighboring blocks or from the merging of the HOGs of the neighboring blocks' neighboring blocks.

[0123] At 1401, when the MHOG of the current block is obtained, the HOG obtained for each of the neighboring blocks of the current block used to obtain the MHOG is normalized based on the distance between the current block and its neighboring blocks. In one variant, this distance is the spatial distance between the current block and its neighboring blocks. In another variant, this distance is the HOG distance between the current block and its neighboring blocks.

[0124] In one variant, at 1401, the MHOG for the current block is obtained by merging one or more HOGs previously obtained for one or more neighboring blocks with the HOG obtained for the current block using a reconstruction sample of the template of the current block.

[0125] In one variant, the MHOG is obtained for the current block by merging a portion of the HOGs mentioned above (e.g., by merging only the indices with the highest magnitude or those with magnitudes higher than a given value).

[0126] Further details and variations of the MHOG used to determine the current block are provided below.

[0127] At 1402, the prediction for the current block is determined using the MHOG obtained for the current block. One or more intra-prediction modes are derived from the MHOG obtained for the current block. The prediction for the current block is obtained by merging all predictions provided by the derived one or more intra-prediction modes. For example, prediction is performed in a similar manner to the above using one or more intra-prediction directions with the highest magnitude in the obtained MHOG. At 1403, the current block is encoded using the prediction, for example, based on determining the residual between the current block and the prediction, and the residual is encoded into a bitstream.

[0128] Figure 15 An example of a method 1500 for decoding an image or video according to one embodiment is illustrated. For example, method 1500 is composed of... Figure 3 The method described is the one performed by the decoder. Consider that at least one current block of the image or video to be decoded will be decoded using DIMD in merge mode. At 1501, in a similar manner to that at the encoder, for example... Figure 14 As in 1401, the merged HOG (MHOG) is obtained for the current block. At 1502, in a similar manner to that at the encoder, for example... Figure 14 As in 1402, the MHOG obtained for the current block is used to determine the prediction for the current block. At 1503, the prediction is used to reconstruct the current block, for example by decoding the residual from the bitstream and adding the decoded residual to the prediction.

[0129] Figure 16 The figure illustrates an example of a method 1600 for obtaining the MHOG of the current block according to one embodiment, wherein each HOG involved in the merging of HOGs is normalized before being combined to form the MHOG of the current block.

[0130] At 1601, for example when deciding whether to encode or decode the current block using DIMD in merge mode, the current... The determination of the block's MHOG is triggered.

[0131] At position 1602, the neighboring blocks of the current block selected for intra-frame prediction using either conventional DIMD or DIMD in merge mode are added to the candidate block set `blockSet`. These neighboring blocks are preferably non-neighboring blocks of the current block. In some variations, neighboring blocks may also include the blocks adjacent to the current block.

[0132] In some variations, the candidate block set can be pruned based on given criteria, and a portion of this set, `blockSet`, can be selected. As an example, the given criterion for eliminating a given block from the set could be based on the distance between the current block and the given block. For instance, if the spatial distance between the given block and the current block is greater than a given value. As another example, the given criterion for eliminating a given block from the set could be based on the difference between the size of the current block and the size of the given block. For instance, if the difference is greater than a given value, or if the current block and the given block do not have the same shape.

[0133] At 1603, as an optional step, the block set blockSet can be reduced by filtering according to the given criteria mentioned above, resulting in a reduced block set blockSetFiltered.

[0134] At position 1604, distance information is collected from the current block and the blocks in the blockSetFiltered set. distInfo As an example, distInfo This could be the SAD (sum of absolute distances) distance between blocks in the `blockSetFiltered` set and the current block. As another example, distInfo It can be the spatial distance between blocks in the blockSetFiltered collection and the current block.

[0135] At 1605, use distInfo And the blocks in the set blockSetFiltered, the HOG of each block in the set blockSetFiltered is normalized, thus producing .

[0136] At position 1606, the normalized HOG It is merged into the MHOG of the current block.

[0137] Normalization relative to spatial distance In one variant, for each HOG involved in the MHOG merged into the current block, the HOG is normalized by its spatial distance d from the current block. For example, the block is not adjacent to the current block.

[0138] The number of blocks involved in obtaining the MHOG can vary with the strategy used to select and eliminate blocks in the candidate block set.

[0139] For example, non-adjacent candidates can be searched around the current block until a given spatial distance is reached. For example, such as Figure 17 As shown, the range used to search for non-adjacent candidates (1701) around the current block (1702) can have a size that depends on the shape of the current block (1702). An example of a non-adjacent block is shown in... Figure 17 They are shown as gray squares. Some of them are listed in... Figure 17 To the left: for example, (1703) is block number 30, (1704) is block number 27, and (1705) is block number 27. For example, the current block (1702) and non-adjacent blocks (e.g., (1703), (1704) and (1705)) can be CU (coding unit).

[0140] In this variant, Figure 16 The distInfo determined at (1604) can be the spatial distance from each non-adjacent block involved in the merge to the current block. Figure 17 In the example shown, d19 corresponds to the spatial distance between the neighboring block (1705) and the current block (1702), d30 corresponds to the spatial distance between the neighboring block (1703) and the current block (1702), and d27 corresponds to the spatial distance between the neighboring block (1704) and the current block (1702).

[0141] Figure 18 An example of a method for determining the merged HOG (MHOG) of the current WxH block (1702) from two HOGs of two non-adjacent blocks (1703, 1705) selected for intra-frame prediction using DIMD. Some blocks (e.g., (1704)) may not select either conventional DIMD or DIMD in merged mode for intra-frame prediction. Another criterion for selecting a non-adjacent block could be the spatial distance between the non-adjacent block and the current block. For example, a non-adjacent block (1704) could be considered too far from the current block (d27). In this case, the non-adjacent block (1704) is not selected. (1704) is an example of an unselected block used to obtain the merged HOG of the current block.

[0142] During the encoding / decoding of the current frame to which the current block (1702) belongs, at 1840, the HOG of block (1705) is obtained and represented as HOG. 19 For example, filling the HOG from the template of the reconstructed pixel (1706). 19 At position 1830, the HOG of block (1703) is obtained and denoted as HOG. 30 For example, filling the HOG from the template of the reconstructed pixel (1708). 30 .

[0143] At 1850, by using HOG 30 Divide each histogram by the spatial distance between block (1703) and the current block (1702) to determine the HOG. 30 Perform normalization. For example, here, the spatial distance is d30, and can be equal to... In one variant, and It can be the center of their respective blocks.

[0144] At 1860, by using HOG 19 Divide each histogram block by the spatial distance between block (1705) and the current block (1702) to determine the HOG. 19 Perform normalization. For example, here, the spatial distance is d19, and it can be equal to... In one variant, and It can be the center of their respective blocks.

[0145] At 1870, the normalized HOG ( ) is obtained by merging blocks (1703, 1705). and To obtain the MHOG of block (1702). For example, for and Take the average to produce the MHOG for the current block (1702).

[0146] Standardization relative to HOG distance In another variation, for each HOG involved in the MHOG merged into the current block, the HOG is normalized by its distance from the HOG of the current block. For example, the block is a neighboring block that is not adjacent to the current block.

[0147] The number of blocks involved in obtaining the MHOG can vary with the strategy of selecting and eliminating blocks from the candidate block set.

[0148] For example, search for non-adjacent candidates within a given range around the current block. In another example, non-adjacent candidates (1701) around the current block (1702) can be selected based on their HOG distance to the current block (1702). For example, if the distance between the HOG of a non-adjacent block and the HOG of the current block is less than a given value, the non-adjacent block is selected and added to the list of candidate neighboring blocks; otherwise, the non-adjacent block is not selected.

[0149] Examples of non-adjacent blocks in Figure 17 They are shown as gray squares. Some of them are listed in... Figure 17To the left: for example, (1703) is block number 30, (1704) is block number 27, and (1705) is block number 27. For example, the current block (1702) and non-adjacent blocks (e.g., (1703), (1704) and (1705)) can be CU (coding unit).

[0150] In this variant, Figure 16 The distInfo determined at (1604) is the sum of absolute distances (SAD) between the HOG of each non-adjacent block involved in the merge and the HOG of the current block (1702). In a variant, the other HOG distance could be the Euclidean distance. Figure 17 As shown, (1706) corresponds to the template of the reconstructed pixel of block (1705) (block number 19), (1708) corresponds to the template of the reconstructed pixel of block (1703) (block number 30), and (1707) corresponds to the template of the reconstructed pixel of block (1704) (block number 27).

[0151] Figure 19 An example of a method is provided for determining the merged HOG (MHOG) of the current WxH block (1702) from the two HOGs of two non-adjacent blocks (1703, 1705) according to this variant, where the two non-adjacent blocks (1703, 1705) select DIMD for intra-frame prediction. Some blocks (e.g., (1704)) may select neither conventional DIMD nor DIMD in merged mode for intra-frame prediction.

[0152] Another criterion for selecting non-adjacent blocks could be the spatial distance between the non-adjacent block and the current block in the variation above, or the SAD distance determined between the templates of the non-adjacent block and the current block, or the SAD distance determined between the HOG (determined using reconstruction samples from the templates of the non-adjacent block) and the HOG (determined using reconstruction samples from the templates of the current block). If the SAD distance is too high compared to other SADs from non-adjacent blocks or compared to the absolute value Th, the non-adjacent block is not selected. Figure 17 Above, block (1704) is an example of a block that is not selected.

[0153] One variation could be to use a combination of SAD distance and spatial distance to select non-adjacent blocks involved in the HOG merging of the current block.

[0154] In one variant, the number of non-adjacent blocks can be limited to a given number; for example, N non-adjacent blocks below Th are selected.

[0155] During the encoding / decoding of the current frame to which the current block belongs, at position 1940, the HOG of block (1703) is obtained and represented as HOG. 19For example, HOG template filling from the reconstructed pixel (1704) of block (1703). 19 At position 1930, the HOG of block (1702) is obtained and denoted as HOG. 30 For example, HOG is filled from the template of the reconstructed pixel (1705) of block 30 (1702). 30 At 1920, the HOG of the current block (1702) is filled with the template of the reconstructed pixel (1902) from the current block (1702), and is denoted as HOG0.

[0156] At 1950, by using the HOG of block (1703) 30 Divide each histogram by the HOG distance to normalize the HOG of block (1703). 30 The HOG distance here is the HOG associated with block (1703). 30 The relationship between and HOG0 associated with the current block (1702) For example, here, HOG is far from... It was identified as in 1990. .

[0157] At 1960, by using the HOG of block (1705) 19 Divide each of the square prisms by HOG of normalized block (1705) 19 ,Should It is in the HOG associated with block (1705) 19 It is determined between HOG0 and the current block (1702). For example, here, the HOG distance is... It was identified as in 1980. .

[0158] In one variant, the HOG distance can be the Euclidean distance between the HOGs of non-adjacent blocks and the HOG of the current block. For example, in this variant, the Euclidean distance can be used instead of the SAD.

[0159] At 1970, the normalized HOG values ​​obtained at 1960 and 1950 were merged. and To determine the MHOG of block (1702). For example, and The average can be taken to produce a merged MHOG for block (1702). In a variant, the SAD distance between the most important indices of the HOG can be considered only in the merge. For example, the SAD distance can be considered only in the HOG corresponding to the most important indices. 19 and HOG 30 The calculation is performed on the five higher-amplitude indexes i.

[0160] In another variant, at 1970, the normalized HOG values ​​obtained at 1960 and 1950 are combined. and And determine the MHOG of block (1702) by determining the HOG0 for the current block at 1920.

[0161] HOG is normalized by combining HOG distance and spatial distance.

[0162] In one variant, spatial distance and SAD can be combined to normalize HOG. For example, regarding HOG 19 The normalization can be achieved as follows: , in, It is a weighted scaling factor for spatial distance.

[0163] It is a weighting factor for the SAD distance.

[0164] Combine the first HOG normalization before spatial distance and / or SAD distance normalization.

[0165] In one variant, before performing spatial distance and / or SAD distance normalization, it can be done through the above regarding... Figure 12 The method described is to normalize HOG.

[0166] Smooth the HOG.

[0167] Smooth the HOGs involved in the merger before merging.

[0168] In one variant, the HOG of each non-adjacent block involved in merging the MHOG of the current block can be filtered by a filter before the merging. Any filter that reduces pixel noise in the template of the reconstructed samples reflected in the HOG histogram can be used.

[0169] For example, a Gaussian filter can be used: It is the mean, and it can be equal to 0. This is the variance of the Gaussian filter kernel. The filtering process can then be represented as follows: .

[0170] The LUT of a Gaussian filter can be used to approximate Gaussian(y).

[0171] In another example, a Laplace filter can be used: It is the mean, and it can be equal to 0. It is a proportional parameter. Then, the filtering process can be expressed as follows. .

[0172] The LUT of the Laplace filter can be used to approximate laplace(y).

[0173] Smoothing of MHOG after the merger In one variation, after merging the HOGs of non-adjacent blocks involved in the merge, the MHOG of the current block is generated, and the MHOG can be filtered. Any filter can be used. For example, in the case of using a Gaussian filter, the filtering process can be expressed as follows: .

[0174] Combination of HOG normalization and HOG smoothing In one variant, related to the above about Figure 14-19 Any embodiment of the normalization-related HOG described herein can be directly combined with any embodiment of the smoothing-related HOG described above.

[0175] exist Figure 20 In the embodiment illustrated, within a transmission context between two remote devices A and B on a communication network NET, device A includes a processor associated with RAM and ROM configured to implement methods for encoding images or video according to any embodiment described herein, and device B includes a processor associated with RAM and ROM configured to implement methods for decoding images or video according to any embodiment described herein. According to one example, the network is a broadcast network suitable for broadcasting / transmitting encoded video from device A to a decoding device including device B.

[0176] Figure 21 An example of the syntax for signals transmitted via a packet-based transport protocol is shown. Each transport packet P includes a header H and a payload PAYLOAD. In some embodiments, the payload PAYLOAD may include data representing at least a portion of an image encoded according to any of the embodiments described above. The payload may also include any signaling required for DIMD prediction modes.

[0177] Various implementations involve decoding. As used herein, "decoding" can encompass all or part of a process performed, for example, on a received encoded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also includes, or alternatively includes, processes performed by a decoder of the various implementations described herein, such as entropy decoding of a sequence of binary symbols to reconstruct image or video data.

[0178] As another example, in one embodiment, "decoding" refers only to entropy decoding; in another embodiment, "decoding" refers only to differential decoding; in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding; and in yet another embodiment, "decoding" refers to the entire image reconstruction process including entropy decoding. It will be clear, and is considered well understood by those skilled in the art, whether the phrase "decoding process" is intended to specifically refer to a subset of operations or generally to a broader decoding process, based on the context of the specific description.

[0179] Various implementations involve encoding. In a manner similar to the discussion above regarding “decoding,” the term “encoding,” as used herein, can encompass all or part of a process performed, for example, on an input video sequence to produce an encoded bitstream. In various embodiments, such a process includes one or more processes typically performed by an encoder, such as segmentation, differential coding, transform, quantization, and entropy coding. In various embodiments, such a process also includes, or alternatively includes, processes performed by an encoder of the various implementations described herein, such as determining resampling filter coefficients and resampling the decoded image.

[0180] As another example, in one embodiment, “encoding” refers only to entropy encoding; in another embodiment, “encoding” refers only to differential encoding; and in yet another embodiment, “encoding” refers to a combination of differential and entropy encoding. It will be clear, and is considered well understood by those skilled in the art, whether the phrase “encoding process” is intended to specifically refer to a subset of operations or generally to a broader encoding process, based on the context of the specific description.

[0181] Note that the grammatical elements used in this article are descriptive terms. Therefore, they do not preclude the use of other grammatical element names.

[0182] This disclosure has described various pieces of information that can be transmitted or stored, such as, for example, syntax. This information can be packaged or arranged in various ways, including those common in video standards, such as placing the information in SPS, PPS, NAL units, headers (e.g., NAL unit headers, picture headers, or slice headers), or SEI messages. Other methods are also available, including those common for system-level or application-level standards, such as placing the information in one or more of the following: a. SDP (Session Description Protocol), a format for describing multimedia communication sessions, used for session announcements and invitations (as described in RFCs) and in conjunction with RTP (Real-Time Transport Protocol) transmission.

[0183] b. DASH MPD (Media Presentation Description) descriptors, such as those used in DASH and transmitted via HTTP, are associated with representations or sets of representations to provide additional features to the content representation.

[0184] c. RTP header extensions, such as those used during RTP streaming.

[0185] d. ISO basic media file formats, such as those used in OMAF, and using boxes as object-oriented building blocks defined by unique type identifiers and lengths (also referred to as "atoms" in some specifications).

[0186] e. An HLS (HTTP Live Streaming) manifest transmitted via HTTP. The manifest can be associated with, for example, versions or sets of versions of content to provide characteristics of the version or set of versions.

[0187] When a diagram is presented as a flowchart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0188] Various implementations involve rate-distortion optimization. Specifically, during the encoding process, a balance or trade-off between bit rate and distortion is typically considered, often taking into account computational complexity constraints. Rate-distortion optimization is generally formulated as minimizing a rate-distortion function, which is a weighted sum of bit rate and distortion. Different approaches exist to address the rate-distortion optimization problem. For example, these approaches can be based on extensive testing of all encoding options, including all considered modes or encoding parameter values, and a complete evaluation of their encoding costs and the associated distortion of the reconstructed signal after encoding and decoding. Faster methods can also be used to save encoding complexity, particularly by calculating approximate distortion based on predicting or predicting the residual signal rather than the reconstructed signal. A hybrid of these approaches can also be used, such as by using approximate distortion only for some possible encoding options and full distortion for others. Other methods evaluate only a subset of possible encoding options. More generally, many methods employ any of a variety of techniques to perform optimization, but optimization is not necessarily a complete evaluation of encoding costs and associated distortion.

[0189] The implementations and aspects described herein can be implemented, for example, in a method or process, apparatus, software program, data stream, or signal. Even if discussed only in the context of a single implementation (e.g., discussed only as a method), the implementation of the features in question can be implemented in other forms (e.g., apparatus or program). An apparatus can be implemented, for example, in suitable hardware, software, and firmware. A method can be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. A processor also includes communication devices, such as, for example, a computer, a cellular phone, a portable / personal digital assistant (“PDA”), and other devices that facilitate communication of information between end-users.

[0190] The references to "an embodiment," "an embodiment," "an implementation," or "an implementation," as well as other variations thereof, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the appearance of the phrases "in an embodiment," "in an embodiment," "in an implementation," or "in an implementation," as well as any other variations appearing throughout this application, do not necessarily refer to the same embodiment.

[0191] Additionally, this application may refer to the term "determined" for various pieces of information. Determined information may include, for example, one or more of the following: estimated information, calculated information, predicted information, or information retrieved from memory.

[0192] In addition, this application may refer to the various pieces of information called "access". Accessed information may include one or more of the following: receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0193] Additionally, this application may refer to the term "receive" for various pieces of information. Like "access," "receive" is intended to be a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory) one or more. Furthermore, "receive" is generally referred to in one manner or another during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0194] It should be understood that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one” is intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As another example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). As will be apparent to those skilled in the art and related fields, this can be extended to as many items as listed.

[0195] Furthermore, as used herein, the term "signaling" specifically refers to instructing the corresponding decoder to do something. In this way, in one embodiment, the same parameters are used on both the encoder and decoder sides. Thus, for example, the encoder can transmit (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. Conversely, if the decoder already has specific parameters as well as other parameters, signaling can be used without transmission (implicit signaling) to simply allow the decoder to know and select specific parameters. Bit savings are achieved in various embodiments by avoiding the transmission of any actual functionality. It should be understood that signaling can be done in various ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. Although the verb form of the term "signaling" has been referred to above, the term "signaling" can also be used as a noun herein.

[0196] As will be apparent to those skilled in the art, implementations can generate various signals that are formatted to carry information, such as information that can be stored or transmitted. This information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, signals may be formatted to carry a bitstream of the described embodiments. Such signals may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is well known that signals can be transmitted via a variety of different wired or wireless links. Signals may be stored on a processor-readable medium.

[0197] Many embodiments have been described above. The features of these embodiments may be provided individually or in any combination across various claim classes and types.

Claims

1. A method comprising: For at least one block of an image, at least one first-direction gradient histogram is obtained based on at least one second-direction gradient histogram obtained for neighboring blocks of the at least one block, wherein the at least one second-direction gradient histogram is normalized based on a distance determined between the at least one block and the neighboring blocks. One or more intra-frame prediction modes are used to determine predictions for the at least one block, the one or more intra-frame prediction modes being determined using the at least one first-direction gradient histogram. The determined prediction is used to encode the at least one block.

2. An apparatus comprising one or more processors, said one or more processors being operable to: For at least one block of an image, at least one first-direction gradient histogram is obtained based on at least one second-direction gradient histogram obtained for neighboring blocks of the at least one block, wherein, Based on the distance determined between the at least one block and the neighboring blocks, the at least one second-direction gradient histogram is normalized. One or more intra-frame prediction modes are used to determine predictions for the at least one block, the one or more intra-frame prediction modes being determined using the at least one first-direction gradient histogram. The determined prediction is used to encode the at least one block.

3. A method comprising: For at least one block of an image, at least one first-direction gradient histogram is obtained based on at least one second-direction gradient histogram obtained for neighboring blocks of the at least one block, wherein the at least one second-direction gradient histogram is normalized based on a distance determined between the at least one block and the neighboring blocks. One or more intra-frame prediction modes are used to determine predictions for the at least one block, the one or more intra-frame prediction modes being determined using the at least one first-direction gradient histogram. The determined prediction is used to decode the at least one block.

4. An apparatus comprising one or more processors, said one or more processors being operable to: For at least one block of an image, at least one first-direction gradient histogram is obtained based on at least one second-direction gradient histogram obtained for neighboring blocks of the at least one block, wherein, Based on the distance determined between the at least one block and the neighboring blocks, the at least one second-direction gradient histogram is normalized. One or more intra-frame prediction modes are used to determine predictions for the at least one block, the one or more intra-frame prediction modes being determined using the at least one first-direction gradient histogram. The determined prediction is used to decode the at least one block.

5. The method according to claim 1 or 3, or the apparatus according to claim 2 or 4, wherein, The neighboring block is a block of the image that is not adjacent to the at least one block.

6. The method according to any one of claims 1, 3, or 5, or the apparatus according to any one of claims 2, 4, or 5, wherein, The distance determined between the at least one block and the neighboring block is the spatial distance in the image between the at least one block and the neighboring block.

7. The method according to any one of claims 1, 3, or 5, or the apparatus according to any one of claims 2, 4, or 5, wherein, The distance determined between the at least one block and the neighboring block is the distance determined between a third-direction gradient histogram determined for the at least one block and at least one second-direction gradient histogram obtained for the neighboring block.

8. The method according to any one of claims 1, 3, or 5-7, or the apparatus according to any one of claims 2, 4-7, wherein, The neighboring blocks are encoded using predictions determined using one or more intra-prediction modes, which are determined using the at least one second-direction gradient histogram.

9. The method according to any one of claims 1, 3, or 5-8, or the apparatus according to any one of claims 2, 4-8, wherein, The at least one first directional gradient histogram is obtained by merging the at least one second directional gradient histogram with at least one other directional gradient histogram obtained for at least one other block of the image.

10. The method or apparatus according to claim 9, wherein, Determine the at least one other directional gradient histogram for the at least one block.

11. The method or apparatus according to claim 9 or 10, wherein, The neighboring block is part of a group of neighboring blocks of the at least one block, and the at least one first directional gradient histogram is obtained by merging the at least one second directional gradient histogram with one or more directional gradient histograms obtained for one or more neighboring blocks of the group, respectively.

12. The method according to any one of claims 1, 3, or 5-11, or the apparatus according to any one of claims 2, 4, or 11, wherein, Obtaining the at least one second directional gradient histogram for the neighboring block includes normalizing the at least one second directional gradient histogram based on the number of reference samples in the template of the neighboring block, or based on the number of pixels in the neighboring block, or based on the magnitude of the directional gradient histogram.

13. The method according to any one of claims 1, 3, or 5-12, or the apparatus according to any one of claims 2, 4-12, wherein, Obtaining the at least one first-direction gradient histogram includes filtering the at least one second-direction gradient histogram.

14. The method according to any one of claims 1, 3, or 5-13, or the apparatus according to any one of claims 2, 4-13, wherein, Filter the obtained gradient histogram in at least one first direction.

15. The method according to any one of claims 1, 3, or 5-14, or the apparatus according to any one of claims 2, 4-14, wherein, Each histogram of the at least one first directional gradient histogram is associated with an index of a directional intra-frame prediction mode.

16. A computer program product comprising instructions for causing one or more processors to perform the method of any one of claims 1, 3, or 5-15.

17. A non-transitory computer-readable medium storing executable program instructions that cause a computer executing the program instructions to perform the method of any one of claims 1, 3, or 5-15.

18. An apparatus comprising: The apparatus according to claim 4; as well as At least one of the following: (i) an antenna configured to receive or transmit a signal including data representing the image; (ii) a band limiter configured to limit the signal to a band including data representing the image; or (iii) a display configured to display the image.

19. The device according to claim 18, wherein, The device includes at least one of a television, a cellular phone, a tablet computer, and a set-top box.

Images