Decoder-side intra-mode derivation (DIMD) merge mode

Abstract

The video coder (encoder or decoder) generates a list of merge candidates for the current block in the current frame. The merge candidates in the list include decoder-side intra-mode derivation (DIMD) information associated with blocks spatially adjacent to the current block. The coder further determines indicators for the merge candidates in the list and encodes the current block based on the DIMD information of the merge candidates in the list according to the indicators.

Description

TECHNICAL FIELD

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 542,821 filed on October 6, 2023, which is hereby incorporated by reference in its entirety into the present specification.

BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Several examples among various embodiments of the present disclosure are described in the present specification with reference to the drawings.

[0003] [Figure 1] Figure 1 illustrates an exemplary video encoding / decoding system in which embodiments of the present disclosure may be implemented. [Figure 2] Figure 2 illustrates an exemplary encoder in which embodiments of the present disclosure may be implemented. [Figure 3] Figure 3 illustrates an exemplary decoder in which embodiments of the present disclosure may be implemented. [Figure 4] Figure 4 illustrates an exemplary quadtree partitioning of a coding tree block (CTB) according to an embodiment of the present disclosure. [Figure 5] Figure 5 illustrates the corresponding quadtree for the exemplary quadtree partitioning of the CTB of Figure 4 according to an embodiment of the present disclosure. [Figure 6] Figure 6 illustrates exemplary binary and ternary tree splits according to an embodiment of the present disclosure. [Figure 7] Figure 7 illustrates an example of quadtree plus multi-type tree partitioning of a CTB according to an embodiment of the present disclosure. [Figure 8] Figure 8 illustrates the corresponding quadtree plus multi-type tree partitioning for the exemplary quadtree plus multi-type tree of the CTB of Figure 7 according to an embodiment of the present disclosure. [Figure 9] Figure 9 illustrates an exemplary set of reference samples determined for intra prediction of a current block to be encoded or decoded in accordance with an embodiment of the present disclosure. [Figure 10A]Figure 10A shows 35 intra-predictive modes supported by HEVC according to embodiments of the present disclosure. [Figure 10B] Figure 10B shows 67 intra-prediction modes supported by HEVC according to embodiments of the present disclosure. [Figure 11] Figure 11 shows the current block and reference sample from Figure 9 in the two-dimensional x, y plane according to an embodiment of the present disclosure. [Figure 12] Figure 12 shows an exemplary angular mode prediction of the current block from Figure 9 according to an embodiment of the present disclosure. [Figure 13A] Figure 13A shows an example of interpretation performed on the current block of the current image being encoded according to an embodiment of the present disclosure. [Figure 13B] Figure 13B shows exemplary horizontal and vertical components of a motion vector according to an embodiment of the present disclosure. [Figure 14] Figure 14 shows an example of a dual prediction performed on the current block according to an embodiment of the present disclosure. [Figure 15A] Figure 15A shows exemplary locations of five spatially adjacent candidate blocks to the current block, which are coded according to embodiments of this disclosure. [Figure 15B] Figure 15B shows exemplary locations of two blocks at the same temporal position relative to the current block encoded according to embodiments of this disclosure. [Figure 16] Figure 16 shows an example of an IBC applied to screen content according to an embodiment of the present disclosure. [Figure 17] Figure 17 shows examples of template matching search areas in some embodiments of the present disclosure. [Figure 18] Figure 18 shows examples of decoder-side intra-mode derivation (DIMD) template zones for calculating a gradient (HoG) histogram according to some embodiments of the present disclosure. [Figure 19A]Figures 19A and 19B show flowcharts illustrating exemplary processes for DIMD predictor derivation according to several embodiments of the present disclosure. [Figure 19B] Figures 19A and 19B show flowcharts illustrating exemplary processes for DIMD predictor derivation according to several embodiments of the present disclosure. [Figure 20] Figure 20 shows an example of predictive fusion by a weighted average of two HoG modes and a planar mode according to some embodiments of the present disclosure. [Figure 21] Figure 21 shows examples of embodiments relating to template-based intra-mode derivation (TIMD) according to some embodiments of the present disclosure. [Figure 22] Figure 22 shows a flowchart of an exemplary process for TIMD predictor generation according to several embodiments of the present disclosure. [Figure 23] Figure 23 shows a flowchart illustrating an exemplary process of the DIMD merge mode according to several embodiments of the present disclosure. [Figure 24] Figure 24 shows examples of spatially adjacent neighboring blocks that are considered in the derivation of a list of DIMD merge candidates according to some embodiments of the present disclosure. [Figure 25] Figure 25 shows examples of spatially non-adjacent neighboring blocks used to derive DIMD merge candidates in some embodiments of the present disclosure. [Figure 26] Figure 26 shows a flowchart illustrating an exemplary process for adding DIMD information from spatially adjacent blocks to a history-based DIMD table, according to some embodiments of the present disclosure. [Figure 27] Figure 27 shows a flowchart illustrating an exemplary process for deriving an index to a list of DIMD merge candidates, according to some embodiments of the present disclosure. [Figure 28]FIG. 28 shows a flowchart of a process for generating a list of DIMD merge candidates without duplicate DIMD information and enforcing predetermined thresholds for the maximum list size and maximum number of candidates from each family of candidates, in accordance with some embodiments of the present disclosure. [Figure 29] FIG. 29 shows a flowchart of a method for DIMD merge mode encoding, in accordance with some embodiments of the present disclosure. [Figure 30] FIG. 30 shows a flowchart of a method for DIMD merge mode decoding, in accordance with some embodiments of the present disclosure. [Figure 31] FIG. 31 shows a block diagram of an example computer system in which embodiments of the present disclosure may be implemented. [Description of Embodiments]

[0004] In the following description, numerous specific details are set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure, including structures, systems, and methods, may be practiced without these specific details. The descriptions and representations herein are the common means used by those skilled in the art to most effectively convey the substance of their work to others skilled in the art. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present disclosure.

[0005] References in the present specification to "one embodiment", "an embodiment", "exemplary embodiment" and the like indicate that the described embodiment may include a particular configuration, structure, or characteristic, but not all embodiments necessarily include the particular configuration, structure, or characteristic. Furthermore, such phrases are not necessarily referring to the same embodiment. Further, where a particular configuration, structure, or characteristic is described in connection with one embodiment, it is submitted that affecting such configuration, structure, or characteristic in connection with other embodiments, whether or not explicitly described, is within the scope understood by those skilled in the art.

[0006] It should be noted that individual embodiments may be described as a process illustrated as a flowchart, flow diagram, data flow diagram, structural diagram, or block diagram. While a flowchart may describe operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, and the like. When a process corresponds to a function, its termination may correspond to the return of the function to the calling function or the main function.

[0007] The term “computer-readable medium” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media that can store, contain, or carry one or more instructions and / or data. Computer-readable medium may include non-temporary media that do not contain carrier waves and / or temporary electronic signals propagating over wireless or wired connections, on which data may be stored. Examples of non-temporary media include, but are not limited to, magnetic disks or optical storage media such as tapes, compact discs (CDs) or digital multipurpose discs (DVDs), flash memory, memory, or memory devices. Computer-readable medium may store code and / or machine-executable instructions that may represent procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuits by passing through and / or receiving information, data, arguments, parameters, or memory content. Information, arguments, parameters, data, etc., may be passed, transferred, or transmitted by any preferred means, including memory sharing, message passing, token passing, network transmission, or similar.

[0008] Furthermore, embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented by software, firmware, middleware, or microcode, the program code or code segments for performing the required task (e.g., a computer program product) may be stored on a computer-readable or machine-readable medium. One or more processors may perform the required task.

[0009] Representing a video sequence in digital format can require a large number of bits. The data size of a video sequence in digital format may be too large for storage and / or transmission in many applications. Video encoding can be used to compress the size of a video sequence to provide more efficient storage and / or transmission. Video decoding can be used to decompress a video sequence that has been compressed for viewing and / or other forms of consumption.

[0010] Figure 1 shows an exemplary video coding / decoding system 100 in which embodiments of the present disclosure may be implemented. The video coding / decoding system 100 comprises a source device 102, a transmission medium 104, and a destination device 106. The source device 102 encodes a video sequence 108 into a bitstream 110 for more efficient storage and / or transmission. The source device 102 may store and / or transmit the bitstream 110 to the destination device 106 via the transmission medium 104. The destination device 106 decodes the bitstream 110 to display the video sequence 108. The destination device 106 may receive the bitstream 110 from the source device 102 via the transmission medium 104. The source device 102 and the destination device 106 may be any one of many different devices, such as a desktop computer, laptop computer, tablet computer, smartphone, wearable device, television, camera, video game console, set-top box, or video streaming device.

[0011] To encode the video sequence 108 into a bitstream 110, the source device 102 may include a video source 112, an encoder 114, and an output interface 116. The video source 112 may provide and / or generate the video sequence 108 from imaging of natural scenes and / or synthetically generated scenes. Synthetically generated scenes may include computer-generated graphics or screen content. The video source 112 may include a video capture device (e.g., a video camera), a video archive containing previously captured natural scenes and / or synthetically generated scenes, a video feed interface for receiving natural scenes and / or synthetically generated scenes captured from a video content provider, and / or a processor for generating synthetic scenes.

[0012] As shown in Figure 1, a video sequence, such as video sequence 108, may contain a series of images (also called frames). By presenting the images of the video sequence sequentially using fixed or variable time intervals, the video sequence may create an impression of motion. An image may contain one or more sample arrays of luminance values. Luminance values ​​may be acquired at a series of regularly spaced positions within the image. A color image includes a luminance sample array and two chrominance sample arrays. The luminance sample array may contain luminance values ​​representing the luminance (or luminance component, Y) of the image. The chrominance sample arrays may contain luminance values ​​representing the blue and red components, respectively, of a separate image (or chrominance component, Cb, and Cr). Other color image sample arrays may be based on different color schemes (e.g., red, green, blue (RGB) color schemes). For a color image, a pixel may refer to all three luminance values ​​for a given position in the three sample arrays used to represent the color image. A monochrome image includes a single luminance sample array. In the case of a monochrome image, a pixel represents the luminance value at a given position in a single luminance sample array, which is used to represent the monochrome image.

[0013] The encoder 114 may encode the video sequence 108 into a bitstream 110. To encode the video sequence 108, the encoder 114 may apply one or more prediction techniques to reduce redundancy information in the video sequence 108. Redundancy information is information that can be predicted by the decoder but does not need to be transmitted to the decoder for accurate decoding of the video sequence. For example, the encoder 114 may apply spatial prediction (e.g., intraframe, or intra-prediction), temporal prediction (e.g., in-frame, or inter-prediction), inter-layer prediction, and / or other prediction techniques to reduce redundancy information in the video sequence 108. Before applying one or more prediction techniques, the encoder 114 may divide the image of the video sequence 108 into rectangular regions called blocks. The encoder 114 may then encode the blocks using one or more of the prediction techniques.

[0014] For time prediction, encoder 114 may search for blocks similar to the blocks to be encoded in another image (also called a reference image) of video sequence 108. Then, it may predict the blocks to be encoded using the blocks determined in the search (also called prediction blocks). For spatial prediction, encoder 114 may form prediction blocks based on data from reconstructed adjacent samples of blocks to be encoded within the same image of video sequence 108. The reconstructed samples refer to the encoded and then decoded samples. Encoder 114 may calculate a prediction error (also called a residual) based on the difference between the blocks to be encoded and the prediction blocks. The prediction error may represent non-redundant information that can be sent to the decoder for accurate decoding of the video sequence.

[0015] The encoder 114 may generate conversion coefficients by applying a transformation (e.g., a discrete cosine transform (DCT)) to the prediction error. The encoder 114 may form the bitstream 110 based on the conversion coefficients and other information used to determine the prediction block (e.g., prediction type, motion vector, and prediction mode). In some embodiments, the encoder 114 may perform one or more of the following on the conversion coefficients and / or other information used to determine the prediction block before forming the bitstream 110: quantization and entropy coding, to further reduce the number of bits required to store and / or transmit the video sequence 108.

[0016] The output interface 116 may be configured to write and / or store the bitstream 110 on the transmission medium 104 for transmission to the destination device 106. In an additional or alternative configuration, the output interface 116 may be configured to transmit, upload, and / or stream the bitstream 110 to the destination device 106 via the transmission medium 104. The output interface 116 may include wired and / or wireless transmitters configured to transmit, upload, and / or stream the bitstream 110 in accordance with one or more proprietary and / or standardized communication protocols (e.g., Digital Video Broadcasting (DVB) standards, Advanced Television Systems Commission (ATSC) standards, Integrated Services Digital Broadcasting (ISDB) standards, Data Over Cable Services Interface Specification (DOCSIS) standards, Third Generation Partnership Project (3GPP®) standards, Institute of Electrical and Electronics Engineers (IEEE) standards, Internet Protocol (IP) standards, and Wireless Application Protocol (WAP) standards).

[0017] The transmission medium 104 may include wireless, wired, and / or computer-readable media. For example, the transmission medium 104 may comprise one or more wires, cables, air interfaces, optical disks, flash memory, and / or magnetic memory. As an additional or alternative configuration, the transmission medium 104 may include one or more networks (e.g., the Internet) or file servers configured to store and / or transmit the encoded video data.

[0018] To decode the bitstream 110 into a video sequence 108 for display, the destination device 106 may comprise an input interface 118, a decoder 120, and a video display 122. The input interface 118 may be configured to read the bitstream 110 stored on the transmission medium 104 by the source device 102. In an additional or alternative configuration, the input interface 118 may be configured to receive, download, and / or stream the bitstream 110 from the source device 102 via the transmission medium 104. The input interface 118 may comprise a wired and / or wireless receiver configured to receive, download, and / or stream the bitstream 110 according to one or more proprietary and / or standardized communication protocols (e.g., as referenced herein).

[0019] Decoder 120 may decode video sequence 108 from encoded bitstream 110. To decode video sequence 108, decoder 120 may generate prediction blocks of images of video sequence 108 in a similar manner to encoder 114 and calculate the prediction error for said blocks. Decoder 120 may generate prediction blocks using prediction type, prediction mode, and / or motion vectors received in bitstream 110, and calculate the prediction error using conversion coefficients also received in bitstream 110. Decoder 120 may calculate the prediction error by using the conversion coefficients to weight a conversion base function. Decoder 120 may decode video sequence 108 by combining the prediction blocks and the prediction error. In some embodiments, decoder 120 may decode a video sequence that approximates video sequence 108, for example, due to loss in the compression of video sequence 108 by encoder 114 and / or errors introduced in the encoded bitstream 110 during transmission to destination device 106.

[0020] The video display 122 can display the video sequence 108 to the user. The video display 122 may include a cathode ray tube (CRT) display, a liquid crystal display (LCD), a plasma display, a light-emitting diode (LED) display, or any other display device suitable for displaying the video sequence 108.

[0021] It should be noted that the video encoding / decoding system 100 is presented as an example and is not limiting. In the example shown in Figure 1, the video encoding / decoding system 100 may include other components and / or arrangements. For example, the video source 112 may be outside the source device 102. Similarly, the video display 122 may be outside the destination device 106, or may be omitted entirely if the video sequence is intended for use on a machine and / or storage device. In other examples, the source device 102 may be configured to further include a video decoder, and the destination device 106 may further include a video encoder. In such examples, the source device 102 may be configured to further receive the encoded bitstream from the destination device 106 to support bidirectional video transmission between the devices.

[0022] In the embodiment shown in Figure 1, the encoder 114 and decoder 120 may operate according to one of several proprietary or industrial video coding standards. For example, the encoder 114 and decoder 120 may operate according to one or more of the following: ITU-T H.263, ITU-T H.264, and MPEG-4 Visual (also known as Advanced Video Coding (AVC)), ITU-T H.265, and MPEG-H Part 2 (also known as High Efficiency Video Coding (HEVC)), ITU-T H.265, and MPEG-I Part 3 (also known as Multipurpose Video Coding (VVC)), WebM VP8 and VP9 codes, and AOMedia Video 1 (AV1).

[0023] Figure 2 shows an exemplary encoder 200 in which embodiments of the present disclosure may be implemented. The encoder 200 encodes a video sequence 202 into a bitstream 204 for more efficient storage and / or transmission. The decoder 200 may be implemented in the video coding / decoding system 100 of Figure 1, or in any one of a number of different devices (such as a desktop computer, laptop computer, tablet computer, smartphone, wearable device, television, camera, video game console, set-top box, and / or video streaming device). The encoder 200 comprises an inter-prediction unit 206, an intra-prediction unit 208, combiners 210 and 212, a transform and quantization unit (TR+Q) 214, an inverse transform and quantization unit (iTR+iQ) 216, an entropy coding unit 218, one or more filters 220, and a buffer 222.

[0024] The encoder 200 may divide the images of the video sequence 202 into blocks and encode the video sequence 202 block by block. The encoder 200 may perform prediction techniques on the encoded blocks using either the inter-prediction unit 206 or the intra-prediction unit 208. The inter-prediction unit 206 may perform inter-prediction by searching for blocks similar to blocks encoded in other reconstructed images (also called reference images) of the video sequence 202. Reconstructed images refer to images that have been encoded and decoded. The encoder may then use the blocks determined during the search (also called prediction blocks) to predict which blocks to encode in order to remove redundancy information. The inter-prediction unit 206 may determine prediction blocks by utilizing temporal redundancy or similarity in the scene content between images of the video sequence 202. For example, the scene content between images of the video sequence 202 may be similar except for differences due to the motion of the screen content over time and / or affine transformations.

[0025] The intra-prediction unit 208 may perform intra-prediction by forming prediction blocks based on data from reconstructed adjacent samples of a block encoded within the same image of the video sequence 202. The reconstructed samples refer to the encoded and then decoded samples. The intra-prediction unit 208 may also determine prediction blocks by utilizing spatial redundancy or similarity in the scene content within the image of the video sequence 202. For example, the texture of a region of scene content in an image may be similar to the texture of a region immediately surrounding that region of scene content in the same image.

[0026] After prediction, the combiner 210 may calculate a prediction error (also called a residual) based on the difference between the blocks to be encoded and the predicted blocks. The prediction error may represent non-redundant information that can be sent to the decoder for accurate decoding of the video sequence.

[0027] The transformation and quantization unit 214 may transform and quantize the prediction error. The transformation and quantization unit 214 may transform the prediction error into transformation coefficients by, for example, applying DCT to reduce the correlation information of the prediction error. The transformation and quantization unit 214 may quantize the coefficients by mapping the data of the transformation coefficients to a predetermined set of representative values. The transformation and quantization unit 214 may quantize the coefficients to reduce irrelevant information in the bitstream 204. Irrelevant information refers to information that can be removed from the coefficients after decoding without generating visible and / or perceptible distortion in the video sequence 202.

[0028] The entropy coding unit 218 may further reduce the bitrate by applying one or more entropy coding methods to the quantization conversion coefficients. For example, the entropy coding unit 218 may apply context-adaptive variable-length encoding (CAVLC), context-adaptive binary arithmetic encoding (CABAC), and syntax-based context-adaptive binary arithmetic encoding (SBAC). The entropy-coded coefficients are packaged to form a bitstream 204.

[0029] The inverse transform and quantization unit 216 may dequantize and inverse transform the quantization transform coefficients for calculating the reconstructed prediction error. The combiner 212 may combine the reconstructed prediction error with the prediction block to form a reconstructed block. One or more filters 220 may filter the reconstructed block, for example, using a deblocking filter and / or a sample-adaptive offset (SAO) filter. The buffer 222 may store the reconstructed block for prediction of one or more other blocks in the same and / or different images of the video sequence 202.

[0030] Although not shown in Figure 2, the encoder 200 further includes an encoder control unit configured to control one or more encoders 200 as shown in Figure 2. The encoder control unit may control one or more encoders 200 so that the bitstream 204 is generated in accordance with one of a number of proprietary standards or industry video coding standards. For example, the encoder control unit may control one or more encoders 200 so that the bitstream 204 is generated in accordance with one or more of the ITU-T H.263, AVC, HEVC, VVC, VP8, VP9, ​​and AV1 video coding standards.

[0031] Within the limits of proprietary standards or industry video coding standards, the encoder control unit may be configured to attempt to minimize or reduce the bitrate of bitstream 204 and / or maximize or increase the reconstructed video quality. For example, the encoder control unit may be configured to attempt to minimize or reduce the bitrate of bitstream 204 given a predetermined level below which the reconstructed video quality must not fall, and / or maximize or increase the reconstructed video quality given a level below which the bitrate of bitstream 204 must not exceed. The encoder control unit may determine / control one or more of the following: dividing the image of video sequence 202 into blocks; whether the blocks are inter-predicted by inter-prediction unit 206 or intra-prediction unit 208; motion vectors for inter-prediction of blocks; intra-prediction mode among a plurality of intra-prediction modes for intra-prediction of blocks; filtering performed by one or more filters 220; and one or more transformation types and / or quantization parameters applied by transformation and quantization unit 214. The encoder control unit may determine / control the above based on how the determination / control of the above affects the velocity distortion measurement for the block or image being encoded. The encoder control unit may determine / control the above to reduce the velocity distortion measurement for the block or image being encoded.

[0032] After determination, the prediction type (intra or inter-predict), prediction information for the block (intra-prediction mode, motion vector, etc., in the case of intra-prediction), and / or transformation and / or quantization parameters used to encode the block are sent to the entropy coding unit 218, where they may be further compressed to reduce the bitrate. For example, the entropy decoding unit 218 may apply context-adaptive variable-length encoding (CAVLC), context-adaptive binary arithmetic encoding (CABAC), and syntax-based context-adaptive binary arithmetic encoding (SBAC) to compress the prediction type used to encode the block (intra or inter-prediction), prediction information for the block (intra-prediction mode, motion vector, etc., in the case of intra-prediction), and transformation and quantization parameters. The prediction type, prediction information, and transformation and quantization parameters may be packaged with the prediction error to form the bitstream 204.

[0033] Note that the encoder 200 is presented as an example and is not limiting. In other examples, the decoder 200 may have other components and / or arrangements. For example, one or more of the components shown in Figure 2 may be included in the encoder 200 (e.g., the entropy coding unit 218 and / or one or more filters 220) as an optional configuration.

[0034] Figure 3 shows an exemplary decoder 300 in which embodiments of the present disclosure may be implemented. The decoder 300 decodes a bitstream 302 into a decoded video sequence 304 for display and / or any other form of consumption. The decoder 300 may be implemented in the video coding / decoding system 100 of Figure 1, or in any one of a number of different devices such as a desktop computer, laptop computer, tablet computer, smartphone, wearable device, television, camera, video game console, set-top box, or video streaming device. The decoder 300 comprises an entropy decoding unit 306, an inverse transform and quantization (iTR+iQ) unit 308, a combiner 310, one or more filters 312, a buffer 314, an inter-prediction unit 316, and an intra-prediction unit 318.

[0035] Although not shown in Figure 3, the decoder 300 further comprises a decoder control unit configured to control one or more decoders 300 as shown in Figure 3. The decoder control unit may control one or more decoders 300 so that the bitstream 302 is decoded in accordance with the requirements of one of a number of proprietary or industry video coding standards. For example, the decoder control unit may control one or more decoders 300 so that the bitstream 302 is decoded in accordance with one or more of ITU-T H.263, AVC, HEVC, VVC, VP8, VP9, ​​and AV1.

[0036] The decoder control unit may determine / control one or more of the following: whether a block is interpredicted by the interprediction unit 316 or intrapredicted by the intraprediction unit 318; the motion vector for the interprediction of the block; the intraprediction mode among a plurality of intraprediction modes for the intraprediction of the block; filtering performed by one or more filters 312; and one or more inverse transform types and / or inverse quantization parameters applied by the inverse transform and quantization unit 308. One or more of the control parameters used by the decoder control unit may be packaged in the bitstream 302.

[0037] The entropy decoding unit 306 may entropy decode the bitstream 302. For example, the entropy decoding unit 306 may apply context-adaptive variable-length encoding (CAVLC), context-adaptive binary arithmetic encoding (CABAC), and syntax-based context-adaptive binary arithmetic encoding (SBAC) to decode the prediction type used to code blocks (intra or inter-prediction), prediction information for the blocks (intra-prediction mode, motion vector, etc., in the case of intra-prediction), and transform and quantization parameters. The inverse transform and quantization unit 308 may dequantize and inverse transform the quantization transform coefficients to calculate the decoded prediction error. The combiner 310 may combine the decoded prediction error with the prediction block to form a decoded block. The prediction block may be generated by the intra-prediction unit 318 or the inter-prediction unit 316, as described above with respect to the encoder 200 in Figure 2. One or more filters 312 may filter the decoded blocks, for example, using a deblocking filter and / or a sample-adaptive offset (SAO) filter. Buffer 314 may store the decoded blocks for prediction of one or more other blocks in the same and / or different images of the video sequence of the bitstream 302. The decoded video sequence 304 may be output from the filters 312 as shown in Figure 3.

[0038] It should be noted that the decoder 300 is presented as an example and is not limiting. In other examples, the decoder 300 may have other components and / or configurations. For example, one or more of the components shown in Figure 3 may be included in the decoder 300 as an optional configuration, such as an entropy decoding unit 306 and one or more filters 312.

[0039] Although not shown in Figures 2 and 3, each of the encoder 200 and decoder 300 may further include an intrablock copy unit in addition to the interprediction and intraprediction units. The intrablock copy unit moves similarly to the interprediction unit, but may predict blocks within the same image. For example, the intrablock copy unit may utilize repeating patterns that appear in the screen content. The screen content may include, for example, computer-generated text, graphics, animations, etc.

[0040] As mentioned above, video encoding and decoding may be performed block by block. The process of dividing an image into blocks may be adapted based on the content of the image. For example, to improve coding efficiency, larger block divisions may be used in areas of the image that have a higher level of homogeneity.

[0041] In HEVC, an image can be divided into non-overlapping square blocks called coding tree blocks (CTBs), each containing samples from the sample array. A CTB may have a size of 2n × 2n samples, where n can be specified by the encoding system's parameters. For example, n may be 4, 5, or 6. A CTB can be further divided into semi-vertical and semi-horizontal coding blocks (CBs) by a recursive quadtree split. The CTB forms the root of the quadtree. CBs that are not further divided as part of a recursive quadtree split may be called leaf CBs of the quadtree, or otherwise, non-leaf CBs of the quadtree. CBs may have a minimum size specified by the encoding system's parameters. For example, CBs may have minimum sample sizes of 4 × 4, 8 × 8, 16 × 16, 32 × 32, or 64 × 64. For inter-prediction and intra-prediction, a CB can be further divided into one or more prediction blocks (PBs) for performing inter-prediction and intra-prediction. PB may be a rectangular block of samples to which the same prediction type / mode may be applied. For transformation, CB may be divided into one or more transformation blocks (TB). TB may be a rectangular block of samples to which the applied transformation size can be determined.

[0042] Figure 4 shows an exemplary quadtree partition of CTB400. Figure 5 shows the corresponding quadtree 500 of the exemplary quadtree partition of CTB400 in Figure 4. As shown in Figures 4 and 5, CTB400 is first partitioned into four CBs of semi-vertical and semi-horizontal size. Three of the CBs obtained as a result of the first level partition of CTB400 are leaf CBs. The three leaf CBs of the first level partition of CTB400 are labeled 7, 8, and 9, respectively, in Figures 4 and 5. The non-leaf CBs of the first level partition of CTB400 are partitioned into four sub-CBs of semi-vertical and semi-horizontal size. Three of the sub-CBs obtained as a result of the second level partition of CTB400 are leaf CBs. The three leaf CBs of the second level partition of CTB400 are labeled 0, 5, and 6, respectively, in Figures 4 and 5. Finally, the non-lobed CB of the second level division of CTB400 is divided into four lobed CBs, each half vertical and half horizontal in size. The four lobed CBs are labeled 1, 2, 3, and 4, respectively, in Figures 4 and 5.

[0043] Overall, CTB400 is divided into 10 leaf CBs, each labeled 0 through 9. The resulting quadtree division of CTB400 may be scanned using z-scans (e.g., left to right, top to bottom) to form a sequence order for encoding / decoding the CB leaf nodes. The numbers of each CB leaf node in Figures 4 and 5 may correspond to an encoding / decoding sequence order, with CB leaf node 0 being encoded / decoded first and CB leaf node 9 being encoded / decoded last. Note that, although not shown in Figures 4 and 5, each CB leaf node may contain one or more PBs and / or TBs.

[0044] In VVC, images can be partitioned in a similar manner to HEVC. An image may first be partitioned into non-overlapping square CTBs. The CTBs may then be partitioned into semi-vertical and semi-horizontal CBs using recursive quadtree partitioning. In VVC, quadtree leaf nodes may be further partitioned into unequal CBs by binary or ternary tree partitioning. Figure 6 shows exemplary binary and ternary tree partitioning. A binary tree partition may divide the parent block in half vertically by 602 or horizontally by 604. The resulting partitions may be half the size of the parent block. A ternary tree partition may divide the parent block into three parts vertically by 606 or horizontally by 608. The intermediate partitions may be twice the size of the other two end partitions in the ternary tree partition.

[0045] Because binary and ternary tree partitioning are added, in VCV, the block partitioning strategy may also be called quadtree + multi-tree partitioning. Figure 7 shows an exemplary quadtree + multi-tree partition for CTB700. Figure 8 shows the corresponding quadtree + multi-tree 800 to the exemplary quadtree + multi-tree partition for CTB700 in Figure 7. In both Figures 7 and 8, quadtree partitions are shown with solid lines, and multi-tree partitions are shown with dashed lines. For ease of explanation, CTB700 is illustrated with the same quadtree partition as CTB400 shown in Figure 4. Therefore, the explanation of the quadtree partition for CTB700 is omitted. The explanation of the additional multi-tree partition for CTB700 is for the three leaf-CBs shown in Figure 4, which have been further partitioned using one or more binary and ternary tree partitions. The three leaf-CBs from Figure 4, shown in Figure 7 as further partitioned, are leaf CB5, 8, and 9.

[0046] Starting with leaf-CB5 in Figure 4, Figure 7 shows this leaf-CB divided into two CBs based on vertical binary tree partitioning. The two resulting CBs are the leaf-CBs numbered 5 and 6 in Figures 7 and 8, respectively. With respect to leaf-CB8 in Figure 4, Figure 7 shows this leaf-CB divided into three CBs based on vertical ternary tree partitioning. Two of the three resulting CBs are the leaf-CBs numbered 9 and 14 in Figures 7 and 8, respectively. The remaining non-leaf CB is first divided into two CBs based on horizontal binary tree partitioning, one of which is the leaf-CB numbered 10, and the other is further divided into three CBs based on vertical ternary tree partitioning. The three resulting CBs are the leaf-CBs numbered 11, 12, and 13 in Figures 7 and 8, respectively. Finally, with respect to leaf-CB9 in Figure 4, Figure 7 shows this leaf-CB divided into three CBs based on horizontal ternary tree partitioning. Two of the three CBs are leaf CBs numbered 15 and 19, respectively, in Figures 7 and 8. The remaining non-leaf CB is divided into three CBs based on another horizontal ternary tree partitioning. All three resulting CBs are leaf CBs numbered 16, 17, and 18, respectively, in Figures 7 and 8.

[0047] Overall, CTB700 can be divided into 20 leaf CBs, each numbered 0 through 19. The resulting quadtree + multitree division of CTB700 can be scanned using a z-scan (left to right, top to bottom) to form a sequence order for encoding / decoding the CB leaf nodes. The numbers of each CB leaf node in Figures 7 and 8 may correspond to an encoding / decoding sequence order in which CB leaf node 0 is encoded / decoded first and CB leaf node 19 is encoded / decoded last. Although not shown in Figures 7 and 8, each CB leaf node may contain one or more PBs and / or TBs.

[0048] In addition to specifying various blocks (e.g., CTB, CB, PB, TB), HEVC and VVC further define various units. A block may consist of a rectangular area of ​​samples from a sample array, but a unit may also consist of collocated blocks of samples from different sample arrays (e.g., luminance and chrominance sample arrays) that form an image, as well as syntactic elements of the block and prediction data. A coding tree unit (CTU) may consist of collocated CTBs from different sample arrays that may form a complete entity within the encoded bitstream. A coding unit (CU) may consist of collocated CBs from different sample arrays and syntactic structures used to code samples in a CB. A prediction unit (PU) may consist of collocated PBs from different sample arrays and syntactic elements used to predict a PB. A transformation unit (TU) may consist of TBs from different sample arrays and syntactic elements used to transform a TB.

[0049] It should be noted that the term "block" may be used in the context of HEVC and VVC to refer to any of CTB, CB, PB, TB, CTU, CU, PU, ​​or TU. It should also be noted that the term "block" may be used in the context of other video coding standards to refer to similar data structures. For example, the term "block" may refer to a macroblock in AVC, a macroblock or subblock in VP8, a superblock or subblock in VP9, ​​and / or a superblock or subblock in AV1.

[0050] In intra-prediction, the samples of the block being encoded (also called the current block) can be predicted from the samples in the column immediately adjacent to the leftmost column of the current block, and from the samples in the row immediately adjacent to the top row of the current block. The samples from the immediately adjacent columns and rows can be collectively referred to as reference samples. Each sample in the current block may also be predicted by projecting the position of the sample within the current block onto an arbitrary point along several reference samples in a given direction (also called intra-prediction mode). If the projection does not fall directly onto the reference samples, the sample may be predicted by interpolation between the two nearest reference samples at the projection point. The prediction error (also called residual) may be calculated for the current block based on the difference between the predicted sample value of the current block and the original sample value.

[0051] In an encoder, this process of predicting a sample and calculating a prediction error based on the difference between the predicted sample and the original sample may be performed (for example, in the encoder) for several different intra-prediction modes (e.g., including non-directional intra-prediction modes). The encoder may select one of the several intra-prediction modes and its corresponding prediction error to encode the current block. The encoder may send a display of the selected prediction mode and its corresponding prediction error to the decoder for decoding the current block. The decoder may decode the current block using the intra-prediction mode indicated by the encoder and combining the predicted sample with the prediction error.

[0052] Figure 9 shows an exemplary set of reference samples 902 determined for the intra-prediction of the current block 904 to be encoded or coded. In Figure 9, the current block 904 corresponds to block 3 of the divided CTB700 shown in Figure 7. As described above, the block numbers 0-19 of the divided CTB700 may also correspond to the sequence order for encoding / decoding the blocks, and are used in this way in the example of Figure 9.

[0053] If the current block 904 is a sample of size w×h, then multiple reference samples 902 may include 2w samples from the row immediately adjacent to the top row of the current block 904, 2h samples from the column immediately adjacent to the leftmost column of the current block 904 (or any other amount of samples), and the top-left adjacent corner sample relative to the current block 904. In the example in Figure 9, the current block 904 is a square, and therefore w=h=s. To construct a set of reference samples 902, available samples from adjacent blocks to the current block 904 may be used. Samples may not be available to construct a set of reference samples 902, for example, if the sample lies outside the image of the current block, the sample is part of a different slice of the current block (e.g., if the concept of slices is used), and / or the sample belongs to a block that is intercoded and for which a constrained intra-prediction is shown. If a constrained intra-prediction is shown, the intra-prediction may not depend on the inter-prediction block.

[0054] In addition to the above configuration, samples that are not available to construct the set of reference sample 902 are those that, based on the sequence order for encoding / decoding, contain samples within a block that have not yet been encoded and reconstructed by the encoder or decoded by the decoder. This limitation may make it possible for both the encoder and decoder to determine the same prediction result. In Figure 9, considering that these blocks are encoded and reconstructed by the encoder and decoded by the decoder before the current block 904 is coded, samples from adjacent blocks 0, 1, and 2 may be available to construct the reference sample 902. This assumes there are no other issues, as mentioned above, that prevent the availability of samples from adjacent blocks 0, 1, and 2. However, portions of the reference sample 902 from adjacent block 6 may not be available due to the sequence order for encoding / decoding.

[0055] Unavailable reference samples among multiple reference samples 902 may be filled with available reference samples 902. For example, an unavailable reference sample may be filled with the nearest available reference sample determined by moving clockwise through reference sample 902 from the location of the unavailable reference. If no available reference samples exist, reference sample 902 may be filled with an intermediate value in the dynamic range of the image being coded.

[0056] Furthermore, reference sample 902 may be filtered based on the size of the current block 904 being coded and the intra-prediction mode applied. It should be further noted that Figure 9 shows only one exemplary determination of the reference sample for intra-prediction of a block. In some proprietary and industry video coding standards, the reference sample may be determined in a different manner than that discussed above. For example, in other cases (e.g., for use in VVC), multiple reference lines may be used.

[0057] After a reference sample 902 is determined and filtered as an optional configuration, the samples of the current block 904 may be intra-predicted based on the reference sample 902. Most encoders / decoders support multiple intra-prediction modes according to one or more video coding standards. For example, HEVC supports 35 intra-prediction modes, including planar mode, DC mode, and 33 angular modes. VVC supports 67 intra-prediction modes, including planar mode, DC mode, and 65 angular modes. Planar mode and DC mode may be used to predict smooth, gradually changing regions of an image. Angular mode may be used to predict directional structure in regions of an image.

[0058] Figure 10A shows the 35 intra-prediction modes supported by HEVC. The 35 intra-prediction modes are indicated by indices 0 to 34. Prediction mode 0 corresponds to the planar mode. Prediction mode 1 corresponds to the DC mode. Prediction modes 2 to 34 correspond to the angular mode. Prediction modes 2 to 18 can be called horizontal prediction modes because the primary prediction source is in the horizontal direction. Prediction modes 19 to 34 can be called vertical prediction modes because the primary prediction source is in the vertical direction.

[0059] Figure 10B shows the 67 intra-prediction modes supported by VVC. The 67 intra-prediction modes are identified by indices 0 to 66. Prediction mode 0 corresponds to the planar mode. Prediction mode 1 corresponds to the DC mode. Prediction modes 2 to 66 correspond to the angular modes. Prediction modes 2 to 34 can be called horizontal prediction modes because the primary prediction source is in the horizontal direction. Prediction modes 35 to 66 can be called vertical prediction modes because the primary prediction source is in the vertical direction. Since the VVC blocks do not need to be square, some of the intra-prediction modes shown in Figure 10B can be adaptively replaced by wide-angle directions.

[0060] To further illustrate the application of the intra-prediction mode for determining the prediction of the current block, refer to Figures 11 and 12. In Figure 11, the current block 904 and reference sample 902 from Figure 9 are shown in the two-dimensional x,y plane, and the sample is,

number

number

number

[0061] The reference sample 902 on the left of the current block 904 is a one-dimensional array.

number

number

[0062] In planar mode, the current position within block 904

number

number

number

number

number

[0063] During the ceremony,

number

number

number

number

[0064] In DC mode, the current position within block 904

number

number

number

[0065] Regarding angle mode, the current position within block 904

number

number

number

[0066] Figure 12 shows the current position within block 904 relative to the vertical prediction mode 906 given by angle φ.

number

number

number

number

number

number

number

number

number

number

number

number

[0067] In horizontal prediction mode, the current sample position [x][y] within block 904 is the reference sample.

number

number

number

number

number

number

number

number

number

[0068] The interpolation functions (7) and (10) may be implemented as a set of two-tap finite impulse response (FIR) filters by an encoder or decoder such as encoder 200 in Figure 2 or decoder 300 in Figure 3. The coefficients of the two-tap FIR filters are, respectively,

number

number

number

number

[0069] In one embodiment, a 2-tap interpolation FIR filter may be used to predict a color difference sample. For luminance samples, different interpolation techniques may be used. For example, for luminance samples, a 4-tap FIR filter may be used to calculate the predicted value of the luminance sample. For example, a 4-tap FIR filter, similar to a 2-tap FIR filter,

number

number

number

number

number

number

number

[0070] The encoder may predict the samples that the current block, such as the current block 904, will encode for multiple intra-prediction modes, as described above. For example, the encoder may predict the samples of the current block for each of the 35 intra-prediction modes in HEVC or the 67 intra-prediction modes in VVC. For each intra-prediction mode applied, the encoder may calculate a prediction error for the current block based on the difference between the predicted sample determined for the intra-prediction mode and the original sample of the current block (e.g., the sum of squared differences (SSD), the sum of absolute differences (SAD), or the sum of absolute transform differences (SATD)). Based on the calculated prediction error, the encoder may select one of the intra-prediction modes to encode the current block. For example, the encoder may select the intra-prediction mode that yields the smallest prediction error for the current block. In another example, the encoder may select an intra-prediction mode to encode the current block based on a velocity distortion measurement (e.g., Lagrangian velocity distortion cost) obtained using the prediction error. The encoder may send the decoder a selected prediction mode and its corresponding intra-prediction error for decoding the current block.

[0071] Similar to encoders, decoders may predict, as described above, that samples of the current block, such as the current block 904, will be decoded for an intra-prediction mode. For example, a decoder may receive an indication of the angular intra-prediction mode from an encoder for a given block. The decoder may construct a set of reference samples and perform an intra-prediction based on the intra-prediction mode indicated by the encoder for that block, as described above for the encoder. The decoder reconstructs the block by adding the predicted values ​​of the samples for that block to the residuals of the block. In other examples, a decoder may not receive an indication of the angular intra-prediction mode from an encoder for a given block. Instead, the decoder may determine the intra-prediction mode via other decoder-side means.

[0072] While the above description has been made primarily in relation to intra predictive modes in HEVC and VVC, it will be understood that the techniques of this disclosure, as described above and below, may be applied to other intra predictive modes, including those of other video coding standards such as VP8, VP9, ​​and AV1.

[0073] As described above, intra-prediction can perform video compression by utilizing correlations between spatially adjacent samples of the same image in a video sequence. Inter-prediction is another coding tool that can be used to perform video compression by utilizing temporal domain correlations between sample blocks of different images in a video sequence. Generally, an object may be visible across multiple images in a video sequence. The object may move (e.g., by some translational motion and / or affine motion) or remain stationary across multiple images. Thus, the current block of a sample in the currently encoded image may have a corresponding block in the decoded image's sample that accurately predicts the current block of that sample. The corresponding block of a sample may be displaced from the current block of the sample due to the object represented in both blocks moving across each image of the block. The decoded image may be referred to as a reference image, and the corresponding block of a sample in the reference image may be referred to as a reference block or motion-compensated prediction. The encoder may use block-matching techniques to estimate the displacement (or motion) of an object and determine the reference block in the reference image.

[0074] Similar to intra-prediction, once the prediction for the current block is determined and / or generated using intra-prediction, the encoder may calculate the difference between the current block and the prediction for the current block. This difference may be referred to as the prediction error or residual. The encoder may then store and / or signal the prediction error and other relevant prediction information in the bitstream for decoding and / or other forms of consumption. The decoder may decode the current block by using the relevant prediction information to predict the samples of the current block by combining the predicted samples with the prediction error.

[0075] Figure 13A shows an example of interpretation performed on the current block 1300 in the current image 1302 being encoded. An encoder, such as the encoder 200 shown in Figure 2, may perform interpretation to determine and / or generate a reference block 1304 in a reference image 1306 and predict the current block 1300. The reference image (e.g., reference image 1306) is a decoded image present in the encoder and / or decoder. The presence or absence of a previously decoded image may depend on whether the decoded image is in the decoded image buffer when the current block 1300 is encoded and / or decoded. The encoder may search for one or more reference images for reference blocks similar to the current block 1300, for example. The encoder may determine the “best-matching” block from the blocks tested during the search process as reference block 1304. The encoder may determine that reference block 1304 is the best-matching reference block based on one or more cost criteria, such as a velocity distortion criterion (e.g., Lagrangian velocity distortion cost). One or more cost criteria may be based, for example, on the difference between the predicted sample in reference block 1304 and the original sample in current block 1300 (e.g., the sum of squared differences (SSD), the sum of absolute differences (SAD), or the sum of absolute transformed differences (SATD)).

[0076] The encoder may search for reference block 1304 within search range 1308. Search range 1308 is located around the collocated block (or location) 1310 of the current block 1300 in reference image 1306. In some cases, search range 1308 may extend at least partially outside reference image 1306. If it extends outside reference image 1306, the boundary extension may be constant so that sample values ​​in rows or columns of reference image 1306 immediately adjacent to the portion of search range 1308 extending outside reference image 1306 can be used for the "sample" location outside reference image 1306. All or a subset of potential locations within search range 1308 may be searched for reference block 1304. The encoder may determine and / or generate reference block 1304 using one of several search implementation examples. For example, the encoder may determine a set of candidate search locations based on motion information of adjacent blocks relative to the current block 1300.

[0077] One or more reference images may be retrieved by the encoder during interpretation to determine and / or generate the most matching reference block. The reference images retrieved by the encoder may be included in one or more reference image lists. For example, HEVC and VVC may use two reference image lists, namely reference image list 0 and reference image list 1. A reference image list may consist of one or more images. The reference image 1306 of reference block 1304 may be represented by a reference index that points to a reference image list containing reference image 1306.

[0078] The displacement between reference block 1304 and current block 1300 can be interpreted as an estimate of the motion between reference block 1304 and current block 1300 across each image. The displacement can be represented by a motion vector 1312. For example, motion vector 1312 can be represented by a horizontal component (MVx) and a vertical component (MVy) relative to the position of current block 1300. Figure 13B shows the horizontal and vertical components of motion vector 1312. Motion vectors such as motion vector 1312 can have fractional or integer resolution. Motion vectors with fractional resolution can point between two samples in the reference image to provide a better estimate of the motion of current block 1300. For example, motion vectors may have fractional sample resolutions of 1 / 2, 1 / 4, 1 / 8, 1 / 16, or 1 / 32. If the motion vector points to a non-integer sample value in the reference image, interpolation between two samples at integer positions may be used to generate the reference block and its corresponding sample at fractional positions. Interpolation may be performed by a filter having two or more taps.

[0079] Once a reference block 1304 is determined and / or generated for the current block 1300 using interprediction, the encoder may calculate the difference between the reference block 1304 and the current block 1300 (e.g., the difference between corresponding samples). The difference may be referred to as the prediction error or residual. The encoder may then store and / or signal the prediction error and associated motion information in a bitstream for decoding or other forms of consumption. The motion information may include a motion vector 1312 and a reference index pointing to a reference image list containing a reference image 1306. In other examples, the motion information may include a representation of the motion vector 1312 and a representation of the reference index pointing to a reference image list containing a reference image 1306. The decoder may use the motion information to decode the current block 1300 by determining and / or generating a reference block 1304 that forms the prediction for the current block 1300, combining the prediction with the prediction error.

[0080] In Figure 13A, interpretation is performed using a single reference image 1306 as the source for the prediction of the current block 1300. Because the prediction of the current block 1300 is derived from a single image, this type of interpretation is called unidirectional prediction. Figure 14 shows another type of interpretation called bipretation, which is performed for the current block 1400. In bipretation, the source for the prediction of the current block 1400 is obtained from two photographs. Bipretation can be useful, for example, when a video sequence contains fast motion, camera panning, zooming, or scene changes. Bipretation can also be useful for capturing fade-outs from one scene or from one scene to another, where the two images are displayed simultaneously and effectively at different levels of brightness.

[0081] Whether unidirectional prediction or both unidirectional and bidirectional prediction are available for performing inter-prediction may depend on the slice type of the current block 1400. For P slices, only unidirectional prediction may be available for performing inter-prediction. For B slices, either unidirectional or bidirectional prediction may be used to perform inter-prediction. When performing unidirectional prediction, the encoder may determine and / or generate a reference block from reference list 0 for predicting the current block 1400. When performing bidirectional prediction, the encoder may determine and / or generate a first reference block from reference image list 0 for predicting the current block 1400, and determine and / or generate a second reference block from reference image list 1 for predicting the current block 1400.

[0082] In Figure 14, mutual prediction is performed using double prediction, where two reference blocks 1402 and 1404 are used to predict the current block 1400. Reference block 1402 may be in one reference image from reference image list 0 or reference image list 1, and reference block 1404 may be in another reference image from reference image list 0 or reference image list 1. As shown in Figure 14, reference block 1402 is in an image that precedes the current image of the current block 1400 with respect to the Image Order Number (POC), and reference block 1402 is in an image that advances the current image of the current block 1400 with respect to the POC. In other examples, both reference images may precede or follow the current image in terms of the POC. The POC is, for example, the order in which images are output from a decoded image buffer, and the order in which images are generally intended to be displayed. However, the output images may not necessarily be displayed but may undergo different processing and / or consumption (e.g., transcoding). In several other examples, the two reference blocks determined and / or generated using dual prediction may originate from the same reference image. In these examples, the reference image may be included in both reference image list 0 and reference image list 1.

[0083] Appropriately configurable weights and / or offset values ​​may be applied to one or more interprediction reference blocks. The encoder may enable the use of weighted predictions using flags in the Image Parameter Set (PPS) and may signal the weighting and offset parameters in the slice segment header for the current block. Different weights and / or offset parameters may be signaled for luminance and / or chrominance components.

[0084] Once reference blocks 1402 and 1404 are determined and / or generated using interpretation for the current block 1400, the encoder may calculate the difference between reference blocks 1402 and 1404 and the current block 1400. This difference may be referred to as the prediction error or residual. The encoder may then store and / or signal the prediction error and the respective associated motion information in the bitstream for decoding or other forms of consumption. Motion information for reference block 1402 may include the motion vector 1406 of the reference image containing reference block 1402 and a reference index to the reference image list. In other examples, motion information for reference block 1402 may include a display of the motion vector 1406 and a display of the reference index to the reference image containing reference block 1402. Motion information for reference block 1404 may include the motion vector 1408 of the reference image containing reference block 1404 and a reference index to the reference image list. In other examples, motion information for reference block 1404 may include a display of motion vector 1408 and a display of a reference index to a list of reference images containing reference block 1404. The decoder may decode the current block 1400 by determining and / or generating reference blocks 1402 and 1404, which together form a prediction for the current block 1400 using their respective motion information and by combining the prediction with the prediction error.

[0085] In HEVC, VVC, and other video compression schemes, motion information can be predictively encoded before it is stored or signaled in the bitstream. Motion information for the current block may be predictively encoded based on the motion information of adjacent blocks. Motion information for adjacent blocks can often correlate with the motion information of the current block because the motion of the object represented in the current block is often the same as or similar to the motion of the object in the adjacent block. Two motion information prediction techniques in HEVC and VVC are advanced motion vector prediction (AMVP) and inter-prediction block merging.

[0086] Encoders such as encoder 200 in Figure 2 may encode the motion vector using AMVP as the difference between the motion vector of the current block to be coded and a motion vector predictor (MVP). The encoder may select an MVP from a list of candidate MVPs. Candidate MVPs may originate from the decoded motion vectors of adjacent blocks in the current image of the current block at the collocated position of the current block in other reference images, or in the vicinity of that position. Both the encoder and the decoder may generate and / or determine a list of candidate MVPs.

[0087] After the encoder selects an MVP from a list of candidate MVPs, the encoder may signal the display of the selected MVP and the Motion Vector Difference (MVD) in the bitstream. The encoder may indicate the selected MVP in the bitstream by an index indicating the list of candidate MVPs. The MVD may be calculated based on the difference between the motion vector of the current block and the selected MVP. For example, for a motion vector represented by a horizontal component (MVx) and a vertical displacement (MVy) relative to the current block's position, the MVD may be represented by two components calculated as follows:

number

[0088] In HEVC and VVC, the AMVP candidate MVP list may include two candidates, called Candidate A and Candidate B. Candidates A and B may include up to two spatial candidate MVPs derived from five spatially adjacent blocks of the current block being encoded, one temporal candidate MVP derived from two temporally coexisting blocks that are either unavailable or identical, or a zero motion vector when no spatial, temporal, or both candidates are available. Figure 15A shows the locations of five spatial candidate neighboring blocks relative to the current block 1500 being encoded. The five spatially adjacent candidate blocks are denoted A0, A1, B0, B1, and B2, respectively. Figure 15B shows the locations of two temporally identical blocks relative to the current block 1500 being encoded. The two temporally identical blocks are denoted C0 and C1 and are contained in a reference picture different from the current picture of the current block 1500.

[0089] Encoders such as encoder 200 in Figure 2 may code motion vectors using an interpredictive block merge tool, also known as merge mode. Using merge mode, the encoder may reuse the same motion information from adjacent blocks for interprediction of the current block. Because the same motion information from adjacent blocks is used, there is no need to signal to the MVD, and the signaling overhead for signaling the motion information of the current block may be small. Similar to AMVP, both the encoder and decoder may generate a candidate list of motion information from the adjacent blocks of the current block. The encoder may then decide to use (or inherit) the motion information of one adjacent block in the candidate list to predict the motion information of the current block to be coded. The encoder may signal an index of the motion information from the candidate list in the bitstream. For example, the encoder may signal an index pointing to the list of candidate motion information to indicate the determined motion information.

[0090] In HEVC and VVC, the list of candidate motion information for the merge mode includes up to four spatial merge candidates, which are derived from five adjacent blocks used in AMVP as shown in Figure 15A, one temporal merge candidate derived from two temporally identical blocks used in AMVP as shown in Figure 15B, and additional merge candidates including dual prediction candidates and zero motion vector candidates.

[0091] Interpretation may be implemented in forms or variations other than those described above. For example, motion information prediction techniques other than AMVP and merge mode may also be used. Furthermore, although the above description has been made primarily in relation to interpretation modes in HEVC and VVC, it will be understood that the techniques of this disclosure, as further described above and below, may be applied to other interpretation modes, including those of other video coding standards such as VP8, VP9, ​​and AV1. Also, configurations that implement history-based motion vector prediction (HMVP), combined intra / interpretation mode (CIIP), and merge mode with motion vector difference (MMVD), as described in VVC, may also be within the scope of this disclosure.

[0092] In interpretation, block matching techniques may be applied to determine reference blocks in an image that are different from those in the current block being encoded. Block matching techniques are also applied to determine reference blocks in the same photograph as the current block being encoded. However, for video captured by a camera, reference blocks in the same image as the current block, determined using block matching, often do not accurately predict the current block. This is usually not the case for screen content video, which may include, for example, computer-generated text, graphics, and animations. Screen content video may often contain repeating patterns (e.g., repeating patterns of text and / or graphics) within the same image. Therefore, block matching techniques applied to determine reference blocks in the same image as the current block being encoded can provide efficient compression of screen content video.

[0093] Both HEVC and VVC include predictive techniques to leverage the correlation between blocks of samples within the same image of screen content video. This technique is called intra-block copying (IBC) or current image reference (CPR). Similar to interpretation, the encoder may apply block matching techniques to determine a displacement vector (referred to as a block vector (BV)) that indicates the relative displacement from the current block to a reference block (or intra-block corrected prediction) that "best matches" the current block. The encoder may determine the best-matching reference block from blocks tested during a search process similar to interpretation. The encoder may determine that a reference block is the best-matching reference block based on one or more cost criteria, such as a velocity distortion criterion (e.g., Lagrangian velocity distortion cost). One or more cost criteria may be based on, for example, the difference between the predicted sample of the reference block and the original sample of the current block (e.g., sum of squared differences (SSD), sum of absolute differences (SAD), or sum of absolute transformed differences (SATD)). The reference block may correspond to a decoded block of the sample in the current image. The reference block may contain decoded blocks of the current image sample before being processed by an in-loop filtering operation such as deblocking or SAO filtering. Figure 16 shows an example of an IBC applied to screen content. The rectangular portions with arrows starting at their boundaries are the current blocks to be encoded, and the rectangular portions indicated by the arrows are reference blocks for predicting the current blocks.

[0094] Once a reference block is determined and / or generated for the current block using IBC, the encoder may determine the difference between the reference block and the current block (e.g., the corresponding sample-by-sample difference). This difference may be referred to as the prediction error or residual. The encoder may then store and / or signal the prediction error and associated prediction information in the bitstream for decoding or other forms of consumption. The prediction information may include BV. In other examples, the prediction information may include a representation of BV. A decoder, such as decoder 300 in Figure 3, may decode the current block by using the prediction information and determining and / or generating a reference block that forms a prediction for the current block, combining the prediction with the prediction error.

[0095] In HEVC, VVC, and other video compression schemes, the BV may be predictively coded before being stored or signaled in the bitstream. For example, the BV for the current block may be predictively coded based on the BVs of blocks adjacent to the current block. For example, an encoder may predictively code the BV using the merge mode described above for interprediction, or a technique similar to AMVP described above for interprediction. Techniques similar to AMVP may be referred to as BV prediction and differential coding.

[0096] For BV prediction and differential coding, an encoder, such as encoder 200 in Figure 2, may code the BV as the difference between the BV of the currently coded block and a block vector predictor (BVP). The encoder may select a BVP from a list of candidate BVPs. The candidate BVPs may be derived from the decoded BVs of adjacent blocks to the current block in the current image. Both the encoder and the decoder may generate or determine the list of candidate BVPs.

[0097] After the encoder selects a BVP from a list of candidate BVPs, the encoder may signal the selected BVP and an index of the BV difference (BVD) in the bitstream. The encoder may indicate the selected BVP in the bitstream by an index indicating the list of candidate BVPs. The BVD may be calculated based on the difference between the BV of the current block and the selected BVP. For example, for a BV represented by a horizontal component (BVx) and a vertical component (BVy) relative to the current block being coded, the BVD may be represented by two components which can be calculated as follows:

number

[0098] In HEVC and VVC, the list of candidate BVPs may include two candidates, referred to as candidate A and candidate B. Candidates A and B may include up to two spatial candidate BVPs derived from five spatially adjacent blocks of the current block being encoded, or one or more of the last two encoded BVs when no spatially adjacent candidates are available (for example, because they are coded in intra-mode or inter-mode). For the locations of the five spatial candidate neighbor blocks relative to the current block coded using IBC, see Figure 15A for inter-mode prediction. The five spatial candidate neighbor blocks are denoted as A0, A1, B0, B1, and B2, respectively.

[0099] The Most Probable Mode (MPM) refers to the intra-predictive mode that is most likely to be the best mode for the current block being encoded or decoded. In current intra-predictive techniques, the MPM is determined by analyzing the intra-predictive modes of neighboring CUs. The VVC uses a list of six MPMs (called the MPM list) for luminance intra-predictive mode. The MPM list is derived from the intra-predictive modes of neighboring CUs and is updated as the encoder progresses through video frames. When encoding a block, the encoder may first check whether the current block is a candidate for any of the MPMs in the MPM list. If so, the encoder compares the prediction errors of each MPM to determine which MPM from the MPM list is the best mode for the current block. If the current block is not a candidate for any of the MPMs in the MPM list, the encoder may then evaluate all intra-predictive modes (e.g., 67 in the VVC) to determine the best mode for the current block.

[0100] The use of MPMs can significantly improve the coding efficiency of VVCs. This is because, if the encoder is one of the MPMs, it does not need to signal the intra-predictive mode for the current block. Instead, the decoder can infer the intra-predictive mode for the current block from the corresponding MPM list.

[0101] Three types of intra-modes are considered to construct the MPM list: default intra-mode, neighboring intra-mode, and derived intra-mode. The unified MPM list is used within a block regardless of whether multiple reference line (MRL) and inter-subpartition (ISP) coding tools are applied. The MPM list for the current block is constructed based on the intra-modes of the left neighboring block and the upper neighboring block of the current block. Assuming that the mode of the left neighboring block is indicated as "left" and the mode of the upper neighboring block is indicated as "up," the unified MPM list may be constructed as follows:When a neighboring block is unavailable, its intra-mode is set to planar mode by default, and both the left and above modes are non-angled modes, the MPM list is set to include {planar, DC, V, H, V - 4, V + 4} (where [V] and [H] refer to vertical and horizontal modes, respectively), if one of the left and above modes is an angled mode and the other is non-angled, the mode max is set as the larger mode in left and above, and the MPM list is set to include {planar, Max, Max - 1, Max + 1, Max -- 2, Max + 2}, if both left and above are angled and different, the mode max and mode min are set as the larger mode in left and above and the smaller mode in left and above, respectively, and then if max - min equals 1, the MPM list is set to {planar, Left, Above, Min - 1, Max + 1, Min - Set to include {planar, Left, Above, Min + 1, Max - 1, Min + 2} if Max - Minimum is 62 or greater, set to include {planar, Left, Above, Min + 1, Min - 1, Max + 1} if Max - Minimum is equal to 2, or otherwise set to include {planar, Left, Above, Min - 1, -Min + 1, Max - 1} if both Left and Above are angles and they are the same, set to include {planar, Left, Left - 1, Left + 1, Left - 2, Left + 2}.

[0102] The encoder may encode an MPM index in the bitstream to indicate to the decoder the location of the selected intra-predictive mode in the MPM list. The decoder may derive the MPM list in the same manner as the encoder and use the MPM index obtained from the bitstream to obtain the intra-predictive mode from the MPM list derived by the decoder. In some examples, the first bin of the MPM index codeword is a CABAC context coded to achieve additional coding efficiency. Three contexts are used, corresponding to whether the current intra-block is MRL enabled, ISP enabled, or a normal intra-block.

[0103] During the six MPM list generation processes, duplicate modes can be removed to ensure that only unique modes are included in the MPM list. The truncated binary code (TBC) can be used for entropy coding of the 61 non-MPM modes (i.e., 67 modes of VC minus 6 MPMs).

[0104] In some implementations, the MPM list is expanded to include 16 additional candidates and is divided into two parts: a primary MPM (PMPM) (6 entries) and a secondary MPM (SMPM) (16 entries). In some implementations, the first input in the general MPM list is the planar mode. The remaining entries consist of intra-modes of adjacent neighboring blocks corresponding to left position (L), top position (A), bottom left position (BL), right position (AR), and left position (AL) (shown in Figure 15A as A1, A1, B1, A0, B0, and B2, for example), as well as intra-mode derivation (DIMD) modes on the decoder side, which are sorted in ascending order of cost, for example, SAD, SSD, SATD. Up to five modes with the lowest cost are added to the MPM list. The cost is calculated between the prediction and a reconstructed sample of the current block's template. The sorted directional modes are added to the general MPM list, followed by the default mode, until a general MPM list containing 22 entries is constructed. In some embodiments, if the CU blocks are oriented vertically, the order of adjacent blocks corresponds to A, L, BL, AR, AL; otherwise, it is L, A, AL, AR, BL.

[0105] IntraTemplate Match Prediction (IntraTMP) is a special intra-prediction mode that copies the best predictive block from the reconfigured portion of the current frame, so that its template (e.g., an L-shaped template, a template-only top, or a left-only template) matches the current template (i.e., the L-shaped template, template-only top, or left-only template of the current block). For a predefined search range, the encoder searches the reconfigured portion of the current frame for the template most similar to the current template and uses the corresponding block as the predictive block. SAD cost, or another cost measure (e.g., SSD, SATD, SSE, etc.), may be used as the cost function to evaluate similarity. The encoder may then signal the use of this mode, and the decoder may perform the same predictive motion.

[0106] In some embodiments, the prediction of the current block is generated by matching the L-shaped, top-only, or left-only causal adjacencies of the current block with other blocks within a predetermined search area (e.g., TMP search area). For example, as shown in Figure 17, the search area may include portions of the current CTU (R1), top-left CTU (R2), top CTU (R3), and left CTU (R4).

[0107] Within each region R1-R4 of the IntraTMP search area, the decoder constructs a list of candidate template-matching block vectors up to a predetermined maximum value (e.g., 19), ranked in ascending order according to their template-matching costs (e.g., SAD, SATD, SSD, SSE, etc.). In some embodiments, the following modes are supported: single predictor, fused multiple predictors, subplane precision, and linear filter model. In single predictor mode, a single predictor is selected from the candidate list. In fused multiple predictor mode, multiple predictors are selected and blended to derive a final predicted block. Blending weights may be calculated from the template-matching costs of each predictor or using a Wiener filter-based weight derivation method. In subplane precision mode, when using a single predictor, subplane precision can be 1 / 2 Pell precision, 1 / 4 Pell precision, or 3 / 4 Pell precision, each having eight possible directions. In linear filter model mode, a linear filter may be derived between a reference template and the current template and applied to a reference block. This mode can be used for a single predictor when sub-PEL accuracy is not used.

[0108] In some embodiments, the dimensions of the region, particularly the width and height of the search range (SearchRange_w, SearchRange_h), may be set proportionally to the block dimensions (BlkW, BlkH) to have a fixed number of cost comparisons per pixel. That is: SearchRange_w = min (64, * BlkW); and SearchRange_h = min (64, * BlkH) (where "a" is a constant controlling the gain / complex tradeoff). In practice, "a" may be set to equal 5.

[0109] In some embodiments, to speed up the template matching process, the search range of all search areas may be subsampled by a factor (e.g., factor 3). After finding the best match, a refinement process is performed. Refinement is carried out through a second template matching search that best matches the reduced range.

[0110] In some embodiments, the IntraTMP tool may be enabled for CUs having a width and height of 64 or less. The maximum CU size for IntraTMP can be configured as appropriate. The IntraTMP prediction mode is signaled at the CU level via a dedicated flag if DIMD is not used for the current CU.

[0111] Decoder-side intra-mode derivation (DIMD) is an intra-coding mode in which the intra-predictive mode is not transmitted within the bitstream (after being determined by the encoder, for example, by a velocity-distortion optimization (RDO) algorithm), but is derived by using gradient analysis of adjacent reconstructed pixels. That is, in DIMD, the intra-predictive mode is implicit to the decoder. DIMD is signaled by a flag, and the intra-predictive mode is derived during the reconstruction process at the decoder in the same manner as the intra-predictive mode is derived at the encoder. If DIMD is not selected at the encoder as the intra-coding mode for the current block, the intra-predictive mode may be analyzed from the bitstream at the decoder, similar to an intra-coding mode process that signals the intra-predictive mode within the bitstream.

[0112] In DIMD, a template region (consisting of left, top, and top-left) with three sample widths (width or height) of the current block is defined, where an edge detection filter adjacent to the current block (e.g., a 3x3 horizontal and vertical Sobel filter) is applied to all 3x3 window positions centered on the midline pixels of the template region to determine the amplitude and brightness direction (orientation) angles for each midline sample in the template region. A gradient histogram (HoG) is calculated, with each entry in the HoG corresponding to its respective conventional in-angle mode. For example, G hor and G ver However, if the intensity is purely horizontal and vertical, respectively, then when calculated using a Sobel filter at one 3x3 window position, the window angle is calculated as follows:

number

number

[0113] Figure 18 shows examples of DIMD template regions (e.g., template regions) for calculating HoG according to several embodiments. The reconstruction region 1806 for the current block 1802 is shown similarly to the template 1804 (e.g., L-shaped template) for the current block 1802. An exemplary 4x4 pixel square current block 1808 is shown in the template region as a 3x3 window 1810. HoG 1812 is the histogram corresponding to the current block 1808. Another example is a rectangular current block 1814 with a 3-pixel deep template region 1816. HoG 1818 corresponds to the rectangular current block 1814. As illustrated, each coordinate position on the x-axis of the illustrated HoG is the respective intra-prediction mode (IPM), and the y-axis represents the cumulative amplitude (e.g., count) for each intra-prediction mode.

[0114] Figures 19A and 19B show a flowchart 1900 of an exemplary DIMD predictor derivation process. The operations in flowchart 1900 may be performed mutually by an encoder (e.g., encoder 200) and a decoder (e.g., decoder 300). In 1902, the availability of samples in the template region is determined, for example, in the upper right, lower left, and upper left CUs of the current block. An exemplary template area 1804 of the current block 1802 is shown in Figure 18. In 1904, the HoG of the available template region is constructed. The HoG structure of the template region is described above in relation to Figure 18. In 1906, a maximum predetermined number (e.g., 5) of the most represented intra-prediction modes is identified based on the highest amplitude of the HoG. In 1908, the position dependence of each identified intra-prediction mode is determined as the ratio of amplitudes in the template region. In some implementations, the position dependence of the identified intra-prediction modes can be determined as follows: The default position dependency is set to 0 (i.e., no position dependency). If the amplitude of the intra-prediction mode within the left template (the sum of the amplitudes of the identified intra-prediction mode from the samples within the template region of the current block) is less than the average amplitude of the intra-prediction mode across all templates (e.g., across three template regions: left template, top template, and top-left template), the position dependency is set to 1 (vertical). Otherwise, if the amplitude of the intra-prediction mode within the above template is less than the average amplitude of the intra-prediction mode across all templates, the position dependency is set to 2 (horizontal). This calculation can be facilitated by generating HoGs by separately accumulating the amplitudes for each of the left template regions, above the template region, and above the left template region in each different HoG, and then summing the three HoGs to determine the combined HoG for the entire template region.

[0115] At 1910, it is determined whether mode blending is enabled. If mode blending is disabled, at 1912, it is decided to use one of the identified intra-prediction modes (e.g., the one corresponding to the highest amplitude) as the DIMD predictor. Alternatively, if it is determined at 1910 that mode blending is possible, at 1914, it is determined how many angular intra-prediction modes to blend. Mode blending may be enabled, for example, when both the first and second identified intra-prediction modes are angular modes. At 1916, it is determined whether the number of modes to blend is greater than 1. If the number of modes to blend is 1 or less, at 1918, the DIMD predictor is determined by blending one identified angular intra-prediction mode and a planar mode (e.g., weights 2 / 3 and 1 / 3, respectively). If it is determined at 1916 that the number of modes to blend is greater than 1, at 1920, the blend weights of the identified intra-prediction modes are determined according to their amplitude ratio. A planar mode may be used for blending with a weight of 1 / 4, for example. In 1922, the selected intra-prediction mode is generated as a result of blending the identified intra-prediction modes and planar modes. As described below in relation to Figure 20, the blending of identified intra-prediction modes and planar modes to derive the selected intra-prediction mode for the current block may include calculating all identified intra-prediction modes and planar modes using adjacent reconstructed samples.

[0116] In some implementations, the selected intra-prediction mode determined in 1922 is used as the DIMD predictor for the current block.

[0117] As an alternative configuration, some implementations may further use position-dependent blend modes to derive DIMD predictors. The blend weights determined in 1920 are applied uniformly to all samples of the predictor and may be called uniform weights. For position-dependent blends, for example in 1924, sample-based weights for predictors (e.g., for each identified DIMD intra-prediction mode) are calculated such that the average weight used within a block is approximately equal to the corresponding uniform weight (e.g., calculated in 1920), and higher weights are used for the left template region or for parts of the block closer to the template region, depending on the position dependency of the DIMD intra-prediction mode. The range is predefined, corresponding to the maximum deviation of the sample-based weights from the corresponding uniform weight. A higher value in the given range results in greater weight variability within the block. After the blend weights for each position dependency (e.g., vertical, horizontal, no position dependency) are calculated in 1924, the predictions for each position dependency are blended in 1926. For example, for each of the horizontal, vertical, and non-positional (i.e., diagonal positional) prediction modes used by any of the identified intra-prediction modes of the DIMD predictor, the identified intra-prediction modes with that positional dependency are blended based on their respective amplitudes. In 1928, the DIMD predictor is obtained by blending together each positional dependency predictor (horizontal, vertical, and non-positional) according to the respective sum of the amplitudes of the identified intra-prediction modes in each positional dependency. This provides directional sample-wise blending that depends on the positional dependency and uses fixed weight deviations calculated in 1924.

[0118] In an exemplary implementation, the DIMD intra-prediction mode determined at 1922 or 1928 may be added to the MPM list along with other intra-prediction modes considered for use in the current block. If, then, rate-distortion cost calculations or similar determine, the DIMD intra-prediction mode is the best (i.e., most cost-effective) intra-prediction mode, the current block is encoded using the DIMD intra-prediction mode, and only a flag indicates that the current block is transmitted on the bitstream using the residuals of the current block. The intra-prediction mode is not transmitted on the bitstream.

[0119] In an exemplary implementation, up to a predetermined number (e.g., 4) of angular intra-prediction modes are selected from the HoG and combined with planar modes having weights derived from the ratio of the selected angular intra-prediction amplitudes (e.g., 1 / 4 with 6-bit integer precision, i.e., a plane with a ratio of 16 / 64) (as described, for example, in relation to 1920 or 1928 above). An example of a combination (also called fusion or blend) of angular intra-prediction modes from the HoG is shown in Figure 20. Figure 20 shows a blend of two angular intra-prediction modes and a planar mode. Intra-prediction modes M1 and M2 are selected for the current block 2006 based on the two highest peaks in HoG2004. The predicted block 2008 for the current block 2006 is determined by a combined (fused) motion 2002 that combines predictions based on intra-modes M1 and M2 as well as the planar mode, based on weights w1, w2, and w3. The preceding weights for each angle intra-prediction mode are determined according to the amplitude of the corresponding gradient in the histogram, and a given weight can be used for the planar mode.

[0120] The DIMD process is performed before the MPM list is constructed because the derived intra-mode is included in the MPM list. The primary derived intra-predicted mode of a DIMD block is stored with the block and used to construct the MPM list of adjacent blocks. When DIMD is signaled, the MPM and MRL are assumed to be 0, but no signal is transmitted.

[0121] Template-based intra-mode derivation (TIMD) is a recently proposed intra-coding mode in which a template zone is set within a reconfigured portion of the current frame and used to derive an intra-predictive mode for the current block. In TIMD mode, the intra-predictive mode is not transmitted in the bitstream (after determination by the encoder, e.g., by a velocity distortion optimization algorithm), and only the use of TIMD is signaled, thereby saving intra-predictive mode signaling bits. More specifically, TIMD use is signaled with a flag at the CU level, and the TIMD-derived intra-predictive mode is derived identically in the encoder and decoder during the reconstruction process once TIMD is enabled. If TIMD is not selected by the encoder, the intra-mode may be parsed from the bitstream by the decoder, similar to an intra-mode in which the intra-predictive mode is signaled in the bitstream.

[0122] For each intra-prediction mode in the MPM list, the SATD cost (or another cost such as SAD, SSD, or SSE cost) between the predicted sample and the reconstructed sample of the template is calculated. The intra-prediction mode with the lowest cost is selected as the TIMD mode and used for intra-prediction of the current block. In TIMD, either a template region (consisting of left, top, and top-left regions) with a width of two samples or four samples (width or height) is defined. The reference sample of the template is used to derive the predictor of the template region. Figure 21 shows an example of the current block 2102 of NxN pixels, with a template 2109 of L pixel width and a corresponding reference sample 2106 of the template.

[0123] Figure 22 shows an exemplary flowchart 2200 of the TIMD process, which includes operation 2202 for deriving TIMD modes and weights and operation 2204 for generating TIMD predictors, according to some embodiments. The operations in flowchart 2200 may be performed on an encoder (e.g., encoder 200) and a decoder (e.g., decoder 300). A list of candidate intra-prediction modes is set (e.g., consisting of an MPM list which may be extended mainly to DC, horizontal, and vertical modes), and each candidate mode is checked as an intra-prediction of the template using a cost function (e.g., SAD, SSD, SSE, or SATD) which calculates a template match score between the reconstructed sample in the template and the predicted sample in the template. In 2204, two intra-prediction modes that yield the lowest cost (see 2206, 2208, and 2210) are selected for refinement, where the angular mode range is extended from 67 modes to 131 modes and two adjacent modes (i.e., + / -1) of each selected intra-prediction mode are checked. Next, in step 2212, the selected candidates are chosen as two TIMD modes. Finally, the two TIMD modes are mixed with weights determined from their respective costs to generate a TIMD predictor.

[0124] Another recent intramode prediction technique proposes merging DIMD HoG information from neighboring blocks to predict the current block. If DIMD-coded neighboring blocks are available, the DIMD histograms of selected neighboring CUs are combined to form a merged histogram (MHoG) of gradients for the current block. DIMD modes and weights are derived from this merged histogram. The HoG of the current block is not used in constructing the MHoG.

[0125] The DIMD merge process is similar to the DIMD process described above in relation to Figure 19, and may differ from that process only by including a series of operations preceding 1910. In the proposed DIMD HoG merge process, up to three adjacent CUs are coded as DIMD and selected from up to 13 available adjacent CUs closest to the current block. The HoG or merged HoG of the selected adjacent CUs is obtained. The merged HoG is calculated from the HoG (or merged HoG) of the selected CUs. The mixed mode is determined according to the fused HoG in the same manner as described above in relation to DIMD. Up to five intra-prediction modes are selected as in the normal DIMD mode (e.g., based on the highest amplitude of the merged HoG) and their positional dependencies are determined. If multiple intra-prediction modes are selected, the positional dependencies are set to 0 (diagonal); otherwise, the positional dependencies are set to the positional dependencies of the selected intra-prediction modes. The DIMD HoG merge mode intra-prediction mode for the current block can be determined by combining the selected intra-prediction modes.

[0126] Decoder-side mode derivation is at least partially attractive because it can reduce the amount of information required to send signals from the encoder to the decoder. However, in previous DIMD techniques, DIMD modes and weights are derived from reconstructed neighbor samples (template region) by calculating gradients of a small 3x3 template size. This often does not result in the best adjacencies, as the further the current block sample is from the template region, the less likely it is that there will be a high spatial correlation between the template region and the current block.

[0127] Furthermore, unlike some other intra-modes, DIMD (see Figure 19) does not reuse neighbor information. Moreover, DIMD HoG merging can only be considered if at least one neighbor uses DIMD mode (or DIMD HoG merge mode). As mentioned above, the DIMD process is used by some intra-prediction modes (particularly non-angle modes, to obtain directional information) and during the MPM list derivation process, even when the neighbor mode is not DIMD. Compression efficiency can be improved by using inter-block correlations. As a result, intra-mode efficiency can be further improved by using DIMD information from neighbor CUs.

[0128] Furthermore, in DIMD HoG merge mode, combining HoGs can dilute the relevance of neighboring HoGs because motion is not processed on the HoGs themselves. HoGs are combined without weighting (e.g., deriving the MHoG using only amplitude averaging) or without processing operations (e.g., delamination, densification), thus allowing unrelated neighboring HoGs to distort the MHoG and dilute the peaks of the relevant modes. In fact, spatial correlations may be significant, but the images may contain different spatial regions with distinct features, and therefore may be unrelated.

[0129] These drawbacks can compromise the reliability and competitiveness of DIMD compared to other intra modes.

[0130] Embodiments of this disclosure relate to an improved approach to DIMD that can more reliably obtain a more optimal intra-mode. These and other embodiments of this disclosure are further described below.

[0131] The intra-prediction process described above provides 67 intra-prediction modes (planar, DC, and 65 angular directions) for each prediction block. Luminance intra-prediction modes are selected for luminance prediction blocks, and chrominance intra-prediction modes are selected for chrominance prediction blocks. Mode-dependent references and prediction sample smoothing are applied to increase prediction efficiency, and intra-prediction modes are coded using either one of the six most likely modes (MPMs) or one of the remaining 61 modes. Note that 28 broad angular modes (rectangular CUs only) may be used for regular mode swapping.

[0132] Exemplary embodiments of this disclosure provide a technique for a decoder to efficiently derive an intra-predictive mode to be applied to the current block without sending a signal from the encoder. More specifically, the technique combines the advantages of DIMD and inter-predictive merge modes to obtain a novel, efficient intra-predictive mode, referred to herein as DIMD-merged mode.

[0133] For each CU predicted using the DIMD merge mode, the embodiment includes constructing a list of DIMD information and selecting the CU to be used to reconstruct the current block. This list, called the “list of DIMD merge candidates,” is constructed from DIMD information from various spatial neighbors (neighboring blocks in the current frame). As mentioned above, since DIMD is used to generate the MPM list, DIMD information may already be available for most in-neighbor CUs (i.e., spatially adjacent blocks). In some embodiments, such neighbor CUs may include CUs that have used DIMD HoG merge.

[0134] The initial list of DIMD merge candidates can be constructed according to a given check order (also called the inclusion order) for each DIMD merge candidate family, for example, DIMD information for spatially adjacent candidates, DIMD information for spatially non-adjacent candidates, DIMD information for history-based spatial candidates, DIMD information for combined candidates, and default DIMD information candidates.

[0135] In some embodiments, the list of DIMD merge candidates may be configured to have a predetermined maximum number of DIMD merge candidates, and the generation process can be stopped when the maximum number is reached (i.e., when the list is full). Furthermore, the list generation can ensure that DIMD merge candidates with duplicate DIMD information are not added to the list. The list generation process can also ensure that the number of DIMD merge candidates from each family of DIMD merge candidates does not exceed a predetermined maximum threshold.

[0136] Once the list is complete, a DIMD merge candidate is selected with the best DIMD information for the current block. In the encoder, this selection may be based on the rate-distortion cost of the selected DIMD merge candidate. The selected DIMD merge candidate may be derived in the decoder according to an indicator signaled by the encoder. This indicator indicates which input (i.e., which candidate from the constructed list) of the list of DIMD merge candidates to use to process the block reconstruction. The list of DIMD merge candidates stores all the DIMD information from spatially adjacent blocks necessary to reconstruct the current block, such as the intra-prediction modes to be blended, the corresponding blend weights, and position-dependency information. This DIMD information defines the DIMD merge candidate and may be called the “DIMD parameter”. In some embodiments, additional information, such as HoG information, may be included in the DIMD parameter.

[0137] Figure 23 shows an exemplary flowchart 2300 of a method for a novel intra-prediction mode, DIMD merge mode, according to some embodiments. The method of flowchart 2300 includes DIMD merge information derivation motions 2302 and 2304, which derive the mixed mode, weights, and position dependencies. This method may be carried out by an encoder, such as encoder 200 in Figure 2, or by a decoder, such as decoder 300 in Figure 3.

[0138] This method begins at 2302. The DIMD merge information derivation in 2302 may be configured to include several operations, such as 2306-2316, each of which may add a family of DIMD merge candidates to the list of DIMD merge candidates, obtained from combinations of DIMD information already in the list or from spatially adjacent blocks (i.e., intrablocks) of the current block, selected from the configured default DIMD information, based on different selection criteria. In some embodiments, the DIMD merge candidates in the list of DIMD merge candidates are arranged as a list or a table, but embodiments are not limited to the arrangement structure of the determination of DIMD merge candidates.

[0139] In 2306, DIMD merge candidates from a family of spatially adjacent blocks are considered to add the DIMD merge candidate to the list of DIMD merge candidates. Figure 24 shows adjacent neighbor samples of a block in a family of spatially adjacent blocks. In some embodiments, DIMD merge candidates from spatially adjacent blocks can be added to the list of DIMD merge candidates according to the inclusion order of the blocks corresponding to the left sample (L), upper sample (A), upper-left sample (AL), upper-right sample (AR), and lower-left sample (BL). In some embodiments, a different inclusion order such as L, A, AR, BL, and AL, or yet another different inclusion order, may be used. Blocks in a family of spatially adjacent blocks are also considered in the derivation of the MPM list. In 2306, candidates 1-5 (all spatially adjacent neighbors) may be added to the DIMD merge candidate list. For the DIMD information of a spatial block in a decision included in the DIMD merge candidate list, the block must already have at least the DIMD information derived for that block (i.e., during the reconstruction of the block in the current frame). DIMD information may be derived for blocks of DIMD or another intra-prediction mode using DIMD parameters such as DIMD HoG merge mode, DIMD merge mode, and MPM list derivation. In some embodiments, a merge candidate is added to the list of DIMD merge candidates only if the corresponding neighboring block is coded or decoded based on DIMD or another intra-prediction mode using DIMD parameters. In some examples, for example, if none of the adjacent spatially adjacent blocks are reconstructed using DIMD mode, DIMD HoG merge mode, or DIMD merge mode, or have no derived DIMD information, 2306 may not add any candidates to the list of DIMD merge candidates.

[0140] In step 2308, DIMD merge candidates from a family of blocks that are not spatially adjacent to the current block are added to the list of DIMD merge candidates. The position and inclusion order of non-adjacent DIMD merge candidates within the same frame as the current block may follow a predetermined sequence, for example, as shown in the configuration in Figure 25. The block patterns and arrangements shown in Figure 25 are the same as those defined in some implementations for inter-merge prediction candidates.

[0141] The numbers 1-5 in Figure 25 refer to spatially adjacent parts of the current block (shown in shadow) in inclusion order L, A, AR, BL, and AL, which are either already added to the merge candidate list in 2306 or are otherwise expected to be included, and therefore do not need to be considered in 2308. The sequence of blocks indicated by the numbers 6-23 may be considered for inclusion in the corresponding DIMD information in the DIMD merge candidate list in 2308. As shown in Figure 25, the non-adjacent parts 6-23 that are considered for inclusion in the DIMD information in 2308 are located within the frame along the left horizontal, top vertical, left diagonal, top right diagonal, and bottom right diagonal directions.

[0142] For spatially non-adjacent blocks of a decision included in the list of DIMD merge candidates, the block must have DIMD to be used for encoding / decoding, or at least have derived DIMD information for that block. The DIMD information may be derived for the block for encoding / decoding based on DIMD or another intra-prediction mode using DIMD parameters such as DIMD HoG merge mode, DIMD merge mode, and MPM list derivation. In some embodiments, a candidate is added to the list of DIMD merge candidates only if the corresponding neighboring block is reconstructed based on DIMD or another intra-prediction mode using DIMD parameters. In some examples, for example, if none of the adjacent spatial candidates were coded / decoded using DIMD, DIMD HoG merge, or DIMD merge, or had derived DIMD information, 2308 may not add any candidates to the list of DIMD merge candidates.

[0143] In step 2310, a family of merge candidates from a history-based table of DIMD information is added to the list of DIMD merge candidates. The history-based table may be maintained to contain DIMD information for recently used internal blocks. This history-based table can be updated each time the CU intra prediction mode uses the DIMD process, i.e., when DIMD information is derived (for example, based on the block using DIMD mode, or MPM-based mode, or DIMD, or DIMD HoG merge mode, or the direction derived from DIMD merge mode). The number of times DIMD information from previous blocks is entered into this table is limited, and duplicate DIMD information is not included.

[0144] For DIMD information of spatial blocks of decisions included in a history-based table, the block must have had a DIMD used for encoding / decoding, or at least had DIMD information derived for that block. DIMD information may be derived for blocks for encoding / decoding based on DIMD or another intra-prediction mode using DIMD parameters such as DIMD HoG merge mode, DIMD merge mode, and MPM list derivation. In some embodiments, input of DIMD information is added to the history-based table only if the corresponding neighboring block has been reconstructed based on DIMD or another intra-prediction mode using DIMD parameters. In some embodiments, exemplary embodiments may consider spatially adjacent and spatially non-adjacent blocks separately according to a predetermined inclusion order, so that only DIMD information for spatial blocks that are neither spatially adjacent nor spatially non-adjacent according to a predetermined inclusion pattern is added to the history-based table.

[0145] When inserting new DIMD information into a history-based table, a constrained first-in, first-out (FIFO) rule can be used, where a redundancy check is applied first to determine if the same DIMD information already exists in the table. If found, the entry containing the same DIMD information is removed from the table, the entries in the table move forward in FIFO order, and the new DIMD information is inserted as the last entry. The table may be reset at intervals, for example, at the beginning of each CTU row. Such resets may be performed due to hardware constraints. Such resets may also help ensure that only the most recent neighboring blocks are present in the table. In some embodiments, the history-based table of DIMD information may be similar to the history-based table of motion vectors (HMVP) in the all-block-based advanced motion vector predictor (AMVP) mode for interprediction in VVC.

[0146] Figure 26 is a flowchart 2600 of a method for adding DIMD information to a history-based table according to several embodiments. The operations in flowchart 2600 may be performed mutually by an encoder (e.g., encoder 200) and a decoder (e.g., decoder 300). The method in flowchart 2600 begins at 2602, considering the input of DIMD information for a DIMD merge candidate into a history-based table. Such considerations may arise whenever spatially adjacent blocks use DIMD or other intra-modes from which DIMD information is derived. At 2602, it is determined whether the DIMD information to be added already exists in the table. If it is determined at 2602 that the DIMD information to be added does not exist in the table, at 2606, it may be determined whether the table is full. This determination may be based on a pre-configured maximum size threshold for the history-based table. If the table is not full, at 2610, the DIMD information to be added is added to the history-based table.

[0147] In step 2602, if it is determined that the DIMD information to be added already exists in the table, then in step 2604, the duplicate DIMD information already in the table is removed from the table.

[0148] If the history-based table is determined to be full in step 2606, the oldest entry in the table is removed in step 2608.

[0149] After either 2604 or 2608, at 2612, the inputs following the deleted input in the table are shifted up or moved sequentially (e.g., in a FIFO sequence). After reconfiguring the table entries at 2612, at 2610, the new DIMD information is added as the last entry in the table.

[0150] According to the exemplary inclusion order, DIMD merge candidates derived from the DIMD information in the history-based table are added to the list of DIMD merge candidates after considering DIMD merge candidates derived from spatially adjacent and spatially non-adjacent blocks of the current block.

[0151] Returning to Figure 23, in 2312, a set of combined DIMD merge candidates, according to some embodiments of the present disclosure, is added to the list of DIMD merge candidates.

[0152] Once all spatially adjacent, spatially non-adjacent, and historically-based spatially neighboring potential DIMD merge candidates, or more specifically, their respective DIMD information, are considered for inclusion in the list of DIMD merge candidates, the combined DIMD information may be added to the list of DIMD merge candidates. This process involves combining DIMD information from DIMD information of DIMD merge candidates already in the list of DIMD merge candidates. DIMD merge candidates already in the list, but based on combinations of other DIMD merge candidates in the list, are not used to derive further combined DIMD merge candidates.

[0153] In some examples, a new combined HoG is created by combining HoGs from at least two DIMD merge candidates in a list of DIMD merge candidates, from which the blend mode and weights are derived to obtain the DIMD information of the new DIMD merge candidate. For example, the HoGs of at least two merge candidates are averaged, and the blend mode and weights are derived from this averaged HoG, similar to the derivation of the blend mode in DIMD, as described with respect to Figure 20, for example. Given the new combined DIMD merge candidate, the position dependency may be set to diagonal (0) as a default value, or it may be set according to the blend mode, i.e., the vertical position dependency is set to the vertical blend mode, or the horizontal position dependency is set to the horizontal blend mode.

[0154] The combined DIMD merge candidates may be derived as follows: a candidate derived from the first and second DIMD merge candidates in the list of DIMD merge candidates; a candidate derived from the first and third DIMD merge candidates in the list of DIMD merge candidates; a candidate derived from the second and third DIMD merge candidates in the list of DIMD merge candidates; a candidate derived from the two first DIMD merge candidates in the list of DIMD merge candidates having vertical position dependency; a candidate derived from the two first DIMD merge candidates in the list of DIMD merge candidates having horizontal position dependency; or a candidate derived from combining the DIMD information of all spatially adjacent blocks of the current block.

[0155] Alternatively or additionally, in some embodiments, the output DIMD information of the DIMD HoG merge mode is added as a combined candidate to the list of DIMD merge candidates.

[0156] In some embodiments, merge candidates are not added to the list if they cannot be derived. For example, if there are only two candidates in the list of DIMD merge candidates before adding a combined candidate, the second combined candidate derived from the first and third candidates will be skipped because there is no third candidate.

[0157] In 2314, a default set of DIMD merge candidates is added to the list of DIMD merge candidates, according to some embodiments of the present disclosure.

[0158] One or more default DIMD information entries are added to the list of DIMD merge candidates as DIMD merge candidates. For example, the DIMD parameters of two exemplary default DIMD information entries are shown in Table 1. [Table 1]

[0159] The tables indicate default mode 0 and default mode 1, each specifying a predetermined set of DIMD parameters. One or more DIMD merge candidates having predetermined default DIMD information may be added to the list of DIMD merge candidates based on the DIMD parameters specified in Table 1.

[0160] If available, optionally, adding merge candidates to the default merge candidates completes the addition of merge candidates to the DIMD merge candidate list.

[0161] Motion 2302, which includes 2306-2314 and generates a list of DIMD merge candidates, may be identical in both the encoder and decoder. After 2302 is completed (i.e., the list of DIMD merge candidates is constructed), in 2304, the mixed modes, weights, and position dependencies are derived. In the encoder, this may include determining the best DIMD merge candidate in the list of DIMD merge candidates based on the cost (e.g., SAD, SATD, etc.) difference between the predicted block obtained using the DIMD predictor and the current block, and / or based on the rate-distortion optimization of each DIMD merge candidate starting from the top of the list of DIMD merge candidates (e.g., the lowest SAD, SATD cost). An index to the position of the best (selected) DIMD merge candidate in the list of DIMD merge candidates is determined. Subsequently, in some embodiments, the selected predictor may be added to the MPM list. If the current block is predicted by the DIMD predictor and encoded for transmission based on the DIMD prediction, the DIMD enabled flag and index may be transmitted on the bitstream.

[0162] In the decoder, 2304 may include parsing the bitstream for one or both of the DIMD-enabled flags and / or indices for a list of DIMD merge candidates. The indices can be used to obtain selected DIMD merge candidates from which a DIMD predictor is derived.

[0163] The DIMD merge mode has the advantage of using available neighbor information (e.g., DIMD information derived particularly for spatially adjacent blocks) to better fit the predicted signal characteristics of the current block. Block consistency is introduced by reusing neighbor DIMD parameters. Furthermore, the DIMD merge mode provides less complex and competitive intra-mode derivation. Once an intra-mode is identified, only the signaling element (e.g., an index to a list of DIMD merge candidates) may be signaled. The parameters of the DIMD candidates can be ordered by relevance (e.g., using SAD, SATD, logic) to reduce the mode signaling cost. The reuse of substantial parts of the DIMD process, e.g., low-level operations and computationally intensive operations such as block pixel reconstruction, remains unchanged, thereby saving hardware logic and potentially facilitating adoption by implementers.

[0164] Figure 27 is a flowchart 2700 of a method for determining an index to a list of DIMD merge candidates, according to some embodiments of the present disclosure. This method can be implemented in a decoder, for example, the decoder 300 in Figure 3.

[0165] When considering the DIMD merge mode, an index is signaled or derived that refers to an entry in the list of DIMD merge candidates (which is derived identically on both the encoder and decoder sides). One or more indicators may be signaled from the encoder to the decoder in order to proceed with signaling the DIMD merge index. The DIMD merge enabled flag is a signaling element that indicates whether the DIMD merge index is signaled, and the DIMD merge index is a signaling element that signals the DIMD merge index.

[0166] The DIMD merge index refers to a candidate (also called a selected candidate) in the DIMD merge candidate list used to reconstruct a block pixel, and may be derived in the decoder for the following two cases: If the DIMD merge candidate list size is 1, i.e., there is only one candidate in the list, the index can be derived to DIMD merge index = 0, thus bypassing index signaling. Furthermore, if the DIMD merge enabled flag is signaled but equal to zero, it can be derived that there is no DIMD merge index to extract from the bitstream, and the DIMD merge index is set to its default value, zero.

[0167] In all other cases, the DIMD merge enabled flag is equal to 1, the DIMD merge candidate list size is greater than 1, but the DIMD merge index is decoded from the bitstream. Optionally, the signaling element of the DIMD merge enabled flag may be discarded. In this case, if the DIMD merge candidate list size is greater than 1, i.e., the DIMD merge index is not derived, the DIMD merge index is read from the bitstream.

[0168] After the list of DIMD merge candidates is derived in 2702, the DIMD merge index is initialized to 0 in 2704. In 2706, it is determined whether the list of DIMD merge candidates has multiple entries. If yes, in 2708, it is determined whether the merge index is signaled. If the response in 2708 is yes, in 2710 the DIMD merge index is read from the bitstream. In 2712, the selected DIMD merge candidate is derived by accessing the merge candidate list using the DIMD merge index.

[0169] In step 2706, if it is determined that the merge candidate list size is 1, the method proceeds to step 2712 to derive the DIMD merge candidates selected based on DIMD merge index 0. In step 2708, if it is determined that the DIMD merge index is not signaled, the method proceeds to step 2712 to derive the DIMD merge candidates selected based on DIMD merge index 0.

[0170] According to some embodiments, the DIMD merge mode operation in a decoder, such as decoder 300 in Figure 3, may be initiated when an encoded video bitstream containing encoded video data and associated DIMD information is received. A DIMD enable flag and, optionally, DIMD indices may be parsed from the bitstream. For each encoded video frame, each CTU in the frame is decoded. For each CU of a CTU, if DIMD merge mode is enabled, a list of DIMD merge candidates is constructed. The construction of the list of DIMD merge candidates may include adding DIMD information from spatially adjacent blocks, spatially non-adjacent blocks, from a history-based table, from combinations of DIMD merge candidates, and / or default DIMD information. In some embodiments, rearranging of the merge candidate list may be performed at will. DIMD parameters are derived from the DIMD merge candidate list using indices signaled (if signaled) in the bitstream. DIMD predictors are then generated according to the determined blend weights, modes, and position-dependent states of the predictors.

[0171] According to some exemplary embodiments, the DIMD merge mode operation may be performed for each encoded video frame by an encoder, such as encoder 200 in Figure 2. For each CU within each CTU in the frame, if DIMD is enabled, a list of DIMD merge candidates is constructed. Constructing the list of DIMD merge candidates may include adding DIMD information from spatially adjacent blocks, spatially non-adjacent blocks, from a history-based table, from combinations of DIMD merge candidates, and / or default DIMD information. The best DIMD merge candidate is then selected. The selection may include generating DIMD predictors for each (or some) DIMD merge candidates from the list of DIMD merge candidates according to the determined blend weights, modes, and predictor position-dependent states. Based on each predictor, the cost difference (e.g., SATD, SAD, etc.) between the predicted and reconstructed samples of the template is calculated, and based on the minimum cost, the best DIMD merge candidate is selected and its corresponding index in the list of DIMD merge candidates is determined. In some embodiments, the best DIMD merge candidate is added to a list of candidate intra-modes (e.g., MPM list) considered for the CU. Subsequently, if the best DIMD information is selected from the candidate list for encoding the CU, the index may be sent in the bitstream along with an indication that DIMD merge mode is enabled (for example, the current block is encoded in DIMD merge mode).

[0172] In some embodiments, to improve the relevance of the list of DIMD merge candidates, a potential DIMD merge candidate is added to the list only if its DIMD information differs from all the DIMD information already in the list. For example, if a candidate is derived from the left block (L) and the left block and the lower left block (BL) have the same DIMD parameters, such as blending mode, blending weight, and position dependency, the DIMD information of the BL block is not added to the list of DIMD merge candidates as a new DIMD merge candidate. As another example, if pre-configured default DIMD information has the same DIMD parameters as a candidate in the list of DIMD merge candidates, the pre-configured default DIMD information is not added to the list.

[0173] In some embodiments, the list of DIMD merge candidates is limited to a pre-configured maximum list size. Additionally or alternatively, each set of DIMD merge candidates (e.g., each family such as spatially adjacent blocks, spatially non-adjacent blocks, history-based tables, candidate combinations, and candidates from default DIMD information) may consist of a maximum number of candidate entries.

[0174] Figure 28 shows a flowchart 2800 of a method for generating a DIMD merge candidate list that does not exceed a pre-configured maximum number of DIMD merge candidates in the list as a whole, nor in each family of merge candidates. The generation of the DIMD merge candidate list by the method of flowchart 2800 may be carried out by an encoder such as the encoder 200 shown in Figure 2, or by a decoder such as the decoder 300 shown in Figure 3.

[0175] Flowchart 2800 begins at 2802. At 2802, the coder (e.g., encoder or decoder) is configured to repeat the subsequent operations 2804–2816 for each family of DIMD merge candidates (which may also be referred to herein as a set of merge candidates). For example, as described above in relation to Figure 23, the list of DIMD merge candidates may be constructed by adding merge candidates from several candidate families, such as spatially adjacent neighbor blocks, spatially non-adjacent neighbor blocks, spatial history-based table blocks, combined entries, and default entries. Thus, in some embodiments, 2804–2816 are repeated for each family of DIMD merge candidates.

[0176] In 2804, the coder is configured to repeat 2806-2816 for all DIMD merge candidate entries in the selected family of merge candidate entries. For example, for each selected family of merge candidate families, such as spatially adjacent neighbor blocks, spatially non-adjacent neighbor blocks, spatial history-based table blocks, combined entries, and default entry DIMD information, 2806-2816 may be repeated until the addition of candidate entries from the selected candidate family is complete.

[0177] At 2806, potential candidates from the selected family are identified. For example, if the selected family is spatially adjacent blocks, each of the blocks is identified as L, A, AR. AL and BL shown in Figure 24 may be considered in a predetermined order. Similarly, when the selected family is the spatially non-adjacent blocks, each of the blocks of the patterns and sequences shown in Figure 25 can be considered for potential candidates, or when the selected family is a history-based table, each entry of the history-based table can be regarded as a potential candidate of FIFO, or when the selected family is a family of combined entries, a predetermined set of entries already in the list of DIMD merge candidates as described above can be considered in sequence for potential candidates, and when the selected family is a family of default candidates, each predetermined default candidate can be regarded as a potential candidate.

[0178] At 2808, it is determined whether potential candidates are available. For example, for spatially adjacent blocks and spatially non-adjacent neighboring blocks, it may be determined whether the identified blocks have been reconstructed. In some instances, even if reconstructed, a block may not be considered available if the block was not predicted or reconstructed in DIMD or a DIMD-related prediction mode, or if DIMD information was not previously derived. For combined candidates, it may be determined whether each of a plurality of candidates combined together according to a preconfigured combination pattern (e.g., a combination of entries 1 and 3 in the list) is available. For example, if a predetermined pattern combines the first two entries in the list of DIMD merge candidates, it may be determined whether each of the first two entries is available in the list of DIMD merge candidates. If the currently selected candidate entry is not available, the method returns to 2804 to select the next candidate from the currently selected family, otherwise the method proceeds to 2810.

[0179] At 2810, it is determined whether the DIMD parameters of a potential DIMD merge candidate are already in the list of DIMD merge candidates. For example, it is determined whether there is already a DIMD merge candidate entry in the list such that the DIMD information, more specifically, the set of DIMD parameters defining the DIMD information, is identical to the DIMD information of the potential DIMD merge candidate. If an input with identical DIMD information is not available, the method returns to 2804 to select the next candidate from the currently selected family; otherwise, the method proceeds to 2812.

[0180] At 2812, the neighbor / candidate DIMD information is added to the list of DIMD merge candidates. Adding may include appending a new candidate to the end of the current list of DIMD merge candidates.

[0181] At 2814, it is determined whether the merge candidate list is full. This determination is based on comparing the total number of DIMD merge candidates currently present in the list with a preset maximum number of entries for the list. If the list is full, the generation of the list is completed. Otherwise, the method proceeds to 2816.

[0182] At 2816, it is determined whether the number of DIMD merge candidates added to the list from the family of currently selected potential DIMD merge candidates has reached a predetermined maximum number of entries for the currently selected family. In some embodiments, a separate maximum number of candidates may be defined for each family. In some other embodiments, the same maximum number of entries may be defined for all candidate families. If the number of DIMD merge candidates of the currently selected family does not exceed the predetermined maximum number for that family, the method returns to 2804 to select the next candidate from the selected family. Otherwise, if the number of entries for that family exceeds the predetermined maximum number for that family, the method proceeds to 2802 to select the next family of potential candidates.

[0183] The method of flowchart 2800 provides a way to generate a list of DIMD merge candidates (for example, both in the encoder and decoder) in a manner that limits the number of DIMD merge candidates in the list, and thus limits the amount of complexity increase. By reducing the size of the list, fewer DIMD merge candidates are derived and evaluated, reducing the in-mode complexity in the decoder and encoder. The list is bounded, for example, the list has a maximum of 6 candidates. In this case, a candidate is added to the list only if the number of candidates in the list is less than a preset maximum DIMD merge candidate list size.

[0184] If the list is full, all remaining steps in the list derivation process can be skipped. For example, if the maximum size of the DIMD merge candidate list is 6 and it already contains 6 candidates, e.g., all spatially adjacent candidates (L, A, AL, AR, BL) and one spatially non-adjacent candidate, the list derivation process can be stopped, and other families of candidates that should be considered according to their inclusion order, e.g., non-adjacent candidates, history-based DIMD candidates, combined candidates, and default DIMD candidates, will not be calculated or considered.

[0185] Furthermore, a limit may be set on the maximum number of candidates that can be added to each candidate family. For example, according to one embodiment, all spatially adjacent candidates are considered (e.g., the maximum number of spatially adjacent candidates in a list is 5). The maximum number of non-adjacent candidates can be combined into 4 candidates, the maximum number of history-based candidates is 2, the maximum number of candidate-combined candidates is 4, and the maximum number of default candidates is 2. If the maximum size of the DIMD merge candidate list is 8 and it already contains 2 candidates, e.g., one spatial candidate (e.g., AR), one non-adjacent candidate, and one history-based candidate, then the maximum number of candidates in this candidate family is 4, so a maximum of 4 combined candidates can be added to the list.

[0186] By bounding the list of DIMD merge candidates and discarding duplicate candidates, the list of DIMD merge candidates contains more diverse and relevant DIMD information, increasing the reliability and competitiveness of DIMD.

[0187] One or more of the features of rejecting duplicates, limiting the maximum size of the list of DIMD merge candidates, and / or limiting the maximum number of merge candidates from each family of DIMD merge candidates may be implemented in generating the list of DIMD merge candidates, for example, as shown in the configuration in Figure 23. For example, when each merge candidate is considered for insertion into the list of DIMD merge candidates, one or more of the following can be implemented: rejection of duplicates (e.g., described in relation to 2810), reaching the maximum list size (e.g., described in relation to 2814), and / or reaching the maximum number of merge candidates for the family of merge candidates.

[0188] According to some embodiments, the DIMD merge mode operation in a decoder, such as decoder 300 in Figure 3, may be initiated when an encoded video bitstream containing encoded video data and associated DIMD information is received. A DIMD enable flag and, optionally, DIMD indices may be parsed from the bitstream. For each encoded video frame, each CTU in the frame is decoded. For each CU of a CTU, if DIMD merge mode is enabled, a DIMD merge candidate list is constructed. The construction of the DIMD merge candidate list may include DIMD information from spatially adjacent blocks, spatially non-adjacent blocks, from a history-based table, from a combination of candidate DIMD information entries, and / or the addition of default DIMD information. DIMD parameters are derived from the DIMD merge list using indices signaled (if signaled) in the bitstream. DIMD predictors are then generated according to the determined blend weights, modes, and position-dependent states of the predictors.

[0189] According to some exemplary embodiments, the DIMD merge mode operation may be performed for each encoded video frame by an encoder, such as encoder 200 in Figure 2. For each CU within each CTU in the frame, if DIMD is enabled, a DIMD merge candidate list is constructed. The construction of the DIMD merge candidate list may include adding DIMD information from combinations of candidate DIMD information entries from spatially adjacent blocks, spatially non-adjacent blocks, and history-based tables, and / or default DIMD information. The best DIMD merge candidate is then selected. The selection may include generating DIMD predictors for each (or some) candidate DIMD information input from the DIMD merge candidate list according to the determined blending weights, modes, and predictor position-dependent states. Based on each predictor, the cost difference (e.g., SATD, SAD, etc.) between the predicted and reconstructed samples of the template is calculated, and based on the minimum cost, the best candidate DIMD information entry is selected and the corresponding index in the DIMD merge candidate list is determined. In some embodiments, the best candidate DIMD information input is added to a list of candidate intra-modes to be considered for the CU. Subsequently, if the best DIMD information is selected from the candidate list for encoding the CU, the index may be sent in the bitstream along with an indication that DIMD merge mode is enabled.

[0190] Figure 29 shows a flowchart 2900 of a method for encoding a current block having a DIMD merge mode according to an embodiment of the present disclosure. The method of flowchart 2900 can be implemented by an encoder such as the encoder 200 in Figure 2.

[0191] The method of flowchart 2900 begins at 2902. At 2902, a list of Intra-Mode Derivation (DIMD) merge candidates on the decoder side is generated for the current block in the current frame. The DIMD merge candidates include at least one DIMD merge candidate that contains DIMD information associated with a spatially adjacent block to the current block.

[0192] A DIMD merge candidate may additionally include a second DIMD merge candidate, which includes one or more DIMD blend modes associated with adjacent neighboring blocks of the current block, non-adjacent neighboring blocks in the current frame, another block in the current frame, and a combination of DIMD merge candidates from a list of DIMD merge candidates. As an alternative configuration, a DIMD merge candidate may additionally include a third DIMD merge candidate containing predetermined DIMD information.

[0193] Each DIMD merge candidate may include one or more DIMD intra-modes and one or more corresponding blend weights. Each DIMD merge candidate may further include one or more positional dependencies corresponding to each of the one or more DIMD intra-modes.

[0194] Generating a list of DIMD merge candidates may include adding each DIMD merge candidate to the list of DIMD merge candidates according to the inclusion order. The inclusion order includes DIMD merge candidates associated with adjacent neighboring blocks, preceding DIMD merge candidates associated with non-adjacent neighboring blocks if present, and DIMD merge candidates associated with adjacent neighboring blocks and non-adjacent neighboring blocks if present, and preceding DIMD merge candidates from history-based buffers if present. The inclusion order further includes merge candidates associated with adjacent neighboring blocks, DIMD merge candidates associated with spatially non-adjacent neighboring blocks if present, and DIMD merge candidates from history-based buffers if present, and IMD merge candidates that include combinations of DIMD information from merge candidates already in the list of DIMD merge candidates if present.

[0195] Adding each DIMD merge candidate to the list of DIMD merge candidates in inclusion order involves adding each potential DIMD merge candidate to the list of DIMD merge candidates for each potential DIMD merge candidate from either adjacent neighbor blocks, non-adjacent neighbor blocks, or history-based buffers, prior to the addition: the DIMD information for the potential DIMD merge candidate does not yet exist in the list of DIMD merge candidates; the total number of DIMD merge candidates in the list of DIMD merge candidates is less than a predetermined maximum list limit; and / or the number of DIMD merge candidates currently in the list of DIMD merge candidates is one of the corresponding adjacent neighbor blocks, and the number of non-adjacent neighbor blocks or history-based buffers is less than or equal to a predetermined maximum family limit.

[0196] The list of DIMD merge candidates may include first and third DIMD merge candidates, second and third DIMD merge candidates, two first DIMD merge candidates with vertical position dependency, two first DIMD merge candidates with horizontal position dependency, or DIMD merge candidates derived from a combination of DIMD information of a set of spatially adjacent neighboring blocks.

[0197] The list of DIMD merge candidates may include at least one DIMD merge candidate from each of the spatially adjacent neighboring blocks and the spatially non-adjacent neighboring blocks. The list of DIMD merge candidates may further include at least one DIMD merge candidate from a history-based buffer.

[0198] Adjacent neighboring blocks may contain one or more of the following in relation to the current block: the left block, the top-left block, the top block, the top-right block, and the bottom-left block. DIMD merge candidates associated with the aforementioned adjacent neighboring blocks are added to the list of DIMD merge candidates in the secondary inclusion order of top-left block, top-left block, top block, top-right block, and bottom-left block.

[0199] A non-adjacent neighboring block may include one or more blocks that are further from the current block than any block in an adjacent neighboring block, and that are located in one of the following directions from the current block: diagonally to the lower left, diagonally to the upper right, diagonally to the upper left, upward, or to the right.

[0200] DIMD merge candidates from the history-based buffer include DIMD merge candidates associated with spatially adjacent blocks for which DIMD information has been determined. The history-based buffer may be a first-in, first-out (FIFO) table. After deletion, DIMD information following the deleted DIMD information may move in FIFO order, and DIMD information associated with blocks for which DIMD information has been determined may be added as the last input to the table. Each block with DIMD information in the history-based table is subjected to pre-intra prediction based on the DIMD mode, or the most probable mode (MPM) based mode, or the DIMD histogram (HoG) merge mode, or based on the direction from which the DIMD originates. The table is reset at the beginning of each coding tree unit (CTU) row.

[0201] In 2904, a DIMD merge candidate is selected from a list of DIMD merge candidates. The selection is based on predicting the current block based on the DIMD information of the DIMD merge candidate. DIMD predictors may be generated from the list of DIMD merge candidates for each (or some) DIMD merge candidates, according to the determined blending weights, modes, and position-dependent states of the predictors. Based on each predictor, a cost difference (e.g., SATD, SAD, etc.) between the predicted and reconstructed samples of the template can be calculated, and based on the minimum cost, the best DIMD merge candidate is selected and the corresponding index in the DIMD merge candidate list is determined. In some embodiments, the best DIMD merge candidate is added to an MPM list. Subsequently, if the best DIMD merge candidate is selected from the MPM list to code the current block, the index may be sent in a bitstream along with an indication that the DIMD merge mode is enabled. The selection may also be based on a rate-distortion optimization (RDO) metric of the DIMD merge candidates.

[0202] At 2906, an indicator indicating that the current block is encoded based on the selected DIMD merge candidate is encoded in the bitstream. The indicator may include at least one of a DIMD merge availability flag and an index for the position of the selected DIMD merge candidate in the list of DIMD merge candidates. When the size of the list of DIMD merge candidates is smaller than a threshold size, only the DIMD merge availability flag may be encoded as the indicator.

[0203] FIG. 30 shows a flowchart 3000 of a method for decoding a current block in a DIMD merge mode according to an embodiment of the present disclosure. The method of flowchart 3000 may be implemented by a decoder, such as decoder 300 of FIG. 3.

[0204] The method of flowchart 3000 begins at 3002. At 3002, a list of decoder-side intra mode derivation (DIMD) merge candidates is generated for a current block in a current frame. The generation of the list of DIMD merge candidates at 3002 may be identical to the generation of the list of DIMD merge candidates at 2902 described above with respect to the method of flowchart 2900.

[0205] At 3004, an indicator of a DIMD merge candidate in the list of DIMD merge candidates is decoded from the bitstream. The indicator may include a flag indicating whether an index for the list of DIMD merge candidates is signaled in the bitstream.

[0206] In 3006, the current block is decoded based on the DIMD information of a DIMD merge candidate selected from a list of DIMD merge candidates according to an indicator. Based on a flag indicating that the index is signaled, the index to the list of DIMD merge candidates may be decoded from the bitstream. Decoding the current block based on the DIMD information of the selected DIMD merge candidate may include generating a predicted block of the current block based on the DIMD information and reconstructing the current block based on the predicted block and the residuals of the current block obtained from the bitstream.

[0207] Embodiments of the present disclosure may be implemented in software using analog and / or digital circuits, through the execution of instructions by one or more general-purpose or decision-making processors, or in hardware as a combination of hardware and software. As a result, embodiments of the present disclosure may be implemented in the environment of a computer system or other processing system. An example of such a computer system 3100 is shown in Figure 31. Blocks shown in the above figures, such as the blocks in Figures 1, 2, and 3, may be executed on one or more computer systems 3100. Furthermore, each step of a flowchart illustrated in the present disclosure may be implemented on one or more computer systems 3100.

[0208] The computer system 3100 includes one or more processors, such as a processor 3104. The processor 3104 may be, for example, a decision-making processor, a general-purpose processor, a microprocessor, or a digital signaling processor. The processor 3104 may be connected to a communication infrastructure 3102 (e.g., a bus or network). The computer system 3100 may also be configured to include main memory 3106, such as random access memory (RAM), and may also include secondary memory 3108.

[0209] The secondary memory 3108 may include, for example, a hard disk drive 3110 and / or a removable storage drive 3112, representing a magnetic tape drive, an optical disk drive, or the like. The removable storage drive 3112 may be read from and / or written to by the removable storage unit 3116 in a known manner. The removable storage unit 3116 represents a magnetic tape, optical disk, or the like that is read from and written to by the removable storage drive 3112. As will be understood by those skilled in one or more related technologies, the removable storage unit 3116 includes a computer-usable storage medium in which computer software and / or data are stored.

[0210] In alternative implementations, the secondary memory 3108 may include other similar means for enabling the loading of computer programs or other instructions into the computer system 3100. Such means may include, for example, a removable storage unit 3118 and an interface 3114. Examples of such means include a program cartridge and cartridge interface (such as those found in video game devices), a removable memory chip (such as an EPROM or PROM) and associated sockets, thumb drives, and USB ports, as well as other removable storage units 3118 and interfaces 3114 that enable the transmission of software and data from the removable storage unit 3118 to the computer system 3100.

[0211] The computer system 3100 may also include a communication interface 3120. The communication interface 3120 enables the transmission of software and data between the computer system 3100 and external devices. Examples of the communication interface 3120 include modems, network interfaces (such as Ethernet® cards), and communication ports. The software and data transmitted through the communication interface 3120 are in the form of signals, which may be electronic, electromagnetic, optical, or other signals that can be received by the communication interface 3120. These signals are provided to the communication interface 3120 via a communication path 3122. The communication path 3122 transmits signals and may be implemented using wires, or cables, optical fibers, telephone lines, cell phone links, RF links, and other communication channels.

[0212] As used herein, the terms “computer program medium” and “computer-readable medium” are used to refer to tangible storage media, such as removable storage units 3116 and 3118, or hard disks mounted on hard disk drive 3110. These computer program products are means for providing software to computer system 3100. Computer programs (also called computer control logic) may be stored in main memory 3106 and / or secondary memory 3108. Computer programs may also be received via communication interface 3120. When executed, such computer programs enable computer system 3100 to implement the disclosure as discussed herein. In particular, when executed, computer programs enable processor 3104 to implement the processes of the disclosure, such as any of the methods described herein. Thus, such computer programs represent the controller of computer system 3100.

[0213] In another embodiment, the configuration of the Disclosure may be implemented in hardware using hardware components such as, for example, an integrated circuit (ASIC) for decision applications and a gate array. Implementations of a hardware state machine for performing the functions described herein will also be apparent to those skilled in the art.

Claims

1. It is a method, To generate a list of merge candidates for the current block in the current frame, wherein the merge candidates in the list include decoder-side intra-mode derivation (DIMD) information associated with spatially adjacent blocks to the current block, and the list is A first merge candidate including one or more DIMD blend modes associated with a combination of merge candidates from the list of merge candidates, an adjacent neighboring block of the current block, a non-adjacent neighboring block in the current frame, another block in the current frame, or To generate a second merge candidate that includes the specified DIMD information, Determining the indicator for the merge candidate in the list of merge candidates, A method comprising coding the current block based on the DIMD information of the merge candidate selected from the list of merge candidates according to the indicator.

2. It is a method, To generate a list of merge candidates for the current block in the current frame, wherein the merge candidates in the list include decoder-side intra-mode derivation (DIMD) information associated with spatially adjacent blocks to the current block. Determining the indicator for the merge candidate in the list of merge candidates, A method comprising coding the current block based on the DIMD information of the merge candidate in the list of merge candidates, according to the indicator.

3. The merge candidates are, A first merge candidate including one or more DIMD blend modes associated with a combination of merge candidates from the list of merge candidates, an adjacent neighboring block of the current block, a non-adjacent neighboring block in the current frame, another block in the current frame, or The method of claim 2, comprising a second merge candidate containing predetermined DIMD information.

4. The method according to any one of claims 1 to 3, wherein each merge candidate of the merge candidates includes one or more DIMD intra modes and one or more corresponding blend weights.

5. The method according to any one of claims 1 to 4, wherein each merge candidate further includes one or more positional dependencies corresponding to one or more DIMD intra modes.

6. The method according to any one of claims 1 to 5, further comprising selecting the merge candidate from the list of merge candidates according to the indicator.

7. The method according to any one of claims 1 to 6, wherein the indicator includes a flag indicating the use of a DIMD merge mode in the current block, and when the flag is set, selecting the merge candidate from the list of merge candidates according to the indicator includes selecting the merge candidate from a predetermined position in the list of merge candidates.

8. The method of claim 7, wherein the indicator further includes an index, and when the flag is set, selecting the merge candidate from the list of merge candidates according to the indicator further includes selecting the merge candidate in the list of merge candidates according to the index.

9. Generating a list of merge candidates includes adding each merge candidate to the list of merge candidates in order of inclusion, and the addition is Before adding any merge candidates associated with non-adjacent neighboring blocks and any merge candidates from the history-based buffer, add one or more merge candidates associated with adjacent neighboring blocks. Before adding any merge candidates from the history-based buffer, add one or more merge candidates associated with the non-adjacent neighboring blocks, or The method according to any one of claims 1 to 9, comprising one or more of the following: adding merge candidates from the history-based buffer.

10. The method of claim 9, wherein adding each merge candidate to the list of merge candidates in inclusion order is performed for each potential merge candidate from any one of adjacent neighbor blocks, non-adjacent neighbor blocks, or history-based buffers, on the basis of a set of conditions that are met before the addition, The DIMD information of the aforementioned potential merge candidate is not yet present in the aforementioned list of merge candidates. The total number of merge candidates in the aforementioned list of merge candidates is less than a predetermined maximum list limit, A method comprising: the number of merge candidates currently in the list of merge candidates; the corresponding one of the adjacent neighbor blocks, the non-adjacent neighbor blocks, or the history-based buffer being less than or equal to a predetermined maximum family limit.

11. The method according to claim 9 or 10, wherein the adjacent neighboring block includes one or more of the left block, upper left block, upper block, upper right block, and lower left block in relation to the current block.

12. The method according to any one of claims 9 to 11, wherein the merge candidates associated with the adjacent neighboring blocks are added to the list of merge candidates in a secondary inclusion order of left block, upper left block, upper block, upper right block, and lower left block.

13. The method according to any one of claims 9 to 12, wherein the non-adjacent neighboring block is further from the current block than any block in the adjacent neighboring block, and includes one or more blocks located in any of the following directions from the current block: diagonally to the lower left, diagonally to the upper right, diagonally to the upper left, upward, and to the right.

14. The method according to any one of claims 9 to 13, wherein the merge candidates from the history-based buffer include merge candidates associated with spatially adjacent blocks for which DIMD information has been determined.

15. The method according to any one of claims 9 to 14, wherein the history-based buffer is a first-in, first-out (FIFO) table.

16. A method according to any one of claims 14 and 15, wherein before adding the DIMD information associated with a block for which the DIMD information has been determined to the table, To determine whether the DIMD information associated with the block is the same as the DIMD information in the table, A method further comprising: removing the DIMD information in the table based on the fact that the DIMD information associated with the block is identical to the DIMD information in the table.

17. The method according to claim 16, further comprising: moving the DIMD information following the removed DIMD information upward in FIFO order after the removal; and adding the DIMD information associated with the block which has been determined to be the last input to the table.

18. The method according to any one of claims 15 to 17, wherein each block having DIMD information in the table is subjected to intraprediction in advance based on the DIMD mode, the most likely mode (MPM) based mode, or the DIMD histogram (HoG) merge mode, or based on the direction originating from DIMD.

19. The method according to any one of claims 15 to 18, wherein the table is reset to the beginning of each coding tree unit (CTU) row.

20. The method according to any one of claims 9 to 19, wherein adding each merge candidate to the list of merge candidates in order of inclusion further comprises adding a combination of DIMD information from merge candidates already present in the list of merge candidates.

21. The method according to claim 20, wherein the merge candidates, which include a combination of DIMD information, include, in the list of merge candidates, first and third merge candidates, second and third DIMD merge candidates, two first merge candidates having vertical position dependency, two first merge candidates having horizontal position dependency, or merge candidates derived from a combination of DIMD information of a set of spatially adjacent neighboring blocks.

22. The method according to any one of claims 9 to 21, wherein the list of merge candidates includes at least one merge candidate from each of spatially adjacent neighboring blocks and spatially non-adjacent neighboring blocks.

23. The method according to any one of claims 9 to 22, wherein the list of merge candidates further comprises at least one merge candidate from the history-based buffer.

24. The method according to any one of claims 1 to 23, wherein the indicator includes a flag indicating whether an index of the merge candidates to the list is signaled in the bitstream.

25. The method according to claim 24, wherein the flag is a DIMD merge enabled flag.

26. The method according to any one of claims 24 to 25, further comprising decoding the index for the list of merge candidates based on the flag indicating that the index is signaled.

27. The current block is coded based on the DIMD information of the merge candidate. Based on the DIMD information, a predicted block for the current block is generated, The method according to any one of claims 1 to 26, comprising: reconstructing the current block based on the predicted block and the residual of the current block obtained from the bitstream.

28. The method according to any one of claims 1 to 23, wherein coding the current block based on the DIMD information of the merge candidate includes encoding an indicator in the bitstream that the current block is encoded based on the merge candidate.

29. Determining the indicator of the merge candidate in the list of DIMD merge candidates is Based on the DIMD information of the merge candidates, the merge candidates are selected from the list of merge candidates based on predicting the current block, The method according to any one of claims 1 to 23, comprising determining the indicator based on the selected merge candidate.

30. The method according to claim 29, wherein the indicator includes at least one of a merge enable flag and an index to the position of the merge candidate in the list of merge candidates.

31. The method according to any one of claims 29 to 30, further comprising selecting based on a rate distortion optimization (RDO) metric of the merge candidates.

32. The method according to any one of claims 28 to 31, wherein the encoding includes encoding only the DIMD merge enable flag as the indicator when the size of the list of merge candidates is less than a threshold size.

33. It is an encoder, One or more processors, An encoder comprising: a memory for storing instructions, wherein, when the instructions are executed by the one or more processors, the memory causes the encoder to execute the method according to any one of claims 1 to 23 and 28 to 32.

34. It is a decoder, One or more processors, A decoder comprising: a memory for storing instructions, wherein the instructions, when executed by one or more processors, cause the decoder to execute the method according to any one of claims 1 to 27.

35. A non-temporary computer-readable medium containing instructions, wherein, when executed by one or more processors of the device, the instructions cause the device to perform the method according to any one of claims 1 to 32.

Images