Decoder-side intra-mode derivation merge mode candidate sorting

Abstract

The video coder (encoder, or decoder) sorts the list of merge candidates. The list of merge candidates includes decoder-side intra-mode derivation (DIMD) information, or template-based intra-mode derivation (TIMD) information, associated with spatially adjacent blocks to the current block within the current frame. The video coder determines the indicators of the merge candidates in the list and, according to the indicators, codes the current block based on the DIMD information of the merge candidates in the list.

Description

[Technical Field]

[0001] (Cross-reference of related applications) This application claims the interests of U.S. Provisional Patent Application No. 63 / 543,217, filed on 9 October 2023, which is incorporated herein by reference in its entirety. [Brief explanation of the drawing]

[0002] Some of the various embodiments of this disclosure are described herein with reference to the drawings.

[0003] [Figure 1] Figure 1 shows an exemplary video coding / decoding system in which embodiments of the present disclosure may be implemented. [Figure 2] Figure 2 shows an exemplary encoder in which an embodiment of the present disclosure may be implemented. [Figure 3] Figure 3 shows an exemplary decoder in which an embodiment of the present disclosure may be implemented. [Figure 4] Figure 4 shows an exemplary quadtree partition of a coding tree block (CTB) according to an embodiment of the present disclosure. [Figure 5] Figure 5 shows the corresponding quadtree of the exemplary quadtree partition of the CTB in Figure 4, according to an embodiment of the present disclosure. [Figure 6] Figure 6 shows exemplary binary and ternary tree partitions according to embodiments of the present disclosure. [Figure 7] Figure 7 shows an example of a quadtree + multitree partitioning of CTB according to an embodiment of the present disclosure. [Figure 8] Figure 8 shows the corresponding quadtree + multitree for the exemplary quadtree + multitree partition of the CTB of Figure 7, according to an embodiment of the present disclosure. [Figure 9] Figure 9 shows an exemplary set of reference samples determined for intra-prediction of the current block, which are coded or decoded according to embodiments of the present disclosure. [Figure 10A]Figure 10A shows 35 intra-prediction modes supported by HEVC according to embodiments of the present disclosure. [Figure 10B] Figure 10B shows 67 intra-prediction modes supported by HEVC according to embodiments of the present disclosure. [Figure 11] Figure 11 shows the current block from Figure 9 and a reference sample in the two-dimensional x,y plane according to an embodiment of the present disclosure. [Figure 12] Figure 12 shows an exemplary angular mode prediction of the current block from Figure 9 according to an embodiment of the present disclosure. [Figure 13A] Figure 13A shows an example of interpretation performed on the current block of the current picture, which is encoded according to an embodiment of the present disclosure. [Figure 13B] Figure 13B shows exemplary horizontal and vertical components of a motion vector according to an embodiment of the present disclosure. [Figure 14] Figure 14 shows an example of a bi prediction performed on the current block according to an embodiment of the present disclosure. [Figure 15A] Figure 15A shows exemplary locations of five spatially adjacent candidate blocks relative to the current block, as coded according to embodiments of the present disclosure. [Figure 15B] Figure 15B shows exemplary locations of two temporal, collocation blocks relative to the current block, as coded according to embodiments of this disclosure. [Figure 16] Figure 16 shows an example of IBC applied to screen content according to an embodiment of the present disclosure. [Figure 17] Figure 17 shows an example of an intra-template matching search area according to several embodiments of the present disclosure. [Figure 18] Figure 18 shows an example of a decoder-side intra-mode derivation (DIMD) template zone for calculating a gradient direction histogram (HoG) according to some embodiments of the present disclosure. [Figure 19A]Figures 19A and 19B show flowcharts of exemplary processes for DIMD predictor derivation according to several embodiments of the present disclosure. [Figure 19B] Figures 19A and 19B show flowcharts of exemplary processes for DIMD predictor derivation according to several embodiments of the present disclosure. [Figure 20] Figure 20 shows examples of predictive fusion by a weighted average of two HoG modes and a planar mode, according to some embodiments of the present disclosure. [Figure 21] Figure 21 shows an example of a template-based intra-mode derivation (TIMD) related to several embodiments of the present disclosure. [Figure 22] Figure 22 shows a flowchart of an exemplary process for TIMD predictor generation according to several embodiments of the present disclosure. [Figure 23] Figure 23 shows an exemplary process flowchart of a DIMD merge mode according to some embodiments of the present disclosure. [Figure 24] Figure 24 shows examples of spatially adjacent neighboring blocks considered in the derivation of a list of DIMD merge candidates according to some embodiments of the present disclosure. [Figure 25] Figure 25 shows examples of spatially non-adjacent neighboring blocks used to derive DIMD merge candidates according to some embodiments of the present disclosure. [Figure 26] Figure 26 shows a flowchart of an exemplary process for adding DIMD information from spatially adjacent blocks to a history-based DIMD table, according to some embodiments of the present disclosure. [Figure 27] Figure 27 shows a flowchart of an exemplary process for deriving an index to a list of DIMD merge candidates, according to some embodiments of the present disclosure. [Figure 28]FIG. 28 shows a flowchart of a process for generating a list of DIMD merge candidates without duplicate DIMD information, and enforcing a predetermined threshold on the maximum list size of candidates from each family of candidates and the maximum number, in accordance with some embodiments of the present disclosure. [Figure 29] FIG. 29 shows a flowchart of an exemplary process for generating a sorted list of DIMD merge candidates, in accordance with some embodiments of the present disclosure. [Figure 30] FIG. 30 shows an example of a template that may be used for template-based sorting of a list of DIMD merge candidates, in accordance with some embodiments of the present disclosure. [Figure 31] FIG. 31 shows a flowchart of an exemplary process for vertical position dependency-based sorting of a list of DIMD merge candidates, in accordance with some embodiments of the present disclosure. [Figure 32] FIG. 32 shows a flowchart of an exemplary process for generating a list of DIMD merge candidates with template-based sorting of a portion of the list, in accordance with some embodiments of the present disclosure. [Figure 33] FIG. 33 shows a flowchart of an exemplary process for generating a list of DIMD merge candidates with addition of DIMD merge candidates derived from Template-based Intra Mode Derivation (TIMD), in accordance with some embodiments of the present disclosure. [Figure 34] FIG. 34 shows a flowchart of a method for DIMD merge mode encoding, in accordance with some embodiments of the present disclosure. The method includes generating a list of DIMD merge candidates, and sorting the list before an intra mode is determined. [Figure 35] FIG. 35 shows a flowchart of another method for DIMD merge mode decoding, in accordance with some embodiments of the present disclosure. The method includes generating a list of DIMD merge candidates, and sorting the list before an intra mode is determined, according to an indicator obtained from a bitstream. [Figure 36]Figure 36 shows a block diagram of an exemplary computer system in which embodiments of the present disclosure may be implemented. [Modes for carrying out the invention]

[0004] The following description includes numerous specific details to provide a complete understanding of the disclosure. However, it will be apparent to those skilled in the art that the disclosure, including structures, systems, and methods, can be implemented without these specific details. The descriptions and expressions herein are common means used by those skilled in the art to most effectively communicate the nature of the work to them. In other cases, well-known methods, procedures, components, and circuits are not described in detail to avoid unnecessarily obscuring aspects of the disclosure.

[0005] References in this specification to “one embodiment,” “one example,” and “exemplary embodiment” indicate that the embodiments described may include certain features, structures, or characteristics, but not all embodiments necessarily include those features, structures, or characteristics. Furthermore, such phrases do not necessarily refer to the same embodiment. In addition, if certain features, structures, or characteristics are described in relation to one embodiment, it is presented that any influence on such features, structures, or characteristics in relation to other embodiments, whether explicitly stated or not, is within the scope of what is understood by those skilled in the art.

[0006] Furthermore, it should be noted that individual embodiments may be described as processes shown as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. While flowcharts can describe operations as sequential processes, many operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operations are complete, but it may have additional steps not shown in the diagram. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. If a process corresponds to a function, its termination may correspond to the return of a function to a calling function or a main function.

[0007] The term “computer-readable media” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media that can store, contain, or carry instructions and / or data. Computer-readable media may also include non-temporary media that do not contain carrier waves and / or temporary electronic signals propagating over wireless or wired connections, on which data may be stored. Examples of non-temporary media include, but are not limited to, magnetic disks or optical storage media such as tapes, compact discs (CDs) or digital multipurpose discs (DVDs), flash memory, memory, or memory devices. Computer-readable media may store code and / or machine-executable instructions that may represent procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuits by passing through and / or receiving information, data, arguments, parameters, or memory content. Information, arguments, parameters, data, etc., may be passed, transferred, or transmitted by any preferred means, including memory sharing, message passing, token passing, network transmission, or similar.

[0008] Furthermore, embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented by software, firmware, middleware, or microcode, the program code or code segments for performing the required task (e.g., a computer program product) may be stored in a computer-readable or machine-readable medium. The processor may then perform the required task.

[0009] Representing a video sequence in digital format can require a large number of bits. The data size of a video sequence in digital format can be too large for storage and / or transmission in many applications. Video encoding can be used to compress the size of a video sequence to provide more efficient storage and / or transmission. Video decoding can be used to decompress a video sequence that has been compressed for viewing and / or other forms of consumption.

[0010] Figure 1 shows an exemplary video coding / decoding system 100 in which embodiments of the present disclosure may be implemented. The video coding / decoding system 100 comprises a source device 102, a transmission medium 104, and a destination device 106. The source device 102 encodes a video sequence 108 into a bitstream 110 for more efficient storage and / or transmission. The source device 102 may store and / or transmit the bitstream 110 to the destination device 106 via the transmission medium 104. The destination device 106 decodes the bitstream 110 to display the video sequence 108. The destination device 106 may receive the bitstream 110 from the source device 102 via the transmission medium 104. The source device 102 and the destination device 106 may be any one of several different devices (e.g., a desktop computer, a laptop computer, a tablet computer, a smartphone, a wearable device, a television, a camera, a video game console, a set-top box, a video streaming device, etc.).

[0011] To encode the video sequence 108 into a bitstream 110, the source device 102 may include a video source 112, an encoder 114, and / or an output interface 116. The video source 112 may provide and / or generate the video sequence 108 from captures of natural scenes and / or synthetically generated scenes. Synthetically generated scenes may be scenes including computer-generated graphics and / or screen content. The video source 112 may comprise a video capture device (e.g., a video camera), a video archive containing previously captured natural scenes and / or synthetically generated scenes, a video feed interface for receiving natural scenes and / or synthetically generated scenes captured from a video content provider, and / or a processor for generating synthetic scenes.

[0012] As shown in Figure 1, a video sequence, such as video sequence 108, may contain a series of pictures (also called frames). A video sequence can achieve an impression of motion by using the continuous presentation of the pictures in the video sequence for a fixed or variable time. A picture may contain one or more sample arrays of luminance values. Luminance values ​​may be obtained at a series of regularly spaced positions within the picture. A color picture typically contains a luminance sample array and two chrominance sample arrays. The luminance sample array may contain luminance values ​​representing the luminance (e.g., luminance component, Y) of the picture. The chrominance sample arrays may contain luminance values ​​representing the blue and red components, respectively, of a separate picture (or chrominance component, Cb, and Cr). Other color picture sample arrays are possible based on different color schemes (e.g., red, green, blue (RGB) color scheme). For a color picture, a pixel may refer to all three luminance values ​​for a given position from the three sample arrays used to represent the color picture. A monochrome picture contains a single luminance sample array. In a monochrome picture, a pixel can refer to a luminance value at a given position on the single luminance sample array used to represent the monochrome picture.

[0013] The encoder 114 can encode the video sequence 108 into a bitstream 110. To encode the video sequence 108, the encoder 114 may apply one or more prediction techniques to reduce redundant information in the video sequence 108. Redundant information may be predicted by the decoder and therefore does not need to be transmitted to the decoder for accurate decoding of the video sequence. For example, the encoder 114 may reduce redundant information in the video sequence 108 by applying spatial prediction (e.g., intraframe, or intra-prediction), temporal prediction (e.g., interframe, or inter-prediction), interlayer prediction, and / or other prediction techniques. Before applying one or more prediction techniques, the encoder 114 may divide the pictures of the video sequence 108 into rectangular regions called blocks. The encoder 114 may then encode the blocks using one or more of the prediction techniques.

[0014] For time prediction, encoder 114 may search for a block similar to the one to be encoded in another picture (also called a reference picture) of video sequence 108. Then, using the block determined during the search (also called a predicted block), it may predict the block to be encoded. For spatial prediction, encoder 114 may form a predicted block based on data from reconstructed adjacent samples of the block to be encoded within the same picture of video sequence 108. The reconstructed sample refers to the encoded and then decoded sample. Encoder 114 may determine a prediction error (also called a residual) based on the difference between the block to be encoded and the predicted block. The prediction error may represent non-redundant information that can be sent to the decoder for accurate decoding of the video sequence.

[0015] The encoder 114 may generate conversion coefficients by applying a transformation (e.g., discrete cosine transform (DCT)) to the prediction error. Based on the conversion coefficients, and other information used to determine the prediction block (e.g., prediction type, motion vector, and / or prediction mode), the encoder 114 may form the bitstream 110. In some embodiments, the encoder 114 may perform one or more of the following on the conversion coefficients and / or other information used to determine the prediction block before forming the bitstream 110: quantization and entropy coding, to further reduce the number of bits required to store and / or transmit the video sequence 108.

[0016] The output interface 116 may be configured to write and / or store the bitstream 110 on the transmission medium 104 for transmission to the destination device 106. In addition, or alternatively, the output interface 116 may be configured to transmit, upload, and / or stream the bitstream 110 to the destination device 106 via the transmission medium 104. The output interface 116 may include wired and / or wireless transmitters configured to transmit, upload, and / or stream the bitstream 110 in accordance with one or more proprietary and / or standardized communication protocols (e.g., Digital Video Broadcasting (DVB) standards, Advanced Television Systems Commission (ATSC) standards, Integrated Services Digital Broadcasting (ISDB) standards, Data over Cable Services Interface Specification (DOCSIS) standards, Third Generation Partnership Project (3GPP®) standards, Institute of Electrical and Electronics Engineers (IEEE) standards, Internet Protocol (IP) standards, Wireless Application Protocol (WAP) standards, and / or any other communication protocols).

[0017] The transmission medium 104 may include wireless, wired, and / or computer-readable media. For example, the transmission medium 104 may comprise one or more wires, cables, air interfaces, optical disks, flash memory, and / or magnetic memory. In addition, or alternatively, the transmission medium 104 may include one or more networks (e.g., the Internet) or file servers configured to store and / or transmit encoded video data.

[0018] To decode the bitstream 110 into a video sequence 108 for display, the destination device 106 may comprise an input interface 118, a decoder 120, and a video display 122. The input interface 118 may be configured to read the bitstream 110 stored on the transmission medium 104 by the source device 102. In addition, or alternatively, the input interface 118 may be configured to receive, download, and / or stream the bitstream 110 from the source device 102 via the transmission medium 104. The input interface 118 may comprise a wired and / or wireless receiver configured to receive, download, and / or stream the bitstream 110 according to one or more proprietary and / or standardized communication protocols (e.g., as referenced above).

[0019] Decoder 120 can decode video sequence 108 from encoded bitstream 110. To decode video sequence 108, decoder 120 can generate prediction blocks of the pictures in video sequence 108 in a similar manner to encoder 114 and determine the prediction error of the blocks. Decoder 120 can generate prediction blocks using the prediction type, prediction mode, and / or motion vector received in bitstream 110, and determine the prediction error using the conversion coefficients also received in bitstream 110. Decoder 120 can determine the prediction error by weighting the conversion base function using the conversion coefficients. Decoder 120 can decode video sequence 108 by combining the prediction blocks and the prediction error. In some embodiments, the decoder 120 may decode a video sequence that approximates the video sequence 108, for example, due to lossy compression of the video sequence 108 by the encoder 114 and / or errors introduced into the encoded bitstream 110 during transmission to the destination device 106.

[0020] The video display 122 can display the video sequence 108 to the user. The video display 122 may include a cathode ray tube (CRT) display, a liquid crystal display (LCD), a plasma display, a light-emitting diode (LED) display, or any other display device suitable for displaying the video sequence 108.

[0021] It should be noted that the video encoding / decoding system 100 is presented as an embodiment and is not limiting. In the embodiment of Figure 1, the video encoding / decoding system 100 may have other components and / or arrangements. For example, the video source 112 may be outside the source device 102. Similarly, the video display 122 may be outside the destination device 106 or may be omitted entirely (for example, if the video sequence is intended for consumption by a machine and / or storage device). In another embodiment, the source device 102 may further comprise a video decoder, and the destination device 106 may further comprise a video encoder. In such an embodiment, the source device 102 may be configured to further receive an encoded bitstream from the destination device 106 to support bidirectional video transmission between the devices.

[0022] In the embodiment shown in Figure 1, the encoder 114 and decoder 120 may operate according to one of several proprietary or industrial video coding standards. For example, the encoder 114 and decoder 120 may operate according to one or more of the following: International Telecommunication Union Telecommunication Standardization Sector (ITU-T) H.263, ITU-T H.264, and the Expert Group on Video Coding (MPEG)-4 Visual (also known as Advanced Video Coding (AVC)), ITU-T H.265, and MPEG-H Part 2 (also known as High Efficiency Video Coding (HEVC)), ITU-T H.265, and MPEG-I Part 3 (also known as Multipurpose Video Coding (VVC)), WebM VP8 and VP9 codes, and AOMedia Video 1 (AV1).

[0023] Figure 2 shows an exemplary encoder 200 in which embodiments of the present disclosure may be implemented. The encoder 200 encodes a video sequence 202 into a bitstream 204 for more efficient storage and / or transmission. The encoder 200 may be implemented in the video coding / decoding system 100 shown in Figure 1, or in any of a variety of different devices such as a desktop computer, laptop computer, tablet computer, smartphone, wearable device, television, camera, video game console, set-top box, or video streaming terminal. The encoder 200 may comprise one or more of the following: an inter-prediction unit 206, an intra-prediction unit 208, combiners 210 and 212, a transform and quantization unit (TR+Q) 214, an inverse transform and quantization unit (iTR+iQ) 216, an entropy coding unit 218, one or more filters 220, and a buffer 222.

[0024] Encoder 200 may divide the pictures of video sequence 202 into blocks and encode video sequence 202 block by block. Encoder 200 may perform prediction techniques on the encoded blocks using either inter-prediction unit 206 or intra-prediction unit 208. Inter-prediction unit 206 may perform inter-prediction by searching for blocks similar to blocks encoded in another reconstructed picture of video sequence 202 (e.g., called a reference picture). The reconstructed picture refers to an encoded and then decoded picture. It may then predict which blocks to encode to remove redundant information using blocks determined during the search (e.g., called prediction blocks). Inter-prediction unit 206 may determine prediction blocks by utilizing temporal redundancy or similarity in the scene content from picture to picture in video sequence 202. For example, the scene content between pictures in video sequence 202 may be similar over time, except for differences due to motion or affine transformations of the screen content.

[0025] The intra-prediction unit 208 may perform intra-prediction by forming prediction blocks based on data from reconstructed adjacent samples of a block encoded within the same picture of the video sequence 202. The reconstructed samples refer to the encoded and then decoded samples. The intra-prediction unit 208 may determine prediction blocks by utilizing spatial redundancy or similarity in the scene content within the picture of the video sequence 202. For example, the texture of a region of scene content within a picture may be similar to the texture of a region of scene content immediately surrounding it within the same picture.

[0026] After prediction, the combiner 210 may determine a prediction error (e.g., called a residual) based on the difference between the blocks to be encoded and the predicted blocks. The prediction error may represent non-redundant information that can be sent to the decoder for accurate decoding of the video sequence.

[0027] The transformation and quantization unit 214 can transform and quantize the prediction error. The transformation and quantization unit 214 can transform the prediction error into transformation coefficients by, for example, applying DCT to reduce the correlation information of the prediction error. The transformation and quantization unit 214 can quantize the transformation coefficients by mapping the data of the transformation coefficients to a predefined set of representative values. The transformation and quantization unit 214 can quantize the coefficients to reduce irrelevant information in the bitstream 204. Irrelevant information refers to information that, after decoding, can be removed from the coefficients without generating visible and / or perceptible distortion in the video sequence 202.

[0028] The entropy coding unit 218 can further reduce the bitrate by applying one or more entropy coding methods to the quantization conversion coefficients. For example, the entropy coding unit 218 can apply context-adaptive variable-length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), and syntax-based context-adaptive binary arithmetic coding (SBAC). The entropy-coded coefficients can be packaged to form a bitstream 204.

[0029] The inverse transform and quantization unit 216 may dequantize and inverse transform the quantization transformation coefficients to determine the reconstructed prediction error. The combiner 212 may combine the reconstructed prediction error with the prediction block to form a reconstructed block. The filter 220 may filter the reconstructed block using, for example, a deblocking filter and / or a sample-adaptive offset (SAO) filter. The buffer 222 may store the reconstructed block for prediction of one or more other blocks in the same and / or different pictures of the video sequence 202.

[0030] Although not shown in Figure 2, the encoder 200 further comprises an encoder control unit configured to control one or more units of the encoder 200 shown in Figure 2. The encoder control unit can control one or more units of the encoder 200 so that the bitstream 204 is generated in accordance with any one requirement of a proprietary or industry video coding standard. For example, the encoder control unit can control one or more units of the encoder 200 so that the bitstream 204 is generated in accordance with one or more of the ITU-T H.263, AVC, HEVC, VVC, VP8, VP9, ​​or AV1 video coding standards.

[0031] Within the constraints of proprietary or industry video coding standards, the encoder control unit may attempt to minimize or reduce the bitrate of bitstream 204 and maximize or improve the quality of the reconstructed video. For example, the encoder control unit may be configured to minimize or reduce the bitrate of bitstream 204 so that the reconstructed video quality does not fall below a certain level / threshold, or to maximize or increase the reconstructed video quality so that the bitrate of bitstream 204 does not exceed a certain level / threshold. The encoder control unit may determine / control one or more of the following: dividing the pictures of the video sequence 202 into blocks; determining whether the blocks are inter-predicted by the inter-prediction unit 206 or intra-predicted by the intra-prediction unit 208; determining the motion vector for the inter-prediction of the blocks; controlling the intra-prediction mode among multiple intra-prediction modes for the intra-prediction of the blocks; filtering performed by the filter 220; and / or determining one or more transformation types and / or quantization parameters applied by the transformation and quantization unit 214. The encoder control unit may determine / control and influence one or more of the above based on velocity distortion measurements for the blocks or pictures being encoded. The encoder control unit may determine / control the above in order to reduce velocity distortion measurements for the blocks or pictures being encoded.

[0032] After determination, the prediction type (intra or inter-predict), prediction information for the block (intra-prediction mode, motion vector, etc., in the case of intra-prediction), and / or transformation and / or quantization parameters used to encode the block are sent to the entropy coding unit 218, where they may be further compressed to reduce the bitrate. For example, the entropy coding unit 218 may apply context-adaptive variable-length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), and syntax-based context-adaptive binary arithmetic coding (SBAC) to compress the prediction type used to code the block (intra or inter-prediction), prediction information for the block (intra-prediction mode, motion vector, etc., in the case of intra-prediction), and transformation and quantization parameters. The prediction type, prediction information, and transformation and quantization parameters may be packaged with prediction errors to form a bitstream 204.

[0033] It should be noted that encoder 200 is presented as an example and is not limiting. In other embodiments, encoder 200 may have other components and / or arrangements. One or more of the components shown in Figure 2 may optionally be included in encoder 200 (e.g., entropy coding unit 218 and / or filter 220).

[0034] Figure 3 shows an exemplary decoder 300 in which embodiments of the present disclosure may be implemented. The decoder 300 decodes a bitstream 302 into a decoded video sequence 304 for display and / or any other form of consumption. The decoder 300 may be implemented in the video coding / decoding system 100 of Figure 1, or in any different device including a desktop computer, laptop computer, tablet computer, smartphone, wearable device, television, camera, video game console, set-top box, and / or video streaming device. The decoder 300 may comprise an entropy decoding unit 306, an inverse transform and quantization (iTR+iQ) unit 308, a combiner 310, one or more filters 312, a buffer 314, an inter-prediction unit 316, and / or an intra-prediction unit 318.

[0035] Although not shown in Figure 3, the decoder 300 further comprises a decoder control unit configured to control one or more units of the decoder 300 shown in Figure 3. The decoder control unit can control one or more units of the decoder 300 so that the bitstream 302 is decoded in accordance with the requirements of one or more proprietary or industry video coding standards. For example, the decoder control unit can control one or more units of the decoder 300 so that the bitstream 302 is decoded in accordance with one or more ITU-T H.263, AVC, HEVC, VVC, VP8, VP9, ​​and AV1 video coding standards.

[0036] The decoder control unit can determine / control one or more of the following: whether a block is inter-predicted by the inter-prediction unit 316 or intra-predicted by the intra-prediction unit 318; determining the motion vector for the inter-prediction of the block; controlling the intra-prediction mode among multiple intra-prediction modes for the intra-prediction of the block; filtering performed by the filter 312; and / or determining the inverse transform, and one or more inverse transform types and / or inverse quantization parameters applied by the quantization unit 308. One or more of the control parameters used by the decoder control unit can be packaged in the bitstream 302.

[0037] The entropy decoding unit 306 can entropy decode the bitstream 302. For example, the entropy decoding unit 306 can apply context-adaptive variable-length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), and syntax-based context-adaptive binary arithmetic coding (SBAC) to decode the prediction type used to code blocks (intra or inter-predictions), prediction information for the blocks (intra-prediction mode, motion vector, etc., in the case of intra-predictions), and transform and quantization parameters. The inverse transform and quantization unit 308 can inverse quantization and inverse transform the quantization transform coefficients to determine the decoded prediction error. The combiner 310 can combine the decoded prediction error with the prediction block to form a decoded block. The prediction block can be generated by the intra-prediction unit 318 or the inter-prediction unit 316, as described above with respect to the encoder 200 in Figure 2. Filter 312 may filter the decoded blocks using, for example, a deblocking filter and / or a sample-adaptive offset (SAO) filter. Buffer 314 may store the decoded blocks for prediction of one or more other blocks in the same and / or different pictures of the video sequence of the bitstream 302. The decoded video sequence 304 may be output from filter 312 as shown in Figure 3.

[0038] It should be noted that the decoder 300 is presented as an example and is not limiting. In other embodiments, the decoder 300 may have other components and / or arrangements. One or more of the components shown in Figure 3 may optionally be included in the decoder 300 (e.g., the entropy decoding unit 306 and / or the filter 312).

[0039] Although not shown in Figures 2 and 3, it should be further noted that each of the encoder 200 and decoder 300 may further comprise an intrablock copy unit in addition to the interprediction and intraprediction units. The intrablock copy unit may perform similarly to the interprediction unit but may predict blocks within the same picture. For example, the intrablock copy unit may utilize repeating patterns appearing in screen content. Screen content may include, for example, computer-generated text, graphics, and animations.

[0040] As described above, video encoding and decoding can be performed block by block. The process of dividing a picture into blocks can be adaptive based on the content of the picture. For example, to improve coding efficiency, larger block divisions may be used in areas of the picture that have a higher level of homogeneity.

[0041] In HEVC, a picture can be divided into non-overlapping square blocks called coding tree blocks (CTBs), which contain samples from the sample array. A CTB may have a size of 2n × 2n samples, where n may be specified by the encoding system's parameters. For example, n may be 4, 5, or 6. A CTB can be further divided into coding blocks (CBs) of semi-vertical and semi-horizontal size by a recursive quadtree split. The CTB forms the root of the quadtree. CBs that are not further divided as part of a recursive quadtree split may be called leaf CBs of the quadtree, otherwise they are called non-leaf CBs of the quadtree. A CB may have a minimum size specified by the encoding system's parameters. For example, a CB may have the smallest sample size among 4 × 4, 8 × 8, 16 × 16, 32 × 32, and 64 × 64. A CB can be further divided into one or more prediction blocks (PBs) to perform inter-prediction and intra-prediction. A PB can be a rectangular block of samples to which the same prediction type / mode can be applied. For transformation, a CB can be divided into one or more transformation blocks (TBs). A TB can be a rectangular block of samples to which the applied transformation size can be determined.

[0042] Figure 4 shows an exemplary quadtree partition of CTB400. Figure 5 shows the corresponding quadtree 500 of the exemplary quadtree partition of CTB400 in Figure 4. As shown in Figures 4 and 5, CTB400 can first be partitioned into four CBs of semi-vertical and semi-horizontal size. Three of the CBs obtained as a result of the first level partition of CTB400 are leaf CBs. The three leaf CBs of the first level partition of CTB400 are labeled 7, 8, and 9 in Figures 4 and 5, respectively. The non-leaf CBs of the first level partition of CTB400 are partitioned into four sub-CBs of semi-vertical and semi-horizontal size. Three of the sub-CBs obtained as a result of the second level partition of CTB400 are leaf CBs. The three leaf CBs of the second level partition of CTB400 are labeled 0, 5, and 6 in Figures 4 and 5, respectively. Finally, the non-lobed CB of the second level division of CTB400 is divided into four lobed CBs, each half vertical and half horizontal in size. The four lobed CBs are labeled 1, 2, 3, and 4, respectively, in Figures 4 and 5.

[0043] Overall, CTB400 can be divided into 10 leaf CBs, each labeled 0 through 9. The resulting quadtree division of CTB400 can be scanned using z-scans (e.g., left to right, top to bottom) to form a sequence order for encoding / decoding the CB leaf nodes. The numerical labels of each CB leaf node in Figures 4 and 5 may correspond to an encoding / decoding sequence order, with CB leaf node 0 being encoded / decoded first and CB leaf node 9 being encoded / decoded last. Note that, although not shown in Figures 4 and 5, each CB leaf node may contain one or more PBs and / or TBs.

[0044] In VVC, pictures can be partitioned in a similar manner to HEVC. A picture can first be partitioned into non-overlapping square CTBs. The CTBs can then be partitioned into semi-vertical and semi-horizontal CBs by recursive quadtree partitioning. In VVC, quadtree leaf nodes can be further partitioned into unequal CBs by binary or ternary tree partitioning. Figure 6 shows exemplary binary and ternary tree partitioning. A binary tree partition can divide the parent block in half vertically by 602 or horizontally by 604. The resulting partitions can be half the size of the parent block. A ternary tree partition can divide the parent block into three parts vertically by 606 or horizontally by 608. The intermediate partitions can be twice the size of the other two end partitions within the ternary tree partition.

[0045] Due to the addition of binary and ternary tree partitions, in VCV, block partitioning strategies may be called quadrutree + multi-tree partitions. Figure 7 shows an exemplary quadrutree + multi-tree partition for CTB700. Figure 8 shows the corresponding quadrutree + multi-tree 800 to the exemplary quadrutree + multi-tree partition for CTB700 in Figure 7. In both Figures 7 and 8, quadrutree partitions are shown with solid lines and multi-tree partitions with dashed lines. For ease of explanation, CTB700 is shown with the same quadrutree partition as CTB400 described in Figure 4. Therefore, the description of the quadrutree partition for CTB700 is omitted. The description of the additional multi-tree partition for CTB700 is done for the three leaf CBs shown in Figure 4, which are further partitioned using one or more binary and ternary tree partitions. The three leaf CBs from Figure 4, shown in Figure 7 as further partitioned, are leaf CB5, 8, and 9.

[0046] Starting with leaf CB5 in Figure 4, Figure 7 shows this leaf CB split into two CBs based on vertical binary tree partitioning. The two resulting CBs are leaf CBs labeled 5 and 6 in Figures 7 and 8, respectively. With respect to leaf CB8 in Figure 4, Figure 7 shows this leaf CB split into three CBs based on vertical ternary tree partitioning. Two of the three resulting CBs are leaf CBs labeled 9 and 14 in Figures 7 and 8, respectively. The remaining non-leaf CB is first split into two CBs based on horizontal binary tree partitioning, one of which is a leaf CB labeled 10, and the other of which is further split into three CBs based on vertical ternary tree partitioning. The three resulting CBs are leaf CBs labeled 11, 12, and 13 in Figures 7 and 8, respectively. Finally, with respect to leaf CB9 in Figure 4, Figure 7 shows this leaf CB split into three CBs based on horizontal ternary tree partitioning. Two of the three CBs are leaf CBs labeled 15 and 19 in Figures 7 and 8, respectively. The remaining non-leaf CB can be split into three CBs based on other horizontal ternary tree partitioning. All three resulting CBs are leaf CBs labeled 16, 17, and 18 in Figures 7 and 8, respectively.

[0047] Overall, CTB700 is divided into 20 leaf CBs, each labeled 0 through 19. The resulting quadtree-plus-multitree division of CTB700 can be scanned using z-scans (left to right, top to bottom) to form a sequence order for encoding / decoding the CB leaf nodes. The numerical labels of each CB leaf node in Figures 7 and 8 may correspond to an encoding / decoding sequence order, with CB leaf node 0 being encoded / decoded first and CB leaf node 19 being encoded / decoded last. Note that, although not shown in Figures 7 and 8, each CB leaf node may contain one or more PBs and TBs.

[0048] In addition to specifying various blocks (e.g., CTB, CB, PB, TB), HEVC and VVC further define various units. A block may contain a rectangular region of samples within a sample array, while a unit may contain collocated blocks of samples from different sample arrays (e.g., luminance and chrominance sample arrays) that form a picture, as well as syntactic elements of the block and prediction data. A coding tree unit (CTU) may contain collocated CTBs from different sample arrays that may form a complete entity within the encoded bitstream. A coding unit (CU) may comprise collocated CBs of different sample arrays and syntactic structures used to code samples in a CB. A prediction unit (PU) may comprise collocated PBs of different sample arrays and syntactic elements used to predict a PB. A transformation unit (TU) may comprise TBs of different sample arrays and syntactic elements used to transform a TB.

[0049] It should be noted that in the context of HEVC and VVC, the term "block" can refer to any of the following: CTB, CB, PB, TB, CTU, CU, PU, ​​or TU. Furthermore, it should be noted that in the context of other video coding standards, the term "block" can refer to a similar data structure. For example, the term "block" can refer to a macroblock in AVC, a macroblock or subblock in VP8, a superblock or subblock in VP9, ​​and / or a superblock or subblock in AV1.

[0050] In intra-prediction, the samples of the block being encoded (also called the current block) can be predicted from the samples in the column immediately adjacent to the leftmost column of the current block, and from the samples in the row immediately adjacent to the top row of the current block. The samples from the immediately adjacent columns and rows can be collectively referred to as reference samples. Each sample in the current block can be predicted by projecting the position of the sample within the current block onto a point along the reference samples in a given direction (for example, referred to as the intra-prediction mode). If the projection does not fall directly onto the reference samples, the sample can be predicted by interpolation between the two nearest reference samples at the projection point. The prediction error (also called the residual) can be determined for the current block based on the difference between the predicted sample value of the current block and the original sample value.

[0051] The encoder predicts a sample and determines a prediction error based on the difference between the predicted sample and the original sample. This process can be performed for several different intra-prediction modes (e.g., including non-directional intra-prediction modes). The encoder may select one of the several intra-prediction modes and its corresponding prediction error to encode the current block. The encoder may send a display of the selected prediction mode and its corresponding prediction error to the decoder for decoding the current block. The decoder may decode the current block using the intra-prediction mode indicated by the encoder and by combining the predicted sample with the prediction error.

[0052] Figure 9 shows an exemplary set of reference samples 902 determined for intra-prediction of the current block 904 to be encoded or decoded. In Figure 9, the current block 904 may correspond to block 3 of the partitioned CTB700, as shown in Figure 7. As described above, the numerical labels 0-19 of the blocks in the partitioned CTB700 may also correspond to the sequence order for encoding / decoding the blocks, and are used in this manner in the embodiment of Figure 9.

[0053] Given a current block 904, where the sample size is w×h, the reference sample 902 may extend across the 2w samples in the row immediately adjacent to the top row of the current block 904, the 2h samples in the column immediately adjacent to the leftmost column of the current block 904, and the top-left adjacent corner sample relative to the current block 904. In the embodiment of Figure 9, the current block 904 is a square, and therefore w=h=s. To construct the set of reference samples 902, available samples from adjacent blocks to the current block 904 may be used. Samples may not be available to construct the set of reference samples 902, for example, if the sample lies outside the picture of the current block, the sample is part of a different slice of the current block (where the concept of a slice is used), and / or the sample belongs to a block in which an intercoded and constrained intra-prediction is shown. If a constrained intra-prediction is shown, the intra-prediction may not depend on the inter-prediction block.

[0054] In addition to the above, samples that are not available to construct the set of reference sample 902 may include samples within a block that have not yet been encoded and reconstructed by the encoder, and / or decoded by the decoder, based on the sequence order for encoding / decoding. This limitation may make it possible for both the encoder and the decoder to determine the same prediction result. In Figure 9, considering that these blocks are encoded and reconstructed by the encoder and decoded by the decoder before the current block 904 is coded, samples from adjacent blocks 0, 1, and 2 may be available to construct the reference sample 902. This assumes that there are no other issues, as described above, that would hinder the availability of samples from adjacent blocks 0, 1, and 2. However, portions of the reference sample 902 from adjacent block 6 may not be available due to the sequence order for encoding / decoding.

[0055] Unavailable reference samples in reference sample 902 may be filled with available reference samples in reference sample 902. For example, an unavailable reference sample may be filled with the nearest available reference sample, which may be determined by moving clockwise through reference sample 902 from the location of the unavailable reference. Reference sample 902 may be filled with, for example, the midpoint of the dynamic range of the picture being coded if no available reference samples exist.

[0056] It should be noted that reference sample 902 may be filtered based on the size of the current block 904 being coded and the intra-prediction mode applied. It should be further noted that Figure 9 shows only one exemplary determination of the reference sample for intra-prediction of a block. In some proprietary and industry video coding standards, the reference sample may be determined in a different manner than described above. For example, multiple reference lines may be used in other cases, e.g., VVC.

[0057] After the reference sample 902 is determined and optionally filtered, the samples in the current block 904 can be intra-predicted based on the reference sample 902. Most encoders / decoders can support multiple intra-prediction modes according to one or more video coding standards. For example, HEVC supports 35 intra-prediction modes, including planar mode, DC mode, and 33 angular modes. VVC supports 67 intra-prediction modes, including planar mode, DC mode, and 65 angular modes. Planar and DC modes can be used to predict smooth, gradually changing regions of a picture. Angular modes can be used to predict directional structures in regions of a picture.

[0058] Figure 10A shows the 35 intra-prediction modes supported by HEVC. The 35 intra-prediction modes can be identified by indices 0 to 34. Prediction mode 0 corresponds to the planar mode. Prediction mode 1 corresponds to the DC mode. Prediction modes 2 to 34 correspond to the angular mode. Prediction modes 2 to 18 can be called horizontal prediction modes because the primary prediction source is in the horizontal direction. Prediction modes 19 to 34 can be called vertical prediction modes because the primary prediction source is in the vertical direction.

[0059] Figure 10B shows 67 intra-prediction modes supported by VVC. The 67 intra-prediction modes can be identified by indices 0 to 66. Prediction mode 0 corresponds to the planar mode. Prediction mode 1 corresponds to the DC mode. Prediction modes 2 to 66 correspond to the angular modes. Prediction modes 2 to 34 can be called horizontal prediction modes because the primary prediction source is in the horizontal direction. Prediction modes 35 to 66 can be called vertical prediction modes because the primary prediction source is in the vertical direction. Since the VVC blocks do not need to be square, some of the intra-prediction modes shown in Figure 10B can be adaptively replaced by wide-angle directions.

[0060] To further illustrate the application of the intra-prediction mode for determining the prediction of the current block, refer to Figures 11 and 12. In Figure 11, the current block 904 and reference sample 902 from Figure 9 are shown in the two-dimensional x,y plane, and the sample is...

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0061] In DC mode, the current position within block 904

number

number

number

[0062] Regarding angle mode, the current position within block 904

number

number

number

[0063] Figure 12 shows the current position within block 904 relative to the vertical prediction mode 906, given by angle φ.

number

number

number

number

number

number

number

number

number

number

number

number

[0064] For horizontal prediction mode, the current sample position [x][y] within block 904 is the reference sample

number

number

number

number

number

number

number

number

[0065] The interpolation functions of (7) and (10) can be implemented by an encoder or decoder, such as encoder 200 in Figure 2 or decoder 300 in Figure 3, as a set of two-tap finite impulse response (FIR) filters. The coefficients of the two-tap FIR filters are, respectively,

number

number

number

number

[0066] In one embodiment, a 2-tap interpolation FIR filter may be used to predict a color difference sample. For luminance samples, different interpolation techniques may be used. For example, for luminance samples, a 4-tap FIR filter may be used to determine the predicted value of the luminance sample. For example, the coefficients of the 4-tap FIR filter are:

number

number

number

number

number

number

number

[0067] As described above, the encoder may predict the samples to be encoded in the current block, such as the current block 904, for multiple intra-prediction modes. For example, the encoder may predict the samples of the current block for each of the 35 intra-prediction modes in HEVC or the 67 intra-prediction modes in VVC. For each intra-prediction mode applied, the encoder may determine a corresponding prediction error for the current block based on the difference (e.g., sum of squared differences (SSD), sum of absolute differences (SAD), or sum of absolute transform differences (SATD)) between the predicted samples determined for the intra-prediction mode and the original samples of the current block. Based on the determined prediction error, the encoder may select one of the intra-prediction modes to code the current block. For example, the encoder may select one of the intra-prediction modes that yields the smallest prediction error for the current block. In another embodiment, the encoder may select an intra-prediction mode to code the current block based on a velocity distortion measurement (e.g., Lagrangian velocity distortion cost) determined using the prediction error. The encoder may send the selected intra-prediction mode and its corresponding prediction error to the decoder for decoding the current block.

[0068] Similar to the encoder, the decoder may predict, as described above, that a sample of the current block, such as the current block 904, will be decoded for an intra-prediction mode. For example, the decoder may receive an indication of the angular intra-prediction mode from the encoder for a block. The decoder may construct a set of reference samples and, in a similar manner (for example, as described above for the encoder), perform an intra-prediction for the current block based on the angular intra-prediction mode indicated by the encoder. The decoder reconstructs the block by adding the predicted values ​​of the block's samples to the block's residuals. In another embodiment, the decoder does not need to receive an indication of the angular intra-prediction mode from the encoder for a block. Instead, the decoder may determine the intra-prediction mode via other decoder-side means.

[0069] While the above description has been made primarily in relation to the intra-predictive modes in HEVC and VVC, it should be understood that the techniques of this disclosure, as described above and further below, may be applicable to other intra-predictive modes, including those of other video coding standards such as VP8, VP9, ​​and AV1.

[0070] As described above, intra-prediction can perform video compression by utilizing correlations between spatially adjacent samples of the same picture in a video sequence. Inter-prediction is another coding tool that can be used to perform video compression by utilizing temporal domain correlations between sample blocks of different pictures in a video sequence. Generally, an object may be visible across multiple pictures in a video sequence. The object may move (e.g., by some translational motion and / or affine motion) or remain stationary across multiple pictures. Thus, the current block of samples in the currently encoded picture may have a corresponding block of samples in a previously decoded picture that accurately predicts the current block of samples. The corresponding block of samples may be displaced from the current block of samples due to the movement of the object represented in both blocks across the respective pictures of the block. The previously decoded picture may be called a reference picture, and the corresponding block of samples in the reference picture may be called a reference block or motion-compensated prediction. The encoder may use block matching techniques to estimate displacement (or motion) and determine the reference block in the reference picture.

[0071] Similar to intra-prediction, once predictions for the current block are determined and / or generated using inter-prediction, the encoder may determine the difference between the current block and the predicted values. This difference may be called the prediction error or residual. The encoder may then store or signal the prediction error and other related prediction information as a bitstream for decoding or other forms of consumption. The decoder may decode the current block by predicting the samples of the current block (e.g., by using the related prediction information) and by combining the predicted samples with the prediction error.

[0072] Figure 13A shows an example of interpretation in which encoding is performed for the current block 1300 within the current picture 1302. The encoder (e.g., encoder 200 shown in Figure 2) may perform interpretation to determine and / or generate a reference block 1304 within a reference picture 1306 in order to predict the current block 1300. A reference picture such as reference picture 1306 may be a previously decoded picture available to the encoder and / or decoder. The availability of a previously decoded picture may depend on whether the previously decoded picture is available in the decoded picture buffer at which point the current block 1300 is encoded or decoded. The encoder may look up one or more reference pictures for a reference block that is similar to the current block 1300, for example. During the search process, the encoder may determine a “best match” reference block from the tested blocks (e.g., reference block 1304). The encoder may determine that reference block 1304 is the best match reference block based on one or more cost criteria, such as a rate distortion criterion (e.g., Lagrangian rate distortion cost). One or more cost criteria may be set based on the difference between the predicted sample of reference block 1304 and the original sample of the current block 1300 (e.g., sum of squared differences (SSD), sum of absolute differences (SAD), or sum of absolute transform differences (SATD)).

[0073] The encoder may search for reference block 1304 within search range 1308. Search range 1308 may be located around the collocated position (or block) 1310 of the current block 1300 in reference picture 1306. In some cases, search range 1308 may extend at least partially outside reference picture 1306. When extending the range outside reference picture 1306, constant boundary extension may be used, in which case the sample values ​​of rows or columns of reference picture 1306 adjacent to the portion of search range 1308 extending outside reference picture 1306 are used for the “sample” positions outside reference picture 1306. All or a subset of potential positions within search range 1308 may be searched for reference block 1304. The encoder may determine and / or generate reference block 1304 using any one of different search implementations. For example, the encoder may determine a set of candidate search locations based on motion information of adjacent blocks relative to the current block 1300.

[0074] One or more reference pictures may be searched by the encoder during interpretation to determine and / or generate the best-matching reference block. The reference pictures searched by the encoder may be contained in one or more reference picture lists. For example, HEVC and VVC may use two reference picture lists, reference picture list 0 and reference picture list 1. A reference picture list may contain one or more pictures. Reference picture 1306 of reference block 1304 may be represented by a reference index pointing to a reference picture list containing reference picture 1306.

[0075] The displacement between reference block 1304 and current block 1300 can be interpreted as an estimate of the motion between reference block 1304 and current block 1300 across each picture. The displacement can be represented by a motion vector 1312. For example, motion vector 1312 can be represented by a horizontal component (MVx) and a vertical component (MVy) with respect to the position of current block 1300. Figure 13B shows the horizontal and vertical components of motion vector 1312. A motion vector, for example motion vector 1312, can have fractional or integer resolution. A motion vector with fractional resolution can point between two samples in the reference picture to provide a better estimate of the motion of current block 1300. For example, a motion vector can have a sample resolution of 1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32, or any other fractional. Interpolation between two samples at integer positions can be used to generate a reference block and its corresponding sample at a decimal position, for example, if the motion vector points to a non-integer sample value in the reference picture. Interpolation can be performed by a filter having two or more taps.

[0076] Once a reference block 1304 is determined and / or generated for the current block 1300 using interpretation, the encoder may determine the difference (e.g., the corresponding sample-by-sample difference) between the reference block 1304 and the current block 1300. This difference may be called the prediction error or residual. The encoder may then store and / or signal the prediction error and associated motion information into a bitstream for decoding or other forms of consumption. The motion information may include a motion vector 1312 and a reference index pointing to a reference picture list containing the reference picture 1306. In other examples, the motion information may include a representation of the motion vector 1312 and a representation of the reference index pointing to a reference picture list containing the reference picture 1306. The decoder may decode the current block 1300 by using the motion information to determine and / or generate a reference block 1304 that forms a prediction for the current block 1300, combining the prediction with the prediction error.

[0077] In Figure 13A, interpretation can be performed using a single reference picture 1306 as the source of prediction for the current block 1300. Since the prediction for the current block 1300 comes from a single picture, this type of interpretation is called unipretation. Figure 14 shows another type of interpretation, called bipretation, performed for the current block 1400. In bipretation, the source of prediction for the current block 1400 comes from two pictures. Bipretation can be useful, for example, when a video sequence contains fast motion, camera panning, zooming, or scene changes. Bipretation may also be useful for capturing a fade-out from one scene or a fade-out from one scene to another, where the two pictures can be displayed simultaneously and effectively at different levels of brightness.

[0078] Whether uni-prediction, or both uni-prediction and bi-prediction, are available for inter-prediction may depend on the slice type of the current block 1400. For P-slice, only uni-prediction may be available for inter-prediction. For B-slice, either uni-prediction or bi-prediction may be used. When uni-prediction is performed, the encoder may determine and / or generate a reference block from reference picture list 0 to predict the current block 1400. When bi-prediction is performed, the encoder may determine and / or generate a first reference block from reference picture list 0 to predict the current block 1400, and a second reference block from reference picture list 1 to predict the current block 1400.

[0079] In Figure 14, interpretation is performed using biprediction, predicting the current block 1400 using two reference blocks 1402 and 1404. Reference block 1402 may be included in a reference image of either reference picture list 0 or 1, and reference block 1404 may be included in a reference image of the other reference picture list 0 or 1. As shown in Figure 14, reference block 1402 is in a picture that precedes the current picture of the current block 1400 with respect to the picture order count (POC), and reference block 1402 is in a picture that advances the current picture of the current block 1400 with respect to the POC. In other examples, the reference pictures may both precede the current picture or both advance the current picture in terms of the POC. The POC is, for example, the order in which pictures are output from the decoded picture buffer, and the order in which pictures are generally intended to be displayed. However, the output picture may not necessarily be displayed, but may undergo different processing and / or consumption, such as transcoding. In other embodiments, two reference blocks, determined and / or generated using biprediction, may originate from the same reference picture. In such cases, the reference picture may be included in both reference picture list 0 and reference picture list 1.

[0080] The configurable weights and offset values ​​can be applied to one or more interprediction reference blocks. The encoder can enable the use of weighted predictions using flags in the Picture Parameter Set (PPS) and signal the weighting and offset parameters in the slice segment header for the current block. Different weighting and / or offset parameters can be signaled for luminance and / or chrominance components.

[0081] Once reference blocks 1402 and 1404 are determined and / or generated for the current block 1400 using interpretation, the encoder may determine the difference between the current block 1400 and each of the reference blocks 1402 and 1404. This difference may be called the prediction error or residual. The encoder may then store and / or signal the prediction error and / or their respective associated motion information within the bitstream for decoding or other forms of consumption. Motion information for reference block 1402 may include the motion vector 1406 and the reference index of the reference picture containing reference block 1402 into the reference picture list. In other cases, motion information for reference block 1402 may include a representation of the motion vector 1406 and a representation of the reference index of the reference picture containing reference block 1402 into the reference picture list. Motion information for reference block 1404 may include motion vector 1408 and a reference index to the reference picture list of the reference picture containing reference block 1404. In other cases, motion information for reference block 1404 may include a representation of motion vector 1408 and a representation of a reference index to the reference picture list of the reference picture containing reference block 1404. The decoder may decode the current block 1400 by determining and / or generating reference blocks 1402 and 1404, which together form a prediction for the current block 1400 using their respective motion information and by combining the prediction with the prediction error.

[0082] In HEVC, VVC, and other video compression schemes, motion information can be predictively encoded before it is stored in the bitstream or signaled. Motion information for the current block can be predictively encoded based on the motion information of one or more blocks adjacent to the current block. In general, the motion information of adjacent blocks can often correlate with the motion information of the current block because the motion of objects represented in the current block is often the same (or similar) to the motion of objects in adjacent blocks. Two motion information prediction techniques in HEVC and VVC may include advanced motion vector prediction (AMVP) and / or interpredictive block merging.

[0083] Encoders, such as encoder 200 in Figure 2, can encode motion vectors using the AMVP tool as the difference between the motion vector of the current block being coded and a motion vector predictor (MVP). The encoder can select an MVP from a list of candidate MVPs. Candidate MVPs may originate from previously decoded motion vectors of blocks adjacent to the current picture of the current block, and / or blocks near the current block's collocation location of other reference pictures. The encoder and decoder can mutually generate or determine lists of candidate MVPs.

[0084] After the encoder selects an MVP from a list of candidate MVPs, it may signal the display of the selected MVP and the Motion Vector Difference (MVD) in the bitstream. The encoder may indicate the selected MVP in the bitstream by an index that indicates the list of candidate MVPs. The MVD may be calculated based on the difference between the motion vector of the current block and the selected MVP. For example, for a motion vector represented by a horizontal component (MVx) and a vertical displacement (MVy) with respect to the current block's position being coded, the MVD may be represented by two components, calculated as follows:

number

[0085] In HEVC and VVC, the AMVP candidate MVP list may include two candidates, called Candidate A and Candidate B. Candidates A and B may include up to two spatial candidate MVPs derived from five spatially adjacent blocks of the current block being coded, one temporal candidate MVP derived from two temporal collocation blocks if both spatial candidate MVPs are unavailable or identical, or a zero motion vector if no spatial, temporal, or both candidates are available. Figure 15A shows the locations of five spatially adjacent candidate blocks relative to the current block 1500 being encoded. The five spatially adjacent candidate blocks may be A0, A1, B0, B1, and B2, respectively. Figure 15B shows the locations of two temporal collocation blocks relative to the current block 1500 being coded. The two temporal collocation blocks are shown as C0 and C1 and are contained in a reference picture different from the current picture of the current block 1500.

[0086] An encoder, for example, encoder 200 shown in Figure 2, may encode motion vectors using an interpredictive block merging tool called merge mode. Using merge mode, the encoder may reuse the same motion information from adjacent blocks for interprediction of the current block. Because the same motion information from adjacent blocks is used, there is no need to signal to MVD, and the signaling overhead for signaling the motion information of the current block can be small. Similar to AMVP, both the encoder and decoder may generate a candidate list of motion information from blocks adjacent to the current block. The encoder may then decide to use (e.g., inherit) the motion information of one adjacent block in the candidate list to encode the motion information of the current block. The encoder may signal the representation of the determined motion information from the candidate list within the bitstream. For example, the encoder may signal an index pointing to the list of candidate motion information to indicate the determined motion information.

[0087] In HEVC and VVC, the candidate motion information list for merge mode may include up to four spatial merge candidates derived from five spatially adjacent blocks used in AMVP, as shown in Figure 15A; one temporal merge candidate derived from two temporal collocation blocks used in AMVP, as shown in Figure 15B; and additional merge candidates, including bi-prediction candidates and zero-motion vector candidates.

[0088] Interpretation can be performed by other methods and variations other than those described above. For example, motion information prediction techniques other than AMVP and merge mode are available. Furthermore, although the above description has been made primarily in relation to interpretation modes in HEVC and VVC, it will be understood that the techniques of this disclosure described above and further described below may be applicable to other interpretation modes, including those of other video coding standards such as VP8, VP9, ​​and AV1. In addition, history-based motion vector prediction (HMVP), combined intra / interpretation modes (CIIP), and / or merge modes with motion vector difference (MMVD) (for example, as described in VVC) can be performed and are within the scope of this disclosure.

[0089] In interpretation, block matching techniques can be applied to determine a reference block in a picture that is different from that of the current block being encoded. Block matching techniques can also be applied to determine a reference block in a picture that is the same as the current block being encoded. However, in the case of video captured by a camera, it has been found that reference blocks within the same picture as the current block, determined using block matching, often fail to accurately predict the current block. This is generally not the case for screen content video, which may include, for example, computer-generated text, graphics, and animations. Screen content often contains repeating patterns (e.g., repeating patterns of text and / or graphics) within the same picture. Therefore, applying block matching techniques to determine a reference block within the same picture as the current block being encoded can provide efficient compression for screen content video.

[0090] Both HEVC and VVC include predictive techniques to leverage inter-block correlations of samples within the same picture of screen content video. This technique is called intra-block copying (IBC) or current-picture referencing (CPR). Similar to inter-prediction, the encoder may apply block-matching techniques to determine a displacement vector (referred to as a block vector (BV)) that indicates the relative displacement from the current block to a reference block (or intra-block corrected prediction) that "best matches" the current block. The encoder may determine the best-matching reference block from the tested blocks during a search process similar to that used for inter-prediction. The encoder may determine that a particular reference block is the best-matching reference block based on one or more cost criteria, such as a rate-distortion criterion (e.g., Lagrangian rate-distortion cost). One or more cost criteria may be based on, for example, the difference between the predicted sample of the reference block and the original sample of the current block (e.g., the sum of squared differences (SSD), the sum of absolute differences (SAD), the sum of absolute transform differences (SATD), or a difference determined based on a hash function). The reference block may correspond to a previously decoded block of the current picture, of a sample. The reference block may contain a decoded block of the current picture, of a sample, before being processed by an in-loop filtering operation, e.g., deblocking, and / or SAO filtering. Figure 16 shows an example of IBC applied to screen content. The rectangular portions with arrows beginning at their boundaries are the current blocks to be encoded, and the rectangular portions indicated by the arrows are reference blocks for predicting the current blocks.

[0091] Using IBC, once a reference block is determined and / or generated for the current block, the encoder may determine the difference between the reference block and the current block (e.g., the corresponding sample-by-sample difference). This difference may be called the prediction error or residual. The encoder may then store and / or signal the prediction error and associated prediction information within the bitstream for decoding or other forms of consumption. The prediction information may include BV. In other cases, the prediction information may include a representation of BV. A decoder, such as decoder 300 in Figure 3, may decode the current block by using the prediction information to determine and / or generate a reference block that forms the prediction for the current block, combining the prediction with the prediction error.

[0092] In HEVC, VVC, and other video compression schemes, the BV (Block Value) can be predictively coded before being stored in or signaled to the bitstream. The BV for the current block can be predictively coded based on the BVs of blocks adjacent to the current block. For example, an encoder may predictively code the BV using merge mode as described above for interprediction, or a technique similar to AMVP as described above for interprediction. Techniques similar to AMVP may be called BV prediction and differential coding.

[0093] BV prediction and differential coding: An encoder (e.g., encoder 200 shown in Figure 2) can code the BV as the difference between the BV of the current block being coded and a BV predictor (BVP). The encoder can select a BVP from a list of candidate BVPs. Candidate BVPs may originate from previously decoded BVs of the current picture, of adjacent blocks to the current block. Both the encoder and the decoder can generate or determine a list of candidate BVPs.

[0094] After the encoder selects a BVP from the list of candidate BVPs, the encoder may signal the selected BVP and an indication of a BV difference (BVD) in the bitstream. The encoder may indicate the selected BVP in the bitstream by an index indicating the list of candidate BVPs. The BVD may be calculated based on the difference between the BV of the current block and the selected BVP. For example, for the position of the current block to be coded, the horizontal component (BV x ), and the vertical component (BV y ), the BVD for the BV represented by these may be represented by two components as follows: [Equation] In the formula, BVD x , and BVD y represent the horizontal component and the vertical component of the BVD, respectively, and BVP x , and BVP y represent the horizontal component and the vertical component of the BVP, respectively. A decoder, for example, the decoder 300 shown in FIG. 3, may decode the BV by adding the BVD to the BVP indicated in the bitstream. Then, the decoder may decode the current block by using the decoded BV to form a prediction of the current block, combining the prediction with a prediction error, determining a reference block, and / or generating the reference block.

[0095] In HEVC and VVC, the list of candidate BVPs may include two candidates, referred to as candidate A and candidate B. Candidates A and B may include at most two spatially adjacent candidate BVPs derived from five spatially adjacent blocks of the current block being encoded, or one or more of the last two encoded BVs when no spatially adjacent candidates are available (for example, because they are to be coded in intra-mode or inter-mode). The positions of the five spatially adjacent candidate blocks relative to the current block being coded using IBC are the same as those shown in Figure 15A for inter-prediction. The five spatially adjacent candidate blocks may be A0, A1, B0, B1, and B2, respectively.

[0096] The Most Probable Mode (MPM) refers to the intra-predictive mode that is most likely to be the best mode for the current block to be encoded or decoded. In current intra-predictive techniques, the MPM is determined by analyzing the intra-predictive modes of adjacent CUs. The VVC uses a list of six MPMs (called the "MPM list") for luminance intra-prediction. The MPM list is derived from the intra-predictive modes of adjacent CUs and is updated as the encoder progresses through video frames. When encoding a block, the encoder may first check whether the current block is a candidate for any of the MPMs in the MPM list. If so, the encoder then compares the prediction errors of each MPM to determine which MPM is the best mode for the current block from the MPM list. If the current block is not a candidate for any of the MPMs in the MPM list, the encoder may then evaluate all intra-predictive modes (e.g., 67 in the VVC) to determine the best mode for the current block.

[0097] The use of MPMs can significantly improve the coding efficiency of VCVs. This is because, if the encoder is one of the MPMs, it does not need to signal the intra-predictive mode for the current block. Instead, the decoder can infer the intra-predictive mode for the current block from the corresponding MPM list.

[0098] Three types of intra-modes are considered to construct the MPM list: default intra-mode, adjacent intra-mode, and derived intra-mode. The six integrated MPM lists are used for intra-blocks, regardless of whether multiple reference lines (MRL) and intra-subpartition (ISP) code tools are applied. The MPM list for the current block is constructed based on the intra-modes of the current block's left neighbors and upper neighbors.Assuming that the mode of the left neighboring block is indicated as "left" and the mode of the upper neighboring block is indicated as "up", the unified MPM list can be constructed as follows: When a neighboring block is unavailable, its intra-mode is set to planar mode by default, and both the left mode and the upper mode are non-angled modes, then the MPM list is set to include {planar, DC, V, H, V - 4, V + 4}, where "V" and "H" refer to the vertical mode and the horizontal mode, respectively, and if one of the left and upper modes is an angled mode, the other is non-angled, and in the left and upper modes, the maximum mode is set as the larger mode, and the MPM list is set to {planar, Max, Max - 1, Max + 1, Max - 2, Max + Set to include {planar, Left, Above, Min-1, Max+1, Min-2}, if both Left and Above are angles and different, set Mode Max and Mode Minimum as larger modes for Left and Above, and Mode Minimum as smaller modes for Left and Above, if Max-Min is equal to 1, then set MPM list to include {planar, Left, Above, Min-1, Max+1, Min-2}, if Max-Min is 62 or greater, then set MPM list to include {planar, Left, Above, Min+1, Max-1, Min+2}, if Max-Min is equal to 2, set MPM list to include {planar, Left, Above, Min+1, Min-1, Max+2}, otherwise set MPM list to include {planar, Left, Above, Min+1, Min-1, Max+ Set the MPM list to include {planar, Left, Left - 1, Left + 1, Left - 2, Left + 2} if both left and top are angles and they are the same.

[0099] The encoder may encode the MPM index in the bitstream to indicate to the decoder the location of the selected intra-predictive mode in the MCM list. The decoder may derive the MPM list in the same manner as the encoder and, using the MPM index obtained from the bitstream, may obtain the intra-predictive mode from the MPM list derived by the decoder. In some cases, the first bin of the MPM index codeword is CABAC context-coded to achieve additional coding efficiency. Three contexts are used, corresponding to whether the current intra-block is MRL enabled, ISP enabled, or a normal intra-block.

[0100] During the six MPM list generation processes, panning may be used to remove duplicate modes so that only unique modes can be included in the MPM list. For the 61 non-MPM modes (i.e., 67 modes of VC minus 6 MPMs), a shortened binary code (TBC) may be used for entropy coding.

[0101] In some implementations, the MPM list is expanded to include 16 additional candidates and is divided into two parts: a primary MPM (PMPM) (6 entries) and a secondary MPM (SMPM) (16 entries). In some implementations, the first entry in the general MPM list is the planar mode. The remaining entries are the intra-modes of adjacent neighboring blocks corresponding to the left (L), top (A), bottom left (BL), top right (AR), and top left (AL) positions (e.g., shown in Figure 15A as A1, B1, A0, B0, and B2), as well as decoder-side intra-mode derivation (DIMD) modes sorted in ascending order of cost, such as SAD, SSD, SATD. Up to five modes with the lowest cost are added to the MPM list. The cost is calculated between the prediction and a reconstructed sample of the current block's template. The sorted directional modes are added to the general MPM list, then the default mode, until a general MPM list containing 22 entries is constructed. In some embodiments, if the CU blocks are oriented vertically, the order of adjacent blocks corresponds to A, L, BL, AR, AL; otherwise, it is L, A, AL, AR, BL.

[0102] Intra-Template Matching Prediction (Intra-TMP) is a special intra-prediction mode that copies the best prediction block from the reconfigured portion of the current frame, whose template (e.g., L-shaped template, top-only template, or left-only template) matches the current template (i.e., the L-shaped template, top-only template, or left-only template of the current block). For a predefined search range, the encoder searches the reconfigured portion of the current frame for the template most similar to the current template and uses the corresponding block as the prediction block. SAD cost, or another cost measure (e.g., SSD, SATD), may be used as a cost function to evaluate similarity. The encoder then signals the use of this mode, and the same prediction operation is performed in the decoder.

[0103] The prediction for the current block is generated by matching the current block's L-shaped, top-only, or left-only causal adjacencies with other blocks within a predefined search area shown in Figure 17, which consists of the current CTU(R1), top-left CTU(R2), top CTU(R3), and left CTU(R4).

[0104] Within each region R1-R4, the decoder constructs a list of candidate template matching block vectors up to a given maximum value (e.g., 19), ranked in ascending order according to their template cost (e.g., SAD cost). The following modes are supported: single predictor, fused multiple predictors, subplane precision, and linear filter model. In single predictor mode, a single predictor is selected from the candidate list. In fused multiple predictor mode, multiple predictors are blended to derive the final predicted block. Blend weights are calculated from the template matching cost of each predictor or using a Wiener filter-based weight derivation method. In subplane precision mode, when using a single predictor, subplane precision can be used with 1 / 2 plane precision, 1 / 4 plane precision, and 3 / 4 plane precision, each with 8 possible directions. In linear filter model mode, a linear filter can be learned between a reference template and the current template and applied to the reference block. This mode can be used for single predictor when subplane precision is not used.

[0105] The dimensions of the region, specifically the width and height of the search range (SearchRange_w, SearchRange_h), are set proportionally to the block dimensions (BlkW, BlkH) and have a fixed number of cost comparisons per pixel. That is: SearchRange_w = min (64, a * BlkW); and SearchRange_h = min (64, a * BlkH), where "a" is a constant that controls the gain / complex tradeoff. In practice, "a" can be set to equal to 5.

[0106] To expedite the template matching process, the search range of all search areas is subsampled by a factor (e.g., factor 3). After finding the best match, a refinement process is performed. Refinement is carried out through a second template matching search around the best match with a reduced range.

[0107] The IntraTMP tool is enabled for CUs with a width and height of 64 or less. The maximum CU size for IntraTMP is configurable. IntraTMP prediction mode is signaled at the CU level via a dedicated flag if DIMD is not used for the current CU.

[0108] Decoder-side intra-mode derivation (DIMD) is an intra-code mode in which the intra-predictive mode is not transmitted within the bitstream (after being determined by the encoder, for example, by a rate-distortion optimization (RDO) algorithm), but is derived by using gradient analysis of adjacent reconstructed pixels. In other words, with DIMD, the intra-predictive mode is implicit to the decoder. DIMD is signaled by a flag, and the intra-predictive mode is derived during the reconstruction process at the decoder in the same manner as the intra-predictive mode is derived at the encoder. If DIMD is not selected at the encoder as the intra-code mode for the current block, the intra-predictive mode may be analyzed from the bitstream at the decoder, similar to an intra-code mode process that signals the intra-predictive mode within the bitstream.

[0109] In DIMD, a template region (consisting of left, top, and top-left regions) with a three-sample width (width, or height) of the current block is defined, where an edge detection filter adjacent to the current block (e.g., a 3x3 horizontal and vertical Sobel filter) is applied to all 3x3 window positions centered on the midline pixels of the template region to determine the amplitude and brightness direction (orientation) angle for each midline sample in the template region. A gradient direction histogram (HoG) is calculated, with each entry in the HoG corresponding to its respective conventional in-angle mode. For example, G hor , and G ver However, if the intensity is purely horizontal and vertical, respectively, the angle is calculated relative to the window as follows, as calculated using a Sobel filter for a single 3x3 window position:

number

number

[0110] Figure 18 shows examples of DIMD template zones for calculating HoG according to several embodiments. The reconstructed region 1806 for the current block 1802 is shown similarly to the L-shaped template 1804 for the current block 1802. It should be understood that the shape of this template does not necessarily have to be L-shaped, and may be, for example, an upper template (e.g., including the upper sample of the current block 1802) or a left template (e.g., including the left sample of the current block 1802). The current block 1808, which is an exemplary 4x4 pixel square, is shown in a 3x3 window 1810 within the template region. HoG 1812 is a histogram corresponding to the current block 1808. Another embodiment is the current block 1814, which is a rectangle with a 3-pixel deep template region 1816. HoG 1818 corresponds to the rectangular current block 1814. As shown, each coordinate position on the x-axis of the shown HoG represents the respective intra-prediction mode (IPM), and the y-axis represents the cumulative amplitude for each intra-prediction mode.

[0111] Figures 19A and 19B show a flowchart 1900 of an exemplary DIMD predictor derivation process. The operation of flowchart 1900 is implemented (e.g., carried out) by an encoder (e.g., encoder 200 in Figure 2) and / or a decoder (e.g., decoder 300 in Figure 3).

[0112] In step 1902, the availability of samples within the template region is determined, for example, in the upper right, lower left, and upper left CUs of the current block. For example, the template region could be an exemplary L-shaped template 1804 of the current block 1802 shown in Figure 18. In step 1904, the HoG of the available template region is constructed. The HoG structure of the template region is as described above with respect to Figure 18. In step 1906, the maximum predetermined number (e.g., 5) of the most represented intra-prediction modes is identified based on the highest amplitude of the HoG. In step 1908, the position dependence of each identified intra-prediction mode is determined as the ratio of amplitudes in the template region. In some implementations, the position dependency of identified intra-predictive modes can be determined as follows: The default position dependency is set to 0 (i.e., no position dependency). If the amplitude of intra-predictive modes in the left template (the sum of the amplitudes of identified intra-predictive modes from samples in the template region of the current block) is smaller than the average amplitude of intra-predictive modes across all templates (e.g., from three template regions: left template, top template, and top-left template), then the position dependency is set to 1 (vertical). Otherwise, if the amplitude of intra-predictive modes in the top template is smaller than the average amplitude of intra-predictive modes across all templates, then the position dependency is set to 2 (horizontal). This calculation can be facilitated by generating HoGs by separately accumulating the amplitudes for the left template region, top template region, and top-left template region in different HoGs, and then summing the three HoGs to determine the combined HoG for the entire template region.

[0113] At 1910, it is determined whether mode blending is enabled. If mode blending is disabled, then at 1912, it is determined to use one of the identified intra-prediction modes (e.g., the one corresponding to the highest amplitude) as the DIMD predictor. Alternatively, at 1910, it is determined that mode blending is enabled, then at 1914, the number of angular intra-prediction modes to be blended is determined. Mode blending may be enabled, for example, when both the first and second identified intra-prediction modes are angular modes. At 1916, it is determined whether the number of modes to blend is greater than 1. If the number of modes to blend is 1 or less, then at 1918, the DIMD predictor is determined by blending one identified angular intra-prediction mode and a planar mode (e.g., weights 2 / 3 and 1 / 3, respectively). At 1916, it is determined that the number of modes to blend is greater than 1, then at 1920, the blend weights of the identified intra-prediction modes are determined according to their amplitude ratio. Planar modes can be used in blends, for example, with 1 / 4 weights. In 1922, the selected intra-prediction mode is generated as a result of blending the identified intra-prediction modes with the planar modes. As described below in relation to Figure 20, the blending of identified intra-prediction modes and planar modes to derive the selected intra-prediction mode for the current block may involve computing all identified intra-prediction modes and planar modes using adjacent reconstructed samples.

[0114] In some implementations, the selected intra-prediction mode determined in 1922 is used as the DIMD predictor for the current block.

[0115] Alternatively, some implementations may further use position-dependent blend modes to derive DIMD predictors. The blend weights determined in 1920 are applied uniformly to all samples of the predictor and may be called "uniform weights". For position-dependent blends, for example in 1924, the sample-based weights for the predictor (e.g., for each identified DIMD intra-prediction mode) are calculated such that the average weight used within the block is approximately equal to the corresponding uniform weight (e.g., calculated in 1920), and that higher weights are used in the left template region or in parts of the block closer to the template region, depending on the position dependency of the DIMD intra-prediction mode. The range is predefined and corresponds to the maximum deviation of the sample-based weights from the corresponding uniform weight. A higher value for the predefined range results in greater weight variability within the block. After the blend weights for each position dependency (e.g., vertical, horizontal, no position dependency) are calculated in 1924, the predictions for each position dependency are blended in 1926. For example, for each of the horizontal, vertical, and non-positional (i.e., diagonal positional) prediction modes used by any of the identified intra-prediction modes of the DIMD predictor, the identified intra-prediction modes with that positional dependency are blended based on their respective amplitudes. In 1928, the DIMD predictor is obtained by blending together each positional dependency predictor (horizontal, vertical, and non-positional) according to the respective sums of the amplitudes of the identified intra-prediction modes for each positional dependency. This provides a directional sample-wise blend that depends on the positional dependency and uses fixed weight deviations calculated in 1924.

[0116] In an exemplary implementation, the DIMD intra-prediction mode, determined at 1922 or 1928, may be added to the MPM list along with other intra-prediction modes whose use in the current block is considered. If, subsequently, rate distortion cost calculations or similar determine that the DIMD intra-prediction mode is the best (i.e., most cost-effective) intra-prediction mode, then the current block is encoded using the DIMD intra-prediction mode, and only a flag indicating that the current block is transmitted on the bitstream using the residuals of the current block is present. The intra-prediction mode is not transmitted on the bitstream.

[0117] In an exemplary implementation, up to a predetermined number (e.g., 4) angular intra-prediction modes are selected from the HoG and combined with planar modes (e.g., in planes with a ratio of 1 / 4, i.e., 16 / 64, with 6-bit integer precision) (as described, for example, in relation to 1920 or 1928 above). An example of combining (also called "fusion" or "blending") angular intra-prediction modes from the HoG is shown in Figure 20. Figure 20 shows a blend of two angular intra-prediction modes and a planar mode. Intra-prediction modes M1 and M2 are selected for the current block 2006 based on the two highest peaks in the HoG 2004. The prediction block 2008 for the current block 2006 is determined by a combination (fusion) operation 2002 that combines predictions based on intra-modes M1 and M2, as well as the planar mode, based on weights w1, w2, and w3. The preceding weights for each angle intra-prediction mode are determined according to the amplitude of the corresponding gradient in the histogram, and a given weight can be used for the planar mode.

[0118] The DIMD process is performed before the MPM list is constructed, as the derived intra-mode is included in the MPM list. The primary derived intra-predicted mode of a DIMD block is saved with the block and used to construct the MPM list of adjacent blocks. When DIMD is signaled, the MPM and MRL are assumed to be 0 and are not signaled.

[0119] Template-based intra-mode derivation (TIMD) is a recently proposed intra-code mode in which a template zone is set within a reconfigured portion of the current frame and used to derive an intra-predictive mode for the current block. In TIMD mode, the intra-predictive mode is not transmitted in the bitstream (after determination by the encoder, e.g., by a rate-distortion optimization algorithm), and only the use of TIMD is signaled, thereby saving intra-predictive mode signaling bits. More specifically, TIMD use is signaled at the CU level by a flag, and the TIMD-derived intra-predictive mode is identically derived in the encoder and decoder during the reconstruction process when TIMD is enabled. If TIMD is not selected in the encoder, the intra-mode can be parsed from the bitstream in the decoder, like an intra-mode in which the intra-predictive mode is signaled in the bitstream.

[0120] For each intra-prediction mode in the MPM list, the SATD cost (or SAD, SSD cost) between the template's predicted sample and the reconstructed sample is calculated. The intra-prediction mode with the lowest cost is selected as the TIMD mode and used for intra-prediction of the current block. In TIMD, either a template region (consisting of left, top, and top-left regions) with a sample width (width or height) of 2 samples or 4 samples is defined. The template's reference sample is used to derive the predictor for the template region. Figure 21 shows an example of the current block 2102, which is NxN pixels, with a template 2109 of L pixel width and a corresponding reference sample 2106 of the template.

[0121] As shown in Figure 22, an embodiment of the TIMD process 2200 includes an operation 2202 for deriving TIMD modes and weights, and an operation 2204 for generating TIMD predictors. The operations of the TIMD process 2200 can be implemented (e.g., performed) by an encoder (e.g., encoder 200 in Figure 2) and / or a decoder (e.g., decoder 300 in Figure 3). A list of candidate intra-prediction modes is set (e.g., consisting of an MPM list, which may be extended mainly to DC, horizontal, and vertical modes), and each candidate mode is checked as an intra-prediction of the template using a cost function (e.g., SAD, SSD, or SATD) which calculates a template matching score between the reconstructed samples in the template and the predicted samples of the template. In 2204, two intra-prediction modes that yield the lowest cost (see 2206, 2208, and 2210) are selected for refinement, with the angular mode range expanded from 67 modes to 131 modes, and two adjacent modes (i.e., + / -1) of each selected intra-prediction mode being checked. Then, in 2212, the selected candidates are chosen as two TIMD modes. Finally, the two TIMD modes are blended with weights determined from their respective costs to generate a TIMD predictor.

[0122] Another recent intra-mode prediction technique proposes merging DIMD HoG information from neighboring blocks to predict the current block. If DIMD-salted neighboring blocks are available, the DIMD histograms of selected neighboring CUs are combined to form a merged gradient direction histogram (MHoG) for the current block. DIMD modes and weights are derived from this merged histogram. The HoG of the current block is not used in constructing the MHoG.

[0123] The DIMD merge process is similar to the DIMD process described above in relation to Figure 19, and may differ from that process only by including a series of operations preceding 1910. In the proposed DIMD HoG merge process, at most three adjacent CUs are coded as intra-DIMD and selected from up to 13 available adjacent CUs closest to the current block. The HoG of the selected neighboring CUs, or the merged HoG, is obtained. The merged HoG is calculated from the HoG (or merged HoG) of the selected CUs. The blend mode is determined in the same manner as described above in relation to DIMD, depending on the merged HoG. Up to five intra-prediction modes are selected as in the normal DIMD mode (e.g., based on the highest amplitude of the merged HoG), and the position dependency is determined. If multiple intra-prediction modes are selected, the position dependency is set to 0 (diagonal); otherwise, the position dependency is set to the position dependency of the selected intra-prediction modes. The current block's DIMD HoG merge mode intra-prediction mode can be determined by combining the selected intra-prediction modes.

[0124] In existing techniques, decoder-side intra-mode derivation is at least partially attractive because it can reduce the amount of information required for signaling from encoder to decoder. However, in previous DIMD techniques, DIMD modes and weights are derived from reconstructed neighboring samples (template region) by calculating gradients, which can be as small as a 3x3 template size. This can often not produce the best adjacency, as the further away from the template region the current block sample is from the template region, the less likely it is that there is a high spatial correlation between the template region and the current block. Furthermore, behavior is not handled in existing DIMD techniques to improve the derived blend modes. For example, HoG-like pruning, or behavior on merge peaks, can add diversity to the derived blend modes, which can be beneficial in providing more accurate intra-predictions.

[0125] Furthermore, unlike some other intra-modes, DIMD (see Figure 19) does not reuse neighbor information. Moreover, DIMD HoG merging may only be considered if at least one neighbor uses the DIMD mode (or DIMD HoG merge mode). As mentioned above, the DIMD process is used by some intra-prediction modes (e.g., non-angle modes in particular for obtaining directional information) and during the MPM derivation process, even if the neighbor mode is not DIMD. Compression efficiency can be improved by using correlations between blocks. As a result, intra-mode efficiency can be further improved by using DIMD information from neighbor CUs.

[0126] Furthermore, in DIMD mode and DIMD HoG merge mode, combining HoGs can dilute the relevance of neighboring HoGs because no operation is processed on the HoGs themselves. HoGs are combined without weights (e.g., using only amplitude averages to derive the MHoG) or without processing operations (e.g., pruning, densification), thus allowing unrelated (or less related) neighboring HoGs to distort the MHoG and dilute the peaks of related modes. In fact, spatial correlations can be significant, but they can also be irrelevant, as pictures may contain different spatial regions with distinct characteristics.

[0127] These drawbacks can compromise the reliability and competitiveness of DIMD compared to other intra modes.

[0128] Embodiments of this disclosure relate to an improved approach to DIMD that can more reliably obtain a more optimal intra-mode. These and other features of this disclosure are further described below.

[0129] The intra-prediction process described above provides 67 intra-prediction modes (planar, DC, and 65 angular directions) for each prediction block. Luminance intra-prediction modes are selected for luminance prediction blocks, and chrominance intra-prediction modes are selected for chrominance prediction blocks. Mode-dependent references and smoothing of prediction samples are applied to increase prediction efficiency, and intra-prediction modes are coded using either one of the 6 most likely modes (MPMs) or one of the remaining 61 modes. Note that 28 wide-angle modes (rectangular CUs only) can be used for regular mode swapping.

[0130] Exemplary embodiments of this disclosure provide a technique for a decoder to efficiently derive an intra-predictive mode to be applied to the current block without signaling from the encoder. More specifically, the technique combines the advantages of DIMD and inter-predictive merge modes to obtain a novel, efficient intra-predictive mode, referred to herein as “DIMD merge mode”. Several embodiments of this disclosure provide enhancements to the DIMD merge mode.

[0131] Using the DIMD merge mode, for each predicted CU, the embodiment includes building a list of DIMD information and selecting the CU to be used to reconstruct the current block. This list, called the “list of DIMD merge candidates,” is built from DIMD information from various spatial neighbors (neighboring blocks in the current frame). As mentioned above, since DIMD is used to generate the MPM list, DIMD information may already be available for most in-neighbor CUs (e.g., spatially adjacent blocks). In some embodiments, such neighbor CUs may include CUs that have used DIMD HoG merge.

[0132] The initial list of DIMD merge candidates may be constructed according to a given check order (also called the "inclusion order") for each DIMD merge candidate family, for example, in the order of DIMD information for spatially adjacent candidates, DIMD information for spatially non-adjacent candidates, DIMD information for history-based spatial candidates, DIMD information for combined candidates, and / or default DIMD information candidates.

[0133] In some embodiments, the list of DIMD merge candidates may have a predetermined maximum number of DIMD merge candidates, and the generation process may be stopped when the maximum number is reached (e.g., when the list is full). Furthermore, the list generation process may ensure that DIMD merge candidates with duplicate DIMD information are not added to the list. The list generation process may also ensure that the number of DIMD merge candidates from each family of DIMD merge candidates does not exceed a predetermined maximum threshold.

[0134] In some embodiments, after an initial version of the list of DIMD merge candidates is constructed, the list may be sorted according to criteria such as template matching-based sorting and / or direction-based sorting. Alternatively, template matching-based sorting may be applied only to a portion of the list of DIMD merge candidates. The template matching cost of the candidate's DIMD information during the sorting process may be measured by the cost difference (e.g., SAD, SSD, SATD, etc.) between the current block's template sample and their corresponding reconstructed adjacent samples. The lower the candidate's template matching cost, the closer the candidate is to the beginning of the list. Sorting the list can improve the encoding efficiency of the DIMD merge mode by selecting smaller indices that are more efficiently encoded. By sorting the list of DIMD merge candidates, the DIMD merge candidates selected by the encoder are more likely to have lower indices that can be signaled more efficiently.

[0135] Once the list is complete and sorted (e.g., arbitrarily), a DIMD merge candidate is selected with the best DIMD information for the current block. In the encoder, this selection may be based on the rate-distortion optimization (RDO) cost of the selected DIMD merge candidate. The selected DIMD merge candidate may be derived in the decoder according to an indicator signaled by the encoder. This indicator shows which entry in the list of DIMD merge candidates (i.e., which candidate from the constructed list) to use to process the block reconstruction. The list of DIMD merge candidates stores all the DIMD information from spatially neighboring blocks necessary to reconstruct the current block, such as the blended intra-prediction modes, corresponding blend weights, and position-dependency information. This DIMD information defines the DIMD merge candidate and may be called a "DIMD parameter." In some embodiments, additional information, such as HoG information, may be included in the DIMD parameter.

[0136] A flowchart 2300 of a method for a novel intra-prediction mode, a DIMD merge mode, according to some embodiments of the present disclosure, is shown in Figure 23. The method of flowchart 2300 includes DIMD merge information derivation operations 2302 and 2304, which derive blend modes, weights, and position dependencies. The method may be performed by an encoder (e.g., encoder 200 in Figure 2) or by a decoder (e.g., decoder 300 in Figure 3).

[0137] The method begins at 2302. The DIMD merge information derivation at 2302 may include several operations, such as those at 2306-2316, each of which may add a family of DIMD merge candidates to the list of DIMD merge candidates, selected from combinations of DIMD information already in the list or from configured default DIMD information, based on different selection criteria, and obtained from spatially neighboring blocks (i.e., intrablocks) of the current block. In some embodiments, the DIMD merge candidates in the list of DIMD merge candidates are arranged as a list or a table, but embodiments may not be limited by a particular arrangement structure of DIMD merge candidates.

[0138] In 2306, DIMD merge candidates from a family of spatially adjacent blocks are considered for addition to the list of DIMD merge candidates. Figure 24 shows adjacent neighbor samples of a block within a family of spatially adjacent blocks. In some embodiments, DIMD merge candidates from spatially adjacent blocks may be added to the list of DIMD merge candidates according to the block inclusion order corresponding to the left sample (L), top sample (A), upper-left sample (AL), upper-right sample (AR), and lower-left sample (BL). In some embodiments, a different inclusion order such as L, A, AR, BL, and AL, or yet another different inclusion order, may be used. Blocks within a family of spatially adjacent blocks are also considered in the derivation of the MPM list. In 2306, candidates 1 to 5 (all spatially adjacent) may be added to the DIMD merge candidate list. For a particular spatial block to be included in the DIMD merge candidate list, the block must already have at least the DIMD information derived for that block (i.e., during the reconstruction of the block in the current frame). DIMD information can be derived for blocks in DIMD or another intra-prediction mode that use DIMD parameters, such as DIMD HoG merge mode, DIMD merge mode, and MPM list derivation. In some embodiments, a merge candidate is added to the list of DIMD merge candidates only if the corresponding neighboring block is coded or decoded based on DIMD or another intra-prediction mode that uses DIMD parameters. In some cases, for example, if none of the adjacent spatially adjacent blocks are reconstructed using DIMD mode, DIMD HoG merge mode, or DIMD merge mode, or have no derived DIMD information, 2306 may not add any candidates to the list of DIMD merge candidates.

[0139] In 2308, DIMD merge candidates from a family of blocks that are not spatially adjacent to the current block are added to the list of DIMD merge candidates. The position and inclusion order of non-adjacent candidates within the same frame as the current block may follow a predetermined order, for example, as shown in Figure 25. The pattern and order of the blocks shown in Figure 25 are the same as those defined for inter-merge prediction candidates in some implementations.

[0140] The numbers 1-5 shown in Figure 25 refer to spatially adjacent neighbors of the current block (shown in shadow) in the inclusion order L, A, AR, BL, and AL, which are either already added to the merge candidate list in 2306 or are otherwise considered to be included, and therefore may not be considered in 2308. The sequence of blocks shown by the numbers 6-23 may be considered in 2308 to include the corresponding DIMD information in the list of DIMD merge candidates. As can be seen from Figure 25, the non-adjacent neighbors 6-23, which were considered for inclusion in the list of DIMD merge candidates in 2308, are located within the frame along the left horizontal, top vertical, top-left diagonal, top-right diagonal, and bottom-right diagonal directions.

[0141] For any DIMD information of a particular spatially non-adjacent block included in the list of DIMD merge candidates, that block must have had a DIMD used for encoding / decoding, or at least had DIMD information derived for that block. DIMD information can be derived for encoding / decoding based on DIMD, or another intra-prediction mode, using DIMD parameters, such as DIMD HoG merge mode, DIMD merge mode, and MPM list derivation. In some embodiments, a candidate is added to the list of DIMD merge candidates only if the corresponding neighboring block has been reconstructed based on DIMD, or another intra-prediction mode, using DIMD parameters. In some cases, for example, if none of the adjacent spatial candidates have been encoded / decoded using DIMD, DIMD HoG merge, or DIMD merge, or if no DIMD information has been derived, 2308 may not add any candidates to the list of DIMD merge candidates.

[0142] In 2310, a family of merge candidates from a history-based table of DIMD information is added to the list of DIMD merge candidates. The history-based table may be maintained to contain recently used DIMD information for intrablocks. This history-based table may be updated each time the CU intraprediction mode uses the DIMD process, i.e., when DIMD information is derived (for example, based on a block using a DIMD mode, or an MPM-based mode, or a DIMD or DIMD HoG merge mode, or a direction derived from a DIMD merge mode). This table has a limited number of entries for DIMD information from previous blocks and does not contain duplicate DIMD information.

[0143] For DIMD information for a particular spatial block included in a history-based table, that block must have had a DIMD used for encoding / decoding, or at least had DIMD information derived for that block. DIMD information can be derived for encoding / decoding based on DIMD or another intra-prediction mode using DIMD parameters, such as DIMD HoG merge mode, DIMD merge mode, and MPM list derivation. In some embodiments, an entry for DIMD information is added to the history-based table only if the corresponding neighboring block has been reconfigured based on DIMD or another intra-prediction mode using DIMD parameters. In some embodiments, exemplary embodiments may consider spatially adjacent blocks and spatially non-adjacent blocks separately according to a predetermined inclusion order, so that only DIMD information for spatial blocks that are neither spatially adjacent nor spatially non-adjacent according to a predetermined inclusion pattern is added to the history-based table.

[0144] When inserting a new entry of DIMD information into a history-based table, a constrained first-in, first-out (FIFO) rule may be used, in which a redundancy check is first applied to determine if identical DIMD information already exists in the table. If found, the entry containing identical DIMD information is removed from the table, and all entries in the table move forward in FIFO order, with the new DIMD information being inserted as the last entry. The table may be reset at intervals, for example, at the beginning of each CTU row. Such resets may be performed due to hardware constraints. Such resets may also help ensure that only the most recent neighboring blocks are present in the table. In some embodiments, the history-based table of DIMD information used may be analogous to the history-based table of motion vectors (HMVP) in the whole-block-based advanced motion vector predictor (AMVP) mode for interpretation in VVC.

[0145] Figure 26 is a flowchart 2600 of a method for adding DIMD information to a history-based table according to several embodiments. The operation of flowchart 2600 can be performed by an encoder (e.g., encoder 200 in Figure 2) or by a decoder (e.g., decoder 300 in Figure 3).

[0146] The method in flowchart 2600 begins in 2602 with considering the DIMD information of DIMD merge candidates for input into a history-based table. Such considerations may arise whenever spatially adjacent blocks use DIMD, or other intra-modes from which DIMD information is derived. In 2602, it is determined whether the DIMD information to be added already exists in the table. If it is determined in 2602 that the DIMD information to be added does not exist in the table, then in 2606, it may be determined whether the table is full. This determination may be based on a pre-configured maximum size threshold for the history-based table. If the table is not full, in 2610, the DIMD information to be added is added to the history-based table.

[0147] In step 2602, if it is determined that the DIMD information to be added is already present in the table, then in step 2604, any duplicate DIMD information already present in the table is removed from the table.

[0148] If, in step 2606, it is determined that the history-based table is full, then in step 2608, the oldest entry in the table is deleted.

[0149] At 2612, after either 2604 or 2608, entries following a deleted entry in the table are shifted up or moved sequentially (i.e., in a FIFO sequence). At 2612, after sorting the table entries, at 2610, the new DIMD information is added as the last entry in the table.

[0150] According to the exemplary inclusion order, DIMD merge candidates derived from the DIMD information in the history-based table are added to the list of DIMD merge candidates after considering DIMD merge candidates derived from blocks that are spatially adjacent to the current block, and blocks that are not spatially adjacent.

[0151] In 2312, a set of combined DIMD merge candidates, according to some embodiments of the present disclosure, is added to the list of DIMD merge candidates.

[0152] When all spatially adjacent, spatially non-adjacent, and historically spatially neighboring potential DIMD merge candidates, or more specifically, their respective DIMD information, are considered for inclusion in the list of DIMD merge candidates, the combined DIMD information may be added to the list of DIMD merge candidates. This process involves combining DIMD information from DIMD information of DIMD merge candidates that are already in the list of DIMD merge candidates. DIMD merge candidates that are already in the list but are based on combinations of other DIMD merge candidates in the list are not used to derive further combined DIMD merge candidates.

[0153] In some cases, the HoGs from at least two DIMD merge candidates in a list of DIMD merge candidates are combined to create a new combined HoG, from which the blend mode and weights are derived to obtain the DIMD information for the new DIMD merge candidate. For example, the HoGs of at least two merge candidates are averaged, and the blend mode and weight coefficients are derived from this averaged HoG in a manner similar to the derivation of the blend mode in DIMD, as described in relation to Figure 20. When considering the new combined DIMD merge candidate, the position dependency may be set to diagonal (0) as a default value, or it may be set according to the blend mode, i.e., the vertical position dependency is set to the vertical blend mode, or the horizontal position dependency is set to the horizontal blend mode.

[0154] A combined DIMD merge candidate can be derived as follows: a candidate derived from the first and second DIMD merge candidates in the list of DIMD merge candidates; a candidate derived from the first and third DIMD merge candidates in the list of DIMD merge candidates; a candidate derived from the second and third DIMD merge candidates in the list of DIMD merge candidates; a candidate derived from the two first DIMD merge candidates in the list of DIMD merge candidates with vertical position dependency; a candidate derived from the two first DIMD merge candidates in the list of DIMD merge candidates with horizontal position dependency; or a candidate derived from combining the DIMD information of all spatially adjacent blocks of the current block.

[0155] Alternatively or additionally, in some embodiments, the output DIMD information of the DIMD HoG merge mode is added as a combined candidate to the list of DIMD merge candidates.

[0156] Note that if a merge candidate cannot be derived, it will not be added to the list. For example, if the list of DIMD merge candidates contains only two candidates before adding a combined candidate, the second combined candidate, derived from the first and third candidates, will be skipped because there is no third candidate.

[0157] In 2314, a default set of DIMD merge candidates, according to some embodiments of the present disclosure, is added to the list of DIMD merge candidates.

[0158] One or more default DIMD information entries are added to the list of DIMD merge candidates as DIMD merge candidates. For example, the DIMD parameters of two exemplary default DIMD information entries are shown in Table 1. [Table 1]

[0159] The table shows default mode 0 and default mode 1, each specifying a predetermined set of DIMD parameters. One or more DIMD merge candidates having predetermined default DIMD information may be added to the list of DIMD merge candidates based on the DIMD parameters specified in Table 1.

[0160] If available, optionally adding a combination of merge candidates following the default merge candidates completes the addition of the merge candidates to the DIMD merge candidate list.

[0161] Operation 2302, which includes steps 2306-2314 and generates a list of DIMD merge candidates, can be identical in both the encoder and the decoder. After 2302 is completed (i.e., the list of DIMD merge candidates is constructed), in 2304 the blend mode, weights, and position dependencies are derived. In the encoder, this may involve determining the best DIMD merge candidate in the list of DIMD merge candidates based on the cost (e.g., SAD, SATD, etc.) difference between the predicted block obtained using the DIMD predictor and the current block, and / or based on the rate-distortion optimization of each DIMD merge candidate, starting from the top of the list of DIMD merge candidates (e.g., the lowest SAD, SATD cost). An index is determined for the position of the best (selected) DIMD merge candidate in the list of DIMD merge candidates. In some embodiments, the selected predictor can then be added to the MPM list. If the current block is predicted by the DIMD predictor and encoded for transmission based on the DIMD prediction, then the DIMD enabled flag and index can be transmitted on the bitstream.

[0162] In the decoder, 2304 may include parsing the bitstream for a DIMD-enabled flag or index, or both, against a list of DIMD merge candidates. Using the index, a selected DIMD merge candidate may be obtained, from which a DIMD predictor is derived.

[0163] The DIMD merge mode benefits from the use of available neighbor information (e.g., DIMD information derived particularly for spatially adjacent blocks) to better fit the signal characteristics of the current block to be predicted. Intra-block consistency is introduced by reusing adjacent DIMD parameters. Furthermore, the DIMD merge mode provides a less complex and competitive intra-mode derivation. Once an intra-mode is identified, only syntactic elements (e.g., an index to a list of DIMD merge candidates) can be signaled. Parameters of DIMD candidates can be arranged by relevance (e.g., using SAD, SATD, logic) to reduce mode signaling costs. The reuse of substantial parts of the DIMD process, e.g., low-level operations and computer-intensive operations such as block pixel reconstruction, remains unchanged, thereby saving hardware logic and potentially facilitating implementation by implementers.

[0164] Figure 27 is a flowchart 2700 of a method for determining an index to a list of DIMD merge candidates, according to some embodiments of the present disclosure. The method may be carried out in a decoder (e.g., decoder 300 in Figure 3).

[0165] When considering DIMD merge mode, an index is signaled, or derived, that points to an entry in a list of DIMD merge candidates (which are derived identically on both the encoder and decoder sides). One or more indicators may be signaled from the encoder to the decoder in order to proceed with signaling the DIMD merge index. The DIMD merge enabled flag is a syntactic element for signaling whether the DIMD merge index is signaled, and the DIMD merge index is a syntactic element for signaling the DIMD merge index.

[0166] The DIMD merge index refers to a candidate in the DIMD merge candidate list, used to reconstruct a block pixel (also called a selected candidate), and can be derived by the decoder in the following two cases: If the DIMD merge candidate list size is 1, i.e., there is only one candidate in the list, then the index can be derived to DIMD merge index = 0, thus bypassing the index signaling. Furthermore, if the DIMD merge enabled flag is signaled but equal to zero, then there is no DIMD merge index to extract from the bitstream, and it can be derived that the DIMD merge index is set to its default value, zero.

[0167] In all other cases, the DIMD merge enabled flag is equal to 1, the DIMD merge candidate list size is greater than 1, but the DIMD merge index is decoded from the bitstream. Optionally, the syntactic elements of the DIMD merge enabled flag may be discarded. In this case, if the DIMD merge candidate list size is greater than 1, i.e., the DIMD merge index is not derived, the DIMD merge index is read from the bitstream.

[0168] A list of DIMD merge candidates is derived, and in 2702, after being optionally sorted, the DIMD merge index is initialized to 0 in 2704. In 2706, it is determined whether the list of DIMD merge candidates has multiple entries. If yes, then in 2708, it is determined whether the merge index is signaled. In 2708, if the response is yes, then in 2710, the DIMD merge index is read from the bitstream. In 2712, the selected DIMD merge candidate is derived by accessing the merge candidate list using the DIMD merge index.

[0169] If, in step 2706, the merge candidate list size is determined to be 1, the method then proceeds to step 2712 to derive a DIMD merge candidate selected based on DIMD merge index 0. If, in step 2708, the DIMD merge index is determined not to be signaled, the method then proceeds to step 2712 to derive a DIMD merge candidate selected based on DIMD merge index 0.

[0170] According to some embodiments, in a decoder such as decoder 300 in Figure 3, for example, DIMD merge mode operation may be initiated when an encoded video bitstream is received, containing encoded video data and associated DIMD information. A DIMD enable flag and, optionally, a DIMD index may be parsed from the bitstream. For each encoded video frame, each CTU in the frame is decoded. For each CU of a CTU, if DIMD merge mode is enabled, a list of DIMD merge candidates is constructed. Constructing the list of DIMD merge candidates may include adding DIMD information from spatially adjacent blocks, spatially non-adjacent blocks, DIMD information from a history-based table, DMD information from combinations of DIMD merge candidates, and / or default DIMD information. In some embodiments, sorting of the merge candidate list may be performed optionally. DIMD parameters are derived from the DIMD merge candidate list using the signaled index (if signaled) in the bitstream. DIMD predictors are then generated according to the determined blend weights, modes, and position-dependent states of the predictors.

[0171] According to several exemplary embodiments, the DIMD merge mode operation in an encoder such as encoder 200 in Figure 2 may be performed for each encoded video frame. For each CU within each CTU in the frame, if DIMD is enabled, a list of DIMD merge candidates is constructed. Constructing the list of DIMD merge candidates may include adding DIMD information from spatially adjacent blocks, spatially non-adjacent blocks, DIMD information from a history-based table, DMD information from combinations of DIMD merge candidates, and / or default DIMD information. In some embodiments, sorting of the list of DIMD merge candidates may be performed optionally. The best DIMD merge candidate is then selected. The selection may include generating DIMD predictors for each (or some) DIMD merge candidates from the list of DIMD merge candidates according to the determined blend weights, modes, and predictor position-dependent states. Based on each predictor, the cost difference (e.g., SATD, SAD, etc.) between the predicted sample of the template and the reconstructed sample is calculated, and the best DIMD merge candidate is selected based on the minimum cost, and the corresponding index in the list of DIMD merge candidates is determined. In some embodiments, the best DIMD merge candidate is added to a list of candidate intra-modes (e.g., an MPM list) that are considered for the CU. If the best DIMD information is then selected from the candidate list for encoding the CU, the index may then be transmitted in the bitstream along with an indication that the DIMD merge mode is enabled (e.g., the current block is encoded in DIMD merge mode).

[0172] In some embodiments, to improve the relevance of the list of DIMD merge candidates, a potential DIMD merge candidate is added to the list only if its DIMD information is different from all DIMD information already in the list. For example, if a candidate is derived from the left block (L), and the left block and the bottom-left block (BL) have the same DIMD parameters, such as blend mode, blend weight, and position dependency, the DIMD information of the BL block is not added to the list of DIMD merge candidates as a new DIMD merge candidate. As another example, if pre-configured default DIMD information has the same DIMD parameters as a candidate in the list of DIMD merge candidates, the pre-configured default DIMD information is not added to the list.

[0173] In some embodiments, the list of DIMD merge candidates is limited to a pre-configured maximum list size. Additionally or alternatively, each respective set of DIMD merge candidates (e.g., each family such as candidates from spatially adjacent blocks, non-spatially adjacent blocks, history-based tables, combinations of candidates, and default DIMD information) may be configured with a maximum number of candidate entries.

[0174] FIG. 28 shows a flowchart 2800 of a method for generating a DIMD merge candidate list that does not exceed the pre-configured maximum number of DIMD merge candidates in the entire list and does not exceed the pre-configured maximum number of DIMD merge candidates in each family of merge candidates, according to some embodiments of the present disclosure. Generation of the list of DIMD merge candidates by the method of flowchart 2800 may be implemented mutually by an encoder (e.g., encoder 200 of FIG. 2) or by a decoder (e.g., decoder 300 of FIG. 3).

[0175] The method of flowchart 2800 begins at 2802. At 2802, a coder (e.g., an encoder or a decoder) is configured to repeat subsequent operations 2804 to 2816 for each family of DIMD merge candidates (also referred to herein as a set of merge candidates). For example, as described above in connection with Figure 23, the list of DIMD merge candidates may be constructed by adding merge candidates from several candidate families, such as spatially adjacent neighbor blocks, non-spatially adjacent neighbor blocks, spatial history-based table blocks, combined entries, and default entries. Accordingly, in some embodiments, 2804 to 2816 are repeated for each family of DIMD merge candidates.

[0176] At 2804, the coder is configured to repeat 2806 to 2816 for all DIMD merge candidate entries of the selected family of merge candidate entries. For example, for each selected family of merge candidate families, such as DIMD information of spatially adjacent neighbor blocks, non-spatially adjacent neighbor blocks, spatial history-based table blocks, combined entries, and default entries, 2806 to 2816 may be repeated until the addition of candidate entries from the selected candidate family is completed.

[0177] In 2806, potential candidates from the selected family are identified. For example, if the selected family consists of spatially adjacent blocks, each block is identified as L, A, and AR. AL and BL, shown in Figure 24, may be considered in a given sequence. Similarly, if the selected family consists of spatially non-adjacent blocks, each block of the pattern and sequence shown in Figure 25 may be considered for potential candidates; or, if the selected family is a history-based table, each entry in the history-based table may be considered a potential candidate for FIFO; or, if the selected family is a family of combined entries, a given set of entries already present in the list of DIMD merge candidates as described above may be considered in order for potential candidates; or, if the selected family is a family of default candidates, each given default candidate may be considered a potential candidate.

[0178] In step 2808, it is determined whether potential candidates are available. For example, for spatially adjacent blocks and spatially non-adjacent neighboring blocks, it may be determined whether the identified block has been reconstructed. In some cases, even if reconstructed, a block may not be considered available if it has not been predicted or reconstructed in DIMD or DIMD-related prediction modes, or if DIMD information has not been previously derived. For combined candidates, it may be determined whether each of multiple candidates that are combined together according to a pre-configured combined pattern (e.g., a combination of entries 1 and 3 in the list) is available. For example, if a given pattern combines the first two entries in the list of DIMD merge candidates, it may be determined whether each of the first two entries is available in the list of DIMD merge candidates. If the currently selected candidate entry is not available, the method then returns to step 2804 to select the next candidate from the currently selected family; otherwise, the method proceeds to step 2810.

[0179] In step 2810, it is determined whether the DIMD parameters of the potential DIMD merge candidate are already in the list of DIMD merge candidates. For example, it is determined whether there is already a DIMD merge candidate entry in the list such that the DIMD information, or more specifically, the set of DIMD parameters defining the DIMD information, is identical to the DIMD information of the potential DIMD merge candidate. If no entry with identical DIMD information is available, the method then returns to step 2804 to select the next candidate from the currently selected family; otherwise, the method proceeds to step 2812.

[0180] In version 2812, neighboring / candidate DIMD information is added to the list of DIMD merge candidates. Adding may involve adding the new candidate to the end of the current list of DIMD merge candidates.

[0181] In step 2814, it is determined whether the merge candidate list is full. This determination is based on comparing the total number of DIMD merge candidates currently present in the list with the maximum number of entries pre-configured for the list. If the list is full, then the list generation is completed. Otherwise, the method proceeds to step 2816.

[0182] In step 2816, it is determined whether the number of DIMD merge candidates to be added to the list from the currently selected family of potential DIMD merge candidates has reached a predetermined maximum number of entries for the currently selected family. In some embodiments, a separate maximum number of candidates may be defined for each family. In some other embodiments, the same maximum number of entries may be defined for all candidate families. If the number of DIMD merge candidates for the currently selected family does not exceed the predefined maximum number for that family, the method loops back to step 2804 to select the next potential candidate from the selected family. Otherwise, if the number of entries for that family exceeds the predefined maximum number for that family, the method then proceeds to step 2802 to select the next family of potential candidates.

[0183] The method of flowchart 2800 provides a way to generate a list of DIMD merge candidates (e.g., both in the encoder and decoder) in a manner that limits the number of DIMD merge candidates in the list, and thus the amount of complexity increase. By reducing the size of the list, fewer DIMD merge candidates are derived and evaluated, reducing the intra-mode complexity in both the decoder and encoder. The list is bounded, for example, the list has a maximum of 6 candidates. In this case, a candidate is added to the list only if the number of candidates included in the list is less than the pre-configured maximum DIMD merge candidate list size.

[0184] If the list becomes full, all remaining steps in the list derivation process may be skipped. For example, if the maximum size of the DIMD merge candidate list is 6 and it already contains 6 candidates, e.g., all spatially adjacent candidates (L, A, AL, AR, BL) and one spatially non-adjacent candidate, the list derivation process may be stopped, and other families of candidates that should be considered according to their inclusion order, e.g., non-adjacent candidates, history-based DIMD candidates, combined candidates, and default DIMD candidates, will not be calculated or considered.

[0185] Furthermore, a limit may be set on the maximum number of candidates that can be added to each candidate family. For example, according to one embodiment, all spatially adjacent candidates are considered (e.g., the maximum number of spatially adjacent candidates in a list is 5), the maximum number of non-adjacent candidates may be bounded to 4 candidates, the maximum number of history-based candidates is 2, the maximum number of combined candidates is 4, and the maximum number of default candidates is 2. If the maximum size of the DIMD merge candidate list is 8 and it already contains 2 candidates, e.g., 1 spatial candidate (e.g., AR), 1 non-adjacent candidate, and a history-based candidate, then since the maximum number of candidates in this candidate family is 4, a maximum of 4 combined candidates can be added to the list.

[0186] By bounding the list of DIMD merge candidates and discarding duplicate candidates, the list of DIMD merge candidates contains more diverse and relevant DIMD information, increasing the reliability and capabilities of DIMD.

[0187] One or more of the features of rejecting duplicates, limiting the maximum size of the list of DIMD merge candidates, and / or limiting the maximum number of merge candidates from each family of DIMD merge candidates can be implemented in the generation of the list of DIMD merge candidates, for example, as shown in Figure 23. For example, when each merge candidate is considered for insertion into the list of DIMD merge candidates, one or more checks can be performed, such as checking for duplicates (e.g., described in relation to 2810), whether the maximum list size has been reached (e.g., described in relation to 2814), and / or whether the maximum number of merge candidates for a particular family of merge candidates has been reached.

[0188] In some embodiments, sorting on the list may be performed to improve the relevance of the DIMD merge candidate list, sorting candidates based on template matching cost. In some embodiments, if template matching is enabled (e.g., by configuration or indication (e.g., flag or syntactic element) that signals to the decoder that a template matching order has been selected or enabled), a template matching cost-based sort is applied after merge candidates have been added to the list (e.g., all merge candidates have been added). The sorted list can then be used to derive a blended DIMD mode, etc. By sorting the list, the likelihood of selected merge candidates having smaller index numbers is likely to increase, and smaller index values ​​may be encoded more efficiently. In some other embodiments, instead of being performed on the entire DIMD merge candidate list, the template matching cost-based sort may be performed on a predetermined number of candidates on the list, or only on a predetermined set of families of candidate entries in the list. If template matching cost sorting is applied to a set of families of candidate entries in a list after adding DIMD information to the list from, for example, spatially adjacent blocks, spatially non-adjacent blocks, and history-based tables, the added DIMD information may be sorted based on the template matching cost before the combined candidates are added to the list. In one embodiment, a given set of families may include only spatially adjacent blocks, only spatially non-adjacent blocks, or only spatially adjacent blocks and non-adjacent blocks. In another embodiment, a given set of families may include only spatially adjacent blocks and non-adjacent blocks, as well as history-based tables of blocks.

[0189] Figure 29 shows a flowchart 2900 of a method for deriving a DIMD merge candidate list using sorting, according to some embodiments of the present disclosure. The operation of flowchart 2900 can be performed by an encoder (e.g., encoder 200 in Figure 2) or by a decoder (e.g., encoder 300 in Figure 3).

[0190] Similar to the method in flowchart 2300 described above, the method in flowchart 2900 includes a DIMD merge candidate list derivation operation 2902, followed by an operation 2904 for deriving the blend mode, weights, and positional dependencies. Operations 2906-2914 in the derivation of the DIMD merge candidate list operation 2902 are identical to operations 2306-2314 in the derivation of the DIMD merge candidate list operation 2302. However, in the method in flowchart 2900, the derivation of the list of DIMD merge candidates includes sorting, following the completion of the construction of the initial list of DIMD merge candidates after 2914.

[0191] For example, step 2915 determines whether template matching-based sorting can be performed. If yes, step 2918 performs template matching-based sorting of the DIMD merge candidate list.

[0192] During the template matching cost-based sorting process, the template matching cost of a DIMD merge candidate is measured by a cost metric (e.g., SAD, SSD, SATD) between the sample template derived from the DIMD merge candidate and the corresponding reconstructed sample. The list is then sorted from the DIMD merge candidate with the lowest template matching cost to the one with the highest template matching cost. The lower the template matching cost, the more relevant the candidate is at the beginning of the list.

[0193] For example, similar to FIG. 30, the template used for sorting may be an L-shaped region 3004 (including left pixels adjacent to the block and upper pixels) having a size corresponding to the size of the current block 3002. The template region indicates adjacent reconstructed pixels that are used to measure the template matching cost. The current block template 3004 forms a set of samples derived from DIMD merge candidates using reference samples 3006. For example, considering a 32×16 block, the template size is formed from an upper portion composed of 32×2 samples and a left portion composed of 2×16 samples.

[0194] Optionally, to reduce the complexity of DIMD merge template-based sorting, several families of candidates are excluded from the sorting, that is, their positions in the DIMD merge candidate list are not changed by the sorting. For example, in some embodiments, candidate families that are likely to produce less relevant candidates may be excluded from sorting due to their low probability of being selected. For example, history-based DIMD merge candidates, combined DIMD merge candidates, and default DIMD merge candidates may be excluded from sorting.

[0195] An example of the template matching sorting process applied to the initial list of DIMD merge candidates is shown in Table 3. The initial list before sorting is shown in Table 2. In this example, template matching sorting is applied to a limited number of DIMD merge candidates, the first three. The order of all other candidates other than the three remains unchanged. [Table 2] [Table 3]

[0196] Furthermore, template-based sorting of the DIMD merge candidate list can be enabled for CUs with a width and height of 64 or less. This maximum CU size for intra-template matching is configurable and signalable. Optionally, template matching sorting can be disabled and signaled.

[0197] Sorting provides a way to consider many candidates, but to reduce the overhead of complexity, it also provides a way to evaluate only a portion of the considered candidates. For example, while a list of 20 DIMD merge candidates can be constructed, only a portion of them (e.g., 6) can be included in the final candidate list, which is evaluated using RDO (RD check).

[0198] The shape and size of the templates used for sorting can be aligned with the size and shape of templates used in other functions of the encoder or decoder, such as TIMD (Template Intra-Mode Derivation), saving hardware logic and, for example, facilitating their adoption by implementers.

[0199] If template matching cost-based sorting is determined to be impossible in 2915, then another sorting method, such as direction-based sorting, is performed in 2916. An example of direction-based sorting is described below with respect to Figure 31.

[0200] Following template matching cost-based sorting in 2918 or direction-based sorting in 2916, the method in flowchart 2900 proceeds to 2904. In 2904, the blend mode, weights, and position dependencies of the selected intra-prediction mode are determined from the DIMD merge candidate list in the same format as described in relation to 2304 in flowchart 2300.

[0201] Figure 31 shows a flowchart 3100 of a method for sorting a list of DIMD merge candidates according to direction. For example, in 2916 above, the list of DIMD merge candidates is sorted according to some embodiments of the present disclosure. For example, the sorting may be carried out according to direction-based sorting. Optionally, sorting on the list may be performed to improve the relevance of the DIMD merge candidate list. According to some embodiments, sorting is performed to sort the candidates based on position dependency. The operation of flowchart 3100 may be carried out by an encoder (e.g., encoder 200 in Figure 2) or by a decoder (e.g., decoder 300 in Figure 3).

[0202] Some implementations support various CU sizes and shapes, particularly rectangular shapes. Considering vertical CUs (e.g., 4x16), horizontal prediction may offer greater compression efficiency than vertical prediction. The list of DIMD merge candidates is then sorted, with preferred DIMD merge candidates having horizontal position dependency over those having vertical position dependency. For horizontal CUs (e.g., 32x8), DIMD merge candidates having vertical position dependency are preferred over those having horizontal position dependency.

[0203] A flowchart 3100 of an exemplary method of vertical position-dependent preferred DIMD merge candidate list sorting, according to several embodiments, is applied when the CU is rectangular and its width (BW) is greater than its height (BH).

[0204] After initializing the counter in 3102, the method in flowchart 3100 proceeds, repeating steps 3104-3114 for each DIMD merge candidate in the list of DIMD merge candidates. In 3104, it is determined whether there are any further merge candidates to process in the list; if there are no further merge candidates, the sorting is considered complete. If there are further merge candidates to process, steps 3106-3114 are then performed. In 3108, the next merge candidate is obtained, and in 3110, it is checked whether its width is greater than its height. If the width of the merge candidate is not greater than its height, then that merge candidate is not processed further, and the method in flowchart 3100 returns to 3104, selecting the next merge candidate if one exists. If the width of the merge candidate is greater than its height, then all merge candidates are shifted by one index from the counter position to the current merge candidate. The current candidate is then added in 3112, and in 3114, the counter is incremented by 1.

[0205] Table 5 reports an example of the orientation-based sorting process applied to the initial DIMD merge candidate list (shown in Table 4). In this example, since the CU is vertically oriented, i.e., its width is less than its height, DIMD merge candidates with horizontal position dependency (e.g., position dependency equal to 2) are preferred and placed first in the list of DIMD merge candidates. [Table 4] [Table 5]

[0206] Optionally, direction-based sorting can be disabled by default and activated as needed. Direction-based sorting can be enabled when template-based sorting is disabled.

[0207] Figure 32 shows a flowchart 3200 of a method, according to some embodiments of the present disclosure, in which a list of DIMD merge candidates, or a portion of a list of DIMD merge candidates, can be sorted according to a template matching cost-based sorting. The operation of flowchart 3200 can be performed by an encoder (e.g., encoder 200 in Figure 2) or by a decoder (e.g., encoder 300 in Figure 3).

[0208] The method in flowchart 3200 can be initiated at 3202. At 3202, no DIMD merge candidates are added from the set of spatially adjacent neighboring blocks of the current block. The addition of DIMD merge candidates from the set of spatially adjacent neighboring blocks is described above at 2306, with respect to Figure 23.

[0209] Next, optionally, after checking whether the list is full, the method proceeds to 3204. In 3204, no DIMD merge candidates are added from the set of spatially non-adjacent neighboring blocks of the current block. The addition of DIMD merge candidates from the set of spatially non-adjacent neighboring blocks is described above in 2308 with respect to Figure 23.

[0210] Next, optionally, after checking whether the list is full, the method proceeds to 3206. In 3206, one or more DIMD merge candidates are not added from the history-based set of candidates. Adding DIMD merge candidates from the history table-based set of candidates is described above in 2310 with respect to Figure 23.

[0211] In 3208, it is determined whether template matching cost-based sorting is possible / enabled. If yes, in 3210, the DIMD merge candidate list is sorted according to template matching cost-based sorting. The template matching cost of the DIMD merge candidates during the sorting process is measured by the template cost difference between the template samples of the current block and their corresponding reference samples. A template contains a set of reconstructed samples adjacent to the current block. The reference samples of a template are located based on the DIMD information of the DIMD merge candidate.

[0212] After sorting at 3210, or if it is decided not to perform sorting at 3208, the method proceeds to add more DIMD merge candidates to the merge candidate list.

[0213] Next, optionally, after checking whether the list is full, the method proceeds to 3212. In 3212, no more than one DIMD merge candidate is added from the DIMD merge candidates in the merge candidate list. Adding DIMD merge candidates from the set of DIMD merge candidates is done in 2312, as described above with respect to Figure 23.

[0214] Next, optionally, after checking whether the list is full, the method proceeds to 3214. In 3214, one or more DIMD merge candidates are not added from the default DIMD information. Adding DIMD merge candidates from the default set of DIMD information is described above in 2314 with respect to Figure 23.

[0215] In 3216, it is determined whether direction-based sorting is possible / enabled. If yes, in 3216, the list of DIMD merge candidates is sorted according to direction-based sorting in 3218. Direction-based sorting is described above in 2316 with respect to Figure 23.

[0216] After sorting in 3218, or if it is determined in 3216 that no sorting will be performed (for example, direction-based sorting may be disabled when template-matching-based sorting is performed), the method proceeds to derive the blend mode, etc., in 3220.

[0217] In 3220, the blend mode, weights, and position dependencies are derived. According to some embodiments, the derivation of the blend mode, weights, and position dependencies can be carried out as described in 2304 with respect to Figure 23.

[0218] According to flowchart 3200, in some embodiments, after DIMD merge candidates are added from spatially adjacent blocks, spatially non-adjacent blocks, and history-based tables, template matching sorting may be performed on those candidates, and then, optionally, once all candidates have been added to the list, the entire list may be sorted using either template matching cost-based sorting or direction-based sorting.

[0219] In the method of flowchart 3200, for each DIMD merge candidate, attention should be paid to one or more of the following: checking for duplicates, checking that the number of merge candidates in the list does not exceed the limit, and checking that the number of merge candidates for a family of merge candidates does not exceed the limit defined for that family.

[0220] Sorting the list of DIMD merge candidates improves the likelihood of selecting a DIMD merge candidate that has a smaller value index and is more efficiently encoded.

[0221] The TIMD described above identifies up to two intra-prediction modes and their corresponding weights, which are merged to derive a TIMD predictor. The TIMD process is different from the DIMD process, although both processes may output several blend modes and their corresponding weights. In some embodiments, the TIMD information (e.g., TIMD blend modes and weights) is reused to derive DIMD information. For example, in some embodiments, several candidates derived from adjacent TIMD information are added to a DIMD merge candidate list.

[0222] The TIMD process requires the derivation of an MPM list, and therefore, blocks using TIMD intra-mode also derive their DIMD information. As a result, neighboring blocks using TIMD mode have their DIMD information available and can be used to derive one or more merge candidates from the list of DIMD merge candidates. However, since TIMD information is an alternative representation of a block, neighboring TIMD information can be used to derive other DIMD merge candidates. In some embodiments of this disclosure, the TIMD information may have fewer intra-modes to blend, so the DIMD merge candidates derived from the TIMD information may be considered less precise, and therefore, as shown in Figure 33, these candidates are added to the DIMD merge candidate list after history-based candidates have been added.

[0223] Figure 33 shows a flowchart 3300 of a method in which a list of DIMD merge candidates may be generated by considering a second set of spatially adjacent neighbor blocks, spatially non-adjacent neighbor blocks, and history-based merge candidates to be added to the list of DIMD merge candidates, in addition to a first set of spatially adjacent neighbor blocks, spatially non-adjacent neighbor blocks, and history-based merge candidates. The operation of flowchart 3300 may be performed by an encoder (e.g., encoder 200 in Figure 2) or by a decoder (e.g., decoder 300 in Figure 3).

[0224] The method in flowchart 3300 can be initiated in 3302. In 3302, one or more DIMD merge candidates are not added from a family of spatially adjacent neighboring blocks of the current block. The addition of DIMD merge candidates from a set of spatially adjacent neighboring blocks is described above in 2306 with respect to Figure 23.

[0225] Next, optionally, after checking whether the list is full, the method proceeds to 3304. In 3304, one or more DIMD merge candidates are not added from families of spatially non-adjacent neighboring blocks of the current block. The addition of DIMD merge candidates from a set of spatially non-adjacent neighboring blocks is described above in 2308 with respect to Figure 23.

[0226] Next, optionally, after checking whether the list is full, the method proceeds to 3306. In 3306, one or more DIMD merge candidates are not added from the history-based family of merge candidates. Adding DIMD merge candidates from the history-based family of candidates is described above in 2310 with respect to Figure 23.

[0227] Next, optionally, the method checks whether the list of DIMD merge candidates is full, and then proceeds to 3308. In 3308, one or more merge candidates are not added to the list of DIMD merge candidates from a second set of merge candidates, each from the family of spatially adjacent neighbor blocks, spatially non-adjacent neighbor blocks, and history-based merge candidates. In an exemplary embodiment, the second set of merge candidates is selected based on the TIMD information of each merge candidate.

[0228] Table 6 shows an example of a list of TIMD information from neighbors that is reused to derive DIMD merge candidates. The results of the derivation are reported in Table 7 using the following methodology (as an example). The TIMD blend mode may be reused and completed in planar mode (e.g., intra mode 0), the TIMD weights may be rescaled to fit the DIMD process, and the position dependency may be set to the default value (0: diagonal). Further constraints may be added so that DIMD merge candidates can be derived from TIMD information, for example, the TIMD information must contain two blend modes. Adding such constraints may change the derived DIMD merge candidates, as reported in Table 8. From Table 6, the fourth candidate (index = 3) is not used to derive a DIMD candidate because it contains only one mode. [Table 6] [Table 7] [Table 8]

[0229] Embodiments of this disclosure are not limited to using two TIMD blend modes to derive DIMD parameters for DIMD merge candidates. In some embodiments, a number of other TIMD blend modes and weights may be used when deriving DIMD parameters from TIMD information.

[0230] After step 3308, the method proceeds to step 3310. In step 3310, no combined DIMD merge candidates are added based on the DIMD merge candidates already in the list of DIMD merge candidates. Adding DIMD merge candidates from those already in the list of DIMD merge candidates is described above in step 2312 with respect to Figure 23.

[0231] Next, optionally, after checking whether the list is full, the method proceeds to 3312. In 3312, one or more DIMD merge candidates are not added from the default DIMD information. Adding DIMD merge candidates from the default set of DIMD information is described above in 2314 with respect to Figure 23.

[0232] In step 3314, it is determined whether and whether a sort is performed. Based on the decision in step 3314, the sort is performed in steps 3316 or 3318. Operations 3314-3318 are identical to operations 2915-2918, which are described in relation to the method in flowchart 2900.

[0233] The sorted list of DIMD merge candidates (or, if sorting is not decided in 3314, the unsorted list of DIMD merge candidates) is used in 3320 to derive the blend mode, weights, and positional dependencies. The derivation of the blend mode, weights, and positional dependencies may be performed in 2304 in the same or similar manner as described above with respect to Figure 23.

[0234] Figure 34 shows a flowchart 3400 of a method for encoding the current block using DIMD merge mode, according to some embodiments of the present disclosure. The method of flowchart 3400 provides generating a sorted list of DIMD merge candidates. The method of flowchart 3400 may be carried out by an encoder (e.g., encoder 200 in Figure 2).

[0235] The method in flowchart 3400 may begin at 3402. At 3402, a sorting is performed on the list of decoder-side intra-mode derivation (DIMD) merge candidates. The DIMD merge candidates include at least one DIMD merge candidate that contains DIMD information associated with spatially adjacent blocks of the current block within the current frame.

[0236] Sorting may involve sorting the list of DIMD merge candidates according to the template matching cost of each DIMD merge candidate. Sorting may apply to all DIMD merge candidates in the list, or to only the first part of the list. If only the first part of the list of DIMD merge candidates is sorted, then all DIMD merge candidates in the list may be sorted accordingly according to the orientation of each DIMD merge candidate. The first part being sorted may include all DIMD merge candidates from any spatially adjacent blocks, and in one implementation, after sorting only the first part of the list of DIMD merge candidates, additions to the list of DIMD merge candidates may be made by combining the DIMD information of two or more DIMD merge candidates in the list. If sorting is based on template matching cost, the template matching cost may be determined based on a set of samples derived from the DIMD merge candidates using adjacent reference samples to the template of the current block.

[0237] Alternatively, sorting may follow the positional dependencies of each DIMD merge candidate within the list of DIMD merge candidates. Sorting according to the positional dependencies of each DIMD merge candidate may involve prioritizing either horizontal or vertical positional dependencies in the reconfigured order based on the dimensions of the current block.

[0238] The list of DIMD merge candidates may include a first set of merge candidates added based on DIMD information, and a second set of DIMD merge candidates added based on template-based intra-mode derivation (TIMD) information. The second set of DIMD merge candidates may include DIMD merge candidates derived from TIMD information derived from blocks spatially adjacent to the current block, blocks not adjacent within the current frame, or other blocks within the current frame.

[0239] A DIMD merge candidate may include a second DIMD merge candidate containing one or more DIMD blend modes associated with adjacent neighboring blocks of the current block, a non-adjacent neighboring block in the current frame, another block in the current frame, a combination of DIMD merge candidates from the list of DIMD merge candidates, or a third DIMD merge candidate containing predetermined DIMD information.

[0240] Each DIMD merge candidate may include one or more DIMD intra-modes and one or more corresponding blend weights. Each DIMD merge candidate may further include one or more positional dependencies, each corresponding to one or more DIMD intra-modes.

[0241] Generating a list of DIMD merge candidates may involve adding each DIMD merge candidate to the list in inclusion order. The inclusion order may include DIMD merge candidates associated with adjacent neighboring blocks, if any; preceding DIMD merge candidates associated with non-adjacent neighboring blocks, if any; DIMD merge candidates associated with adjacent neighboring blocks, if any; DIMD merge candidates associated with non-adjacent neighboring blocks, if any; and DIMD merge candidates from history-based buffers, if any.

[0242] When adding each DIMD merge candidate to the DIMD merge candidate list according to the inclusion order, the potential DIMD merge candidate is added to the DIMD merge candidate list as a DIMD merge candidate if the following conditions are met before adding: the DIMD information for the potential DIMD merge candidate does not already exist in the list of DIMD merge candidates; the total number of DIMD merge candidates in the list of DIMD merge candidates is less than a predetermined maximum list limit; and / or a number of DIMD merge candidates currently in the list of DIMD merge candidates are corresponding to one of adjacent neighboring blocks, and the number of non-adjacent neighboring blocks or history-based buffers is less than or equal to a predetermined maximum family limit. Examples of multiple families of potential DIMD merge candidates considered in the inclusion order are shown in 2306-2314 in the method of flowchart 2300.

[0243] In 3404, DIMD merge candidates are selected from a list of DIMD merge candidates based on predicting the current block based on the DIMD information of the DIMD merge candidates. Selection may include selecting based on the rate-distortion optimization (RDO) metric of the DIMD merge candidates.

[0244] In 3406, an indicator is encoded in the bitstream that shows the current block will be encoded based on the selected DIMD merge candidate.

[0245] The indicator may include at least one of the following: a DIMD merge enabled flag and an index to the position of the selected DIMD merge candidate in the list of DIMD merge candidates. Encoding may include encoding only the DIMD merge enabled flag as the indicator if the size of the list of DIMD merge candidates is less than a threshold size.

[0246] Figure 35 shows a flowchart 3500 of a method for decoding the current block in DIMD merge mode, according to some embodiments of the present disclosure. The method of flowchart 3500 provides generating a sorted list of DIMD merge candidates. The method of flowchart 3500 may be implemented by a decoder (e.g., decoder 300 in Figure 3).

[0247] The method of flowchart 3500 may begin at 3502. At 3502, a sort is performed on the list of decoder-side intra-mode derivation (DIMD) merge candidates. The DIMD merge candidates include at least one DIMD merge candidate that contains DIMD information associated with spatially adjacent blocks of the current block within the current frame. The sort at 3502 may be identical to sort 3402 described above in relation to the method of flowchart 3400.

[0248] In 3504, the indicators for DIMD merge candidates in the list of DIMD merge candidates are decoded from the bitstream.

[0249] In 3506, the current block is decoded based on the DIMD information of the DIMD merge candidate, which is selected from a list of DIMD merge candidates according to the indicator.

[0250] The indicator may include a flag indicating the use of DIMD merge mode in the current block. When the flag is set, the indicator's selection of a DIMD merge candidate from a list of DIMD merge candidates may include selecting a DIMD merge candidate from a given position in the list of DIMD merge candidates.

[0251] If the indicator includes a flag and an index, and the flag is set, the selection of DIMD merge candidates may include selecting a DIMD merge candidate from the list of DIMD merge candidates according to the index.

[0252] Decoding the current block may involve generating a predicted block of the current block based on DIMD information, and reconstructing the current block based on the predicted block and the residuals of the current block obtained from the bitstream.

[0253] Embodiments of the present disclosure may be implemented in software using analog and / or digital circuits, through the execution of instructions by one or more general-purpose or application processors, or in hardware as a combination of hardware and software. As a result, embodiments of the present disclosure may be implemented in the environment of a computer system or other processing system. An example of such a computer system 3600 is shown in Figure 36. Blocks shown in the above figures, such as the blocks in Figures 1, 2, and 3, may be executed on one or more computer systems 3600. Furthermore, each step of the flowcharts shown in the present disclosure may be implemented on one or more computer systems 3600.

[0254] The computer system 3600 includes one or more processors, such as a processor 3604. The processor 3604 may be, for example, an application-specific processor, a general-purpose processor, a microprocessor, or a digital signaling processor. The processor 3604 may be connected to a communication infrastructure 3602 (e.g., a bus or network). The computer system 3600 may also include main memory 3606, such as random access memory (RAM), and secondary memory 3608.

[0255] The secondary memory 3608 may include, for example, a hard disk drive 3610 representing a magnetic tape drive, an optical disk drive, or the like, and / or a removable storage drive 3612. The removable storage drive 3612 may be read from and / or written to by the removable storage unit 3616 in a well-known manner. The removable storage unit 3616 represents a magnetic tape, an optical disk, or the like, which is read from and written to by the removable storage drive 3612. As will be understood by those skilled in the art, the removable storage unit 3616 includes a computer-usable storage medium in which computer software and / or data are stored.

[0256] In alternative implementations, the secondary memory 3608 may include other similar means for enabling the loading of computer programs or other instructions into the computer system 3600. Such means may include, for example, a removable storage unit 3618 and an interface 3614. Examples of such means include a program cartridge and cartridge interface (e.g., those found in video game devices), a removable memory chip (e.g., an EPROM or PROM), and associated sockets, thumb drives, and USB ports, as well as other removable storage units 3618 and interfaces 3614 that enable the transmission of software and data from the removable storage unit 3618 to the computer system 3600.

[0257] The computer system 3600 may also include a communication interface 3620. The communication interface 3620 enables the transmission of software and data between the computer system 3600 and external devices. Embodiments of the communication interface 3620 may include a modem, a network interface (such as an Ethernet® card), a communication port, and the like. The software and data transmitted through the communication interface 3620 are in the form of signals, which may be electronic, electromagnetic, optical, or other signals that can be received by the communication interface 3620. These signals are provided to the communication interface 3620 via a communication path 3622. The communication path 3622 transmits signals and may be implemented using wires, or cables, optical fibers, telephone lines, cell phone links, RF links, and other communication channels.

[0258] As used herein, the terms “computer program medium” and “computer-readable medium” are used to refer to tangible storage media, such as removable storage units 3616 and 3618, or hard disks mounted on hard disk drive 3610. These computer program products are means for providing software to computer system 3600. Computer programs (also called computer control logic) may be stored in main memory 3606 and / or secondary memory 3608. Computer programs may also be received via communication interface 3620. When executed, such computer programs enable computer system 3600 to implement the disclosure as discussed herein. In particular, when executed, computer programs enable processor 3604 to implement the processes of the disclosure, such as any of the methods described herein. Thus, such computer programs represent the controller of computer system 3600.

[0259] In another embodiment, the features of this disclosure may be implemented in hardware using hardware components such as application-specific integrated circuits (ASICs) and gate arrays. Implementations of hardware state machines for performing the functions described herein will also be apparent to those skilled in the art.

Claims

1. It is a method, Reordering a list of merge candidates, wherein the list of merge candidates includes a list of merge candidates that includes decoder-side intra-mode derivation (DIMD) information or template-based intra-mode derivation (TIMD) information associated with spatially adjacent blocks of the current block in the current frame, The aforementioned list of merge candidates includes a first set of merge candidates added based on DIMD information and a second set of merge candidates added based on TIMD information, The second set of merge candidates is sorted to include merge candidates derived from TIMD information, derived from spatially adjacent blocks of the current block, non-adjacent blocks in the current frame, or other blocks in the current frame. To determine the indicator for the merge candidate in the list of merge candidates, A method comprising encoding the current block based on the DIMD information of the merge candidate in the list of DIMD merge candidates, according to the indicator.

2. It is a method, Reordering a list of merge candidates, wherein the list of merge candidates includes merge candidates that include decoder-side intra-mode derivation (DIMD) information or template-based intra-mode derivation (TIMD) information associated with spatially adjacent blocks of the current block in the current frame. To determine the indicator for the merge candidate in the list of merge candidates, A method comprising coding the current block based on the DIMD information of the merge candidate in the list of merge candidates, according to the indicator.

3. The method according to any one of claims 1 and 2, wherein the sorting includes sorting the list of merge candidates according to the template matching cost of each merge candidate.

4. The method according to claim 3, wherein the template matching cost is determined based on a set of samples derived from merge candidates using reference samples adjacent to the template of the current block.

5. The method according to any one of claims 3 and 4, wherein the sorting further comprises sorting all merge candidates in the list of merge candidates.

6. The method according to any one of claims 3 to 5, wherein the sorting further comprises sorting only the first portion of the list of merge candidates.

7. The method according to any one of claims 3 to 6, wherein the sorting further comprises sorting only the first portion of the list of merge candidates according to the template matching cost, and then sorting all of the merge candidates in the list of merge candidates according to the direction of each merge candidate.

8. The method according to claim 7, wherein the first portion includes all merge candidates from any spatially adjacent blocks.

9. The method according to any one of claims 7 and 8, further comprising adding a merge candidate to the list of merge candidates by combining the DIMD information of two or more merge candidates within the list of merge candidates, after sorting only the first portion of the list of merge candidates.

10. The method according to any one of claims 1 and 2, wherein the sorting includes sorting according to the positional dependency of each merge candidate in the list of merge candidates.

11. The method according to claim 10, wherein sorting according to the positional dependency of each merge candidate includes, in the sorted order, prioritizing one of the horizontal positional dependency or the vertical positional dependency based on the dimensions of the current block.

12. The method according to any one of claims 2 to 11, wherein the list of merge candidates includes a first set of merge candidates added based on DIMD information and a second set of merge candidates added based on TIMD information.

13. The method according to claim 12, wherein the second set of merge candidates includes merge candidates derived from TIMD information, which are derived from spatially adjacent blocks of the current block, non-adjacent blocks in the current frame, or other blocks in the current frame.

14. The merge candidates are, A first merge candidate, which includes one or more DIMD blend modes associated with adjacent neighboring blocks of the current block, non-adjacent neighboring blocks in the current frame, another block in the current frame, or a combination of merge candidates from the list of merge candidates, or The method according to any one of claims 1 to 13, comprising a second merge candidate which includes predetermined DIMD information.

15. The method according to any one of claims 1 to 14, wherein each merge candidate of the merge candidates includes one or more DIMD intra modes and one or more corresponding blend weights.

16. The method according to claim 15, wherein each merge candidate further includes one or more positional dependencies, each corresponding to one or more DIMD intra modes.

17. The method according to any one of claims 1 to 16, further comprising selecting the merge candidate from the list of merge candidates according to the indicator.

18. The method according to any one of claims 1 to 17, wherein the indicator includes a flag indicating the use of a DIMD merge mode in the current block, and when the flag is set, the selection of the merge candidate from the list of merge candidates according to the indicator includes selecting the merge candidate from a predetermined position in the list of merge candidates.

19. The method according to any one of claims 1 to 18, wherein the indicator further includes an index, and when the flag is set, selecting the merge candidate from the list of merge candidates according to the indicator further includes selecting the merge candidate in the list of merge candidates according to the index.

20. The list of merge candidates is generated by adding each merge candidate to the list of merge candidates in the order of inclusion, and the addition is Before adding any merge candidates associated with non-adjacent neighboring blocks, and any merge candidates from the history-based buffer, add one or more merge candidates associated with adjacent neighboring blocks. From the aforementioned history-based buffer, before adding any merge candidate, add one or more merge candidates associated with the non-adjacent neighboring blocks, or The method according to any one of claims 1 to 19, comprising one or more of adding merge candidates from the history-based buffer.

21. The process of adding each merge candidate to the list of merge candidates in inclusion order, for each potential merge candidate from either adjacent neighbor blocks, non-adjacent neighbor blocks, or a history-based buffer, is performed based on a set of conditions that are met before the addition, and the set of conditions is The DIMD information of the aforementioned potential merge candidate is not yet present in the list of merge candidates. The total number of merge candidates in the aforementioned list of merge candidates is less than a predetermined maximum list limit, The method according to claim 20, comprising: the number of merge candidates currently in the list of merge candidates; the corresponding one of the adjacent neighbor blocks, the non-adjacent neighbor blocks, or the history-based buffer being less than or equal to a predetermined maximum family limit.

22. The method according to any one of claims 1 to 21, wherein determining the indicator of the merge candidate includes decoding the indicator from a bitstream.

23. The method according to any one of claims 1 to 22, wherein coding the current block based on the DIMD information of the merge candidate comprises decoding the current block based on the DIMD information of the merge candidate according to the indicator.

24. Based on the DIMD information of the merge candidate, the current block is decoded as follows: Based on the DIMD information, a predicted block for the current block is generated, The method according to claim 23, comprising: reconstructing the current block based on the predicted block and the residual of the current block obtained from the bitstream.

25. Based on the DIMD information of the merge candidate, the current block is coded as follows: The method according to any one of claims 1 to 24, comprising encoding an indicator in a bitstream that indicates that the current block is encoded based on the merge candidate.

26. Determining the indicator of the merge candidate in the list of merge candidates is Based on the DIMD information of the merge candidates, and based on predicting the current block, a merge candidate is selected from the list of merge candidates. The method according to any one of claims 1 to 25, comprising determining the indicator based on the selected merge candidate.

27. The method according to any one of claims 1 to 26, wherein the indicator includes at least one of a DIMD merge enable flag and an index to the position of the selected merge candidate in the list of merge candidates.

28. The method according to any one of claims 26 and 27, further comprising selecting based on a rate distortion optimization (RDO) index of the merge candidates.

29. The method according to any one of claims 25 to 28, wherein the encoding includes encoding only the DIMD merge enable flag as the indicator when the size of the list of merge candidates is less than a threshold size.

30. It is an encoder, One or more processors, An encoder comprising: a memory for storing instructions, wherein, when the instructions are executed by one or more processors, the memory causes the encoder to perform the method according to any one of claims 1 to 21 and 25 to 29.

31. It is a decoder, One or more processors, A decoder comprising: a memory for storing instructions, the memory, when executed by one or more processors, causes the decoder to perform the method according to any one of claims 1 to 24.

32. A non-temporary computer-readable medium containing instructions, wherein when the instructions are executed by one or more processors of the device, the device causes the device to carry out the method according to any one of claims 1 to 29.

Images