Method and device for decoder-side intra-mode derivation

Abstract

A method, an apparatus, and a non-transitory computer-readable storage medium for video decoding are provided. In one method, a decoder determines a fusion scheme to be applied in a decoder-side intra-mode derivation (DIMD) mode and a template-based intra-mode derivation (TIMD) mode, and the fusion scheme is applied as a weighted average of predictors in the DIMD mode and the TIMD mode.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on and claims priority to Provisional Application No. 63 / 238,774, filed August 30, 2021, the entire contents of which are incorporated herein by reference for all purposes. FIELD OF THE DISCLOSURE This disclosure relates to video encoding and compression, and more particularly, to decoder-side intra mode derivation (DIMD). [Background technology]

[0002] Digital video is supported by a variety of electronic devices, including digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video gaming consoles, smartphones, video teleconferencing devices, and video streaming devices. Electronic devices transmit and receive or otherwise communicate digital video data over communication networks and / or store the digital video data on storage devices. Due to limited bandwidth capacity in communication networks and limited memory resources in storage devices, video coding may be used to compress video data according to one or more video coding standards before the video data is communicated or stored. Examples of video coding standards include Versatile Video Coding (VVC), Joint Exploration Test Model (JEM), High-Efficiency Video Coding (HEVC / H.265), Advanced Video Coding (AVC / H.264), and Moving Picture Expert Group (MPEG) coding. Video coding generally utilizes prediction methods (e.g., inter-prediction, intra-prediction, etc.) that exploit the redundancy inherent in video data. Video coding aims to compress video data into a format that uses a lower bitrate while avoiding or minimizing degradation of video quality. Summary of the Invention [Problem to be solved by the invention]

[0003] Examples of this disclosure provide methods and apparatus for video encoding using intra prediction. [Means for solving the problem]

[0004] According to a first aspect of the present disclosure, there is provided a method for video decoding. The method may include determining, by a decoder, a fusion scheme to be applied in a decoder-side intra-mode derivation (DIMD) mode and a template-based intra-mode derivation (TIMD) mode, where the fusion scheme is applied as a weighted average of predictors in the DIMD mode and the TIMD mode.

[0005] According to a second aspect of the present disclosure, a method for video decoding using decoder-side intra mode derivation (DIMD) is provided. The method may include determining, by a decoder, whether a fusion scheme is applied in a DIMD mode, where the fusion scheme is applied as a weighted average of predictors in the DIMD mode, and determining, by the decoder, whether to derive an intra mode from the DIMD mode to a list of intra most probable modes (MPMs).

[0006] According to a third aspect of the present disclosure, there is provided a method for video decoding using template-based intra mode derivation (TIMD). The method may include determining, by a decoder, whether to derive an intra mode from a TIMD mode to a list of intra most probable modes (MPMs).

[0007] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to be restrictive of the present disclosure.

[0008] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram illustrating an example system for encoding and decoding video blocks according to some implementations of the present disclosure. [Figure 2]FIG. 1 is a block diagram illustrating an example video encoder according to some implementations of the present disclosure. [Figure 3] FIG. 2 is a block diagram illustrating an example video decoder according to some implementations of the present disclosure. [Figure 4A] A block diagram showing how a frame is recursively partitioned into multiple video blocks of different sizes and shapes in accordance with some implementations of this disclosure. [Figure 4B] A block diagram showing how a frame is recursively partitioned into multiple video blocks of different sizes and shapes in accordance with some implementations of this disclosure. [Figure 4C] A block diagram showing how a frame is recursively partitioned into multiple video blocks of different sizes and shapes in accordance with some implementations of this disclosure. [Figure 4D] A block diagram showing how a frame is recursively partitioned into multiple video blocks of different sizes and shapes in accordance with some implementations of this disclosure. [Figure 4E] A block diagram showing how a frame is recursively partitioned into multiple video blocks of different sizes and shapes in accordance with some implementations of this disclosure. [Figure 5A] FIG. 10 is a diagram illustrating sample definitions used by the PDPC applied to prediction modes according to some implementations of the present disclosure. [Figure 5B] FIG. 10 is a diagram illustrating sample definitions used by the PDPC applied to prediction modes according to some implementations of the present disclosure. [Figure 5C] FIG. 10 is a diagram illustrating sample definitions used by the PDPC applied to prediction modes according to some implementations of the present disclosure. [Figure 5D] FIG. 10 is a diagram illustrating sample definitions used by the PDPC applied to prediction modes according to some implementations of the present disclosure. [Figure 6] FIG. 10 illustrates examples of allowed GPM divisions according to some implementations of the present disclosure. [Figure 7]FIG. 10 illustrates an example of selected pixels on which gradient analysis is performed according to some implementations of the present disclosure. [Figure 8] FIG. 1 illustrates a convolution process according to some implementations of the present disclosure. [Figure 9] FIG. 10 illustrates prediction fusion by weighted average of two HoG modes and one planar mode according to some implementations of the present disclosure. [Figure 10] A diagram showing a template and its reference sample used in a TIMD according to some implementations of the present disclosure. [Figure 11A] FIG. 1 is a block diagram illustrating a video decoding process using TIMD according to some implementations of the present disclosure. [Figure 11B] FIG. 1 is a block diagram illustrating a video decoding process using TIMD according to some implementations of the present disclosure. [Figure 11C] FIG. 1 is a block diagram illustrating a video decoding process using TIMD according to some implementations of the present disclosure. [Figure 12A] FIG. 1 is a block diagram illustrating a video decoding process using DIMD according to some implementations of the present disclosure. [Figure 12B] FIG. 1 is a block diagram illustrating a video decoding process using DIMD according to some implementations of the present disclosure. [Figure 12C] FIG. 1 is a block diagram illustrating a video decoding process using DIMD according to some implementations of the present disclosure. [Figure 13] FIG. 1 is a block diagram illustrating a computing environment coupled with a user interface according to some implementations of the present disclosure. [Figure 14] FIG. 2 is a block diagram illustrating a video decoding process according to some implementations of the present disclosure. [Figure 15] FIG. 1 is a block diagram illustrating a video decoding process using DIMD according to some implementations of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings, in which like numbers in different drawings represent the same or similar elements unless otherwise stated. The implementations described in the following description of the exemplary embodiments do not represent all implementations consistent with the present disclosure. Rather, the implementations are merely examples of apparatus and methods consistent with aspects related to the present disclosure as set forth in the appended claims.

[0011] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It is also to be understood that the term "and / or," as used herein, is intended to mean and include any and all possible combinations of one or more of the associated listed items.

[0012] Terms such as "first," "second," and "third" may be used herein to describe various pieces of information, but it should be understood that these terms are not intended to limit the information. These terms are used only to distinguish one category of information from another. For example, first information may be referred to as second information, and similarly, second information may be referred to as first information, without departing from the scope of this disclosure. The term "if," as used herein, may be understood to mean "when," "in the event of," or "at the discretion of," depending on the context.

[0013] Various video coding techniques may be used to compress video data. Video coding is performed according to one or more video coding standards. For example, today, well-known video coding standards include Versatile Video Coding (VVC), High Efficiency Video Coding (HEVC, also known as H.265 or MPEG-H Part 2), and Advanced Video Coding (AVC, also known as H.264 or MPEG-4 Part 10), which are jointly developed by ISO / IEC MPEG and ITU-T VECG. AOMedia Video 1 (AV1) was developed by the Alliance for Open Media (AOM) as a successor to its predecessor, VP9. Audio-Video Coding (AVS), referring to digital audio and digital video compression standards, is another video compression standard series developed by the China Audio and Video Coding Standards Workgroup. Most of the existing video coding standards are built on the well-known hybrid video coding framework, i.e., using block-based prediction methods (e.g., inter-prediction, intra-prediction) to reduce redundancy present in a video image or sequence, and using transform coding to compact the energy of the prediction error. An important goal of video coding techniques is to compress video data into a format that uses a lower bitrate while avoiding or minimizing degradation of video quality.

[0014] The first generation of AVS standards includes the Chinese national standards "Information Technology, Advanced Audio-Video Coding, Part 2: Video" (known as AVS1) and "Information Technology, Advanced Audio-Video Coding, Part 16: Radio, Television, and Video" (known as AVS+). Compared to the MPEG-2 standard, this achieves approximately 50% bitrate savings at the same perceptual quality. The video portion of the AVS1 standard was promulgated as a Chinese national standard in February 2006. The second generation of AVS standards includes a series of Chinese national standards, "Information Technology, Efficient Multimedia Coding" (known as AVS2), primarily aimed at transmitting additional HD TV programs. AVS2's coding efficiency is twice that of AVS+. AVS2 was published as a Chinese national standard in May 2016. Meanwhile, the video portion of the AVS2 standard has been proposed by the Institute of Electrical and Electronics Engineers (IEEE) as one of the international standards for applications. The AVS3 standard is one of a new generation of video coding standards for UHD video applications, aiming to exceed the coding efficiency of the latest international standard, HEVC. In March 2019, at the 68th AVS Conference, the AVS3-P2 baseline was finalized, achieving approximately 30% bitrate savings compared to the HEVC standard. Currently, there is one reference software, called the High Performance Model (HPM), maintained by the AVS group to demonstrate a reference implementation of the AVS3 standard.

[0015] 1 is a block diagram illustrating an example system 10 for encoding and decoding video blocks in parallel according to some implementations of the present disclosure. As shown in FIG. 1, system 10 includes a source device 12 that generates and encodes video data that is subsequently decoded by a destination device 14. Source device 12 and destination device 14 may include any of a wide variety of electronic devices, including desktop or laptop computers, tablet computers, smartphones, set-top boxes, digital televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, etc. In some implementations, source device 12 and destination device 14 are equipped with wireless communication capabilities.

[0016] In some implementations, destination device 14 may receive encoded video data to be decoded via link 16. Link 16 may comprise any type of communication medium or device capable of moving encoded video data from source device 12 to destination device 14. In one example, link 16 may comprise a communication medium that enables source device 12 to transmit encoded video data directly to destination device 14 in real time. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to destination device 14. The communication medium may include any wireless or wired communication medium, such as the radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful in facilitating communication from source device 12 to destination device 14.

[0017] In some other implementations, the encoded video data may be transmitted from output interface 22 to storage device 32. The encoded video data in storage device 32 may then be accessed by destination device 14 via input interface 28. Storage device 32 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray disc, a digital versatile disc (DVD), a compact disc read-only memory (CD-ROM), flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In a further example, storage device 32 may correspond to a file server or another intermediate storage device capable of holding the encoded video data generated by source device 12. Destination device 14 may access the stored video data from storage device 32 via streaming or download. The file server may be any type of computer capable of storing encoded video data and transmitting the encoded video data to destination device 14. Exemplary file servers include web servers (e.g., for websites), File Transfer Protocol (FTP) servers, Network Attached Storage (NAS) devices, or local disk drives. Destination device 14 may access the encoded video data through any standard data connection, including a wireless channel (e.g., a Wireless Fidelity (Wi-Fi) connection), a wired connection (e.g., a Digital Subscriber Line (DSL), a cable modem, etc.), or a combination of both suitable for accessing encoded video data stored on a file server. Transmission of the encoded video data from storage device 32 may be a streaming transmission, a download transmission, or a combination of both.

[0018] As shown in FIG. 1 , source device 12 includes video source 18, video encoder 20, and output interface 22. Video source 18 may include a source such as a video capture device, e.g., a video camera, a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as source video, or a combination of such sources. As an example, if video source 18 is a video camera in a security surveillance system, source device 12 and destination device 14 may form a camera phone or video phone. However, implementations described herein may be applicable to video encoding generally and may be applicable to wireless and / or wired applications.

[0019] The captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video data may be transmitted directly to destination device 14 via output interface 22 of source device 12. The encoded video data may also (or alternatively) be stored in storage device 32 for later access by destination device 14 or other devices for decoding and / or playback. Output interface 22 may further include a modem and / or a transmitter.

[0020] Destination device 14 includes an input interface 28, a video decoder 30, and a display device 34. Input interface 28 may include a receiver and / or a modem and receive encoded video data via link 16. The encoded video data communicated via link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included within the encoded video data transmitted over a communication medium, stored on a storage medium, or stored on a file server.

[0021] In some implementations, destination device 14 may include a display device 34, which may be an integrated display device or an external display device configured to communicate with destination device 14. Display device 34 displays the decoded video data to a user and may comprise any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

[0022] Video encoder 20 and video decoder 30 may operate in accordance with proprietary or industry standards, such as VVC, HEVC, MPEG-4, Part 10, AVC, or extensions of such standards. It should be understood that the present application is not limited to a particular video encoding / decoding standard and may be applicable to other video encoding / decoding standards. It is generally contemplated that video encoder 20 of source device 12 may be configured to encode video data in accordance with any of these current or future standards. Similarly, it is generally contemplated that video decoder 30 of destination device 14 may be configured to decode video data in accordance with any of these current or future standards.

[0023] Video encoder 20 and video decoder 30 may each be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. If implemented partially in software, the electronic device may store instructions for the software on a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the video encoding / decoding operations disclosed in this disclosure. Each of video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder / decoder (CODEC) within the respective device.

[0024] 2 is a block diagram illustrating an example video encoder 20 according to some implementations described herein. Video encoder 20 may perform intra-predictive and inter-predictive coding of video blocks within video frames. Intra-predictive coding relies on spatial prediction to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-predictive coding relies on temporal prediction to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence. It should be noted that the term "frame" is sometimes used synonymously with the terms "image" or "picture" in the field of video coding.

[0025] 2, video encoder 20 includes a video data memory 40, a prediction processing unit 41, a decoded picture buffer (DPB) 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. Prediction processing unit 41 further includes a motion estimation unit 42, a motion compensation unit 44, a partitioning unit 45, an intra-prediction processing unit 46, and an intra-block copy (BC) unit 48. In some implementations, video encoder 20 also includes an inverse quantization unit 58 for video block reconstruction, an inverse transform processing unit 60, and an adder 62. An in-loop filter 63, such as a deblocking filter, may be disposed between adder 62 and DPB 64 to filter block boundaries and remove blocky artifacts from the reconstructed video. In addition to the deblocking filter, another in-loop filter, such as a Sample Adaptive Offset (SAO) filter and / or an Adaptive in-loop filter (ALF), may also be used to filter the output of summer 62. In some examples, the in-loop filter may be omitted, and the decoded video block may be provided directly by summer 62 to DPB 64. Video encoder 20 may take the form of a fixed or programmable hardware unit, or may be divided into one or more of the illustrated fixed or programmable hardware units.

[0026] Video data memory 40 may store video data to be encoded by components of video encoder 20. The video data in video data memory 40 may be obtained, for example, from video source 18 shown in FIG. 1. DPB 64 is a buffer that stores reference video data (e.g., reference frames or reference pictures) for use in encoding video data by video encoder 20 (e.g., in an intra-predictive coding mode or an inter-predictive coding mode). Video data memory 40 and DPB 64 may be formed by any of a variety of memory devices. In various examples, video data memory 40 may be on-chip with other components of video encoder 20 or may be off-chip relative to those components.

[0027] As shown in FIG. 2, after receiving video data, partitioning unit 45 in prediction processing unit 41 partitions the video data into video blocks. This partitioning may also include partitioning the video frame into slices, tiles (e.g., sets of video blocks), or other larger coding units (CUs) according to a predefined division structure, such as a quad-tree (QT) structure associated with the video data. A video frame may be or be considered as a two-dimensional array or matrix of samples having sample values. The samples in the array may also be referred to as pixels or pels. The number of samples in the horizontal and vertical directions (or axes) of the array or picture defines the size and / or resolution of the video frame. The video frame may be divided into multiple video blocks, for example, by using QT partitioning. A video block may also be or be considered as a two-dimensional array or matrix of samples having sample values, although with smaller dimensions than a video frame. The number of samples in the horizontal and vertical directions (or axes) of a video block defines the size of the video block. A video block may be further partitioned into one or more block partitions or sub-blocks (which may again form blocks), e.g., by repeatedly using QT partitioning, binary-tree (BT) partitioning, or triple-tree (TT) partitioning, or any combination thereof. Note that the term “block” or “video block” as used herein may be a portion of a frame or picture, particularly a rectangular (square or non-square) portion. For example, with reference to HEVC and VVC, a block or video block may be or correspond to a coding tree unit (CTU), CU, prediction unit (PU), or transform unit (TU), and / or may be or correspond to a corresponding block, e.g., a coding tree block (CTB), coding block (CB), prediction block (PB), or transform block (TB), and / or sub-block.

[0028] Prediction processing unit 41 may select one of multiple possible predictive coding modes, such as one of multiple intra-predictive coding modes or one of multiple inter-predictive coding modes, for the current video block based on the error result (e.g., the coding rate and level of distortion). Prediction processing unit 41 may provide the resulting intra-predictively coded block or inter-predictively coded block to summer 50 to generate a residual block and to summer 62 to reconstruct the coded block for later use as part of a reference frame. Prediction processing unit 41 also provides syntax elements, such as motion vectors, intra-mode indicators, partition information, and other such syntax information, to entropy coding unit 56.

[0029] To select an appropriate intra-prediction coding mode for a current video block, intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction coding of the current video block relative to one or more neighboring blocks in the same frame as the current block to be coded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-prediction coding of the current video block relative to one or more predictive blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple coding passes, for example, to select an appropriate coding mode for each block of video data.

[0030] In some implementations, motion estimation unit 42 determines the inter-prediction mode for a current video frame by generating motion vectors that indicate the displacement of video blocks in the current video frame relative to predictive blocks in a reference video frame according to a predetermined pattern in the sequence of video frames. Motion estimation performed by motion estimation unit 42 is the process of generating motion vectors that estimate the motion of video blocks. The motion vectors may indicate, for example, the displacement of video blocks in the current video frame or picture relative to predictive blocks in a reference frame relative to the current block being coded in the current frame. The predetermined pattern may designate video frames in the sequence as P-frames or B-frames. Intra BC unit 48 may determine vectors, e.g., block vectors, for intra BC coding in a manner similar to the determination of motion vectors by motion estimation unit 42 for inter prediction, or may utilize motion estimation unit 42 to determine block vectors.

[0031] A prediction block for a video block may be or correspond to a block or reference block of a reference frame that is deemed to closely match the video block to be encoded in terms of pixel differences, which may be determined by sum of absolute differences (SAD), sum of square differences (SSD), or other difference metrics. In some implementations, video encoder 20 may calculate values for sub-integer pixel positions of the reference frame stored in DPB 64. For example, video encoder 20 may interpolate values for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference frame. Thus, motion estimation unit 42 may perform motion searches for whole pixel positions and fractional pixel positions and output motion vectors with fractional pixel precision.

[0032] Motion estimation unit 42 calculates a motion vector for a video block in an inter-predictively coded frame by comparing the position of the video block to the position of a predictive block of a reference frame selected from a first reference frame list (List 0) or a second reference frame list (List 1), each of which identifies one or more reference frames stored in DPB 64. Motion estimation unit 42 sends the calculated motion vector to motion compensation unit 44 and then to entropy coding unit 56.

[0033] The motion compensation performed by motion compensation unit 44 may include fetching or generating a predictive block based on the motion vector determined by motion estimation unit 42. Upon receiving the motion vector for the current video block, motion compensation unit 44 may locate the predictive block to which the motion vector points in one of the reference frame lists, obtain the predictive block from DPB 64, and forward the predictive block to summer 50. Summer 50 then forms a residual video block of pixel difference values by subtracting pixel values of the predictive block provided by motion compensation unit 44 from pixel values of the current video block being coded. The pixel difference values forming the residual video block may include luma difference components, chroma difference components, or both. Motion compensation unit 44 may also generate syntax elements associated with the video blocks of the video frame for use by video decoder 30 in decoding the video blocks of the video frame. The syntax elements may include, for example, syntax elements defining the motion vector used to identify the predictive block, any flags indicating a prediction mode, or any other syntax information described herein. It should be noted that motion estimation unit 42 and motion compensation unit 44 may be highly integrated, but are shown separately for conceptual purposes.

[0034] In some implementations, the intra BC unit 48 may generate a vector and fetch a predictive block in a manner similar to that described above in connection with the motion estimation unit 42 and the motion compensation unit 44, except that the predictive block is in the same frame as the current block being coded, and the vector is referred to as a block vector rather than a motion vector. Specifically, the intra BC unit 48 may determine an intra prediction mode to use to code the current block. In some examples, the intra BC unit 48 may code the current block using various intra prediction modes, e.g., during separate coding passes, and test their performance through rate-distortion analysis. The intra BC unit 48 may then select an appropriate intra prediction mode to use from the various tested intra prediction modes and generate an intra mode indicator accordingly. For example, the intra BC unit 48 may calculate rate-distortion values using the rate-distortion analysis for the various tested intra prediction modes and select the intra prediction mode with the best rate-distortion characteristics from the tested modes as the appropriate intra prediction mode to use. Rate-distortion analysis generally determines the amount of distortion (or error) between a coded block and the original uncoded block that was coded to create the coded block, as well as the bit rate (i.e., number of bits) used to create the coded block. Intra BC unit 48 may calculate ratios from the distortions and rates of the various coded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for that block.

[0035] In other examples, intra BC unit 48 may use, in whole or in part, motion estimation unit 42 and motion compensation unit 44 to perform such functions for intra BC prediction according to implementations described herein. In either case, for intra block copying, the predictive block may be a block that is deemed to closely match the block to be coded in terms of pixel differences, which may be determined by SAD, SSD, or other difference metrics, and identifying the predictive block may include calculating values for sub-integer pixel positions.

[0036] Regardless of whether the predictive block is a block from the same frame via intra prediction or a block from a different frame via inter prediction, video encoder 20 may form a residual video block by subtracting pixel values of the predictive block from pixel values of the current video block being coded to form pixel difference values. The pixel difference values that form the residual video block may include both luma and chroma component differences.

[0037] Intra-prediction processing unit 46 may intra-predict the current video block as an alternative to inter-prediction performed by motion estimation unit 42 and motion compensation unit 44 or intra-block copy prediction performed by intra BC unit 48, as described above. Specifically, intra-prediction processing unit 46 may determine an intra-prediction mode to use to encode the current block. To do so, intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, e.g., during separate encoding passes, and intra-prediction processing unit 46 (or, in some examples, a mode selection unit) may select an appropriate intra-prediction mode to use from the tested intra-prediction modes. Intra-prediction processing unit 46 may provide information indicating the selected intra-prediction mode for the block to entropy coding unit 56. Entropy coding unit 56 may encode the information indicating the selected intra-prediction mode into the bitstream.

[0038] After prediction processing unit 41 determines a predictive block for the current video block by inter-prediction or intra-prediction, adder 50 forms a residual video block by subtracting the predictive block from the current video block. The residual video data in the residual block, which may be included in one or more TUs, is provided to transform processing unit 52. Transform processing unit 52 converts the residual video data into residual transform coefficients using a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform.

[0039] Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54, which quantizes the transform coefficients to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization unit 54 may then perform a scan of a matrix containing the quantized transform coefficients. Alternatively, entropy coding unit 56 may perform the scan.

[0040] Following quantization, entropy coding unit 56 entropy codes the quantized transform coefficients into a video bitstream using, for example, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding methodology or technique. The coded bitstream may then be transmitted to video decoder 30 as shown in FIG. 1 or archived to storage device 32 as shown in FIG. 1 for later transmission to or retrieval by video decoder 30. Entropy coding unit 56 may also entropy code motion vectors and other syntax elements of the current video frame being coded.

[0041] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain to generate reference blocks for prediction of other video blocks. As described above, motion compensation unit 44 may generate a motion-compensated prediction block from one or more reference blocks of a frame stored in DPB 64. Motion compensation unit 44 may also apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values for use in motion estimation.

[0042] Summer 62 adds the reconstructed residual block to the motion compensated prediction block produced by motion compensation unit 44 to create a reference block for storage in DPB 64. The reference block may then be used by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 as a prediction block for inter predicting another video block in a subsequent video frame.

[0043] 3 is a block diagram illustrating an exemplary video decoder 30 according to some implementations of the present application. Video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, and a DPB 92. Prediction processing unit 81 further includes a motion compensation unit 82, an intra prediction unit 84, and an intra BC unit 85. Video decoder 30 may perform a decoding process that is generally the reverse of the encoding process described above for video encoder 20 in connection with FIG. 2. For example, motion compensation unit 82 may generate prediction data based on a motion vector received from entropy decoding unit 80, while intra prediction unit 84 may generate prediction data based on an intra prediction mode indicator received from entropy decoding unit 80.

[0044] In some examples, units of video decoder 30 may be tasked with performing implementations of the present application. Also, in some examples, implementations of the present disclosure may be divided among one or more of the units of video decoder 30. For example, intra BC unit 85 may perform implementations of the present application alone or in combination with other units of video decoder 30, such as motion compensation unit 82, intra prediction unit 84, and entropy decoding unit 80. In some examples, video decoder 30 may not include intra BC unit 85, and the functions of intra BC unit 85 may be performed by other components of prediction processing unit 81, such as motion compensation unit 82.

[0045] Video data memory 79 may store video data, such as an encoded video bitstream, to be decoded by other components of video decoder 30. The video data stored in video data memory 79 may be obtained, for example, from storage device 32, from a local video source such as a camera, via wired or wireless network communication of video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk). Video data memory 79 may include a coded picture buffer (CPB) that stores coded video data from the coded video bitstream. A data buffer buffer (DPB) 92 of video decoder 30 stores reference video data for use in decoding video data by video decoder 30 (e.g., in an intra-predictive coding mode or an inter-predictive coding mode). Video data memory 79 and DPB 92 may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous dynamic random access memory (SDRAM), magnetoresistive random access memory (MRAM), resistive random access memory (RRAM), or other types of memory devices. 3, video data memory 79 and DPB 92 are depicted as two separate components of video decoder 30. However, it will be apparent to those skilled in the art that video data memory 79 and DPB 92 may be provided by the same memory device or separate memory devices. In some examples, video data memory 79 may be on-chip with other components of video decoder 30 or may be off-chip relative to those components.

[0046] During the decoding process, video decoder 30 receives an encoded video bitstream representing video blocks and associated syntax elements of encoded video frames. Video decoder 30 may receive the syntax elements at the video frame level and / or the video block level. Entropy decoding unit 80 of video decoder 30 entropy decodes the bitstream to generate quantized coefficients, motion vectors or intra-prediction mode indicators, and other syntax elements. Entropy decoding unit 80 then forwards the motion vectors or intra-prediction mode indicators and other syntax elements to prediction processing unit 81.

[0047] When a video frame is coded as an intra-predictively coded (I) frame or for intra-coded predictive blocks within other types of frames, intra prediction unit 84 of prediction processing unit 81 may generate predictive data for video blocks of the current video frame based on a signaled intra-prediction mode and reference data from previously decoded blocks of the current frame.

[0048] When a video frame is coded as an inter-predictive (i.e., B or P) frame, motion compensation unit 82 of prediction processing unit 81 creates one or more predictive blocks for video blocks of the current video frame based on the motion vectors and other syntax elements received from entropy decoding unit 80. Each of the predictive blocks may be created from a reference frame in one of the reference frame lists. Video decoder 30 may construct the reference frame lists, List 0 and List 1, using a default construction technique based on the reference frames stored in DPB 92.

[0049] In some examples, when a video block is encoded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 creates a predictive block of the current video block based on the block vectors and other syntax elements received from entropy decoding unit 80. The predictive block may be within the same reconstructed region of the picture as the current video block defined by video encoder 20.

[0050] Motion compensation unit 82 and / or intra BC unit 85 determine prediction information for video blocks of the current video frame by analyzing motion vectors and other syntax elements, and then use the prediction information to create predictive blocks for the current video block being decoded. For example, motion compensation unit 82 uses some of the received syntax elements to determine the prediction mode (e.g., intra-prediction or inter-prediction) used to encode the video blocks of the video frame, the inter-prediction frame type (e.g., B or P), construction information regarding one or more of the frame's reference frame list, the motion vectors of each inter-predictively coded video block of the frame, the inter-prediction status of each inter-predictively coded video block of the frame, and other information for decoding video blocks in the current video frame.

[0051] Similarly, intra BC unit 85 may use some of the received syntax elements, such as flags, to determine that the current video block was predicted using intra BC mode, construction information regarding which video blocks of the frame are within the reconstruction region and should be stored in DPB 92, block vectors for each intra BC predicted video block of the frame, the intra BC prediction status for each intra BC predicted video block of the frame, and other information for decoding the video blocks in the current video frame.

[0052] Motion compensation unit 82 may also perform interpolation using an interpolation filter used by video encoder 20 during encoding of the video block to calculate sub-integer pixel interpolated values of the reference block. In this case, motion compensation unit 82 may determine the interpolation filter used by video encoder 20 from the received syntax element and use the interpolation filter to create the predictive block.

[0053] Inverse quantization unit 86 uses the same quantization parameter calculated by video encoder 20 for each video block in a video frame to inverse quantize the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80 to determine the degree of quantization. Inverse transform processing unit 88 applies an inverse transform, e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients in order to reconstruct residual blocks in the pixel domain.

[0054] After motion compensation unit 82 or intra BC unit 85 generates a predictive block for the current video block based on the vectors and other syntax elements, adder 90 reconstructs a decoded video block for the current video block by adding a residual block from inverse transform processing unit 88 and the corresponding predictive block generated by motion compensation unit 82 and intra BC unit 85. To further process the decoded video block, an in-loop filter 91, such as a deblocking filter, an SAO filter, and / or an ALF, may be disposed between adder 90 and the DPB. In some examples, in-loop filter 91 may be omitted, and the decoded video block may be provided directly to DPB 92 by adder 90. The decoded video block in a given frame is then stored in DPB 92, which stores reference frames used for subsequent motion compensation of the next video block. DPB 92 or a memory device separate from DPB 92 may store the decoded video for later presentation on a display device, such as display device 34 of FIG. 1.

[0055] In a typical video coding process, a video sequence typically includes an ordered set of frames or pictures. Each frame may include three sample arrays, denoted SL, SCb, and SCr. SL is a two-dimensional array of luma samples. SCb is a two-dimensional array of Cb chroma samples. SCr is a two-dimensional array of Cr chroma samples. In other examples, a frame may be monochromatic and therefore include only one two-dimensional array of luma samples.

[0056] As shown in FIG. 4A, video encoder 20 (or, more specifically, partitioning unit 45) generates an encoded representation of a frame by first partitioning the frame into a set of CTUs. A video frame may include an integer number of CTUs, sequentially ordered from left to right and top to bottom in raster scan order. Each CTU is the largest logical coding unit, and the width and height of the CTU are signaled by video encoder 20 in a sequence parameter set so that all CTUs in a video sequence have the same size, which may be one of 128×128, 64×64, 32×32, and 16×16. However, it should be noted that the present application is not necessarily limited to a particular size. As shown in FIG. 4B, each CTU may include one CTB of luma samples, two corresponding coding tree blocks of chroma samples, and syntax elements used to encode the samples in the coding tree blocks. The syntax elements describe how a video sequence may be reconstructed in video decoder 30, including characteristics of different types of units of coded blocks of pixels, as well as inter- or intra-prediction, intra-prediction mode, motion vectors, and other parameters. For a monochrome picture or a picture with three separate color planes, a CTU may include a single coding tree block and syntax elements used to code samples of the coding tree block. A coding tree block may be an N×N block of samples.

[0057] To achieve better performance, video encoder 20 may recursively perform tree partitioning, such as binary tree partitioning, ternary tree partitioning, quad tree partitioning, or a combination thereof, on the coding tree blocks of the CTU to divide the CTU into smaller CUs. As depicted in FIG. 4C , 64×64 CTU 400 is first partitioned into four smaller CUs, each with a block size of 32×32. Of the four smaller CUs, CU 410 and CU 420 are each partitioned into four 16×16 CUs by block size. Two 16×16 CUs, CU 430 and CU 440, are each further partitioned into four 8×8 CUs by block size. FIG. 4D depicts a quad tree data structure showing the final result of the partitioning process for CTU 400 depicted in FIG. 4C , where each leaf node of the quad tree corresponds to one CU of a respective size ranging from 32×32 to 8×8. Similar to the CTU depicted in FIG. 4B, each CU may include a CB for luma samples, two corresponding coding blocks for chroma samples of the same size frame, and syntax elements used to encode the samples of the coding block. In a monochrome picture or a picture with three separate color planes, a CU may include a single coding block and syntax structures used to encode the samples of the coding block. Note that the quadtree partitioning depicted in FIGS. 4C and 4D is for illustrative purposes only, and a CTU may be divided into CUs based on quadtree / ternary tree / binary tree partitioning to accommodate various local characteristics. In a multi-type tree structure, a CTU is partitioned by a quadtree structure, and each quadtree leaf CU may be further partitioned by a binary tree structure and a ternary tree structure. As shown in FIG. 4E, there are five possible partition types of a coding block with width W and height H: quad-partition, horizontal 2-partition, vertical 2-partition, horizontal 3-partition, and vertical 3-partition.

[0058] In some implementations, video encoder 20 may further partition the coding blocks of a CU into one or more M×N PBs. A PB is a rectangular (square or non-square) block of samples to which the same prediction, either inter or intra, is applied. A PU of a CU may include a PB of luma samples, two corresponding PBs of chroma samples, and syntax elements used to predict the PB. In a monochrome picture or a picture with three separate color planes, a PU may include a single PB and syntax structures used to predict the PB. Video encoder 20 may generate predictive luma blocks, predictive Cb blocks, and predictive Cr blocks for the luma, Cb, and Cr PBs of each PU of a CU.

[0059] Video encoder 20 may generate the predictive blocks of a PU using intra prediction or inter prediction. If video encoder 20 generates the predictive blocks of a PU using intra prediction, video encoder 20 may generate the predictive blocks of the PU based on decoded samples of a frame associated with the PU. If video encoder 20 generates the predictive blocks of a PU using inter prediction, video encoder 20 may generate the predictive blocks of the PU based on decoded samples of one or more frames other than the frame associated with the PU.

[0060] After video encoder 20 generates the predictive luma block, the predictive Cb block, and the predictive Cr block for one or more PUs of a CU, video encoder 20 may generate the luma residual block of the CU by subtracting the predictive luma block of the CU from its original luma coding block, such that each sample in the luma residual block of the CU indicates a difference between a luma sample in one of the predictive luma blocks of the CU and a corresponding sample in the original luma coding block of the CU. Similarly, video encoder 20 may generate the Cb residual block and the Cr residual block of the CU, such that each sample in the Cb residual block of the CU indicates a difference between a Cb sample in one of the predictive Cb blocks of the CU and a corresponding sample in the original Cb coding block of the CU, and such that each sample in the Cr residual block of the CU indicates a difference between a Cr sample in one of the predictive Cr blocks of the CU and a corresponding sample in the original Cr coding block of the CU.

[0061] Further, as shown in FIG. 4C , video encoder 20 may use quadtree partitioning to decompose the luma residual block, Cb residual block, and Cr residual block of a CU into one or more luma transform blocks, Cb transform blocks, and Cr transform blocks, respectively. A transform block is a rectangular (square or non-square) block of samples to which the same transform is applied. A TU of a CU may include a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax elements used to transform the transform block samples. Thus, each TU of a CU may be associated with a luma transform block, a Cb transform block, and a Cr transform block. In some examples, the luma transform block associated with a TU may be a sub-block of the luma residual block of the CU. The Cb transform block may be a sub-block of the Cb residual block of the CU. The Cr transform block may be a sub-block of the Cr residual block of the CU. In a monochrome picture or a picture with three separate color planes, a TU may include a single transform block and syntax structures used to transform the samples of the transform block.

[0062] Video encoder 20 may apply one or more transforms to a luma transform block of a TU to generate a luma coefficient block of the TU. The coefficient block may be a two-dimensional array of transform coefficients. The transform coefficients may be scalar quantities. Video encoder 20 may apply one or more transforms to a Cb transform block of the TU to generate a Cb coefficient block of the TU. Video encoder 20 may apply one or more transforms to a Cr transform block of the TU to generate a Cr coefficient block of the TU.

[0063] After generating a coefficient block (e.g., a luma coefficient block, a Cb coefficient block, or a Cr coefficient block), the video encoder 20 may quantize the coefficient block. Quantization generally refers to a process in which transform coefficients are quantized to potentially reduce the amount of data used to represent the transform coefficients, thereby achieving further compression. After the video encoder 20 quantizes the coefficient block, the video encoder 20 may entropy encode syntax elements indicating the quantized transform coefficients. For example, the video encoder 20 may perform CABAC on the syntax elements indicating the quantized transform coefficients. Finally, the video encoder 20 may output a bitstream including a sequence of bits forming a representation of the encoded frame and associated data, which may be stored on a storage device 32 or transmitted to a destination device 14.

[0064] After receiving the bitstream generated by video encoder 20, video decoder 30 may parse the bitstream to obtain syntax elements from the bitstream. Video decoder 30 may reconstruct frames of video data based at least in part on the syntax elements obtained from the bitstream. The process of reconstructing video data is generally the reverse of the encoding process performed by video encoder 20. For example, video decoder 30 may perform an inverse transform on coefficient blocks associated with TUs of the current CU to reconstruct residual blocks associated with the TUs of the current CU. Video decoder 30 also reconstructs coding blocks of the current CU by adding samples of predictive blocks of PUs of the current CU to corresponding samples of transform blocks of TUs of the current CU. After reconstructing coding blocks for each CU of the frame, video decoder 30 may reconstruct the frame.

[0065] As mentioned above, video coding achieves video compression using two main modes: intra-frame prediction (or intra-prediction) and inter-frame prediction (or inter-prediction). Note that IBC can be considered as intra-frame prediction or a third mode. Of the two modes, inter-frame prediction contributes more to coding efficiency than intra-frame prediction because it uses motion vectors to predict a current video block from a reference video block.

[0066] However, as video data capture technology is constantly improving and video block sizes become finer to preserve details of the video data, the amount of data required to represent the motion vectors of the current frame is also increasing significantly. One way to overcome this challenge is to benefit from the fact that a group of adjacent CUs in both the spatial and temporal domains not only have similar video data for prediction purposes, but also have similar motion vectors between these adjacent CUs. Therefore, the motion information of spatially adjacent CUs and / or temporally co-located CUs can be used as an approximation of the motion information (e.g., motion vector) of the current CU by investigating their spatial and temporal correlations, which is also called the "motion vector predictor (MVP)" of the current CU.

[0067] Instead of encoding the actual motion vector of the current CU determined by motion estimation unit 42 into the video bitstream as described above in connection with Figure 3, the motion vector predictor of the current CU is subtracted from the actual motion vector of the current CU to create a Motion Vector Difference (MVD) for the current CU. By doing so, the motion vector determined for each CU of a frame by motion estimation unit 42 does not need to be encoded into the video bitstream, and the amount of data used to represent motion information in the video bitstream may be significantly reduced.

[0068] Similar to the process of choosing a predictive block in a reference frame during inter-frame prediction of a codeblock, a set of rules needs to be adopted by both video encoder 20 and video decoder 30 to construct a motion vector candidate list (also called a “merge list”) for the current CU using potential candidate motion vectors associated with spatially neighboring CUs and / or temporally co-located CUs of the current CU, and then select one element from the motion vector candidate list as the motion vector predictor for the current CU. By doing so, the motion vector candidate list itself does not need to be transmitted from video encoder 20 to video decoder 30; the index of the selected motion vector predictor in the motion vector candidate list is sufficient for video encoder 20 and video decoder 30 to use the same motion vector predictor in the motion vector candidate list to encode and decode the current CU.

[0069] Position-dependent intra-prediction combination In VVC, the results of intra prediction for DC mode, planar mode, and some angle modes are further modified by the position dependent intra prediction combination (PDPC) method. PDPC is an intra prediction method that invokes the combination of boundary reference samples and HEVC-style intra prediction using filtered boundary reference samples. PDPC applies to the following intra modes without signaling: planar, DC, horizontal intra angles less than or equal to 100°, and vertical intra angles less than or equal to 80°. PDPC is not applied if the current block is in Bdpcm mode or the MRL index is greater than 0.

[0070] Using a linear combination of intra prediction modes (DC, planar, angular) and reference samples, the following equation pred(x',y')=Clip(0,(1<<BitDepth)-1,(wL×R-1,y’+wT×Rx’,-1+(64-wL-wT)×pred(x’,y’)+32)> >6) The predicted sample pred(x',y') is predicted according to, where Rx,-1, R-1,y represent the reference samples located at the upper and left boundaries of the current sample (x,y), respectively.

[0071] When PDPC is applied to DC, planar, horizontal, and vertical intra modes, it does not require the additional boundary filters required in the case of the HEVC DC mode boundary filter or horizontal / vertical mode edge filter. PDPC processing for DC mode and planar mode is identical. For angular modes, if the current angular mode is HOR_IDX or VER_IDX, the left or top reference sample is not used, respectively. PDPC weights and scale factors depend on the prediction mode and block size. PDPC is applied to blocks with both width and height equal to or greater than 4.

[0072] 5A to 5D show definitions of PDPC reference samples (Rx,-1 and R-1,y) applied to various prediction modes. FIG. 5A shows an example of a diagonal-top-right mode. FIG. 5B shows an example of a diagonal-bottom-left mode. FIG. 5C shows an example of an adjacent diagonal-top-right mode. FIG. 5D shows an example of an adjacent diagonal-bottom-left mode. The prediction sample pred(x',y') is located at (x',y') within the prediction block. As an example, in the diagonal mode, the coordinate x of the reference sample Rx,-1 is given by x=x'+y'+1, and the coordinate y of the reference sample R-1,y is similarly given by y=x'+y'+1. In other angular modes, the reference samples Rx,-1 and R-1,y may be located at fractional sample positions. In this case, the sample value of the nearest integer sample position is used.

[0073] As mentioned above, intra prediction samples are generated from a set of unfiltered or filtered neighboring reference samples, which may cause discontinuities along block boundaries between the current coding block and its neighboring blocks. To solve such problems, HEVC applies boundary filtering by combining the first row / column of prediction samples in DC prediction mode, horizontal prediction mode (i.e., mode 18), and vertical prediction mode (i.e., mode 50) with the unfiltered reference samples using a 2-tap filter (for DC mode) or a gradient-based smoothing filter (for horizontal prediction mode and vertical prediction mode).

[0074] Gradient PDPC In VVC, in some scenarios, PDPC may not be applied because the secondary reference sample is not available. Gradient-based PDPC, extended from horizontal / vertical mode, is applied. The PDPC weights (wT / wL) and the nScale parameter, which determines the attenuation in the PDPC weight with respect to the distance from the left / top boundary, are set equal to the corresponding parameters in horizontal / vertical mode, respectively. If the secondary reference sample is at a fractional sample position, bilinear interpolation is applied.

[0075] Geometric partition mode (GPM) VVC supports geometric partitioning mode for inter prediction, which is signaled by a CU-level flag as a special merge mode. In the current GPM design, a total of 64 partitions are supported by GPM modes for each possible CU size where both width and height are between 8 and 64, excluding 8x64 and 64x8.

[0076] When this mode is used, a CU is divided into two parts by a geometrically positioned line, as shown in Figure 6. The position of the division line is mathematically derived from the angle and offset parameters of a particular partition. Each part of a geometric partition within a CU is inter-predicted using its own motion, and only uni-prediction is allowed for each partition; that is, each part has one motion vector and one reference index. Similar to traditional bi-prediction, a uni-prediction motion constraint is applied to ensure that only two motion-compensated predictions are required for each CU. When the geometric partition mode is used for the current CU, a geometric partition index (angle and offset) and two merge indices (one for each partition) indicating the partition mode of the geometric partition are also signaled. The number of maximum GPM candidate sizes is explicitly signaled at the sequence level.

[0077] Blending along the edges of geometric divisions After each geometric partition is obtained using its unique motion, blending is applied to the two uni-predictive signals to derive samples around the edges of the geometric partition. The blending weights for each position of the CU are derived based on the distance from the individual sample position to the corresponding partition edge.

[0078] GPM signal transmission design According to the current GPM design, the use of GPM is indicated by signaling one flag at the CU level. The flag is signaled only if the current CU is coded by merge mode or skip mode. Specifically, if the flag is equal to 1, the flag indicates that the current CU is predicted by GPM. Otherwise (if the flag is equal to 0), the CU is coded by another merge mode, such as normal merge mode, merge mode with motion vector differential, or combined inter- and intra-prediction. If GPM is enabled for the current CU, one syntax element, namely, merge_gpm_partition_idx, is further signaled to indicate the applied geometric partition mode (which specifies the direction and offset of the line from the CU center that divides the CU into two partitions, as shown in FIG. 6). Then, two syntax elements, merge_gpm_idx0 and merge_gpm_idx1, are signaled to indicate the indices of the unipredictive merge candidates used for the first and second GPM partitions. More specifically, these two syntax elements are used to determine the unidirectional MVs of two GPM partitions from the unipredictive merge list, as described in the section "Constructing a Unipredictive Merge List." According to the current GPM design, in order to make the two unidirectional MVs more different, the two indices cannot be the same. Based on such prior knowledge, the unipredictive merge index of the first GPM partition is first signaled and used as a predictor to reduce the signaling overhead of the unipredictive merge index of the second GPM partition. In particular, if the second unipredictive merge index is smaller than the first unipredictive merge index, its original value is directly signaled. Otherwise (if the second unipredictive merge index is larger than the first unipredictive merge index), its value is subtracted by 1 before being signaled to the bitstream. On the decoder side, the first unipredictive merge index is first decoded.Then, for decoding the second unipredictive merge index, if the parsed value is less than the first unipredictive merge index, the second unipredictive merge index is set equal to the parsed value, otherwise (if the parsed value is greater than or equal to the first unipredictive merge index), the second unipredictive merge index is set equal to the parsed value plus 1. Table 1 shows the existing syntax elements used for GPM mode in the current VVC specification.

[0079] [Table 1]

[0080] On the other hand, in the current GPM design, a truncated unary code is used for binarization of the two unipredictive merge indexes, namely, merge_gpm_idx0 and merge_gpm_idx1. Furthermore, since the two unipredictive merge indexes cannot be the same, different maximum values are used for truncating the codewords of the two unipredictive merge indexes, and the maximum values are set to be equal to MaxGPMMergeCand-1 and MaxGPMMergeCand-2 for merge_gpm_idx0 and merge_gpm_idx1, respectively. MaxGPMMergeCand is the number of candidates in the unipredictive merge list.

[0081] When GPM / AWP mode is applied, two different binarization methods are applied to convert the syntax merge_gpm_partition_idx into a string of binary bits. Specifically, the syntax element is binarized by fixed length code and truncated binary code in the VVC and AVS3 standards, respectively. On the other hand, in the AWP mode of AVS3, a different maximum value is used to binarize the syntax element.

[0082] Spatial angular weighted prediction (SAWP) AVS uses the Spatial Angle Weighted Prediction (SAWP) mode, which extends the GPM mode to intra blocks. In SAWP mode, instead of weighting two inter-predicted blocks, two intra-predicted blocks are weighted. The two intra-predicted blocks are predicted using two different intra-prediction modes selected from the intra-prediction modes. The intra-prediction modes are selected from angular modes 5 to 30. The maximum size is 32x32. To derive the MPM for SAWP mode, the two most probable modes (MPMs) of the regular intra-modes are used.

[0083] Multi-direction intra prediction (MDIP) designs follow the same design spirit as SAWP, but with some minor differences in certain design details.

[0084] Decoder-side Intra-mode Derivation (DIMD) DIMD is an intra-coding tool in which the luma intra-prediction mode (IPM) is not transmitted via the bitstream. Instead, the IPM is derived using previously coded / decoded pixels in the same manner at the encoder and decoder. The DIMD method performs texture gradient processing to derive two optimal modes. These two modes and the planar mode are then applied to the block, and their predictors are weighted-averaged. The DIMD selection result is signaled in the bitstream of the intra-coded block using a flag. At the decoder, if the DIMD flag is true, the intra-prediction mode is derived in the reconstruction process using the same previously coded neighboring pixels. If not true, the intra-prediction mode is parsed from the bitstream in the same way as a classical intra-coding mode.

[0085] To derive the intra prediction mode for a block, we must first select a set of neighboring pixels on which we will perform gradient analysis. For standardity purposes, these pixels should be within the pool of decoded / reconstructed pixels. As shown in Figure 7, we choose a template that surrounds the current block by T pixels to the left and T pixels above. We then perform gradient analysis on the template's pixels. This allows us to determine the dominant angular orientation of the template, which we assume is likely to be identical to one of the current blocks (a core premise of our method). Therefore, we convolve the following matrix with the template:

number

[0086] For each pixel in the template, we multiply each of these two matrices by a 3x3 window centered on the current pixel and consisting of its eight immediate neighbors, and then sum the results, thus obtaining two values Gx (from multiplication with Mx) and Gy (from multiplication with My), which correspond to the horizontal and vertical gradients at the current pixel, respectively.

[0087] Figure 8 illustrates the convolution process. The blue pixel is the current pixel. The red pixels (including blue) are pixels for which gradient analysis is possible. The gray pixels are pixels for which gradient analysis is not possible due to lack of neighbors. The purple pixels are available (reconstructed) pixels outside the considered template and are used for the gradient analysis of the red pixel. If a purple pixel is not available (for example, because the block is too close to the picture boundary), gradient analysis of all red pixels that use this purple pixel is not performed. For each red pixel, the gradient strength (G) and orientation (O) are calculated using Gx and Gy:

number

[0088] The gradient orientation is then converted to an intra-angle prediction mode and used to index into a histogram (initialized to zero). The histogram value for that intra-angle mode is incremented by G. Once all red pixels in the template have been processed, the histogram will contain the cumulative gradient magnitudes for each intra-angle mode. For the current block, the IPM corresponding to the two highest histogram bars is selected. If the maximum value in the histogram is 0 (meaning that gradient analysis could not be performed or the area that makes up the template is flat), DC mode is selected as the intra-prediction mode for the current block.

[0089] The two IPMs corresponding to the two highest HoG bars are combined with a planar mode. In one or more instances, predictive fusion is applied as a weighted average of the three predictors mentioned above. For this purpose, the weight of the planar mode is fixed at 21 / 64 (approximately 1 / 3). The remaining weight, 43 / 64 (approximately 2 / 3), is then shared between the two HoG IPMs in proportion to the amplitude of the HoG bars. Figure 9 visualizes this process.

[0090] The derived intra modes are included in the primary list of intra most probable modes (MPMs), so DIMD processing is performed before the MPM list is built. The primary derived intra modes of a DIMD block are stored with the block and are used to build the MPM lists of neighboring blocks.

[0091] Template-based intra mode derivation (TIMD) For each intra mode in MPM, the sum of absolute transformed difference (SATD) between the predicted samples and reconstructed samples of the template region shown in Figure 10 is calculated, and the intra mode with the first two modes with the smallest SATD cost is selected, and then they are fused with weights, and such weighted intra prediction is used to encode the current CU.

[0092] The costs of the two selected modes are compared to a threshold and a cost factor of 2 is applied in the test as follows: costMode2<2*costMode1

[0093] If this condition is true, fusion is applied, otherwise only mode1 is used.

[0094] The weights of modes are calculated from their SATD costs as follows: weight1=costMode2 / (costMode1+costMode2) weight2=1-weight1

[0095] Although DIMD mode can improve intra prediction efficiency, there is still room for further improvement in its performance. Meanwhile, some parts of the existing DIMD mode also need to be simplified for efficient codec hardware implementation or improved for better coding efficiency. Furthermore, the trade-off between its implementation complexity and its coding efficiency benefits needs to be further improved.

[0096] After the finalization of VVC, the JVET group continued to explore compression efficiencies beyond VVC. JVET maintained a single reference software called the Extended Compression Model (ECM) by integrating several additional coding tools onto the VVC Test Model (VTM). In the current ECM, PDPC is used depending on the intra mode. For DIMD mode, PDPC is used depending on each intra mode. Two different positions of the PDPC method are used and applied to each intra mode in DIMD mode. For intra prediction using angular mode in DIMD mode, PDPC is applied before prediction fusion. For intra prediction using DC mode or planar mode in DIMD mode, PDPC is applied after prediction fusion. Such non-uniform designs may not be optimal from a standardization perspective.

[0097] Similarly, two different designs of the fusion scheme are available, applicable to DIMD and TIMD, respectively. Each different fusion design is associated with different candidate and weight calculations. For blocks to which DIMD is applied, the two IPMs and plane modes corresponding to the two highest HoG bars are selected for fusion. The plane weight is fixed at 21 / 64 (approximately 1 / 3). The remaining weight, 43 / 64 (approximately 2 / 3), is then shared between the two HoG IPMs in proportion to the amplitude of the HoG bars. For blocks to which TIMD is applied, the intra-mode with the first two modes with the smallest SATD cost is selected, and the mode weights are calculated from those SATD costs. Such non-uniform designs may not be optimal from a standardization perspective. In addition to the above, there is room for further improvement in performance through various fusion schemes.

[0098] In the current ECM design, intra-modes derived from DIMD are included in the primary list of intra-most probable modes (MPMs), regardless of whether the derived intra-modes are already used in DIMD. There is room for further improvement in its performance.

[0099] In this disclosure, methods are provided to simplify and / or further improve existing designs of DIMD modes to address previously pointed out problems. In general, the main features of the techniques proposed in this disclosure are summarized as follows:

[0100] 1) Unify the PDPC used under angular mode and DC / planar mode in DIMD mode by applying PDPC to all intra predictions before prediction fusion. An example of such a method is shown in the block diagram of Figure 12A.

[0101] 2) Unify the PDPC used under angular mode and DC / planar mode in DIMD mode by applying PDPC to all intra predictions after prediction fusion. An example of such a method is shown in the block diagram of Figure 12B.

[0102] 3) Unify the PDPC used under angular mode and DC / planar mode in DIMD mode by disabling PDPC for all intra prediction in DIMD mode. An example of such a method is shown in the block diagram of Figure 12C.

[0103] 4) The fusion method used under the DIMD mode is applied to the TIMD mode, thereby unifying the fusion methods used under the DIMD mode and the TIMD mode.

[0104] 5) The fusion method used under the DIMD mode and the TIMD mode is unified by applying the fusion method used under the TIMD mode to the DIMD mode.

[0105] 6) Unify the fusion methods used under DIMD mode and TIMD mode by signaling the selection result of the fusion method.

[0106] 7) Derive intra-modes from the DIMD into a list of intra-most probable modes (MPMs), taking into account whether the derived intra-modes are already used in the DIMD.

[0107] 8) Derive intra-modes from TIMD into a list of intra-most probable modes (MPMs).

[0108] It should be noted that the proposed method can also be applied to other intra-prediction coding modes such as TIMD / MDIP. Another set of examples applied to TIMD mode is shown in the block diagrams of Figures 11A to 11C. Figure 11A shows an example of applying all PDPC processing before TIMD fusion processing. Figure 11B shows an example of applying all PDPC processing after TIMD fusion processing. Figure 11C shows an example of disabling all PDPC processing in TIMD.

[0109] It should be noted that the proposed method can also be applied to other combined intra- and inter-prediction coding modes, such as combined inter and intra prediction (CIIP).

[0110] It should be noted that the disclosed methods can be applied singly or in combination.

[0111] Matching of PDPCs used for angular mode and DC / planar mode in DIMD

[0112] According to one or more embodiments of the present disclosure, the same PDPC position is applied for both angular and DC / planar modes under DIMD mode. Various methods may be used to achieve this goal.

[0113] In one example of the present disclosure, it is proposed to apply PDPC calculation before prediction fusion in DIMD mode, as shown in Figure 12A. In other words, before prediction fusion in DIMD mode, each intra prediction mode is applied to PDPC based on its intra mode.

[0114] In another example of the present disclosure, as shown in FIG. 12B, it is proposed to apply a PDPC operation after prediction fusion in DIMD mode. In other words, a weighted combination of three predictors is applied to PDPC based on a specific mode, e.g., DC mode or planar mode. In one example, the specific mode is planar mode, and then PDPC with the planar mode is applied after prediction fusion in DIMD mode. In another example, the IPM corresponding to the highest histogram bar is selected as the specific mode, and then PDPC with the specific mode is applied after prediction fusion in DIMD mode. In yet another example, the IPM corresponding to the second highest histogram bar is selected as the specific mode, and then PDPC with the specific mode is applied after prediction fusion in DIMD mode.

[0115] In yet another example of the present disclosure, it is proposed to disable PDPC calculation in DIMD mode, in other words, PDPC calculation is not used in DIMD mode, as shown in FIG.

[0116] Harmonization of fusion methods used in DIMD and TIMD modes According to one or more embodiments of the present disclosure, the same fusion scheme is applied to both DIMD and TIMD modes. To achieve this goal, various methods may be used, as shown in the block diagram of Figure 14. The fusion scheme is applied as a weighted average of predictors in DIMD and TIMD modes.

[0117] In one example of the present disclosure, we propose to apply the fusion method used under the DIMD mode to the TIMD mode, as shown in step 1404 of Figure 14. In other words, for the TIMD mode, the first two modes with the smallest SATD cost and the plane mode are selected as predictors for fusion, and a weighted average of the predictors is calculated. The weight of the plane mode is fixed at 2 1 / 64 (approximately 1 / 3). The remaining weight, 4 3 / 64 (approximately 2 / 3), is then shared between the other two modes in proportion to the amplitude of the SATD cost.

[0118] In another example of the present disclosure, it is proposed to apply the fusion method used under the TIMD mode to the DIMD mode, as shown in step 1406 of Figure 14. In other words, for the DIMD mode, the first two modes with the highest HoG bars are selected as predictors for fusion, and the mode weights are calculated from the HoG IPM in proportion to the amplitudes of the HoG bars. If the maximum value in the histogram is 0 (meaning that gradient analysis could not be performed or the area constituting the template is flat), one default mode, such as DC mode or planar mode, is selected as the intra prediction mode for the current block.

[0119] In yet another example of the present disclosure, it is proposed to signal the selection result of the fusion scheme in TIMD and / or DIMD mode, as shown in step 1408 of Figure 14. In one example, for a given CU, a flag is signaled to the decoder to indicate whether the block uses DIMD mode or not. If coded using DIMD mode, a flag is further signaled to the decoder to indicate which fusion scheme is used, for example, as the first fusion method or the second fusion method described above.

[0120] Corrected DIMD mode used in MPM lists Another aspect of the present disclosure proposes deriving an intra mode from a DIMD into a list of intra most probable modes (MPMs) depending on whether the derived intra mode is already used in the DIMD. According to one or more embodiments of the present disclosure, if a fusion scheme is used in the DIMD mode, the intra mode derived from the DIMD may be used as a candidate for the MPM list. In other words, if a fusion scheme is not used in the DIMD mode, the intra mode derived from the DIMD cannot be used as a candidate for the MPM list. Figure 15 shows an example of this process. In step 1502, the decoder may determine whether a fusion scheme is applied in the DIMD mode. In step 1504, the decoder may determine whether to derive an intra mode from the DIMD mode into a list of intra most probable modes (MPMs).

[0121] TIMD mode used in MPM list Another aspect of the present disclosure proposes deriving intra modes from TIMD into a list of intra most probable modes (MPMs). Generally, VVC has 67 intra prediction modes, including non-directional modes (planar, DC) and 65 angular modes, which efficiently model various directional structures typically present in video and image content. In one or more embodiments of the present disclosure, intra modes derived from TIMDs may be used as candidates for the MPM list. In one example, intra modes derived from DIMDs cannot be used as candidates for the MPM list, but intra modes derived from TIMDs may be used as candidates for the MPM list.

[0122] Another aspect of the present disclosure proposes deriving an intra mode from a TIMD to a list of intra most probable modes (MPMs) depending on whether the derived intra mode is already used in the TIMD. According to one or more embodiments of the present disclosure, if a fusion scheme is used in the TIMD mode, the intra mode derived from the TIMD may be used as a candidate for the MPM list. In other words, if a fusion scheme is not used in the TIMD mode, the intra mode derived from the TIMD cannot be used as a candidate for the MPM list. In one or more examples, in response to determining that the fusion scheme is not applied in the TIMD mode, the decoder does not add the intra mode derived from the TIMD mode as a candidate for the MPM list.

[0123] The above methods may be implemented using an apparatus including one or more circuits, including an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic components. The apparatus may use the circuits in combination with other hardware or software components to perform the above-described methods. Each module, sub-module, unit, or sub-unit disclosed above may be implemented at least in part using one or more circuits.

[0124] 13 shows a computing environment 1610 coupled with a user interface 1650. The computing environment 1610 may be part of a data processing server. The computing environment 1610 includes a processor 1620, a memory 1630, and an input / output (I / O) interface 1640.

[0125] The processor 1620 typically controls the overall operation of the computing environment 1610, such as operations related to display, data acquisition, data communication, and image processing. The processor 1620 may include one or more processors for executing instructions called for performing all or some of the steps in the methods described above. Additionally, the processor 1620 may include one or more modules that facilitate interaction between the processor 1620 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single-chip machine, a graphics processing unit (GPU), etc.

[0126] Memory 1630 is configured to store various types of data to support the operation of computing environment 1610. Memory 1630 may include predefined software 1632. Examples of such data include instructions for any applications or methods run on computing environment 1610, video data sets, image data, etc. Memory 1630 may be implemented using any type of volatile or non-volatile memory device, or combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0127] The I / O interface 1640 provides an interface between the processor 1620 and peripheral interface modules such as a keyboard, click wheel, buttons, etc. The buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. The I / O interface 1640 may be coupled to an encoder and a decoder.

[0128] In one embodiment, a non-transitory computer-readable storage medium is also provided that includes, e.g., in memory 1630, a plurality of programs executable by processor 1620 in computing environment 1610 for performing the methods described above. Alternatively, the non-transitory computer-readable storage medium may store a bitstream or datastream including encoded video information (e.g., video information including one or more syntax elements) generated by, e.g., an encoder (e.g., video encoder 20 of FIG. 2) using the encoding method described above, for use by a decoder (e.g., video decoder 30 of FIG. 3) in decoding video data. The non-transitory computer-readable storage medium may be, e.g., a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0129] In one embodiment, a computing device is also provided that includes one or more processors (e.g., processor 1620) and a non-transitory computer-readable storage medium or memory 1630 storing a plurality of programs executable by the one or more processors, wherein the one or more processors are configured, upon execution of the plurality of programs, to perform the methods described above.

[0130] In one embodiment, a computer program product is also provided that includes a plurality of programs, e.g., in memory 1630, executable by processor 1620 in computing environment 1610 to perform the methods described above. For example, the computer program product may include a non-transitory computer-readable storage medium.

[0131] In one embodiment, the computing environment 1610 may be implemented using one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0132] The description of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or limiting of the disclosure. Many modifications, variations and alternative implementations will be apparent to one skilled in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings.

[0133] Unless otherwise specified, the order of steps of the method according to the present disclosure is intended to be illustrative only, and the steps of the method according to the present disclosure are not limited to the order specifically described above, and may be changed according to actual conditions. Furthermore, at least one of the steps of the method according to the present disclosure may be adjusted, combined, or deleted according to actual requirements.

[0134] The examples have been chosen and described to explain the principles of the disclosure, to enable those skilled in the art to understand the disclosure in various implementations, and to make best use of the underlying principles and various implementations with various modifications as suited to the particular use contemplated. Therefore, it should be understood that the scope of the disclosure is not limited to the particular examples of implementations disclosed, and that modifications and other implementations are intended to be included within the scope of the disclosure.

Claims

1. 1. A method for video decoding, comprising: determining, by a decoder, a fusion scheme to be applied in a decoder-side intra-mode derivation (DIMD) mode and a template-based intra-mode derivation (TIMD) mode, wherein the fusion scheme is applied as a weighted average of predictors in the DIMD mode and the TIMD mode; applying the fusion method employed in the DIMD mode to the TIMD mode, or applying the fusion method employed in the TIMD mode to the DIMD mode; Applying the fusion method employed in the DIMD mode to the TIMD mode includes: selecting a planar mode and two modes with the smallest sum of absolute transformed difference (SATD) costs in the TIMD mode as the predictors of the fusion scheme; determining the weighted average of the planar mode and the two modes with the smallest SATD cost, wherein a weight of the planar mode is fixed in the fusion scheme, and a remaining weight is shared between the two modes based on the SATD cost of each of the two modes; Applying the fusion method employed in the TIMD mode to the DIMD mode includes: selecting the two modes with the highest Histogram of Oriented Gradients (HoG) bars for the DIMD modes as the predictors for the fusion scheme; determining the weighted average of the two modes in the fusion scheme based on the amplitudes of the HoG bars of the two modes.

2. 2. The method for video decoding of claim 1, further comprising: in response to determining that the highest HoG maximum value is equal to zero, selecting a default mode as the intra-prediction mode for a current block.

3. The method for video decoding of claim 1 , further comprising receiving a selection result of the fusion scheme in the TIMD mode or the DIMD mode.

4. receiving, by the decoder, a signaled flag indicating whether the DIMD mode applies to a block; receiving, by the decoder, in response to determining that the DIMD mode is applied to the block, another signaling flag indicating the merging scheme to be applied to the block; The method for video decoding of claim 3 further comprising: determining, by the decoder, whether a fusion scheme is applied in the DIMD mode; and determining, by the decoder, whether to add an intra mode derived from the DIMD mode to a list of intra most probable modes (MPMs) based on whether the fusion scheme is applied in the DIMD mode; The method for video decoding of claim 1 further comprising:

6. determining whether an intra mode derived from the DIMD mode should be added to the list of MPMs; adding the intra mode derived from the DIMD mode as a candidate to the list of MPMs in response to determining that the fusion scheme is applied in the DIMD mode; 6. The method for video decoding of claim 5, comprising: in response to determining that the fusion scheme does not apply in the DIMD mode, not adding the intra mode derived from the DIMD mode as the candidate to the list of MPMs. determining, by the decoder, whether an intra mode derived from the TIMD mode should be added to a list of intra most probable modes (MPMs); The method for video decoding of claim 1 further comprising:

8. The method for video decoding described in claim 7, further comprising adding the intra mode derived from the TIMD mode as the candidate in the list of MPM in response to a determination that the intra mode derived from the DIMD mode is not added as a candidate in the list of MPM.

9. 8. The method for video decoding of claim 7, further comprising: determining, by the decoder, whether to add an intra mode derived from the TIMD mode to a list of MPMs based on whether a fusion scheme is applied in the TIMD mode.

10. adding the intra mode derived from the TIMD mode as a candidate to the list of MPMs in response to determining that the fusion scheme is applied in the TIMD mode; 10. The method for video decoding of claim 9, further comprising: in response to determining that the fusion scheme does not apply in the TIMD mode, not adding the intra mode derived from the TIMD mode as the candidate to the list of MPMs.

11. 1. An apparatus for video decoding, comprising: one or more processors; a memory configured to store instructions executable by the one or more processors and a bitstream to be decoded; 11. An apparatus, wherein the one or more processors, upon execution of the instructions, are configured to perform the method of any of claims 1 to 10 to decode the bitstream.

12. 11. A non-transitory computer-readable storage medium for video decoding storing computer-executable instructions and a bitstream to be decoded, the computer-executable instructions, when executed by one or more computer processors, causing the one or more computer processors to perform the method of any of claims 1 to 10 and decode the bitstream.

13. 1. A method for storing a bitstream, comprising: generating a bitstream by performing an encoding method; storing the bitstream on a non-transitory computer-readable storage medium; The encoding method comprises: determining a fusion scheme to be applied in a decoder-side intra-mode derivation (DIMD) mode and a template-based intra-mode derivation (TIMD) mode, wherein the fusion scheme is applied as a weighted average of predictors in the DIMD mode and the TIMD mode; applying the fusion method adopted in the DIMD mode to the TIMD mode, or applying the fusion method adopted in the TIMD mode to the DIMD mode; When the fusion method adopted in the DIMD mode is applied to the TIMD mode, the encoding method is selecting a planar mode and two modes with the smallest sum of absolute transformed difference (SATD) costs in the TIMD mode as the predictors of the fusion scheme; determining the weighted average of the planar mode and the two modes with the smallest SATD cost, wherein a weight of the planar mode is fixed in the fusion scheme, and a remaining weight is shared between the two modes based on the SATD cost of each of the two modes; When the fusion method adopted in the TIMD mode is applied to the DIMD mode, the encoding method is selecting the two modes with the highest Histogram of Oriented Gradients (HoG) bars for the DIMD modes as the predictors for the fusion scheme; determining the weighted average of the two modes in the fusion scheme based on the amplitudes of the HoG bars of the two modes; method.

14. A computer program comprising instructions that, when executed by one or more computer processors, cause said one or more computer processors to perform the method according to any of claims 1 to 10.

Images