Methods, apparatuses, and media for encoding video data

Abstract

An electronic device performs a method of encoding video data. The method includes, in response to determining that a merge mode with motion vector difference (MMVD) is enabled for one or more coding units and MMVD is applied to a respective coding unit of the one or more coding units: determining whether an inter prediction filtering (InterPF) mode is enabled for the respective coding unit; and in response to determining that the InterPF mode is enabled for the respective coding unit: generating a syntax element, wherein the syntax element identifies an InterPF mode of a plurality of InterPF modes for the respective coding unit; and reconstructing the respective coding unit based on the identified InterPF mode.

Description

[0001] This application is a divisional application of Chinese Patent Application No. 202180003819.1, which is the Chinese national phase entry of International Patent Application No. PCT / US2021 / 022606, filed March 16, 2021, which claims priority to U.S. Patent Application No. 62 / 989,900, filed March 16, 2020. TECHNICAL FIELD

[0002] This application relates generally to video coding and compression, and more specifically to methods and apparatuses to improve coding efficiency of merge mode with motion vector difference (MMVD) (also known as advanced motion vector expression (UMVE) mode). BACKGROUND

[0003] Digital video is supported by a variety of electronic devices, such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video gaming consoles, smartphones, video teleconferencing devices, video streaming devices, etc. The electronic devices transmit, receive, encode, decode, and / or store digital video data by implementing video compression / decompression standards. Some well-known video coding standards include Versatile Video Coding (VVC), High Efficiency Video Coding (HEVC, also known as H.265 or MPEG-H Part 2), and Advanced Video Coding (AVC, also known as H.264 or MPEG-4 Part 10), which are jointly developed by ISO / IEC MPEG and ITU-T VECG. AOMedia Video 1 (AV1) is developed by the Alliance for Open Media (AOM) as a successor to its previous standard, VP9. Audio Video Coding Standard (AVS), which refers to a series of digital audio and digital video compression standards, is another video compression standard developed by the Audio and Video Coding Standard Workgroup of China.

[0004] Video compression typically includes performing spatial (intra) and / or temporal (inter) prediction to reduce or remove redundancy inherent in video data. For block-based video coding, a video frame is partitioned into one or more slices, each having multiple video blocks that can also be referred to as coding tree units (CTUs). Each CTU can contain one coding unit (CU) or recursively split into smaller CUs until a predefined minimum CU size is reached. Each CU, also referred to as a leaf CU, contains one or more transform units (TUs) and each CU also contains one or more prediction units (PUs). Each CU can be coded in intra, inter, or IBC modes. Video blocks in intra coded (I) slices of a video frame are encoded using spatial prediction relative to reference samples in neighboring blocks within the same video frame. Video blocks in inter coded (P or B) slices of a video frame can be encoded using spatial prediction relative to reference samples in neighboring blocks within the same video frame or temporal prediction relative to reference samples in other reference video frames.

[0005] Spatial or temporal prediction is performed with respect to previously coded reference blocks (e.g., neighboring blocks) to generate a predictive block for a current video block to be coded. The process of finding the reference blocks can be done by block matching algorithms. Residual data representing pixel differences between the current block to be coded and the predictive block is referred to as residual block or prediction error. Inter coded blocks are coded according to the motion vector pointing to a reference block in a reference frame that forms the predictive block, and the residual block. The process of determining the motion vector is typically referred to as motion estimation. Intra coded blocks are coded according to an intra prediction mode and the residual block. For further compression, the residual block is transformed from the pixel domain to a transform domain, e.g., frequency domain, resulting in residual transform coefficients, which can then be quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, can be scanned into a one-dimensional vector, and then entropy coded into the video bitstream for more compression.

[0006] The coded video bitstream is then saved in a computer-readable storage medium (e.g., flash memory) to be accessed by another electronic device with digital video capability, or directly transmitted to the electronic device via a wired or wireless link. The electronic device then performs video decompression by, e.g., parsing the coded video bitstream to obtain syntax elements from the bitstream and reconstructing the digital video data to its original format from the coded video bitstream based at least in part on the syntax elements obtained from the bitstream, and renders the reconstructed digital video data on a display of the electronic device.

[0007] As digital video quality goes from high definition to 4Kx2K or even 8Kx4K, the amount of video data to be encoded / decoded grows exponentially. There is always a challenge how to efficiently encode / decode the video data while keeping the image quality of the decoded video data. SUMMARY

[0008] The present application describes implementations related to video data encoding and decoding, and more specifically, implementations related to methods and apparatuses to improve coding efficiency of merge mode with motion vector difference (MMVD), also known as advanced motion vector expression (UMVE) mode.

[0009] According to a first aspect of the present application, a method of decoding video data includes receiving a first control flag from a bitstream, wherein the first control flag indicates whether a merge mode with motion vector difference (MMVD) is enabled for one or more coding units in a video sequence; receiving a first syntax element from the video data according to a determination that the MMVD is enabled from the first control flag, wherein the first syntax element identifies one of a plurality of sets of motion vector difference (MVD) offset quantities; receiving a second control flag of the video data from the bitstream, the second control flag corresponding to a respective coding unit of the one or more coding units, the second control flag indicating whether the MMVD is applied to the respective coding unit; receiving a second syntax element of the respective coding unit and a third syntax element of the respective coding unit according to a determination that the MMVD is applied to the respective coding unit from the second control flag, the second syntax element selecting a MVD offset quantity from the identified set of MVD offset quantities, the third syntax element selecting a MVD direction corresponding to the selected MVD offset quantity; forming a MVD based on the selected MVD offset quantity and the selected MVD direction; and reconstructing the respective coding unit by generating a motion vector of the respective coding unit using the formed MVD.

[0010] According to a second aspect of the present application, a method of decoding video data includes receiving a first control flag from a bitstream, wherein the first control flag indicates whether a mode of merge with motion vector difference (MMVD) is enabled for one or more coding units; receiving a second control flag of the video data from the bitstream according to a determination that the MMVD is enabled from the first control flag, the second control flag corresponding to a respective coding unit of the one or more coding units, the second control flag indicating whether the MMVD is applied to the respective coding unit; receiving a third control flag from the video data according to a determination that the MMVD is applied to the respective coding unit from the second control flag, wherein the third control flag indicates whether an inter prediction filtering (InterPF) mode is enabled for the respective coding unit; receiving a fourth syntax element from the video data according to a determination that the InterPF mode is enabled for the respective coding unit from the third control flag, wherein the fourth syntax element identifies one InterPF mode of a plurality of InterPF modes of the respective coding unit; and reconstructing the respective coding unit by applying the MMVD and the identified InterPF mode to video data corresponding to the respective coding unit.

[0011] According to a third aspect of the present application, an electronic device includes one or more processing units, memory, and a plurality of programs stored in the memory. The programs, when executed by the one or more processing units, cause the electronic device to perform the method of decoding video data as described above.

[0012] According to a fourth aspect of the present application, a non-transitory computer-readable storage medium stores a plurality of programs for execution by an electronic device having one or more processing units. The programs, when executed by the one or more processing units, cause the electronic device to perform the method of decoding video data as described above.

[0013] According to a fifth aspect of the present application, a computer program product includes a computer program, wherein the computer program, when executed by a processor, implements the method of decoding video data as described above. BRIEF DESCRIPTION OF DRAWINGS

[0014] The accompanying drawings, which are included to provide a further understanding of the implementations and are incorporated in and constitute a part of this specification, illustrate the described implementations and together with the description serve to explain the basic principles. Like reference numerals refer to corresponding parts throughout.

[0015] Figure 1 is a block diagram illustrating an exemplary video encoding and decoding system, in accordance with some implementations of the present disclosure.

[0016] Figure 2 is a block diagram illustrating an exemplary video encoder, in accordance with some implementations of the present disclosure. is a block diagram illustrating an exemplary video encoder, in accordance with some implementations of the present disclosure.

[0017] Figure 3 FIG. 1 is a block diagram illustrating an exemplary video decoder, in accordance with some embodiments of the present disclosure.

[0018] Figures 4A to 4E FIG. 2 is a block diagram illustrating how a frame is recursively divided into multiple video blocks of different sizes and shapes, in accordance with some embodiments of the present disclosure.

[0019] Figure 5A Figure 5B FIG. 3 is a block diagram illustrating exemplary offset values added to the horizontal or vertical components of a starting motion vector (MV), in accordance with some embodiments of the present disclosure.

[0020] Figure 6 FIG. 4 is a block diagram illustrating exemplary decoder-side motion vector refinement (DMVR), in accordance with some embodiments of the present disclosure.

[0021] Figure 7 FIG. 5 is a block diagram illustrating some exemplary integer search candidates for the DMVR mode, in accordance with some embodiments of the present disclosure.

[0022] Figure 8 FIG. 6 is a flowchart illustrating an exemplary process for determining a motion vector difference (MVD) offset from a plurality of MVD offset sets when a merge mode with MVD (MMVD) is enabled, in accordance with some embodiments of the present disclosure. DETAILED DESCRIPTION

[0023] Reference will now be made in detail to specific implementations, examples of which are illustrated in the accompanying drawings. In the following detailed description of embodiments, numerous specific details are set forth in order to provide a thorough understanding of the subject matter presented herein. However, it will be apparent to one of ordinary skill in the art that the subject matter presented can be practiced without the specific details presented. For example, it will be apparent to one of ordinary skill in the art that the subject matter presented herein can be implemented on many types of electronic devices with digital video capabilities.

[0024] ​The first generation of AVS standards includes the Chinese national standards "Information technology - Advanced audio video coding - Part 2: Video" (referred to as AVSl) and "Information technology - Advanced audio video coding - Part 16: Broadcasting television video" (referred to as AVS+). Compared with the MPEG-2 standard, the first generation of AVS standards can provide about 50% bit rate saving at the same perceptual quality. The second generation of AVS standards includes the Chinese national standards "Information technology - High efficiency video coding" (referred to as AVS2) series, which is mainly targeted for the transmission of additional HD TV programs. The coding efficiency of AVS2 is twice that of AVS+. Meanwhile, the AVS2 standard video part is submitted by the Institute of Electrical and Electronics Engineers (IEEE) as an international application standard. The AVS3 standard is a new generation of video coding standard for UHD video applications, aiming to surpass the coding efficiency of the latest international standard HEVC, and the AVS3 standard provides about 30% bit rate saving over the HEVC standard.

[0025] The focus of the present disclosure is to improve the coding performance of the merge mode with motion vector difference (MMVD) tool used in both VVC and AVS3 standards. In AVS3, this tool is referred to as advanced motion vector expression (UMVE). In some embodiments disclosed herein, although the basic MMVD method is illustrated with the MMVD design in the AVS3 standard as an example, the methods described in the present disclosure can also be applied to other MMVD designs or other coding tools with the same or similar design for those skilled in the art of video coding.

[0026] Figure 1 is a block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel according to some embodiments of the present disclosure. As shown, the system 10 includes a source device 12 that generates and encodes video data to be decoded at a later time by a destination device 14. The source device 12 and the destination device 14 can include any of a variety of electronic devices, including desktop or laptop computers, tablet computers, smart phones, set-top boxes, digital televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, etc. In some embodiments, the source device 12 and the destination device 14 are equipped with wireless communication capabilities. Figure 1

[0027] ​In some implementations, destination device 14 can receive, via link 16, the encoded video data to be decoded. Link 16 can comprise any type of communication medium or device capable of transmitting encoded video data from source device 12 to destination device 14. In one example, link 16 can comprise a communication medium to enable source device 12 to transmit encoded video data directly to destination device 14 in real-time. The encoded video data can be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to destination device 14. The communication medium can comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium can form part of a packet-based network, such as a local area network, a wide-area network, or a global network such as the Internet. The communication medium can include routers, switches, base stations, or any other equipment that can be used to facilitate communication from source device 12 to destination device 14.

[0028] In some other implementations, encoded video data can be transmitted from output interface 22 to storage device 32. Subsequently, encoded video data in storage device 32 can be accessed by destination device 14 via input interface 28. Storage device 32 can include any of a variety of distributed or locally accessed data storage media such as a hard drive, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data. In a further example, storage device 32 can correspond to a file server or another intermediate storage device that can hold the encoded video data generated by source device 12. Destination device 14 can access stored video data from storage device 32 via streaming or download. The file server can be any type of computer

[0029] As Figure 1As shown, source device 12 includes a video source 18, video encoder 20 and an output interface 22. Video source 18 can include a source such as a video capture device, e.g., a video camera, a video archive containing previously captured video, a video feed interface to receive video from a video content provider, and / or a computer graphics system for generating computer graphics data as the source video, or a combination of such sources. As one example, if video source 18 is a video camera of a security surveillance system, source device 12 and destination device 14 can form a still or video camera phone. However, the implementations described in this application can be applicable to video coding in general and can apply to wireless and / or wired applications.

[0030] The captured, pre-captured, or computer-generated video can be encoded by video encoder 20. The encoded video data can be transmitted directly to destination device 14 via output interface 22 of source device 12. The encoded video data can also (or alternatively) be stored onto storage device 32 for later access by destination device 14 or other devices, e.g., for decoding and / or playback. Output interface 22 can further include a modem and / or a transmitter.

[0031] Destination device 14 includes an input interface 28, a video decoder 30, and a display device 34. Input interface 28 can include a receiver and / or modem and receive encoded video data through link 16. The encoded video data transmitted through link 16, or provided on storage device 32, can include a variety of syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements can be included with the encoded video data transmitted on a communication medium, stored on a storage medium, or stored on a file server.

[0032] In some implementations, destination device 14 can include a display device 34, which can be an integrated display device and an external display device configured to communicate with destination device 14. Display device 34 displays the decoded video data to a user and can include any of a variety of display devices such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

[0033] Video encoder 20 and video decoder 30 can operate according to a proprietary or industry standard such as VVC, HEVC, MPEG-4, Part 10, Advanced Video Coding (AVC), or extensions of such standards. It should be understood that the application is not limited to a specific video coding / decoding standard and can be applicable to other video coding / decoding standards. It is generally contemplated that video encoder 20 of source device 12 can be configured to encode video data according to any of these current or future standards. Similarly, it is also generally contemplated that video decoder 30 of destination device 14 can be configured to decode video data according to any of these current or future standards.

[0034] Video encoder 20 and video decoder 30 each can be implemented as any of a variety of suitable encoder circuitry, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware or any combinations thereof. When implemented partially in software, an electronic device can store instructions for the software in a suitable, non- transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the video coding / decoding operations disclosed in the present disclosure. Each of video encoder 20 and video decoder 30 can be included in one or more encoders or decoders, either of which can be integrated as part of a combined encoder / decoder (CODEC) in a respective device.

[0035] Figure 2 FIG. 1 is a block diagram illustrating an example video encoder 20, in accordance with some embodiments described in the present application. Video encoder 20 can perform intra-prediction encoding and inter-prediction encoding of video blocks within a video frame. Intra-prediction encoding relies on spatial prediction to reduce or remove spatial redundancy of video data within a given video frame or image. Inter-prediction encoding relies on temporal prediction to reduce or remove temporal redundancy of video data within adjacent video frames or images of a video sequence.

[0036] As Figure 2As shown, video encoder 20 includes video data memory 40, prediction processing unit 41, decoded picture buffer (DPB) 64, summer 50, transform processing unit 52, quantization unit 54, and entropy encoding unit 56. Prediction processing unit 41 further includes motion estimation unit 42, motion compensation unit 44, partition unit 45, intra-prediction processing unit 46, and intra block copy (BC) unit 48. In some implementations, video encoder 20 also includes inverse quantization unit 58, inverse transform processing unit 60, and summer 62 for video block reconstruction. A deblocking filter (not shown) can be positioned between summer 62 and DPB 64 to filter block boundaries to remove blockiness artifacts from reconstructed video. In addition to the deblocking filter, an in-loop filter (not shown) can be used to filter the output of summer 62. Video encoder 20 can take the form of fixed or programmable hardware units, or can be divided among one or more of the illustrated fixed or programmable hardware units.

[0037] Video data memory 40 can store video data to be encoded by the components of video encoder 20. The video data in video data memory 40 can be obtained, for example, from video source 18. DPB 64 is a buffer that stores reference video data for use in encoding video data by video encoder 20 (e.g., in intra- or inter-coding modes). Video data memory 40 and DPB 64 can be formed by any of a wide variety of memory devices. In various examples, video data memory 40 can be on-chip with other components of video encoder 20, or off-chip relative to those components.

[0038] As Figure 2 As shown, after receiving video data, partition unit 45 within prediction processing unit 41 partitions the video data into video blocks. This partitioning can also include partitioning of video frames into slices, tiles, or other larger coding units (CUs) according to a predefined partitioning structure, such as a quadtree structure associated with the video data. A video frame can be partitioned into a plurality of video blocks (or sets of video blocks referred to as tiles). Prediction processing unit 41 can select one of a plurality of possible predictive encoding modes, such as one of a plurality of intra-prediction encoding modes or one of a plurality of inter-prediction encoding modes, for the current video block based on error results (e.g., coding rate versus distortion level). Prediction processing unit 41 can provide the resulting intra- or inter-predicted block to summer 50 to generate a residual block, and to summer 62 to reconstruct the encoded block for use as part of a reference frame at a later time. Prediction processing unit 41 also provides syntax elements, such as motion vectors, intra-mode indicators, partition information, and other such syntax information, to entropy encoding unit 56.

[0039] To select an appropriate intra-prediction coding mode for a current video block, intra-prediction processing unit 46 within prediction processing unit 41 can perform intra-prediction coding of the current video block relative to one or more neighboring blocks in the same frame as the current block to be coded to provide a spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 perform inter-prediction coding of the current video block relative to one or more prediction blocks in one or more reference frames to provide a temporal prediction. Video encoder 20 can perform multiple passes of encoding, for example, in order to select an appropriate coding mode for each block of video data.

[0040] In some implementations, motion estimation unit 42 determines an inter-prediction mode for a current video frame by generating a motion vector that indicates a displacement of a prediction unit (PU) of a video block within the current video frame relative to a prediction block within a reference video frame according to a predetermined pattern within a sequence of video frames. Motion estimation performed by motion estimation unit 42 is a process of generating motion vectors that estimates the motion of video blocks. A motion vector, for example, can indicate a displacement of a PU of a video block within a current video frame or image relative to a prediction block (or other coding unit) within a reference frame that is a close match to the current block (or other coding unit) being coded within the current frame. The predetermined pattern can designate video frames in the sequence as P-frames or B-frames. Intra-BC unit 48 can determine vectors, e.g., block vectors, for intra-BC coding in a similar manner as motion vectors are determined by motion estimation unit 42 for inter-prediction, or can utilize motion estimation unit 42 to determine block vectors.

[0041] A prediction block is a block of a reference frame that is considered to be a close match to a PU of a video block to be coded in terms of pixel differences, which can be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics. In some implementations, video encoder 20 can calculate values for sub-integer pixel positions of reference frames stored in DPB 64. For example, video encoder 20 can interpolate values for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of reference frames. Accordingly, motion estimation unit 42 can perform a motion search relative to full-pixel positions and fractional-pixel positions and output motion vectors with fractional-pixel precision.

[0042] Motion estimation unit 42 computes motion vectors for PUs of video blocks in an inter-prediction coded frame by comparing the locations of the PUs to the locations of prediction blocks of reference frames selected from a first reference frame list (List 0) or a second reference frame list (List 1), each of which identifies one or more reference frames stored in DPB 64. Motion estimation unit 42 sends the computed motion vectors to motion compensation unit 44 and then to entropy encoding unit 56.

[0043] Motion compensation performed by motion compensation unit 44 can involve fetching or generating a prediction block based on a motion vector determined by motion estimation unit 42. Upon receiving a motion vector for a PU of a current video block, motion compensation unit 44 can locate the prediction block pointed to by the motion vector in one of the reference frame lists, retrieve the prediction block from DPB 64 and forward the prediction block to summer 50. Summer 50 then forms a residual video block having pixel difference values by subtracting the pixel values of the prediction block provided by motion compensation unit 44 from the pixel values of the encoded current video block. The pixel difference values forming the residual video block can include luma difference components or chroma difference components or both. Motion compensation unit 44 can also generate syntax elements associated with the video block of the video frame for use by video decoder 30 when decoding the video block of the video frame. The syntax elements can include, for example, syntax elements defining the motion vector used to identify the prediction block, any flags indicating the prediction mode, or any other syntax information described herein. Note that motion estimation unit 42 and motion compensation unit 44 can be highly integrated, but are illustrated separately for conceptual purposes.

[0044] In some implementations, intra BC unit 48 can generate vectors and fetch prediction blocks in a manner similar to that described above in connection with motion estimation unit 42 and motion compensation unit 44, but where the prediction block is in the same frame as the encoded current block, and where the vector is referred to as a block vector relative to the motion vector. Specifically, intra BC unit 48 can determine an intra prediction mode to use for encoding the current block. In some examples, intra BC unit 48 can encode the current block using various intra prediction modes, e.g., during a separate encoding pass, and test their performance through rate-distortion analysis. Next, intra BC unit 48 can select an appropriate intra prediction mode to use among the various tested intra prediction modes and generate an intra mode indicator accordingly. For example, intra BC unit 48 can use the rate-distortion analysis for the various tested intra prediction modes to compute rate-distortion values and select the intra prediction mode having the best rate-distortion characteristics among the tested modes as the appropriate intra prediction mode to use. Rate-distortion analysis generally determines the amount of distortion (or error) between an encoded block and the original, unencoded block (encoded to produce the encoded block) and the bit rate (i.e., number of bits) used to produce the encoded block. Intra BC unit 48 can compute a ratio of the distortion and rate for each encoded block to determine which intra prediction mode exhibits the best rate-distortion values for the block.

[0045] In other examples, the intra BC unit 48 can use the motion estimation unit 42 and the motion compensation unit 44, in whole or in part, to perform such functionality for intra BC prediction in accordance with the implementations described herein. In either case, for intra block copy, the prediction block can be a block that is considered to closely match the block to be encoded in terms of pixel differences, which can be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics, and the identification of the prediction block can include calculating values for sub-integer pixel positions.

[0046] Whether the prediction block is from the same frame according to intra prediction or from a different frame according to inter prediction, the video encoder 20 can form a residual video block by subtracting the pixel values of the prediction block from the pixel values of the encoded current video block, thereby forming pixel difference values. The pixel difference values forming the residual video block can include luma component differences and chroma component differences.

[0047] As described above, the intra prediction processing unit 46 can intra predict the current video block as an alternative to inter prediction performed by the motion estimation unit 42 and the motion compensation unit 44, or intra block copy prediction performed by the intra BC unit 48. In particular, the intra prediction processing unit 46 can determine an intra prediction mode to use for encoding the current block. To this end, the intra prediction processing unit 46 can encode the current block using various intra prediction modes, e.g., during a separate encoding pass, and the intra prediction processing unit 46 (or in some examples a mode selection unit) can select an appropriate intra prediction mode to use from among the tested intra prediction modes. The intra prediction processing unit 46 can provide information indicative of the selected intra prediction mode for the block to the entropy encoding unit 56. The entropy encoding unit 56 can encode the information indicative of the selected intra prediction mode in the bitstream.

[0048] After the prediction processing unit 41 determines the prediction block for the current video block via inter prediction or intra prediction, the adder 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block can be included in one or more transform units (TUs) and provided to the transform processing unit 52. The transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform such as a discrete cosine transform (DCT) or a conceptually similar transform.

[0049] The transform processing unit 52 can send the resulting transform coefficients to a quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce bit rate. The quantization process can also reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting a quantization parameter. In some examples, the quantization unit 54 can then perform a scan of the matrix including the quantized transform coefficients. Alternatively, the entropy encoding unit 56 can perform the scan.

[0050] Following quantization, the entropy encoding unit 56 entropy encodes the quantized transform coefficients using, for example, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), Probability Interval Partitioning Entropy (PIPE) coding or other entropy encoding methodologies or techniques, into a video bitstream. The encoded bitstream can then be transmitted to the video decoder 30 or archived, e.g., on a storage device 32, for later transmission to or retrieval by the video decoder 30. The entropy encoding unit 56 can also entropy encode motion vectors and other syntax elements of the encoded current video frame.

[0051] The inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain to generate a reference block used to predict other video blocks. As described above, the motion compensation unit 44 can generate a motion compensated predicted block from one or more reference blocks stored in the frames in the DPB 64. The motion compensation unit 44 can also apply one or more interpolation filters to the predicted block to calculate sub-integer pixel values for use in motion estimation.

[0052] The summer 62 adds the reconstructed residual block to the motion compensated predicted block produced by the motion compensation unit 44 to produce a reference block for storage in the DPB 64. The reference block can then be used by the intra BC unit 48, the motion estimation unit 42, and the motion compensation unit 44 as a prediction block to inter predict another video block in a subsequent video frame.

[0053] Figure 3 FIG. 1 is a block diagram illustrating an example video encoder 20 that can implement some embodiments of the present application. The video encoder 20 includes a video data memory 29, a prediction processing unit 22, a transform processing unit 24, a quantization unit 26, and an entropy encoding unit 28. The prediction processing unit 22 further includes an intra prediction processing unit 23 and an inter prediction processing unit 25. The video encoder 20 can perform generally the same functions as the video encoder 10 described above in connection with FIG. 2. Figure 2The decoding process is the inverse of the encoding process described with respect to video encoder 20. For example, motion compensation unit 82 can generate prediction data based on motion vectors received from entropy decoding unit 80, while intra-prediction unit 84 can generate prediction data based on intra-prediction mode indicators received from entropy decoding unit 80.

[0054] In some examples, the units of video decoder 30 can be assigned tasks to perform embodiments of the application. Also, in some examples, embodiments of the disclosure can be divided among one or more units of video decoder 30. For example, intra BC unit 85 can perform embodiments of the application alone or in combination with other units of video decoder 30, such as motion compensation unit 82, intra-prediction processing unit 84, and entropy decoding unit 80. In some examples, video decoder 30 can not include intra BC unit 85, and the functionality of intra BC unit 85 can be performed by other components of prediction processing unit 81, such as motion compensation unit 82.

[0055] Video data memory 79 can store video data, such as encoded video bitstreams, to be decoded by other components of video decoder 30. The video data stored in video data memory 79 can be obtained, for example, from storage device 32 via wired or wireless network transmission, from a local video source, such as a camera, by accessing physical data storage media (e.g., a flash drive or hard disk), or by any combination of these. Video data memory 79 can include a coded picture buffer (CPB) that stores coded video data from an encoded video bitstream. Decoded picture buffer (DPB) 92 of video decoder 30 stores reference video data for use in decoding video data by video decoder 30 (e.g., in intra- or inter-coding modes). Video data memory 79 and DPB 92 can be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magneto resistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. For Figure 3 Video data memory 79 and DPB 92 are depicted as two distinct components of video decoder 30 in FIG. 3 for illustrative purposes. It will be readily apparent to one of ordinary skill in the art, however, that video data memory 79 and DPB 92 can be provided by same memory device or separate memory devices. In some examples, video data memory 79 can be on-chip with other components of video decoder 30, or off-chip relative to those components.

[0056] During the decoding process, video decoder 30 receives an encoded video bitstream that represents encoded video frames and associated syntax elements. Video decoder 30 can receive the syntax elements at the video frame level and / or the video block level. Entropy decoding unit 80 of video decoder 30 entropy decodes the bitstream to generate quantized coefficients, motion vectors, or intra-prediction mode indicators, and other syntax elements. Entropy decoding unit 80 then forwards the motion vectors and other syntax elements to prediction processing unit 81.

[0057] When a video frame has been encoded as an intra-predicted coded (I) frame or an intra-coded prediction block in another type of frame, intra-prediction processing unit 84 of prediction processing unit 81 can generate prediction data for a video block of the current video frame based on the signaled intra-prediction mode and reference data from previously decoded blocks of the current frame.

[0058] When a video frame has been encoded as an inter-predicted coded (i.e., B or P) frame, motion compensation unit 82 of prediction processing unit 81 produces one or more prediction blocks for a video block of the current video frame based on the motion vectors and other syntax elements received from entropy decoding unit 80. Each prediction block can be produced from a reference frame within one of the reference frame lists. Video decoder 30 can construct the reference frame lists: List 0 and List 1, using default construction techniques based on reference frames stored in DPB 92.

[0059] In some examples, when a video block is coded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 produces a prediction block for the current video block based on the block vectors and other syntax elements received from entropy decoding unit 80. The prediction block can be within a reconstructed area of the same picture as the current video block defined by video encoder 20.

[0060] Motion compensation unit 82 and / or intra BC unit 85 determine the prediction information for a video block of the current video frame by parsing the motion vectors and other syntax elements, and then use the prediction information to produce a prediction block for the decoded current video block. For example, motion compensation unit 82 uses some of the received syntax elements to determine the prediction mode used to encode the video block of the video frame (e.g., intra-prediction or inter-prediction), the inter-prediction frame type (e.g., B or P), the construction information for one or more of the reference frame lists for the frame, the motion vectors for each inter-predicted coded video block of the frame, the inter-prediction status for each inter-predicted coded video block of the frame, and other information used to decode the video block in the current video frame.

[0061] Similarly, intra BC unit 85 can use some of the received syntax elements (e.g., flags) to determine that the current video block is predicted using: an intra BC mode, construction information that the video block of the frame is in a reconstructed region and should be stored in DPB 92, a block vector for each intra BC predicted video block of the frame, an intra BC prediction status for each intra BC predicted video block of the frame, and other information used to decode the video block in the current video frame.

[0062] Motion compensation unit 82 can also perform interpolation using an interpolation filter to calculate interpolated values for sub-integer pixels of the reference block as used by video encoder 20 during encoding of the video block. In this case, motion compensation unit 82 can determine the interpolation filter used by video encoder 20 from the received syntax elements and use the interpolation filter to produce the prediction block.

[0063] Inverse quantization unit 86 inverse quantizes quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80 using the same quantization parameter calculated by video encoder 20 for each video block in the video frame to determine a degree of quantization. Inverse transform processing unit 88 applies an inverse transform, e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients in order to reconstruct the residual blocks in the pixel domain.

[0064] After motion compensation unit 82 or intra BC unit 85 generates the prediction block for the current video block based on the vectors and other syntax elements, summer 90 reconstructs the decoded video block for the current video block by summing the residual block from inverse transform processing unit 88 and the corresponding prediction block generated by motion compensation unit 82 and intra BC unit 85. In-loop filter (not shown) can be positioned between summer 90 and DPB 92 to further process the decoded video block. The decoded video blocks in a given frame are then stored in DPB 92, which stores the reference frames used for subsequent motion compensation of the following video blocks. DPB 92 or a memory device separate from DPB 92 can also store the decoded video for later presentation on a display device, such as display device 34 of FIG. 1. Figure 1

[0065] In a typical video coding process, a video sequence typically includes an ordered set of frames or pictures. Each frame can include three arrays of samples, denoted as SL, SCb, and SCr, respectively. SL is a two-dimensional array of luma samples. SCb is a two-dimensional array of Cb chroma samples. SCr is a two-dimensional array of Cr chroma samples. In other instances, the frames can be monochrome and thus include only one two-dimensional array of luma samples.

[0066] As Figure 4A ​As shown, video encoder 20 (or more specifically, partition unit 45) generates an encoded representation of a frame by first partitioning the frame into a set of coding tree units (CTUs). A video frame can include an integer number of CTUs ordered consecutively in a raster scan order from left to right and from top to bottom. Each CTU is the largest logical coding unit and the width and height of the CTU are signaled by video encoder 20 in a sequence parameter set such that all CTUs in a video sequence have the same size, i.e., one of 128x128, 64x64, 32x32, and 16x16. It should be noted, however, that the present application is not necessarily limited to a particular size. As Figure 4B As shown, each CTU can include one coding tree block (CTB) of luma samples, two corresponding coding tree blocks of chroma samples, and syntax elements used to code the samples of the coding tree blocks. The syntax elements describe properties of different types of units of a coded block of pixels and how the video sequence can be reconstructed at video decoder 30, including inter or intra prediction, intra prediction mode, motion vectors, and other parameters. In monochrome images or images having three separate color planes, a CTU can include a single coding tree block and syntax elements used to code the samples of the coding tree block. A coding tree block can be an NxN block of samples.

[0067] To achieve better performance, video encoder 20 can recursively perform tree partitioning, such as binary tree partitioning, ternary tree partitioning, quad tree partitioning, or a combination of the two, on the coding tree blocks of a CTU and partition the CTU into smaller coding units (CUs). As Figure 4C As depicted, a 64x64 CTU 400 is first partitioned into four smaller CUs, each having a block size of 32x32. Of the four smaller CUs, CUs 410 and 420 are each partitioned into four 16x16 CUs by block size. Two 16x16 CUs 430 and 440 are each further partitioned into four 8x8 CUs by block size. Figure 4D A quad tree data structure illustrating the final result of the partitioning process of CTU 400 as depicted in Figure 4C A quad tree data structure illustrating the final result of the partitioning process of CTU 400 as depicted in Figure 4B As depicted, each CU can include a coding block (CB) of luma samples and two corresponding coding blocks of chroma samples of the same size of the frame, and syntax elements used to code the samples of the coding blocks. In monochrome images or images having three separate color planes, a CU can include a single coding block and syntax elements used to code the samples of the coding block. It should be noted that Figure 4C and Figure 4DThe depicted quad-tree partitioning is for illustration purposes only, and one CTU can be partitioned into multiple CUs to adapt to different local characteristics based on quad-tree / triple-tree / binary-tree partitioning. In the multi-type tree structure, one CTU is partitioned by a quad-tree structure, and each quad-tree leaf CU can be further partitioned by a binary tree structure or a triple tree structure. As shown in Figure 4E there are five types of partitioning, i.e., quad partitioning, horizontal binary partitioning, vertical binary partitioning, horizontal triple partitioning, and vertical triple partitioning.

[0068] In some implementations, video encoder 20 can further partition the coded block of a CU into one or more MxN prediction blocks (PBs). A prediction block is a rectangular (square or non-square) block of samples on which the same prediction (inter or intra) is applied. A prediction unit (PU) of a CU can include a prediction block of luma samples, two corresponding prediction blocks of chroma samples, and syntax elements used to perform prediction on the prediction blocks. In monochrome images or images having three separate color planes, a PU can include a single prediction block and syntax structures used to predict the prediction block. Video encoder 20 can generate predicted luma, Cb, and Cr blocks for the luma, Cb, and Cr prediction blocks, respectively, of each PU of a CU.

[0069] Video encoder 20 can use intra prediction or inter prediction to generate the prediction blocks for a PU. If video encoder 20 uses intra prediction to generate the prediction blocks for a PU, video encoder 20 can generate the prediction blocks for the PU based on decoded samples of the frame associated with the PU. If video encoder 20 uses inter prediction to generate the prediction blocks for a PU, video encoder 20 can generate the prediction blocks for the PU based on decoded samples of one or more frames other than the frame associated with the PU.

[0070] After video encoder 20 generates the predicted luma, Cb, and Cr blocks for one or more PUs of a CU, video encoder 20 can generate luma residual blocks for the CU by subtracting the predicted luma blocks for the CU from the original luma coded block for the CU, such that each sample in the luma residual blocks for the CU indicates a difference between a luma sample in one of the predicted luma blocks for the CU and a corresponding sample in the original luma coded block for the CU. Similarly, video encoder 20 can generate Cb and Cr residual blocks for the CU, respectively, such that each sample in the Cb residual blocks for the CU indicates a difference between a Cb sample in one of the predicted Cb blocks for the CU and a corresponding sample in the original Cb coded block for the CU, and each sample in the Cr residual blocks for the CU can indicate a difference between a Cr sample in one of the predicted Cr blocks for the CU and a corresponding sample in the original Cr coded block for the CU.

[0071] Furthermore, as Figure 4CAs illustrated, video encoder 20 can use quad-tree partitioning to divide the luma, Cb, and Cr residual blocks of a CU into one or more luma, Cb, and Cr transform blocks. A transform block is a rectangular (square or non-square) block of samples to which the same transform is applied. A transform unit (TU) of a CU can include a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax elements used to transform the transform block samples. Thus, each TU of a CU can be associated with a luma transform block, a Cb transform block, and a Cr transform block. In some examples, the luma transform block associated with a TU can be a sub-block of the luma residual block of the CU. The Cb transform block can be a sub-block of the Cb residual block of the CU. The Cr transform block can be a sub-block of the Cr residual block of the CU. In monochrome images or images having three separate color planes, a TU can include a single transform block and syntax structures used to transform the samples of the transform block.

[0072] Video encoder 20 can apply one or more transforms to the luma transform block of a TU to generate a luma coefficient block of the TU. A coefficient block can be a two- dimensional array of transform coefficients. A transform coefficient can be a scalar. Video encoder 20 can apply one or more transforms to the Cb transform block of a TU to generate a Cb coefficient block of the TU. Video encoder 20 can apply one or more transforms to the Cr transform block of a TU to generate a Cr coefficient block of the TU.

[0073] After generating a coefficient block (e.g., a luma coefficient block, a Cb coefficient block, or a Cr coefficient block), video encoder 20 can quantize the coefficient block. Quantization generally refers to a process that quantizes transform coefficients to possibly reduce the amount of data used to represent the transform coefficients, providing further compression. After video encoder 20 quantizes a coefficient block, video encoder 20 can entropy encode syntax elements indicating the quantized transform coefficients. For example, video encoder 20 can perform Context- Adaptive Binary Arithmetic Coding (CABAC) on the syntax elements indicating the quantized transform coefficients. Ultimately, video encoder 20 can output a bitstream that includes a sequence of bits that forms a representation of encoded frames and associated data, which is saved in storage device 32 or transmitted to destination device 14.

[0074] After receiving the bitstream generated by video encoder 20, video decoder 30 can parse the bitstream to obtain syntax elements from the bitstream. Video decoder 30 can reconstruct the frames of the video data based at least in part on the syntax elements obtained from the bitstream. The process of reconstructing the video data is generally reciprocal to the encoding process performed by video encoder 20. For example, video decoder 30 can perform inverse transforms on the coefficient blocks associated with the TUs of the current CU to reconstruct the residual blocks associated with the TUs of the current CU. Video decoder 30 also reconstructs the coding blocks of the current CU by adding the samples of the prediction blocks of the PUs of the current CU to corresponding samples of the transform blocks of the TUs of the current CU. After reconstructing the coding blocks of each CU of a frame, video decoder 30 can reconstruct the frame.

[0075] In some embodiments, there are different ways to signal the prediction mode. In VVC, each CU can be coded as either skip mode or non-skip mode. For a CU coded as non-skip mode, further syntax elements are signaled to indicate whether the current CU is coded as intra mode, inter mode, intra block copy (IBC) mode, or palette (PLT) mode. These modes are referred to as “MODE INTRA”, “MODE INTER”, “MODE IBC”, and “MODE PLT” in the VVC specification, respectively. For a CU coded as inter mode, the prediction signal can be generated from pixels from reference pictures that are not the current picture. In addition, one flag is further signaled for a CU coded as inter mode to indicate whether the current CU is in merge mode. For a CU coded in merge mode, several different types of merge modes are further signaled using additional syntax elements. These different types of merge modes include regular merge mode, subblock merge mode, merge mode with MV difference (MMVD), combined inter and intra prediction (CIIP), and triangle merge mode. The following sections explain these merge modes.

[0076] In some embodiments, in VVC, the merge candidate list is constructed by including the following five types of candidates in order: spatial MVPs (i.e., motion vector predictors) from spatial neighboring CUs, temporal MVPs from co-located CUs, history-based MVPs from a FIFO table, pair-wise average MVPs, and zero MV.

[0077] The size of the merge list is signaled in the slice header, and the maximum allowed size of the merge list is 6 in VVC. For each CU coded in merge mode, the index of the best merge candidate is coded using truncated unary (TU). The first bin of the merge index is coded with a context, and the other bins are coded using bypass coding. In the following context of this disclosure, this extended merge mode is also referred to as regular merge mode, because its concept is the same as the merge mode used in HEVC.

[0078] In some embodiments, in addition to the merge mode that derives the motion information of a current block from its spatial / temporal neighbors, the MMVD / UMVE mode is introduced in VVC and AVS standards as a special merge mode. Specifically, in VVC and AVS3, this mode is signaled at the block level by a MMVD flag. In MMVD mode, the first two candidates in the merge list of regular merge mode are selected as the two base merge candidates of MMVD. After a base merge candidate is selected and signaled, additional syntax elements are signaled to indicate the motion vector difference (MVD) added to the motion of the selected merge candidate. The MMVD syntax elements include a merge candidate flag for selecting a base merge candidate, a distance index for specifying the MVD magnitude, and a direction index for indicating the MVD direction.

[0079] In the MMVD design, the distance index specifies the MVD magnitude, which is defined based on a set of predefined starting point offsets. Figure 5A and Figure 5B is a block diagram illustrating exemplary offsets added to the horizontal or vertical component of the starting motion vector (MV), i.e., the MV of the selected base merge candidate, according to some embodiments of the present disclosure. Reference picture list L0 and reference picture list L1 are illustrated.

[0080] Table 1 illustrates the relationship of the applied distance index and the predefined MVD offsets.

[0081]

[0082] Table 1: Relationship of distance index and predefined MVD offsets

[0083] The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate the four directions as shown in Table 2. It should be noted that the meaning of the MVD sign can change depending on the information of the starting MV. When the starting MV is a uni-prediction MV or a bi-prediction MV and both lists of the bi-prediction MV point to the same side of the current picture (i.e., both the picture order count (POC) of the two references are greater than the POC of the current picture or both are less than the POC of the current picture), the sign in Table 2 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bi-prediction MV and the two MVs point to different sides of the current picture (i.e., one reference has a picture order count (POC) greater than the POC of the current picture and the other reference has a POC less than the POC of the current picture), the sign in Table 2 specifies the sign of the MVD added to the list 0 (L0 reference) MV component of the starting MV, and the sign of the list 1 (LI reference) MVD has the opposite value.

[0084] Direction IDX 00 01 10 11 x-axis + - N / A N / A y-axis N / A N / A + -

[0085] Table 2: Sign of MVD specified by direction index

[0086] In some embodiments, to increase the accuracy of the MVs of the regular merge mode, a decoder-side motion vector refinement (DMVR) based on bilateral matching is applied. In the bi-prediction operation, a refined MV is searched around the initial MV in the reference picture list L0 and the reference picture list LI. The method calculates the distortion between two candidate blocks in the reference picture list L0 and the list LI. Figure 6 FIG. 1 is a block diagram illustrating an example decoder-side motion vector refinement (DMVR) according to some embodiments of the present disclosure. As shown in FIG. 1, the DMVR is applied to the initial MV 100. The DMVR searches for a refined MV 102 around the initial MV 100. The refined MV 102 is used to generate the bi-predicted signal 104. Figure 6 As illustrated in FIG. 2, the SAD between the block 202 and the block 204 is calculated based on each MV candidate around the initial MV. The MV candidate with the lowest SAD becomes the refined MV, and is used to generate the bi-predicted signal.

[0087] In DVMR, the search point is the surrounding integer sample pointed by the initial MV and the MV offset that is considered to comply with the mirror rule. In other words, any MV refinement examined by the DMVR should satisfy the following two equations:

[0088] MV0' = MV0 + MV_offset (1)

[0089] MV1' = MV1 - MV_offset (2)

[0090] where MV_offset represents the refinement offset between the initial MV and the refined MV in one of the reference pictures. The refinement search range is two integer luma samples from the initial MV. The search includes an integer sample search phase and a fractional sample refinement phase.

[0091] In some embodiments, in the integer sample search stage, SAD is checked for 21 integer sample positions as indicated Figure 7 including the integer sample position corresponding to the initial MV. Figure 7 is a block diagram illustrating some exemplary integer search candidates for the DMVR mode according to some embodiments of the present disclosure. In Figure 7 , the black center triangle is the integer sample position associated with the initial MV, and the white triangles are the neighboring integer sample positions. The SAD of the initial MV pair is first calculated. The integer offset that minimizes the SAD value is selected as the integer sample offset in the integer sample search stage.

[0092] The integer sample search is followed by the fractional sample refinement. To reduce the computational complexity, the fractional sample refinement is obtained by using a parametric error surface method instead of additional search using SAD comparison. In the parametric error surface based sub-pixel offset estimation, the center position cost and the costs of the four neighboring positions of the center are used to fit a 2-D parabolic error surface equation as follows:

[0093] E(x, y) = A(x - x min ) 2 + B(y - y min ) 2 + C (3)

[0094] where (x min， , y min ) corresponds to the fractional position with the lowest cost, and C corresponds to the minimum cost value. By solving the above equation using the cost values of the five search points, (x min , y min ) is calculated as:

[0095] x min = (R(-1, 0) - E(1, 0)) / (2(R(-1, 0) + E(1, 0) - 2E(0, 0))) (4)

[0096] y min = (E(0, -1) - E(0, 1)) / (2((E(0, -1) + E(0, 1) - 2E(0, 0))) (5)

[0097] Since all cost values are positive and the minimum value is R(0, 0), the values of x min and y min are automatically limited between -8 and 8. The calculated fractional (x min , y min ) is added to the integer distance refined MV to obtain the sub-pixel precision refinement AMV.

[0098] In some embodiments, to further improve the prediction efficiency, an inter prediction filtering (InterPF) is applied which combines the intra prediction of a CU and the inter prediction based on the merge mode of the CU. Specifically, for each merge CU, an additional flag is signaled to indicate whether the InterPF mode is enabled for the current CU. When the flag is false, the InterPF is disabled and the regular inter prediction is applied to generate the prediction samples of the current CU. When the flag is true, the InterPF is applied to the current CU and another flag is signaled to further indicate which of the two InterPF modes is used. The two InterPF modes are described as follows:

[0099] InterPF mode one: the combined prediction samples are derived from the weighted average of the inter prediction samples and the five neighboring reconstructed samples from the left, right, above, and below, as shown in the following equation set (6):

[0100] Pred(x, y) = (Pred_inter(x, y) * 5 + Pred_Q(x, y) * 3) » 3

[0101] Pred_Q(x, y) = (Pred_V(x, y) + Pred_H(x, y) + 1) » 2

[0102] Pred_V(x, y) = ((h - 1 - y) * Rec(x, -1) + (y + 1) * Rec(-1, h) + (h » 1)) » log2(h)

[0103] Pred_H(x, y) = ((w - 1 - x) * Rec(-1, y) + (x + 1) * Rec(w, -1) + (w » 1)) » log2(w)

[0104] InterPF mode two: the combined prediction samples are derived from the weighted average of the inter prediction samples and the three neighboring reconstructed samples from the left and above:

[0105] P'(x, y) = f(x) P(-1, y) + f(y) P(x, -1) + (1 - f(x) - f(y)) P(x, y) (7)

[0106] In (7), the weight parameters f(x) and f(y) are predefined as a lookup table (LUT) which is determined based on the size of the current CU and the intra prediction direction. Additionally, according to the existing AVS3 design, the InterPF mode is always disabled when the current CU is coded by the MMVD mode.

[0107] Although the MMVD mode can efficiently improve the efficiency of inter prediction, several aspects of its design can still be further improved. Specifically, the following issues in the current MMVD design in the AVS3 standard are found.

[0108] As previously mentioned, in the MMVD design, the allowed set of MVD offsets that can be selected for one MMVD CU is fixed, which includes 1 / 4-pel (pixel / picture element), 1 / 2-pel, 1-pel, 2-pel, and 4-pel as illustrated in Table 1. This design can not always be optimal for video contents with different characteristics. For example, video sequences with higher resolutions tend to have larger MVs for inter CUs compared to video sequences with lower resolutions like VGA and WQVGA. Accordingly, for video sequences with relatively large motion, in the MMVD mode, large MVD offsets are more often selected than small MVD offsets. Therefore, to maximize the coding gain of the MMVD mode, it is more beneficial to adaptively adjust the allowed MVD offset values according to the specific characteristics of the coded video.

[0109] In AVS3, the DMVR mode is only applied to CUs coded by the regular merge mode. For MMVD CUs, DMVR is always disabled. However, given that the MMVD mode is just an extension of the regular merge mode with signaled MVD, it can also be beneficial to apply DMVR to inter CUs coded in the MMVD mode. This combination can further improve the inter prediction efficiency of the MMVD mode without increasing the actual hardware codec design complexity.

[0110] As discussed above, the InterPF mode is always disabled when coding one current CU by the MMVD mode. However, similar to the regular merge mode, the derived MV in the MMVD mode can not be as accurate as the MV derived in the non-merge mode (in which the MV of one inter CU is directly signaled in the bitstream). In this case, the combination of the inter prediction samples of one MMVD CU and the intra prediction samples from its neighboring reconstructed samples can potentially improve the prediction accuracy and improve the coding efficiency. Based on this analysis, it would be beneficial to allow the combination of the InterPF mode and the MMVD mode.

[0111] In some embodiments, methods and systems are implemented to further improve the coding efficiency of the MMVD mode as disclosed herein. Specifically, the main aspects of the implemented methods are summarized as follows.

[0112] The signaling method is used to support adaptation of the allowed MVD offsets for MMVD mode at various coding levels. In addition, an encoder-side method is used to adaptively determine the optimal MVD offset to apply for MMVD mode.

[0113] A method is used in the DMVR process to further refine the MVs of MMVD mode.

[0114] A method is implemented to enable MMVD mode to be combined with InterPF mode.

[0115] In some embodiments, a method of adaptation of MVD offset values for MMVD mode is implemented.

[0116] In some MMVD designs, the allowed MVD offset values are fixed to 1 / 4-pel, 1 / 2-pel, 1-pel, 2-pel, and 4-pel. This design is suboptimal for video content with high resolution or severe motion. In these cases, the MVs tend to be much larger, making the above MVD offset values defined in MMVD mode possibly not optimal for capturing the true motion of those blocks. To further improve the coding performance of MMVD mode, adaptation of the MVD offsets allowed to be selected by MMVD mode is supported.

[0117] According to some embodiments, a number of MVD offset value sets can be predefined for MMVD mode, and the selected MVD offset set is allowed to vary over time based on the varying motion characteristics in the video content. As one example of the present disclosure, Table 3 shows two predefined MVD offset sets for entropy coding and their corresponding binarized codewords.

[0118]

[0119] Table 3: MVD offset sets for MMVD mode.

[0120] As can be seen from Table 3, the second set of MVD offset is a superset of the first set, and introduces three additional MVD offsets. The additional MVD offsets are 8-pel, 16-pel, and 32-pel. The newly added MVD offsets make the second set more suitable for coding video blocks with larger motion. In addition, in Table 3, different binarization codewords are assigned for each set to accommodate different probability distributions of MVD offset value usage. For example, the binarization used for the first set assigns shorter codewords for small MVD offsets, which makes the first set more efficient for coding pictures / regions of MMVD CUs with small MVD offsets that are frequently selected. Likewise, the second set of MVD offsets is more efficient for coding pictures / regions of MMVD CUs with medium MVD offset values that are frequently selected.

[0121] In the above example, only two sets of MVD offsets are allowed to be selected for MMVD mode. However, in practice, multiple sets of MVD offsets (i.e., more than two sets) and corresponding codewords can be derived offline according to the specific motion characteristics of different video sequences by the methods disclosed herein. The encoder can select the best set of MVD offsets and signal the corresponding index of the selected set to the decoder.

[0122] According to some embodiments, the adaptive selection of the predefined sets of MVD offsets is enabled by signaling one set identifier (or id) to indicate which set is selected. The id can be signaled at different coding levels (e.g., sequence level, picture level, and / or slice level) and at the same time at the signaling level (e.g., at the coding block group level and / or coding block level). The MMVD mode can only select MVD offsets in the identified set. Assuming the adaptation of allowed MVD offsets is performed at the picture level, Table 4 shows an example syntax table where the syntax element is signaled at the picture header.

[0123]

[0124] Table 4: Modified syntax table with MMVD offset control flag

[0125] mmvd_dist_set_id specifies the set of MVD offsets allowed for the selection of the MVD for the coding blocks coded in MMVD mode in the current picture. For the example shown in Table 3, where two sets are predefined, mmvd_dist_set_id becomes a flag. When the flag takes one value, e.g., 0, it indicates that the MVD offsets allowed for MMVD mode include ¼-pel, ½-pel, 1-pel, 2-pel, and 4-pel. When the flag takes the opposite value, e.g., 1, it indicates that the MVD offsets allowed for MMVD mode include ¼-pel, ½-pel, 1-pel, 2-pel, 4-pel, 8-pel, 16-pel, and 32-pel.

[0126] In Table 4, the variable MMVD Enable Flag is a control flag to indicate whether the MMVD tool is enabled for the coding blocks in the video sequence. The variable can be controlled by a control flag at the sequence level, e.g., in the sequence parameter set.

[0127] In another embodiment, instead of using fixed MVD offsets, the MVD offset values are directly signaled in the bitstream. This gives the encoder more freedom in determining the required MVD offset values for the current picture / slice on the fly.

[0128] In yet another embodiment, the selection of the MVD offset value set is determined on the fly based on certain statistics without explicitly signaling the mmvd_dist_set_id. Such certain statistics can include, but are not limited to, the probability distribution of the computed MVD offset value usage based on previously coded pictures, slices and / or coded blocks. The selection of the MVD offset value set can be re-determined and / or updated at various frequency levels. For example, the selection can be re-determined and / or updated each time a CU is coded in MMVD mode. In another example, the selection can be re-determined and / or updated each time multiple CUs (e.g., 8 or 16) are coded in MMVD mode. In another example, the selection of the allowed MVD offset set can be updated once after multiple CUs (e.g., 8 or 16) are coded in inter mode. In this case, the average / maximum / minimum MV of the inter blocks can be computed and used to determine which MVD offset set should be selected for the following coded blocks coded in MMVD mode. According to this approach, the resetting operation of the set selection can be performed at certain points, e.g., at the beginning of the current frame coding, at the beginning of the current GOP (i.e., group of pictures) coding, or when a frame that supports random access functionality, e.g., an IDR frame (IDR frame specifies that any frame after the IDR frame cannot refer to any frame before it) is coded, etc., where the default MVD offset value set is always selected.

[0129] In some embodiments, encoder logic is implemented for selecting the best MVD offset for MMVD mode. In the MMVD approach discussed above, multiple MVD offset sets are allowed to be selected by MMVD mode. In some examples, a method is provided for adaptively selecting the best MVD offset set at the encoder side. Specifically, considering the strong content characteristic correlation between pictures in the same video sequence, the MVD offset selected by one inter picture can be very similar to that of its previously coded neighboring picture. Based on this consideration, the average MVD offset of the MMVD CUs of the previously coded picture is used to determine the MVD offset set for MMVD mode in the current picture. Assuming that the two MVD offset sets exemplified in Table 3 are applied to MMVD mode, and there are M (M is a positive integer) CUs in the previous picture coded by MMVD mode and the MVD offset values are V0, V1,..., VM-1, the average MVD offset of the MMVD mode in the previous picture is calculated as follows: M-1

[0130]

[0131] Based on (8), the index of the MVD offset set used by the current picture is determined as: ​

[0132]

[0133] In some embodiments, DMVR mode is enabled for MMVD mode. In some MMVD designs, DMVR tends not to be applied together. However, because MMVD mode is an extended merge mode with additionally signaled MVD offset, allowing DMVR on top of MMVD can further improve the accuracy of MVs derived by MMVD mode. Additionally, the combination of two coding tools does not cause additional codec implementation complexity of hardware / software. Based on this consideration, the following two methods / embodiments are implemented to enable DMVR process for MMVD mode.

[0134] In the first embodiment, DMVR is applied to refine all MVs derived by MMVD mode as long as the initial MVs derived by MMVD mode are bi-directional and satisfy the common conditions to trigger DMVR process.

[0135] In the second embodiment, DMVR is applied to MMVD CUs only when the signaled MMVD offset is greater than a threshold. When the corresponding MMVD offset is equal to or smaller than the threshold, DMVR is always bypassed. In practice, to maximize the coding performance, different thresholds can be applied according to the specific characteristics of the coded video. In one example, a fixed threshold (e.g., 1-pel) is used. In another embodiment, the encoder adaptively determines the optimal threshold and signals it to the decoder at various coding levels (e.g., sequence level, picture level, slice level, and / or coding block level, etc.).

[0136] In practice, the second embodiment discussed above can provide better coding performance than the first embodiment. Based on the existing DMVR design, its search range is + / - 2-pel from the initial MV. However, as shown before, the allowed MVD offset used by MMVD mode can also cover this search range, and the selection of MVD offset in MMVD is based on more accurate and reliable rate-distortion optimization (RDO) at the encoder side rather than the decoder side bi-directional matching search in DMVR. Therefore, within a small local area around the initial MV, the MVD offset signaled by MMVD should be more accurate than that of DMVR. Disabling DMVR in this case can reduce the signaling overhead and improve the coding efficiency.

[0137] In some embodiments, InterPF is enabled for MMVD mode. As mentioned above, InterPF mode does not allow its inter prediction samples to come from MMVD mode. However, similar to regular merge mode, the MV derived from MMVD mode can not be accurate. For example, when this happens, there can be an undesirable discontinuity between the inter prediction samples of the current block and its neighboring reconstructed samples, which can lead to visible block artifacts along the neighboring block boundaries. In this case, the combination of the inter prediction samples of one MMVD CU with the intra prediction samples from its neighboring reconstructed samples can potentially reduce such artifacts and improve the prediction efficiency. Meanwhile, there is no increase in the complexity of hardware / software codec implementation when the combination of two coding tools is enabled. Based on this consideration, in one embodiment, InterPF mode is enabled for MMVD CUs, i.e., the combination of the inter prediction samples of one MMVD CU with the intra prediction samples from its neighboring reconstructed samples is enabled. Specifically, taking the InterPF design as an example (as shown in equations (6) and (7)), when this combination is enabled, a first control flag indicating whether MMVD mode is applied to one coding unit will be received at the decoder. In the case that MMVD is enabled for the coding unit, a second control flag will be received at the decoder to further specify whether InterPF mode is further applied to the MMVD coding unit. If the second control flag is 1 / enabled, a third control flag is signaled to select one of the two InterPF filters for the CU (as shown in equations (6) and (7)).

[0138] Figure 8 is a flowchart illustrating an exemplary process 800 of determining a motion vector difference (MVD) offset from a plurality of MVD offset sets when merge mode with MVD (MMVD) is enabled, according to some embodiments of the present disclosure.

[0139] The video decoder 30 receives a first control flag from the bitstream, where the first control flag indicates whether merge mode with motion vector difference (MMVD) is enabled for one or more coding units in a video sequence (810). In some embodiments, the first control flag is equal to 1 when MMVD is enabled for one or more coding units in the video sequence.

[0140] In accordance with a determination from the first control flag that MMVD is enabled, the video decoder 30 receives a first syntax element from the video data, where the first syntax element identifies one MVD offset set of a plurality of motion vector difference (MVD) offset sets (820).

[0141] The video decoder 30 receives, from the bitstream, a second control flag of the video data, the second control flag corresponding to a respective coding unit of the one or more coding units, the second control flag indicating whether MMVD is applied to the respective coding unit (830). In some embodiments, the second control flag is equal to 1 when MMVD is applied to the respective coding unit.

[0142] In accordance with a determination that MMVD is applied to the respective coding unit from the second control flag, the video decoder 30 receives a second syntax element of the respective coding unit and a third syntax element of the respective coding unit, the second syntax element selecting a MVD offset from the identified set of MVD offsets, the third syntax element selecting a MVD direction corresponding to the selected MVD offset (840).

[0143] The video decoder 30 forms a MVD based on the selected MVD offset and the selected MVD direction (850).

[0144] The video decoder 30 reconstructs the respective coding unit by generating a motion vector of the respective coding unit using the formed MVD (860).

[0145] In some embodiments, the multiple sets of MVD offsets include a first set of MVD offsets and a second set of MVD offsets.

[0146] In some embodiments, the first set of MVD offsets is a subset of the second set of MVD offsets.

[0147] In some embodiments, a first set of binarized codewords is assigned to the first set of MVD offsets, and a second set of binarized codewords is assigned to the second set of MVD offsets. In some embodiments, the first set of binarized codewords is different from the second set of binarized codewords for a same offset value to accommodate different probability distributions of MVD offset value usage.

[0148] In some embodiments, the first set of MVD offsets includes offset values 1 / 4, 1 / 2, 1, 2, 4 in samples, and the second set of MVD offsets includes offset values 1 / 4, 1 / 2, 1, 2, 4, 8, 16, and 32 in samples.

[0149] In some embodiments, the first MVD offset set includes offset values 1 / 4, 1 / 2, 1, 2, 4 in samples, and binarized codewords 1, 01, 001, 0001, and 0000 corresponding to the offset values 1 / 4, 1 / 2, 1, 2, 4, respectively, and the second MVD offset set includes offset values 1 / 4, 1 / 2, 1, 2, 4, 8, 16, and 32 in samples, and binarized codewords 000, 001, 011, 010, 10, 110, 1110, and 1111 corresponding to the offset values 1 / 4, 1 / 2, 1, 2, 4, 8, 16, and 32, respectively.

[0150] In some embodiments, the first syntax element is signaled in one or more of different coding levels, including a sequence level, a picture level, and a slice level.

[0151] In another aspect, the process of combining InterPF mode and MMVD mode is disclosed as follows.

[0152] The video decoder 30 receives a first control flag from the bitstream, where the first control flag indicates whether a merge mode with motion vector difference (MMVD) is enabled for one or more coding units. In some embodiments, the first control flag is equal to 1 when the MMVD is enabled for the one or more coding units in the video sequence.

[0153] According to a determination from the first control flag that the MMVD is enabled, the video decoder 30 receives a second control flag from the bitstream for the video data, the second control flag corresponding to a respective coding unit of the one or more coding units, the second control flag indicating whether the MMVD is applied to the respective coding unit. In some embodiments, the second control flag is equal to 1 when the MMVD is applied to the respective coding unit.

[0154] According to a determination from the second control flag that the MMVD is applied to the respective coding unit, the video decoder 30 receives a third control flag from the video data, where the third control flag indicates whether an inter prediction filtering (InterPF) mode is enabled for the respective coding unit. In some embodiments, the third control flag is equal to 1 when the InterPF mode is enabled for the respective coding unit.

[0155] According to a determination from the third control flag that the InterPF mode is enabled for the respective coding unit, the video decoder 30 receives a fourth syntax element from the video data, where the fourth syntax element identifies one InterPF mode of a plurality of InterPF modes for the respective coding unit.

[0156] The video decoder 30 reconstructs the respective coding unit by applying the MMVD and the identified InterPF mode to video data corresponding to the respective coding unit.

[0157] In some embodiments, the plurality of InterPF modes includes at least two InterPF modes.

[0158] In one or more examples, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media can include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer- readable media generally can correspond to (1) tangible computer-readable storage media which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and / or data structures for implementation of the techniques described in this application. A computer program product can include a computer-readable medium.

[0159] The terminology used in the description of the implementations herein is for the purpose of describing particular implementations only and is not intended to limit the scope of the claims. As used in the description of the implementations and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “comprises” and / or “comprising,” when used in this specification, specify the presence of stated features, elements, and / or components, but do not preclude the presence or addition of one or more other features, elements, components, and / or groups thereof.

[0160] It will also be understood that, although the terms first, second, etc. can be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first electrode could be termed a second electrode, and, similarly, a second electrode could be termed a first electrode, without departing from the scope of the present implementations. The first electrode and the second electrode are both electrodes, but they are not the same electrode.

[0161] Throughout this specification, the use of the term "example" or "exemplary" means "serving as an example, instance, or illustration." Any implementation described herein as "example" or "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations. Likewise, the use of the terms "first", "second" or "third," etc. does not imply a chronological or logical order. For example, a process, structure, or device discussed as second can very well be performed, created, or completed prior to a process, structure, or device discussed as first. Likewise, a process, structure, or device discussed as third can very well be performed, created, or completed chronologically or logically after a process, structure, or device discussed as second. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner on an embodiment or embodiments.

[0162] The description of the application has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the application in the form disclosed. Many modifications, variations, and alternative embodiments will be apparent to those of ordinary skill in the art based on the teachings and concepts presented throughout this specification and the attached drawings. Embodiments were chosen and described in order to best explain the principles of the application and to enable others skilled in the art to understand the various embodiments of the application and the various modifications that are suited to the particular use contemplated. Accordingly, others skilled in the art will readily appreciate that the scope of the claims is not limited to the examples disclosed and that modifications and other embodiments are intended to be included within the scope of the claims.

Claims

1. A method of encoding video data, the method comprising: in response to determining that a merge mode with motion vector difference (MMVD) is enabled for one or more coding units and the MMVD is applied to a respective coding unit of the one or more coding units: determining whether an inter prediction filtering (InterPF) mode is enabled for the respective coding unit; and in response to determining that an inter prediction filtering (InterPF) mode is enabled for the respective coding unit: generating a syntax element, wherein the syntax element identifies an InterPF mode of a plurality of InterPF modes for the respective coding unit; and reconstructing the respective coding unit based on the identified InterPF mode, wherein the reconstructing the respective coding unit comprises: in accordance with a determination that the InterPF mode identified by the syntax element is a first InterPF mode, for a respective sample in the respective coding unit, obtaining a reconstructed sample for the respective sample from a weighted average of an inter prediction sample for the respective sample and neighboring reconstructed samples from left and above of the respective sample.

2. The method of claim 1, wherein, the plurality of InterPF modes comprises at least two InterPF modes.

3. The method of claim 1 or 2, wherein, the reconstructing the respective coding unit comprises: in accordance with a determination that the InterPF mode identified by the syntax element is a second InterPF mode, for a respective sample in the respective coding unit, obtaining a reconstructed sample for the respective sample from a weighted average of an inter prediction sample for the respective sample and neighboring reconstructed samples from left, right, above, and below of the respective sample.

4. An electronic device, comprising: one or more processing units; memory coupled to the one or more processing units; and a plurality of programs stored in the memory that, when executed by the one or more processing units, cause the electronic device to perform the method of any of claims 1-3.

5. A non-transitory computer-readable storage medium storing a plurality of programs for execution by an electronic apparatus having one or more processing units, wherein, the plurality of programs, when executed by the one or more processing units, cause the electronic device to perform the method of any of claims 1-3 to generate a video bitstream and transmit the video bitstream.

6. A computer program product comprising a computer program, wherein, the computer program, when executed on a processor, performs the method of any of claims 1-3.

7. A method for storing a bitstream, comprising: performing the encoding method of any of claims 1-3 to generate a bitstream; and storing the bitstream. ​

Images