Methods, apparatuses, and media for encoding video data

Abstract

The application relates to a method, device and medium for encoding video data. A method for encoding video data is disclosed, comprising: in response to determining that a merge mode with motion vector difference (MMVD) is enabled for one or more coding units in a video sequence, generating a first syntax element; and in response to determining that MMVD is applied to a respective coding unit of the one or more coding units, generating a second syntax element of the respective coding unit and a third syntax element of the respective coding unit, the second syntax element selecting a MVD offset from a set of identified MVD offsets, and the third syntax element selecting a MVD direction corresponding to the selected MVD offset; and reconstructing the respective coding unit by generating a motion vector of the respective coding unit by applying the selected MVD offset and the selected MVD direction.

Description

[0001] This application is a divisional application of Chinese Patent Application No. 202180003819.1, which is the Chinese national phase entry of International Patent Application No. PCT / US2021 / 022606, filed March 16, 2021, which claims priority to U.S. Patent Application No. 62 / 989,900, filed March 16, 2020. TECHNICAL FIELD

[0002] This application relates generally to video coding and compression, and more specifically to methods and apparatuses to improve coding efficiency of merge mode with motion vector difference (MMVD) (also known as advanced motion vector expression (UMVE) mode). BACKGROUND

[0003] A variety of electronic devices perform video compression / decompression according to one or more video coding standards. These devices include digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video gaming consoles, smart phones, video teleconferencing devices, video streaming devices, etc. The video devices implement video compression / decompression standards to code / decode digital video data to achieve good visual quality while minimizing the use of bandwidth and storage space. Some well-known video coding standards include Versatile Video Coding (VVC), High Efficiency Video Coding (HEVC, also known as H.265 or MPEG-H Part 2), and Advanced Video Coding (AVC, also known as H.264 or MPEG-4 Part 10), which are jointly developed by ISO / IEC MPEG and ITU-T VECG. AOMedia Video 1 (AV1) is developed by the Alliance for Open Media (AOM) as a successor to its previous standard, VP9. Audio Video Coding Standard (AVS), which refers to a series of digital audio and digital video compression standards, is another video compression standard developed by the Audio and Video Coding Standard Workgroup of China.

[0004] Video compression typically includes performing spatial (intra) and / or temporal (inter) prediction to reduce or remove redundancy inherent in video data. For block-based video coding, a video frame is partitioned into one or more slices, each having multiple video blocks that can also be referred to as coding tree units (CTUs). Each CTU can contain one coding unit (CU) or recursively split into smaller CUs until a predefined minimum CU size is reached. Each CU, also referred to as a leaf CU, contains one or more transform units (TUs) and each CU also contains one or more prediction units (PUs). Each CU can be coded in intra, inter, or IBC modes. Video blocks in intra coded (I) slices of a video frame are encoded using spatial prediction relative to reference samples in neighboring blocks within the same video frame. Video blocks in inter coded (P or B) slices of a video frame can be encoded using either spatial prediction relative to reference samples in neighboring blocks within the same video frame or temporal prediction relative to reference samples in other previously coded and / or future coded video frames.

[0005] Spatial or temporal prediction is performed with respect to previously coded reference blocks (e.g., neighboring blocks) to generate a predictive block for a current video block to be coded. The process of finding the reference blocks can be done by block matching algorithms. Residual data representing pixel differences between the current block to be coded and the predictive block is referred to as residual block or prediction error. Inter coded blocks are coded according to the motion vector pointing to a reference block in a reference frame used to form the predictive block, and the residual block. The process of determining the motion vector is typically referred to as motion estimation. Intra coded blocks are coded according to an intra prediction mode and the residual block. For further compression, the residual block is transformed from the pixel domain to a transform domain, e.g., frequency domain, resulting in residual transform coefficients, which can then be quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, can be scanned into a one-dimensional vector, and then entropy coded into the video bitstream for more compression.

[0006] The coded video bitstream is then saved in a computer-readable storage medium (e.g., flash memory) to be accessed by another electronic device with digital video capability, or directly transmitted to the electronic device via a wired or wireless connection. The electronic device then performs video decompression (which is an opposite process to video compression described above) by, e.g., parsing the coded video bitstream to obtain syntax elements from the bitstream and reconstructing the digital video data from the coded video bitstream into its original format based at least in part on the syntax elements obtained from the bitstream, and renders the reconstructed digital video data on a display of the electronic device.

[0007] As digital video quality goes from high definition to 4Kx2K or even 8Kx4K, the amount of video data to be encoded / decoded grows exponentially. There is always a challenge how to efficiently encode / decode the video data while keeping the image quality of the decoded video data. SUMMARY

[0008] This application describes implementations related to video data encoding and decoding, and more specifically, implementations related to methods and apparatuses to improve coding efficiency of merge mode with motion vector difference (MMVD), also known as advanced motion vector expression (UMVE) mode.

[0009] According to a first aspect of the present application, a method of decoding video data includes receiving a first control flag from a bitstream, wherein the first control flag indicates whether a merge mode with motion vector difference (MMVD) is enabled for one or more coding units in a video sequence; receiving a first syntax element from the video data according to a determination that the MMVD is enabled from the first control flag, wherein the first syntax element identifies one of a plurality of sets of motion vector difference (MVD) offset quantities; receiving a second control flag of the video data from the bitstream, the second control flag corresponding to a respective coding unit of the one or more coding units, the second control flag indicating whether the MMVD is applied to the respective coding unit; receiving a second syntax element of the respective coding unit and a third syntax element of the respective coding unit according to a determination that the MMVD is applied to the respective coding unit from the second control flag, the second syntax element selecting a MVD offset quantity from the identified set of MVD offset quantities, the third syntax element selecting a MVD direction corresponding to the selected MVD offset quantity; forming a MVD based on the selected MVD offset quantity and the selected MVD direction; and reconstructing the respective coding unit by generating a motion vector of the respective coding unit using the formed MVD.

[0010] According to a second aspect of the present application, a method of decoding video data includes receiving a first control flag from a bitstream, wherein the first control flag indicates whether a mode of merge with motion vector difference (MMVD) is enabled for one or more coding units; receiving a second control flag of the video data from the bitstream according to a determination that the MMVD is enabled from the first control flag, the second control flag corresponding to a respective coding unit of the one or more coding units, the second control flag indicating whether the MMVD is applied to the respective coding unit; receiving a third control flag from the video data according to a determination that the MMVD is applied to the respective coding unit from the second control flag, wherein the third control flag indicates whether an inter prediction filtering (InterPF) mode is enabled for the respective coding unit; receiving a fourth syntax element from the video data according to a determination that the InterPF mode is enabled for the respective coding unit from the third control flag, wherein the fourth syntax element identifies one InterPF mode of a plurality of InterPF modes of the respective coding unit; and reconstructing the respective coding unit by applying the MMVD and the identified InterPF mode to video data corresponding to the respective coding unit.

[0011] According to a third aspect of the present application, an electronic device includes one or more processing units, memory, and a plurality of programs stored in the memory. The programs, when executed by the one or more processing units, cause the electronic device to perform the method of decoding video data as described above.

[0012] According to a fourth aspect of the present application, a non-transitory computer-readable storage medium stores a plurality of programs for execution by an electronic device having one or more processing units. The programs, when executed by the one or more processing units, cause the electronic device to perform the method of decoding video data as described above.

[0013] According to a fifth aspect of the present application, a computer program product includes a computer program, wherein the computer program, when executed by a processor, implements the method of decoding video data as described above. BRIEF DESCRIPTION OF DRAWINGS

[0014] The accompanying drawings, which are included to provide a further understanding of the implementations and are incorporated in and constitute a part of this specification, illustrate the described implementations and together with the description serve to explain the basic principles. Like reference numerals refer to corresponding parts throughout.

[0015] Figure 1 is a block diagram illustrating an exemplary video encoding and decoding system, in accordance with some implementations of the present disclosure.

[0016] Figure 2 is a block diagram illustrating an exemplary video encoder, in accordance with some implementations of the present disclosure. is a block diagram illustrating an exemplary video encoder, in accordance with some implementations of the present disclosure.

[0017] Figure 3 FIG. 1 is a block diagram illustrating an exemplary video decoder, in accordance with some embodiments of the present disclosure.

[0018] Figures 4A to 4E FIG. 2 is a block diagram illustrating how a frame is recursively divided into multiple video blocks of different sizes and shapes, in accordance with some embodiments of the present disclosure.

[0019] Figure 5A and Figure 5B FIG. 3 is a block diagram illustrating exemplary offset values added to the horizontal or vertical components of a starting motion vector (MV), in accordance with some embodiments of the present disclosure.

[0020] Figure 6 FIG. 4 is a block diagram illustrating exemplary decoder-side motion vector refinement (DMVR), in accordance with some embodiments of the present disclosure.

[0021] Figure 7 FIG. 5 is a block diagram illustrating some exemplary integer search candidates for the DMVR mode, in accordance with some embodiments of the present disclosure.

[0022] Figure 8 FIG. 6 is a flowchart illustrating an exemplary process for determining a motion vector difference (MVD) offset from a plurality of MVD offset sets when a merge mode with MVD (MMVD) is enabled, in accordance with some embodiments of the present disclosure. DETAILED DESCRIPTION

[0023] Reference will now be made in detail to specific implementations, examples of which are illustrated in the accompanying drawings. In the following detailed description of embodiments, numerous specific details are set forth in order to provide a thorough understanding of the subject matter presented herein. However, it will be apparent to one of ordinary skill in the art that the subject matter presented can be practiced without these specific details. For example, it will be apparent to one of ordinary skill in the art that the subject matter presented herein can be implemented on many types of electronic devices with digital video capabilities.

[0024] The first generation of AVS standards includes the Chinese national standards "Information technology - Advanced audio video coding - Part 2: Video" (referred to as AVSl) and "Information technology - Advanced audio video coding - Part 16: Broadcasting television video" (referred to as AVS+). Compared with the MPEG-2 standard, the first generation of AVS standards can provide about 50% bit rate saving at the same perceptual quality. The second generation of AVS standards includes the Chinese national standards "Information technology - High efficiency video coding" (referred to as AVS2) series, which is mainly targeted for the transmission of additional HD TV programs. The coding efficiency of AVS2 is twice that of AVS+. Meanwhile, the AVS2 standard video part is submitted by the Institute of Electrical and Electronics Engineers (IEEE) as an international application standard. The AVS3 standard is a new generation of video coding standard for UHD video applications, aiming to surpass the coding efficiency of the latest international standard HEVC, and the AVS3 standard provides about 30% bit rate saving over the HEVC standard.

[0025] The focus of the present disclosure is to improve the coding performance of the merge mode with motion vector difference (MMVD) tool used in both VVC and AVS3 standards. In AVS3, this tool is referred to as advanced motion vector expression (UMVE). In some embodiments disclosed herein, although the basic MMVD method is illustrated with the MMVD design in the AVS3 standard as an example, the methods described in the present disclosure can also be applied to other MMVD designs or other coding tools with the same or similar design for those skilled in the art of video coding.

[0026] Figure 1 is a block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel according to some embodiments of the present disclosure. As shown, the system 10 includes a source device 12 that generates and encodes video data to be decoded at a later time by a destination device 14. The source device 12 and the destination device 14 can include any of a variety of electronic devices, including desktop or laptop computers, tablet computers, smart phones, set-top boxes, digital televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, etc. In some embodiments, the source device 12 and the destination device 14 are equipped with wireless communication capabilities. Figure 1

[0027] ​In some implementations, destination device 14 can receive, via link 16, the encoded video data to be decoded. Link 16 can comprise any type of communication medium or device capable of transmitting encoded video data from source device 12 to destination device 14. In one example, link 16 can comprise a communication medium to enable source device 12 to transmit encoded video data directly to destination device 14 in real-time. The encoded video data can be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to destination device 14. The communication medium can comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium can form part of a packet-based network, such as a local area network, a wide-area network, or a global network such as the Internet. The communication medium can include routers, switches, base stations, or any other equipment that can be used to facilitate communication from source device 12 to destination device 14.

[0028] In some other implementations, encoded video data can be transmitted from output interface 22 to storage device 32. Subsequently, encoded video data in storage device 32 can be accessed by destination device 14 via input interface 28. Storage device 32 can include any of a variety of distributed or locally accessed data storage media such as a hard drive, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data. In a further example, storage device 32 can correspond to a file server or another intermediate storage device that can hold the encoded video data generated by source device 12. Destination device 14 can access stored video data from storage device 32 via streaming or download. The file server can be any type of computer

[0029] As Figure 1As shown, source device 12 includes a video source 18, video encoder 20 and an output interface 22. Video source 18 can include a source such as a video capture device, e.g., a video camera, a video archive containing previously captured video, a video feed interface to receive video from a video content provider, and / or a computer graphics system for generating computer graphics data as the source video, or a combination of such sources. As one example, if video source 18 is a video camera of a security surveillance system, source device 12 and destination device 14 can form a still or video camera phone. However, the implementations described in this application can be applicable to video coding in general and can apply to wireless and / or wired applications.

[0030] The captured, pre-captured, or computer-generated video can be encoded by video encoder 20. The encoded video data can be transmitted directly to destination device 14 via output interface 22 of source device 12. The encoded video data can also (or alternatively) be stored onto storage device 32 for later access by destination device 14 or other devices, e.g., for decoding and / or playback. Output interface 22 can further include a modem and / or a transmitter.

[0031] Destination device 14 includes an input interface 28, a video decoder 30, and a display device 34. Input interface 28 can include a receiver and / or modem and receive encoded video data through link 16. The encoded video data transmitted through link 16, or provided on storage device 32, can include a variety of syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements can be included with the encoded video data transmitted on a communication medium, stored on a storage medium, or stored on a file server.

[0032] In some implementations, destination device 14 can include a display device 34, which can be an integrated display device and an external display device configured to communicate with destination device 14. Display device 34 displays the decoded video data to a user and can include any of a variety of display devices such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

[0033] Video encoder 20 and video decoder 30 can operate according to proprietary or industry standards such as VVC, HEVC, MPEG-4 Part 10, Advanced Video Codec (AVC), or extensions of such standards. It should be understood that this application is not limited to any particular video encoding / decoding standard and can be applied to other video encoding / decoding standards. It is generally contemplated that the video encoder 20 of source device 12 can be configured to encode video data according to any of these current or future standards. Similarly, it is generally contemplated that the video decoder 30 of destination device 14 can be configured to decode video data according to any of these current or future standards.

[0034] The video encoder 20 and video decoder 30 can each be implemented as any of a variety of suitable encoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When implemented in part in software, the electronic device may store instructions for software in a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the video encoding / decoding operations disclosed in this disclosure. Each of the video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder / decoder (CODEC) in the respective device.

[0035] Figure 2 This is a block diagram illustrating an exemplary video encoder 20 according to some embodiments described in this application. The video encoder 20 can perform intra-frame predictive coding and inter-frame predictive coding on video blocks within a video frame. Intra-frame predictive coding relies on spatial prediction to reduce or remove spatial redundancy of video data within a given video frame or image. Inter-frame predictive coding relies on temporal prediction to reduce or remove temporal redundancy of video data within adjacent video frames or images of a video sequence.

[0036] like Figure 2As shown, the video encoder 20 includes a video data memory 40, a prediction processing unit 41, a decoded image buffer (DPB) 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 further includes a motion estimation unit 42, a motion compensation unit 44, a partitioning unit 45, an intra-frame prediction processing unit 46, and an intra-frame block copying (BC) unit 48. In some embodiments, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and an adder 62 for video block reconstruction. A deblocking filter (not shown) may be located between the adder 62 and the DPB 64 to filter block boundaries to remove block artifacts from the reconstructed video. In addition to the deblocking filter, a loop filter (not shown) may be used to filter the output of the adder 62. The video encoder 20 may be in the form of fixed or programmable hardware units, or may be partitioned within one or more of the fixed or programmable hardware units illustrated.

[0037] Video data memory 40 can store video data to be encoded by components of video encoder 20. The video data in video data memory 40 can be obtained, for example, from video source 18. DPB 64 is a buffer that stores reference video data for encoding the video data by video encoder 20 (e.g., in intra-frame predictive coding mode or inter-frame predictive coding mode). Video data memory 40 and DPB 64 can be formed from any of a variety of memory devices. In various examples, video data memory 40 can be on-chip along with other components of video encoder 20, or off-chip relative to those components.

[0038] like Figure 2 As shown, after receiving video data, the partitioning unit 45 within the prediction processing unit 41 partitions the video data into video blocks. This partitioning may also include dividing the video frame into stripes, tiles, or other larger coding units (CUs) according to a predefined partitioning structure (such as a quadtree structure associated with the video data). A video frame can be partitioned into multiple video blocks (or sets of video blocks referred to as tiles). The prediction processing unit 41 can select one of several possible predictive coding modes for the current video block based on error results (e.g., coding rate and distortion level), such as one of several intra-frame predictive coding modes or one of several inter-frame predictive coding modes. The prediction processing unit 41 can provide the resulting intra-frame or inter-frame predictive coding block to adder 50 to generate a residual block, and to adder 62 to reconstruct the coding block for subsequent use as part of a reference frame. The prediction processing unit 41 also provides syntax elements such as motion vectors, intra-frame mode indicators, partitioning information, and other such syntax information to the entropy coding unit 56.

[0039] To select an appropriate intra-prediction coding mode for a current video block, intra-prediction processing unit 46 within prediction processing unit 41 can perform intra-prediction coding of the current video block relative to one or more neighboring blocks in the same frame as the current block to be coded to provide a spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 perform inter-prediction coding of the current video block relative to one or more prediction blocks in one or more reference frames to provide a temporal prediction. Video encoder 20 can perform multiple passes of encoding, for example, in order to select an appropriate coding mode for each block of video data.

[0040] In some implementations, motion estimation unit 42 determines an inter-prediction mode for a current video frame by generating a motion vector that indicates a displacement of a prediction unit (PU) of a video block within the current video frame relative to a prediction block within a reference video frame according to a predetermined pattern within a sequence of video frames. Motion estimation performed by motion estimation unit 42 is a process of generating motion vectors that estimates the motion of video blocks. A motion vector, for example, can indicate a displacement of a PU of a video block within a current video frame or image relative to a prediction block (or other coding unit) within a reference frame that is a close match to the current block (or other coding unit) being coded within the current frame. The predetermined pattern can designate video frames in the sequence as P-frames or B-frames. Intra-BC unit 48 can determine vectors, e.g., block vectors, for intra-BC coding in a similar manner as motion vectors are determined by motion estimation unit 42 for inter-prediction, or can utilize motion estimation unit 42 to determine block vectors.

[0041] A prediction block is a block of a reference frame that is considered to be a close match to a PU of a video block to be coded in terms of pixel differences, which can be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics. In some implementations, video encoder 20 can calculate values for sub-integer pixel positions of reference frames stored in DPB 64. For example, video encoder 20 can interpolate values for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of reference frames. Accordingly, motion estimation unit 42 can perform a motion search relative to full-pixel positions and fractional-pixel positions and output motion vectors with fractional-pixel precision.

[0042] Motion estimation unit 42 computes motion vectors for PUs of video blocks in an inter-prediction coded frame by comparing the locations of the PUs to the locations of prediction blocks of reference frames selected from a first reference frame list (List 0) or a second reference frame list (List 1), each of which identifies one or more reference frames stored in DPB 64. Motion estimation unit 42 sends the computed motion vectors to motion compensation unit 44 and then to entropy encoding unit 56.

[0043] Motion compensation performed by motion compensation unit 44 can involve fetching or generating a prediction block based on a motion vector determined by motion estimation unit 42. Upon receiving a motion vector for a PU of a current video block, motion compensation unit 44 can locate the prediction block pointed to by the motion vector in one of the reference frame lists, retrieve the prediction block from DPB 64 and forward the prediction block to summer 50. Summer 50 then forms a residual video block having pixel difference values by subtracting the pixel values of the prediction block provided by motion compensation unit 44 from the pixel values of the encoded current video block. The pixel difference values forming the residual video block can include luma difference components or chroma difference components or both. Motion compensation unit 44 can also generate syntax elements associated with the video block of the video frame for use by video decoder 30 when decoding the video block of the video frame. The syntax elements can include, for example, syntax elements defining the motion vector used to identify the prediction block, any flags indicating the prediction mode, or any other syntax information described herein. Note that motion estimation unit 42 and motion compensation unit 44 can be highly integrated, but are illustrated separately for conceptual purposes.

[0044] In some implementations, intra BC unit 48 can generate vectors and fetch prediction blocks in a manner similar to that described above in connection with motion estimation unit 42 and motion compensation unit 44, but where the prediction block is in the same frame as the encoded current block, and where the vector is referred to as a block vector relative to the motion vector. Specifically, intra BC unit 48 can determine an intra prediction mode to use for encoding the current block. In some examples, intra BC unit 48 can encode the current block using various intra prediction modes, e.g., during a separate encoding pass, and test their performance through rate-distortion analysis. Next, intra BC unit 48 can select an appropriate intra prediction mode to use among the various tested intra prediction modes and generate an intra mode indicator accordingly. For example, intra BC unit 48 can use the rate-distortion analysis for the various tested intra prediction modes to compute rate-distortion values and select the intra prediction mode having the best rate-distortion characteristics among the tested modes as the appropriate intra prediction mode to use. Rate-distortion analysis generally determines the amount of distortion (or error) between an encoded block and the original, unencoded block (encoded to produce the encoded block) and the bit rate (i.e., number of bits) used to produce the encoded block. Intra BC unit 48 can compute a ratio of the distortion and the rate for each encoded block to determine which intra prediction mode exhibits the best rate-distortion value for the block.

[0045] In other examples, the intra BC unit 48 can use the motion estimation unit 42 and the motion compensation unit 44, in whole or in part, to perform such functionality for intra BC prediction in accordance with the implementations described herein. In either case, for intra block copy, the prediction block can be a block that is considered to closely match the block to be encoded in terms of pixel differences, which can be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics, and the identification of the prediction block can include calculating values for sub-integer pixel positions.

[0046] Whether the prediction block is from the same frame according to intra prediction or from a different frame according to inter prediction, the video encoder 20 can form a residual video block by subtracting the pixel values of the prediction block from the pixel values of the encoded current video block, thereby forming pixel difference values. The pixel difference values forming the residual video block can include luma component differences and chroma component differences.

[0047] As described above, the intra prediction processing unit 46 can intra predict the current video block as an alternative to inter prediction performed by the motion estimation unit 42 and the motion compensation unit 44, or intra block copy prediction performed by the intra BC unit 48. In particular, the intra prediction processing unit 46 can determine an intra prediction mode to use for encoding the current block. To this end, the intra prediction processing unit 46 can encode the current block using various intra prediction modes, e.g., during a separate encoding pass, and the intra prediction processing unit 46 (or in some examples a mode selection unit) can select an appropriate intra prediction mode to use from among the tested intra prediction modes. The intra prediction processing unit 46 can provide information indicative of the selected intra prediction mode for the block to the entropy encoding unit 56. The entropy encoding unit 56 can encode the information indicative of the selected intra prediction mode in the bitstream.

[0048] After the prediction processing unit 41 determines the prediction block for the current video block via inter prediction or intra prediction, the adder 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block can be included in one or more transform units (TUs) and provided to the transform processing unit 52. The transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform such as a discrete cosine transform (DCT) or a conceptually similar transform.

[0049] The transform processing unit 52 can send the resulting transform coefficients to a quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce bit rate. The quantization process can also reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting a quantization parameter. In some examples, the quantization unit 54 can then perform a scan of the matrix including the quantized transform coefficients. Alternatively, the entropy encoding unit 56 can perform the scan.

[0050] Following quantization, the entropy encoding unit 56 entropy encodes the quantized transform coefficients using, for example, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), Probability Interval Partitioning Entropy (PIPE) coding or other entropy encoding methodologies or techniques, into the video bitstream. The encoded bitstream can then be transmitted to the video decoder 30 or archived, such as in the storage device 32, for later transmission to or retrieval by the video decoder 30. The entropy encoding unit 56 can also entropy encode motion vectors and other syntax elements of the encoded current video frame.

[0051] The inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain to generate a reference block used to predict other video blocks. As described above, the motion compensation unit 44 can generate a motion compensated predicted block from one or more reference blocks stored in the frames in the DPB 64. The motion compensation unit 44 can also apply one or more interpolation filters to the predicted block to calculate sub-integer pixel values for use in motion estimation.

[0052] The summer 62 adds the reconstructed residual block to the motion compensated predicted block produced by the motion compensation unit 44 to produce a reference block for storage in the DPB 64. The reference block can then be used by the intra BC unit 48, the motion estimation unit 42, and the motion compensation unit 44 as a prediction block to inter predict another video block in a subsequent video frame.

[0053] Figure 3 FIG. 1 is a block diagram illustrating an example video encoder 20 that can implement some embodiments of the present application. The video encoder 20 includes a video data memory 29, a prediction processing unit 22, a transform processing unit 24, a quantization unit 26, and an entropy encoding unit 28. The prediction processing unit 22 further includes an intra prediction processing unit 23 and an inter prediction processing unit 25. The video encoder 20 can perform generally the same functions as the video encoder 10 described above in connection with FIG. 2. Figure 2The decoding process is the inverse of the encoding process described with respect to video encoder 20. For example, motion compensation unit 82 can generate prediction data based on motion vectors received from entropy decoding unit 80, while intra-prediction unit 84 can generate prediction data based on intra-prediction mode indicators received from entropy decoding unit 80.

[0054] In some examples, the units of video decoder 30 can be assigned tasks to perform embodiments of the application. Also, in some examples, embodiments of the disclosure can be divided among one or more units of video decoder 30. For example, intra BC unit 85 can perform embodiments of the application alone or in combination with other units of video decoder 30, such as motion compensation unit 82, intra-prediction processing unit 84, and entropy decoding unit 80. In some examples, video decoder 30 can not include intra BC unit 85, and the functionality of intra BC unit 85 can be performed by other components of prediction processing unit 81, such as motion compensation unit 82.

[0055] Video data memory 79 can store video data, such as encoded video bitstreams, to be decoded by other components of video decoder 30. The video data stored in video data memory 79 can be obtained, for example, from storage device 32 via wired or wireless network transmission, from a local video source, such as a camera, by accessing physical data storage media (e.g., a flash drive or hard disk), or by any combination of these. Video data memory 79 can include a coded picture buffer (CPB) that stores coded video data from an encoded video bitstream. Decoded picture buffer (DPB) 92 of video decoder 30 stores reference video data for use in decoding video data by video decoder 30 (e.g., in intra- or inter-coding modes). Video data memory 79 and DPB 92 can be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magneto resistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. For Figure 3 illustrative purposes, video data memory 79 and DPB 92 are depicted as two distinct components of video decoder 30. However, it will be readily apparent to one of ordinary skill in the art that video data memory 79 and DPB 92 can be provided by same memory device or separate memory devices. In some examples, video data memory 79 can be on-chip with other components of video decoder 30, or off-chip relative to those components.

[0056] During the decoding process, video decoder 30 receives an encoded video bitstream that represents encoded video frames and associated syntax elements. Video decoder 30 can receive the syntax elements at the video frame level and / or the video block level. Entropy decoding unit 80 of video decoder 30 entropy decodes the bitstream to generate quantized coefficients, motion vectors, or intra-prediction mode indicators, and other syntax elements. Entropy decoding unit 80 then forwards the motion vectors and other syntax elements to prediction processing unit 81.

[0057] When a video frame has been encoded as an intra-predicted coded (I) frame or an intra-coded prediction block in another type of frame, intra-prediction processing unit 84 of prediction processing unit 81 can generate prediction data for a video block of the current video frame based on the signaled intra-prediction mode and reference data from previously decoded blocks of the current frame.

[0058] When a video frame has been encoded as an inter-predicted coded (i.e., B or P) frame, motion compensation unit 82 of prediction processing unit 81 produces one or more prediction blocks for a video block of the current video frame based on the motion vectors and other syntax elements received from entropy decoding unit 80. Each prediction block can be produced from a reference frame within one of the reference frame lists. Video decoder 30 can construct the reference frame lists: List 0 and List 1, using default construction techniques based on reference frames stored in DPB 92.

[0059] In some examples, when a video block is coded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 produces a prediction block for the current video block based on the block vectors and other syntax elements received from entropy decoding unit 80. The prediction block can be within a reconstructed area of the same picture as the current video block defined by video encoder 20.

[0060] Motion compensation unit 82 and / or intra BC unit 85 determine the prediction information for a video block of the current video frame by parsing the motion vectors and other syntax elements, and then use the prediction information to produce a prediction block for the decoded current video block. For example, motion compensation unit 82 uses some of the received syntax elements to determine the prediction mode used to encode the video block of the video frame (e.g., intra-prediction or inter-prediction), the inter-prediction frame type (e.g., B or P), the construction information for one or more of the reference frame lists for the frame, the motion vectors for each inter-predicted coded video block of the frame, the inter-prediction status for each inter-predicted coded video block of the frame, and other information used to decode the video block in the current video frame.

[0061] Similarly, intra BC unit 85 can use some of the received syntax elements (e.g., flags) to determine that the current video block is predicted using: an intra BC mode, construction information that the video block of the frame is in a reconstructed region and should be stored in DPB 92, a block vector for each intra BC predicted video block of the frame, an intra BC prediction status for each intra BC predicted video block of the frame, and other information used to decode the video block in the current video frame.

[0062] Motion compensation unit 82 can also perform interpolation using an interpolation filter to calculate interpolated values for sub-integer pixels of the reference block as used by video encoder 20 during encoding of the video block. In this case, motion compensation unit 82 can determine the interpolation filter used by video encoder 20 from the received syntax elements and use the interpolation filter to produce the prediction block.

[0063] Dequantization unit 86 dequantizes the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80 using the same quantization parameter calculated by video encoder 20 for each video block in the video frame to determine the degree of quantization. Inverse transform processing unit 88 applies an inverse transform, e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients in order to reconstruct the residual block in the pixel domain.

[0064] After motion compensation unit 82 or intra BC unit 85 generates the prediction block for the current video block based on the vectors and other syntax elements, summer 90 reconstructs the decoded video block for the current video block by summing the residual block from inverse transform processing unit 88 and the corresponding prediction block generated by motion compensation unit 82 and intra BC unit 85. In-loop filter (not shown) can be positioned between summer 90 and DPB 92 to further process the decoded video block. The decoded video block in a given frame is then stored in DPB 92, which stores the reference frames used for subsequent motion compensation of the following video blocks. DPB 92 or a memory device separate from DPB 92 can also store the decoded video for later presentation on a display device, such as display device 34 of FIG. 1. Figure 1

[0065] In a typical video coding process, a video sequence typically includes an ordered set of frames or pictures. Each frame can include three arrays of samples, denoted as SL, SCb, and SCr, respectively. SL is a two-dimensional array of luma samples. SCb is a two-dimensional array of Cb chroma samples. SCr is a two-dimensional array of Cr chroma samples. In other instances, the frames can be monochrome and thus include only one two-dimensional array of luma samples.

[0066] As Figure 4A ​As shown, video encoder 20 (or more specifically, partition unit 45) generates an encoded representation of a frame by first partitioning the frame into a set of coding tree units (CTUs). A video frame can include an integer number of CTUs ordered consecutively in a raster scan order from left to right and from top to bottom. Each CTU is the largest logical coding unit and the width and height of the CTU are signaled by video encoder 20 in a sequence parameter set such that all CTUs in a video sequence have the same size, i.e., one of 128x128, 64x64, 32x32, and 16x16. It should be noted, however, that the present application is not necessarily limited to a particular size. As Figure 4B As shown, each CTU can include one coding tree block (CTB) of luma samples, two corresponding coding tree blocks of chroma samples, and syntax elements used to code the samples of the coding tree blocks. The syntax elements describe properties of different types of units of a coded block of pixels and how the video sequence can be reconstructed at video decoder 30, including inter or intra prediction, intra prediction mode, motion vectors, and other parameters. In monochrome images or images having three separate color planes, a CTU can include a single coding tree block and syntax elements used to code the samples of the coding tree block. A coding tree block can be an NxN block of samples.

[0067] To achieve better performance, video encoder 20 can recursively perform tree partitioning, such as binary tree partitioning, ternary tree partitioning, quad tree partitioning, or a combination of the two, on the coding tree blocks of a CTU and partition the CTU into smaller coding units (CUs). As Figure 4C As depicted, a 64x64 CTU 400 is first partitioned into four smaller CUs, each having a block size of 32x32. Of the four smaller CUs, CUs 410 and 420 are each partitioned into four 16x16 CUs by block size. Two 16x16 CUs 430 and 440 are each further partitioned into four 8x8 CUs by block size. Figure 4D A quad tree data structure illustrating the final result of the partitioning process of CTU 400 as depicted in Figure 4C A quad tree data structure illustrating the final result of the partitioning process of CTU 400 as depicted in Figure 4B As depicted, each CU can include a coding block (CB) of luma samples and two corresponding coding blocks of chroma samples of the same size of the frame, and syntax elements used to code the samples of the coding blocks. In monochrome images or images having three separate color planes, a CU can include a single coding block and syntax elements used to code the samples of the coding block. It should be noted that Figure 4C and Figure 4DThe depicted quad-tree partitioning is for illustration purposes only, and one CTU can be partitioned into multiple CUs to adapt to different local characteristics based on quad-tree / triple-tree / binary-tree partitioning. In the multi-type tree structure, one CTU is partitioned by a quad-tree structure, and each quad-tree leaf CU can be further partitioned by a binary tree structure or a triple tree structure. As shown in Figure 4E there are five types of partitioning, i.e., quad partitioning, horizontal binary partitioning, vertical binary partitioning, horizontal triple partitioning, and vertical triple partitioning.

[0068] In some implementations, video encoder 20 can further partition the coded block of a CU into one or more MxN prediction blocks (PBs). A prediction block is a rectangular (square or non-square) block of samples on which the same prediction (inter or intra) is applied. A prediction unit (PU) of a CU can include a prediction block of luma samples, two corresponding prediction blocks of chroma samples, and syntax elements used to perform prediction on the prediction blocks. In monochrome images or images having three separate color planes, a PU can include a single prediction block and syntax structures used to predict the prediction block. Video encoder 20 can generate predicted luma, Cb, and Cr blocks for the luma, Cb, and Cr prediction blocks, respectively, of each PU of a CU.

[0069] Video encoder 20 can use intra prediction or inter prediction to generate the prediction blocks for a PU. If video encoder 20 uses intra prediction to generate the prediction blocks for a PU, video encoder 20 can generate the prediction blocks for the PU based on decoded samples of the frame associated with the PU. If video encoder 20 uses inter prediction to generate the prediction blocks for a PU, video encoder 20 can generate the prediction blocks for the PU based on decoded samples of one or more frames other than the frame associated with the PU.

[0070] After video encoder 20 generates the predicted luma, Cb, and Cr blocks for one or more PUs of a CU, video encoder 20 can generate luma residual blocks for the CU by subtracting the predicted luma blocks for the CU from the original luma coded block for the CU, such that each sample in the luma residual blocks for the CU indicates a difference between a luma sample in one of the predicted luma blocks for the CU and a corresponding sample in the original luma coded block for the CU. Similarly, video encoder 20 can generate Cb and Cr residual blocks for the CU, respectively, such that each sample in the Cb residual blocks for the CU indicates a difference between a Cb sample in one of the predicted Cb blocks for the CU and a corresponding sample in the original Cb coded block for the CU, and each sample in the Cr residual blocks for the CU can indicate a difference between a Cr sample in one of the predicted Cr blocks for the CU and a corresponding sample in the original Cr coded block for the CU.

[0071] Furthermore, as Figure 4CAs illustrated, video encoder 20 can use quad-tree partitioning to divide the luma, Cb, and Cr residual blocks of a CU into one or more luma, Cb, and Cr transform blocks. A transform block is a rectangular (square or non-square) block of samples to which the same transform is applied. A transform unit (TU) of a CU can include a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax elements used to transform the transform block samples. Thus, each TU of a CU can be associated with a luma transform block, a Cb transform block, and a Cr transform block. In some examples, the luma transform block associated with a TU can be a sub-block of the luma residual block of the CU. The Cb transform block can be a sub-block of the Cb residual block of the CU. The Cr transform block can be a sub-block of the Cr residual block of the CU. In monochrome images or images having three separate color planes, a TU can include a single transform block and syntax structures used to transform the samples of the transform block.

[0072] Video encoder 20 can apply one or more transforms to the luma transform block of a TU to generate a luma coefficient block of the TU. A coefficient block can be a two- dimensional array of transform coefficients. A transform coefficient can be a scalar. Video encoder 20 can apply one or more transforms to the Cb transform block of a TU to generate a Cb coefficient block of the TU. Video encoder 20 can apply one or more transforms to the Cr transform block of a TU to generate a Cr coefficient block of the TU.

[0073] After generating a coefficient block (e.g., a luma coefficient block, a Cb coefficient block, or a Cr coefficient block), video encoder 20 can quantize the coefficient block. Quantization generally refers to a process that quantizes transform coefficients to possibly reduce the amount of data used to represent the transform coefficients, providing further compression. After video encoder 20 quantizes a coefficient block, video encoder 20 can entropy encode syntax elements indicating the quantized transform coefficients. For example, video encoder 20 can perform Context- Adaptive Binary Arithmetic Coding (CABAC) on the syntax elements indicating the quantized transform coefficients. Ultimately, video encoder 20 can output a bitstream that includes a sequence of bits that forms a representation of encoded frames and associated data, which is saved in storage device 32 or transmitted to destination device 14.

[0074] After receiving the bitstream generated by video encoder 20, video decoder 30 can parse the bitstream to obtain syntax elements from the bitstream. Video decoder 30 can reconstruct the frames of the video data based at least in part on the syntax elements obtained from the bitstream. The process of reconstructing the video data is generally reciprocal to the encoding process performed by video encoder 20. For example, video decoder 30 can perform inverse transforms on the coefficient blocks associated with the TUs of the current CU to reconstruct the residual blocks associated with the TUs of the current CU. Video decoder 30 also reconstructs the coding blocks of the current CU by adding the samples of the prediction blocks of the PUs of the current CU to corresponding samples of the transform blocks of the TUs of the current CU. After reconstructing the coding blocks of each CU of a frame, video decoder 30 can reconstruct the frame.

[0075] In some embodiments, there are different ways to signal the prediction mode. In VVC, each CU can be coded as either skip mode or non-skip mode. For a CU coded as non-skip mode, further syntax elements are signaled to indicate whether the current CU is coded as intra mode, inter mode, intra block copy (IBC) mode, or palette (PLT) mode. These modes are referred to as “MODE INTRA”, “MODE INTER”, “MODE IBC”, and “MODE PLT” in the VVC specification, respectively. For a CU coded as inter mode, the prediction signal can be generated from pixels from reference pictures that are not the current picture. In addition, one flag is further signaled for a CU coded as inter mode to indicate whether the current CU is in merge mode. For a CU coded in merge mode, several different types of merge modes are further signaled using additional syntax elements. These different types of merge modes include regular merge mode, subblock merge mode, merge mode with MV difference (MMVD), combined inter and intra prediction (CIIP), and triangle merge mode. The following sections explain these merge modes.

[0076] In some embodiments, in VVC, the merge candidate list is constructed by including the following five types of candidates in order: spatial MVPs (i.e., motion vector predictors) from spatial neighboring CUs, temporal MVPs from co-located CUs, history-based MVPs from a FIFO table, pair-wise average MVPs, and zero MV.

[0077] The size of the merge list is signaled in the slice header, and the maximum allowed size of the merge list is 6 in VVC. For each CU coded in merge mode, the index of the best merge candidate is coded using truncated unary (TU). The first bin of the merge index is coded with a context, and the other bins are coded using bypass coding. In the following context of this disclosure, this extended merge mode is also referred to as regular merge mode, because its concept is the same as the merge mode used in HEVC.

[0078] In some embodiments, in addition to the merge mode that derives the motion information of a current block from its spatial / temporal neighbors, the MMVD / UMVE mode is introduced in VVC and AVS standards as a special merge mode. Specifically, in VVC and AVS3, this mode is signaled at the block level by a MMVD flag. In MMVD mode, the first two candidates in the merge list of regular merge mode are selected as the two base merge candidates of MMVD. After a base merge candidate is selected and signaled, additional syntax elements are signaled to indicate the motion vector difference (MVD) added to the motion of the selected merge candidate. The MMVD syntax elements include a merge candidate flag for selecting a base merge candidate, a distance index for specifying the MVD magnitude, and a direction index for indicating the MVD direction.

[0079] In the MMVD design, the distance index specifies the MVD magnitude, which is defined based on a set of predefined starting point offsets. Figure 5A and Figure 5B is a block diagram illustrating exemplary offsets added to the horizontal or vertical component of the starting motion vector (MV), i.e., the MV of the selected base merge candidate, according to some embodiments of the present disclosure. Reference picture list L0 and reference picture list L1 are illustrated.

[0080] Table 1 illustrates the relationship of the applied distance index and the predefined MVD offsets.

[0081]

[0082] Table 1: Relationship of distance index and predefined MVD offsets

[0083] The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate the four directions as shown in Table 2. It should be noted that the meaning of the MVD sign can change depending on the information of the starting MV. When the starting MV is a uni-predicted MV or a bi-predicted MV and both lists of the bi-predicted MV point to the same side of the current picture (i.e., both the picture order count (POC) of the two references are greater than the POC of the current picture or both are less than the POC of the current picture), the sign in Table 2 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bi-predicted MV and the two MVs point to different sides of the current picture (i.e., one reference has a picture order count (POC) greater than the POC of the current picture and the other reference has a POC less than the POC of the current picture), the sign in Table 2 specifies the sign of the MVD added to the list 0 (L0 reference) MV component of the starting MV, and the sign of the list 1 (LI reference) MVD has the opposite value.

[0084] Direction IDX 00 01 10 11 x-axis + - N / A N / A y-axis N / A N / A + -

[0085] Table 2: Sign of MVD specified by direction index

[0086] In some embodiments, to increase the accuracy of the MVs of the regular merge mode, a decoder-side motion vector refinement (DMVR) based on bilateral matching is applied. In bi-prediction operation, a refined MV is searched around the initial MV in the reference picture list L0 and the reference picture list LI. The method calculates the distortion between two candidate blocks in the reference picture list L0 and the list LI. Figure 6 FIG. 6 is a block diagram illustrating an exemplary decoder-side motion vector refinement (DMVR) according to some embodiments of the present disclosure. As shown in FIG. 6, the SAD between block 602 and block 604 is calculated based on each MV candidate around the initial MV. The MV candidate with the lowest SAD becomes the refined MV and is used to generate the bi-predicted signal. Figure 6

[0087] In DVMR, the search point is the surrounding integer sample pointed by the initial MV and the MV offset considered to comply with the mirror rule. In other words, any MV refinement examined by DMVR should satisfy the following two equations:

[0088] MV0' = MV0 + MV_offset (1)

[0089] MV1' = MV1 - MV_offset (2)

[0090] where MV_offset represents the refinement offset between the initial MV and the refined MV in one of the reference pictures. The refinement search range is two integer luma samples from the initial MV. The search includes an integer sample search phase and a fractional sample refinement phase.​

[0091] In some embodiments, in the integer sample search stage, SAD is checked for 21 integer sample positions as indicated Figure 7 including the integer sample position corresponding to the initial MV. Figure 7 is a block diagram illustrating some exemplary integer search candidates for DMVR mode according to some embodiments of the present disclosure. In Figure 7 the black center triangle is the integer sample position associated with the initial MV, and the white triangles are neighboring integer sample positions. The SAD of the initial MV pair is first calculated. The integer offset that minimizes the SAD value is selected as the integer sample offset in the integer sample search stage.

[0092] The integer sample search is followed by fractional sample refinement. To reduce the computational complexity, the fractional sample refinement is obtained by using a parametric error surface method instead of additional search using SAD comparison. In the parametric error surface based sub-pixel offset estimation, the center position cost and the costs of the four neighboring positions of the center are used to fit a 2-D parabolic error surface equation as follows:

[0093] E(x, y) = A(x - x min ) 2 + B(y - y min ) 2 + C (3)

[0094] where (x min , y min ) corresponds to the fractional position with the lowest cost, and C corresponds to the minimum cost value. By solving the above equation using the cost values of the five search points, (x min , y min ) is calculated as:

[0095] x min = (E(-1, 0) - E(1, 0)) / (2(E(-1, 0) + E(1, 0) - 2E(0, 0))) (4)

[0096] y min = (E(0, -1) - E(0, 1)) / (2((E(0, -1) + E(0, 1) - 2E(0, 0))) (5)

[0097] Since all cost values are positive and the minimum value is E(0, 0), the values of x min and y min are automatically limited between -8 and 8. The calculated fractional (x min , y min ) is added to the integer distance refined MV to obtain the sub-pixel precision refinement AMV.

[0098] In some embodiments, to further improve the prediction efficiency, an inter prediction filtering (InterPF) is applied which combines the intra prediction of a CU and the inter prediction based on the merge mode of the CU. Specifically, for each merge CU, an additional flag is signaled to indicate whether the InterPF mode is enabled for the current CU. When the flag is false, the InterPF is disabled and the regular inter prediction is applied to generate the prediction samples of the current CU. When the flag is true, the InterPF is applied to the current CU and another flag is signaled to further indicate which of the two InterPF modes is used. The two InterPF modes are described as follows:

[0099] InterPF mode one: the combined prediction samples are derived from the weighted average of the inter prediction samples and the five neighboring reconstructed samples from the left, right, above, and below, as shown in the following equation set (6):

[0100] Pred(x, y) = (Pred_inter(x, y) * 5 + Pred_Q(x, y) * 3) » 3

[0101] Pred_Q(x, y) = (Pred_V(x, y) + Pred_H(x, y) + 1) » 2

[0102] Pred_V(x, y) = ((h - 1 - y) * pRec(x, -1) + (y + 1) * Rec(-1, h) + (h » 1)) » log2(h)

[0103] Pred_H(x, y) = ((w - 1 - x) * pRec(-1, y) + (x + 1) * Rec(w, -1) + (w » 1)) » log2(w)

[0104] InterPF mode two: the combined prediction samples are derived from the weighted average of the inter prediction samples and the three neighboring reconstructed samples from the left and above:

[0105] P'(x, y) = f(x) * P(-1, y) + f(y) * P(x, -1) + (1 - f(x) - f(y)) * P(x, y) (7)

[0106] In (7), the weight parameters f(x) and f(y) are predefined as a lookup table (LUT) which is determined based on the size of the current CU and the intra prediction direction. Additionally, according to the existing AVS3 design, the InterPF mode is always disabled when the current CU is coded by the MMVD mode.

[0107] Although the MMVD mode can efficiently improve the efficiency of inter prediction, several aspects of its design can still be further improved. Specifically, the following issues in the current MMVD design in the AVS3 standard are found.

[0108] As previously mentioned, in the MMVD design, the allowed set of MVD offsets that can be selected for one MMVD CU is fixed, which includes 1 / 4-pel (pixel / picture element), 1 / 2-pel, 1-pel, 2-pel, and 4-pel as illustrated in Table 1. This design can not always be optimal for video contents with different characteristics. For example, video sequences with higher resolutions tend to have larger MVs for inter CUs compared to video sequences with lower resolutions like VGA and WQVGA. Accordingly, for video sequences with relatively large motion, in the MMVD mode, large MVD offsets are more often selected than small MVD offsets. Therefore, to maximize the coding gain of the MMVD mode, it is more beneficial to adaptively adjust the allowed MVD offset values according to the specific characteristics of the coded video.

[0109] In AVS3, the DMVR mode is only applied to CUs coded by the regular merge mode. For MMVD CUs, DMVR is always disabled. However, given that the MMVD mode is just an extension of the regular merge mode with signaled MVD, it can also be beneficial to apply DMVR to inter CUs coded in the MMVD mode. This combination can further improve the inter prediction efficiency of the MMVD mode without increasing the actual hardware codec design complexity.

[0110] As discussed above, the InterPF mode is always disabled when coding one current CU by the MMVD mode. However, similar to the regular merge mode, the derived MV in the MMVD mode can not be as accurate as the MV derived in the non-merge mode (in which the MV of one inter CU is directly signaled in the bitstream). In this case, the combination of the inter prediction samples of one MMVD CU and the intra prediction samples from its neighboring reconstructed samples can potentially improve the prediction accuracy and improve the coding efficiency. Based on this analysis, it would be beneficial to allow the combination of the InterPF mode and the MMVD mode.

[0111] In some embodiments, methods and systems are implemented to further improve the coding efficiency of the MMVD mode as disclosed herein. Specifically, the main aspects of the implemented methods are summarized as follows.

[0112] The signaling method is used to support adaptation of the allowed MVD offsets for MMVD mode at various coding levels. In addition, an encoder-side method is used to adaptively determine the optimal MVD offset to apply for MMVD mode.

[0113] A method is used in the DMVR process to further refine the MVs of MMVD mode.

[0114] A method is implemented to enable MMVD mode to be combined with InterPF mode.

[0115] In some embodiments, a method of adaptation of MVD offset values for MMVD mode is implemented.

[0116] In some MMVD designs, the allowed MVD offset values are fixed to 1 / 4-pel, 1 / 2-pel, 1-pel, 2-pel, and 4-pel. This design is suboptimal for video content with high resolution or severe motion. In these cases, the MVs tend to be much larger, making the above MVD offset values defined in MMVD mode possibly not optimal for capturing the true motion of those blocks. To further improve the coding performance of MMVD mode, adaptation of the MVD offsets allowed to be selected by MMVD mode is supported.

[0117] According to some embodiments, a number of MVD offset value sets can be predefined for MMVD mode, and the selected MVD offset set is allowed to vary over time based on the varying motion characteristics in the video content. As one example of the present disclosure, Table 3 shows two predefined MVD offset sets for entropy coding and their corresponding binarized codewords.

[0118]

[0119] Table 3: MVD offset sets for MMVD mode.

[0120] As can be seen from Table 3, the second set of MVD offset is a superset of the first set, and introduces three additional MVD offsets. The additional MVD offsets are 8-pel, 16-pel, and 32-pel. The newly added MVD offsets make the second set more suitable for coding video blocks with larger motion. In addition, in Table 3, different binarization codewords are assigned for each set to accommodate different probability distributions of MVD offset value usage. For example, the binarization used for the first set assigns shorter codewords for small MVD offsets, which makes the first set more efficient for coding pictures / regions of MMVD CUs with small MVD offsets that are frequently selected. Likewise, the second set of MVD offsets is more efficient for coding pictures / regions of MMVD CUs with medium MVD offset values that are frequently selected.

[0121] In the above example, only two sets of MVD offsets are allowed to be selected for MMVD mode. However, in practice, multiple sets of MVD offsets (i.e., more than two sets) and corresponding codewords can be derived offline according to the specific motion characteristics of different video sequences by the methods disclosed herein. The encoder can select the best set of MVD offsets and signal the corresponding index of the selected set to the decoder.

[0122] According to some embodiments, the adaptive selection of the predefined sets of MVD offsets is enabled by signaling one set identifier (or id) to indicate which set is selected. The id can be signaled at different coding levels (e.g., sequence level, picture level, and / or slice level) and at the same time at the signaling level (e.g., at the coding block group level and / or coding block level). The MMVD mode can only select MVD offsets in the identified set. Assuming the adaptation of allowed MVD offsets is performed at the picture level, Table 4 shows an example syntax table where the syntax element is signaled at the picture header.

[0123]

[0124] Table 4: Modified syntax table with MMVD offset control flag

[0125] mmvd_dist_set_id specifies the set of MVD offsets allowed for the selection of the MVD for the coding blocks coded in MMVD mode in the current picture. For the example shown in Table 3, where two sets are predefined, mmvd_dist_set_id becomes a flag. When the flag takes one value, e.g., 0, it indicates that the MVD offsets allowed for MMVD mode include ¼-pel, ½-pel, 1-pel, 2-pel, and 4-pel. When the flag takes the opposite value, e.g., 1, it indicates that the MVD offsets allowed for MMVD mode include ¼-pel, ½-pel, 1-pel, 2-pel, 4-pel, 8-pel, 16-pel, and 32-pel.

[0126] In Table 4, the variable MMVD Enable Flag is a control flag to indicate whether the MMVD tool is enabled for the coding blocks in the video sequence. The variable can be controlled by a control flag at the sequence level, e.g., in the sequence parameter set.

[0127] In another embodiment, instead of using fixed MVD offsets, the MVD offset values are directly signaled in the bitstream. This gives the encoder more freedom in determining the required MVD offset values for the current picture / slice on the fly.

[0128] In yet another embodiment, the selection of the MVD offset value set is determined in operation based on specific statistics, without explicitly signaling `mmvd_dist_set_id`. Such specific statistics may include, but are not limited to, a probability distribution of MVD offset values ​​used based on previously encoded / decoded images, stripes, and / or codec blocks. The selection of the MVD offset value set can be redetermined and / or updated at various frequency levels. For example, the selection can be redetermined and / or updated each time a CU is encoded / decoded in MMVD mode. In another example, the selection can be redetermined and / or updated whenever multiple CUs (e.g., 8 or 16) are encoded / decoded in MMVD mode. In yet another example, the selection of the allowed MVD offset set can be updated once after multiple CUs (e.g., 8 or 16) are encoded / decoded in inter-frame mode. In this case, the average / maximum / minimum MV of the inter-frame blocks can be calculated and used to determine which MVD offset set should be selected for subsequent codec blocks encoded / decoded in MMVD mode. According to this method, the set selection reset operation can be performed at some point, such as at the beginning of the current frame encoding / decoding, at the beginning of the current GOP (i.e., group of pictures) encoding / decoding, or when a frame that supports random access, such as an IDR frame (an IDR frame specifies that no frame after the IDR frame can reference any frame before it), is being encoded / decoded, etc., where the default MVD offset value set is always selected.

[0129] In some embodiments, encoder logic is implemented for selecting the optimal MVD offset for an MMVD mode. In the MMVD methods discussed above, multiple sets of MVD offsets can be selected by the MMVD mode. In some examples, a method is provided for adaptively selecting the optimal set of MVD offsets at the encoder end. Specifically, considering the strong content characteristic correlation between images in the same video sequence, the MVD offset selected by an inter-frame image may be very similar to the MVD offsets of its previously encoded and decoded neighboring images. Based on this consideration, the average MVD offset of the MMVD CUs of the previously encoded and decoded images is used to determine the set of MVD offsets for the MMVD mode in the current image. Assume that the two MVD offset sets illustrated in Table 3 are applied to the MMVD mode, and there are M (M is a positive integer) CUs encoded and decoded by the MMVD mode in the previous images with MVD offset values ​​V0, V1, ..., V... M-1 The average MVD offset of the MMVD mode in the previous image is calculated as follows:

[0130]

[0131] Based on (8), the index of the MVD offset set used by the current image is determined as:

[0132]

[0133] In some embodiments, DMVR mode is enabled for MMVD mode. In some MMVD designs, DMVR tends not to be applied together. However, because MMVD mode is an extended merge mode with additionally signaled MVD offset, allowing DMVR on top of MMVD can further improve the accuracy of MVs derived by MMVD mode. Additionally, the combination of two coding tools does not cause additional codec implementation complexity of hardware / software. Based on this consideration, the following two methods / embodiments are implemented to enable DMVR process for MMVD mode.

[0134] In the first embodiment, DMVR is applied to refine all MVs derived by MMVD mode as long as the initial MVs derived by MMVD mode are bi-directional and satisfy the common conditions to trigger DMVR process.

[0135] In the second embodiment, DMVR is applied to MMVD CUs only when the signaled MMVD offset is greater than a threshold. When the corresponding MMVD offset is equal to or smaller than the threshold, DMVR is always bypassed. In practice, to maximize the coding performance, different thresholds can be applied according to the specific characteristics of the coded video. In one example, a fixed threshold (e.g., 1-pel) is used. In another embodiment, the encoder adaptively determines the optimal threshold and signals it to the decoder at various coding levels (e.g., sequence level, picture level, slice level, and / or coding block level, etc.).

[0136] In practice, the second embodiment discussed above can provide better coding performance than the first embodiment. Based on the existing DMVR design, its search range is + / - 2-pel from the initial MV. However, as shown before, the allowed MVD offset used by MMVD mode can also cover this search range, and the selection of MVD offset in MMVD is based on more accurate and reliable rate-distortion optimization (RDO) at the encoder side rather than the decoder side bi-directional matching search in DMVR. Therefore, within a small local area around the initial MV, the MVD offset signaled by MMVD should be more accurate than that of DMVR. Disabling DMVR in this case can reduce the signaling overhead and improve the coding efficiency.

[0137] In some embodiments, InterPF is enabled for MMVD mode. As mentioned above, InterPF mode does not allow its inter prediction samples to come from MMVD mode. However, similar to regular merge mode, the MV derived from MMVD mode can not be accurate. For example, when this happens, there can be an undesirable discontinuity between the inter prediction samples of the current block and its neighboring reconstructed samples, which can lead to visible block artifacts along the neighboring block boundaries. In this case, the combination of the inter prediction samples of one MMVD CU with the intra prediction samples from its neighboring reconstructed samples can potentially reduce such artifacts and improve the prediction efficiency. Meanwhile, there is no increase in the complexity of hardware / software codec implementation when the combination of two coding tools is enabled. Based on this consideration, in one embodiment, InterPF mode is enabled for MMVD CUs, i.e., the combination of the inter prediction samples of one MMVD CU with the intra prediction samples from its neighboring reconstructed samples is enabled. Specifically, taking the InterPF design as an example (as shown in equations (6) and (7)), when this combination is enabled, a first control flag indicating whether MMVD mode is applied to one coding unit will be received at the decoder. In the case that MMVD is enabled for the coding unit, a second control flag will be received at the decoder to further specify whether InterPF mode is further applied to the MMVD coding unit. If the second control flag is 1 / enabled, a third control flag is signaled to select one of the two InterPF filters for the CU (as shown in equations (6) and (7)).

[0138] Figure 8 is a flowchart illustrating an exemplary process 800 of determining a motion vector difference (MVD) offset from a plurality of MVD offset sets when merge mode with MVD (MMVD) is enabled, according to some embodiments of the present disclosure.

[0139] The video decoder 30 receives a first control flag from the bitstream, where the first control flag indicates whether merge mode with motion vector difference (MMVD) is enabled for one or more coding units in a video sequence (810). In some embodiments, the first control flag is equal to 1 when MMVD is enabled for one or more coding units in the video sequence.

[0140] In accordance with a determination from the first control flag that MMVD is enabled, the video decoder 30 receives a first syntax element from the video data, where the first syntax element identifies one MVD offset set of a plurality of motion vector difference (MVD) offset sets (820).

[0141] The video decoder 30 receives, from the bitstream, a second control flag of video data, the second control flag corresponding to a respective coding unit of the one or more coding units, the second control flag indicating whether MMVD is applied to the respective coding unit (830). In some embodiments, the second control flag is equal to 1 when MMVD is applied to the respective coding unit.

[0142] In accordance with a determination that MMVD is applied to the respective coding unit from the second control flag, the video decoder 30 receives a second syntax element of the respective coding unit and a third syntax element of the respective coding unit, the second syntax element selecting a MVD offset from the identified set of MVD offsets, the third syntax element selecting a MVD direction corresponding to the selected MVD offset (840).

[0143] The video decoder 30 forms a MVD based on the selected MVD offset and the selected MVD direction (850).

[0144] The video decoder 30 reconstructs the respective coding unit by generating a motion vector of the respective coding unit using the formed MVD (860).

[0145] In some embodiments, the multiple sets of MVD offsets include a first set of MVD offsets and a second set of MVD offsets.

[0146] In some embodiments, the first set of MVD offsets is a subset of the second set of MVD offsets.

[0147] In some embodiments, a first set of binarized codewords is assigned to the first set of MVD offsets, and a second set of binarized codewords is assigned to the second set of MVD offsets. In some embodiments, the first set of binarized codewords is different from the second set of binarized codewords for a same offset value to accommodate different probability distributions of MVD offset value usage.

[0148] In some embodiments, the first set of MVD offsets includes offset values 1 / 4, 1 / 2, 1, 2, 4 in samples, and the second set of MVD offsets includes offset values 1 / 4, 1 / 2, 1, 2, 4, 8, 16, and 32 in samples.

[0149] In some embodiments, the first MVD offset set includes offset values 1 / 4, 1 / 2, 1, 2, 4 in samples, and binarized codewords 1, 01, 001, 0001, and 0000 corresponding to the offset values 1 / 4, 1 / 2, 1, 2, 4, respectively, and the second MVD offset set includes offset values 1 / 4, 1 / 2, 1, 2, 4, 8, 16, and 32 in samples, and binarized codewords 000, 001, 011, 010, 10, 110, 1110, and 1111 corresponding to the offset values 1 / 4, 1 / 2, 1, 2, 4, 8, 16, and 32, respectively.

[0150] In some embodiments, the first syntax element is signaled in one or more of different coding levels, including a sequence level, a picture level, and a slice level.

[0151] In another aspect, the process of combining InterPF mode and MMVD mode is disclosed as follows.

[0152] The video decoder 30 receives a first control flag from the bitstream, where the first control flag indicates whether a merge mode with motion vector difference (MMVD) is enabled for one or more coding units. In some embodiments, the first control flag is equal to 1 when the MMVD is enabled for the one or more coding units in the video sequence.

[0153] According to a determination from the first control flag that the MMVD is enabled, the video decoder 30 receives a second control flag from the bitstream for the video data, the second control flag corresponding to a respective coding unit of the one or more coding units, the second control flag indicating whether the MMVD is applied to the respective coding unit. In some embodiments, the second control flag is equal to 1 when the MMVD is applied to the respective coding unit.

[0154] According to a determination from the second control flag that the MMVD is applied to the respective coding unit, the video decoder 30 receives a third control flag from the video data, where the third control flag indicates whether an inter prediction filtering (InterPF) mode is enabled for the respective coding unit. In some embodiments, the third control flag is equal to 1 when the InterPF mode is enabled for the respective coding unit.

[0155] According to a determination from the third control flag that the InterPF mode is enabled for the respective coding unit, the video decoder 30 receives a fourth syntax element from the video data, where the fourth syntax element identifies one InterPF mode of a plurality of InterPF modes for the respective coding unit.

[0156] The video decoder 30 reconstructs the respective coding unit by applying the MMVD and the identified InterPF mode to video data corresponding to the respective coding unit.

[0157] In some embodiments, the plurality of InterPF modes includes at least two InterPF modes.

[0158] In one or more examples, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media can include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer- readable media generally can correspond to (1) tangible computer-readable storage media which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and / or data structures for implementation of the techniques described in this application. A computer program product can include a computer-readable medium.

[0159] The terminology used in the description of the implementations herein is for the purpose of describing particular implementations only and is not intended to limit the scope of the claims. As used in the description of the implementations and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “comprises” and / or “comprising,” when used in this specification, specify the presence of stated features, elements, and / or components, but do not preclude the presence or addition of one or more other features, elements, components, and / or groups thereof.

[0160] It will be also understood that, although the terms first, second, etc. can be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first electrode could be termed a second electrode, and, similarly, a second electrode could be termed a first electrode, without departing from the scope of the present implementations. The first electrode and the second electrode are both electrodes, but they are not the same electrode.

[0161] Throughout this specification, the use of the term "example" or "exemplary" means "serving as an example, instance, or illustration." Any implementation described herein as "example" or "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations. Likewise, the use of the terms "first," "second," or "third," etc. does not generally limit the number for such components. These terms are merely used to distinguish one element from another. Thus, these terms can be used interchangeably throughout this disclosure to refer to a same or different thing. Furthermore, the use of the terms "comprising," "including," "containing," or "having" and the like, are used herein to mean including at least the recited item, whereas the use of the terms "consisting of" and "consisting essentially of" are used to mean including at least the recited item and nothing more, unless otherwise noted. Unless otherwise noted, the use of the term "about" with respect to a recited value means that the value is ANACLOSE TO the recited value as would be understood by one of ordinary skill in the art. Unless otherwise noted, the use of the term "substantially" with respect to a recited value means that the value is ANACLOSE TO the recited value as would be understood by one of ordinary skill in the art.

[0162] The description of the application has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the application in the form disclosed. Many modifications, variations, and alternative embodiments will be apparent to those of ordinary skill in the art, in view of the foregoing description. The embodiments were chosen and described in order to best explain the principles of the application and the practical application, and to enable others skilled in the art to best utilize the application, and various embodiments with various modifications as are suited to the particular use contemplated. Therefore, it is intended that the scope of the application be defined by the following claims and their equivalents.

Claims

1. A method for encoding video data, the method comprising: In response to determining that MMVD (Multiplication Mode with Motion Vector Difference) is enabled for one or more coding units in a video sequence, Generate a first syntax element, wherein the first syntax element identifies one of a plurality of motion vector difference (MVD) offset sets, wherein the plurality of MVD offset sets include a first MVD offset set and a second MVD offset set, wherein a first binary codeword set is specified for the first MVD offset set, and a second binary codeword set is specified for the second MVD offset set, wherein for the same offset value, the codeword corresponding to the offset value in the first binary codeword set and the codeword corresponding to the offset value in the second binary codeword set are different; and In response to determining that the MMVD is applied to the corresponding coding unit in the one or more coding units, Generate a second syntax element and a third syntax element for the corresponding coding unit, wherein the second syntax element selects an MVD offset from the identified MVD offset set, and the third syntax element selects an MVD direction corresponding to the selected MVD offset; and The corresponding coding unit is reconstructed by generating the motion vector of the corresponding coding unit by applying the selected MVD offset and the selected MVD direction.

2. The method according to claim 1, wherein, The first MVD offset set is a subset of the second MVD offset set.

3. The method according to claim 1, wherein, The first MVD offset set includes offset values ​​of 1 / 4, 1 / 2, 1, 2, and 4 in units of samples.

4. The method according to claim 1, wherein, The second MVD offset set includes offset values ​​of 1 / 4, 1 / 2, 1, 2, 4, 8, 16, and 32 in sample units.

5. The method according to claim 1, wherein, The first MVD offset set includes offset values ​​of 1 / 4, 1 / 2, 1, 2, 4 in sample units, and binary codewords 1, 01, 001, 0001, and 0000 corresponding to offset values ​​of 1 / 4, 1 / 2, 1, 2, 4, respectively.

6. The method according to claim 1, wherein, The second MVD offset set includes offset values ​​of 1 / 4, 1 / 2, 1, 2, 4, 8, 16 and 32 in sample units, and binary codewords 000, 001, 011, 010, 10, 110, 1110 and 1111 corresponding to offset values ​​of 1 / 4, 1 / 2, 1, 2, 4, 8, 16 and 32, respectively.

7. The method according to claim 1, wherein, The first syntax element is signaled at one or more different codec levels, including sequence level, image level and stripe level.

8. The method according to claim 1, further comprising: A first control flag is generated, wherein the first control flag indicates whether the MMVD is enabled for the one or more coding units, and when the MMVD is enabled, the first control flag is equal to 1.

9. An electronic device comprising: One or more processing units; A memory coupled to the one or more processing units; as well as A plurality of programs stored in the memory, which, when executed by the one or more processing units, cause the electronic device to perform the method as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium storing a plurality of programs for execution by an electronic device having one or more processing units, wherein, When executed by the one or more processing units, the plurality of programs cause the electronic device to perform the method as described in any one of claims 1 to 8 to generate a video bitstream and transmit the video bitstream.

11. A computer program product comprising instructions for execution by a computing device having one or more processors, wherein, when executed by the one or more processors, the computing device performs the method as described in any one of claims 1 to 8.

12. A method for storing a bit stream, comprising: Perform the encoding method according to any one of claims 1 to 8 to generate a bit stream; as well as Store the bit stream.

Images