Model-based motion vector difference derivation and template matching prediction for video coding.

Abstract

An exemplary device for decoding video data includes a memory configured to store the video data and one or more processors, where the one or more processors are configured to decode data representing an initial motion vector for a current block of the video data, the initial motion vector having integer motion vector differential (MVD) precision, determine a search range around a reference area identified by the initial motion vector in a reference picture, perform a template matching search process in the search range to identify a best matching region, determine error values ​​of neighboring pixels to the best matching region, perform a model-based fractional-pixel motion vector refinement using the error values ​​of the neighboring pixels to derive motion vector difference values, apply at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block, and decode the current block using the improved motion vector.

Description

[Technical Field]

[0001] This application claims priority to U.S. Patent Application No. 17 / 586,492, filed on 27 January 2022, and U.S. Provisional Application No. 63 / 143,585, filed on 29 January 2021, the entire contents of these applications being incorporated herein by reference. U.S. Patent Application No. 17 / 586,492, filed on 27 January 2022, claims the benefit of U.S. Provisional Application No. 63 / 143,585, filed on 29 January 2021.

[0002] This disclosure relates to video coding, including video encoding and video decoding. [Background technology]

[0003] Digital video capabilities can be incorporated into a wide range of devices, including digital television, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite radio phones, so-called "smartphones," video teleconferencing devices, and video streaming devices. Digital video devices implement video coding techniques such as those described in standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265 / High Efficiency Video Coding (HEVC), and extensions of such standards. By implementing such video coding techniques, video devices can transmit, receive, encode, decode, and / or store digital video information more efficiently.

[0004] Video coding techniques include spatial (intra-picture) and / or temporal (inter-picture) predictions to reduce or eliminate redundancy inherent in video sequences. In block-based video coding, a video slice (e.g., a video picture, or a portion of a video picture) may be divided into video blocks, which may also be called coding tree units (CTUs), coding units (CUs), and / or coding nodes. A video block in an intra-coded (I) slice of a picture is coded using spatial predictions for reference samples in adjacent blocks within the same picture. A video block in an inter-coded (P or B) slice of a picture may use spatial predictions for reference samples in adjacent blocks within the same picture or temporal predictions for reference samples in other reference pictures. A picture may be called a frame, and a reference picture may be called a reference frame. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] U.S. Provisional Application No. 63 / 131,676 [Overview of the Initiative] [Means for solving the problem]

[0006] In general, this disclosure describes techniques for decoder-side motion vector derivation (DMVD). A video coder (such as a video encoder and / or video decoder) may be configured to perform DMVD using one or more different techniques, such as template matching, bilateral prediction, and decoder-side motion vector refinement. These techniques may be applied to existing video codecs, such as HEVC (High Efficiency Video Coding), VVC (Versatile Video Coding), Essential Video Coding (EVC), or other existing codec / video coding standards. These techniques may also be implemented in future video coding standards / codecs.

[0007] For example, a method for decoding video data includes the steps of: determining an initial motion vector for the current block of video data, wherein the initial motion vector has integer motion vector difference (MVD) precision; determining a search range around a reference area identified by the initial motion vector in a reference picture; performing a template matching search process within the search range to identify the best matching region; determining the error value of adjacent pixels for the best matching region; using the error value of adjacent pixels to perform a model-based fractional pixel motion vector improvement to derive a motion vector difference value; applying at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block; and decoding the current block using the improved motion vector.

[0008] In another example, a device for decoding video data includes a memory configured to store the video data and one or more processors implemented in circuitry, the one or more processors being configured to determine an initial motion vector for the current block of video data, the initial motion vector having integer motion vector difference (MVD) precision, determine a search range around a reference area identified by the initial motion vector in a reference picture, perform a template matching search process within the search range to identify the best matching region, determine the error value of adjacent pixels for the best matching region, use the error value of adjacent pixels to perform a model-based fractional pixel motion vector improvement to derive a motion vector difference value, apply at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block, and decode the current block using the improved motion vector.

[0009] In another example, a computer-readable storage medium stores instructions that, when executed, cause the device's processor to decode video data to determine an initial motion vector for the current block of video data, that the initial motion vector has integer motion vector difference (MVD) precision, determine a search range around a reference area identified by the initial motion vector in a reference picture, perform a template matching search process within the search range to identify the best matching region, determine the error values ​​of adjacent pixels for the best matching region, perform a model-based fractional pixel motion vector improvement using the error values ​​of adjacent pixels to derive motion vector difference values, apply at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block, and decode the current block using the improved motion vector.

[0010] In another example, a device for decoding video data includes means for determining an initial motion vector for a current block of video data, wherein the initial motion vector has integer motion vector difference (MVD) precision; means for determining a search range around a reference area identified by the initial motion vector in a reference picture; means for performing a template matching search process within the search range to identify the best matching region; means for determining error values ​​of adjacent pixels for the best matching region; means for using the error values ​​of adjacent pixels to perform a model-based fractional pixel motion vector improvement to derive motion vector difference values; means for applying at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block; and means for decoding the current block using the improved motion vector.

[0011] Details of one or more examples are described in the accompanying drawings and the following description. Other features, purposes, and advantages will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0012] [Figure 1] Block diagram shows an exemplary video encoding and decoding system capable of performing the techniques of this disclosure. [Figure 2A] This is a conceptual diagram illustrating an exemplary quadrow-binary tree (QTBT) structure. [Figure 2B] This is a conceptual diagram showing a coding tree unit (CTU) corresponding to the QTBT structure. [Figure 3A] This is a conceptual diagram showing candidate spatially adjacent motion vectors for merge mode. [Figure 3B] This is a conceptual diagram showing candidate spatially adjacent motion vectors for Advanced Motion Vector Prediction (AMVP). [Figure 4A] This is a conceptual diagram showing candidate time motion vector predictions (TMVP). [Figure 4B]This is a conceptual diagram illustrating motion vector scaling for TMVP. [Figure 5] This is a conceptual diagram illustrating template matching performed on a search area around a block using an initial motion vector. [Figure 6] This is a conceptual diagram illustrating an example of bilateral matching prediction where two motion vector difference (MVD) values ​​are proportional based on time distance. [Figure 7] This is a conceptual diagram illustrating an example of bilateral matching prediction where two MVDs are mirrored regardless of time distance. [Figure 8] This is a conceptual diagram showing a 3x3 square search pattern within the search range. [Figure 9] This is a conceptual diagram illustrating the Decoder-Side Motion Vector Improvement (DMVR). [Figure 10] This is a conceptual diagram illustrating the merge mode with motion vector difference (MMVD). [Figure 11] A block diagram illustrating a video encoder capable of performing the techniques of this disclosure. [Figure 12] A block diagram illustrating an exemplary video decoder capable of performing the techniques of this disclosure. [Figure 13] This flowchart shows an exemplary method for encoding the current block using the technique of the present disclosure. [Figure 14] This flowchart shows an exemplary method for decoding the current block using the technique of the present disclosure. [Figure 15] This flowchart shows an exemplary method for encoding or decoding video data using the technique of the present disclosure. [Modes for carrying out the invention]

[0013] Video coding (e.g., video encoding and / or video decoding) typically involves predicting blocks of video data from either already coded blocks of video data within the same picture (e.g., intra-prediction) or already coded blocks of video data within different pictures (e.g., inter-prediction). In some cases, the video encoder also calculates residual data by comparing the predicted blocks to the original blocks. Thus, the residual data represents the difference between the predicted blocks and the original blocks. To reduce the number of bits required to signal the residual data, the video encoder transforms and quantizes the residual data and signals the transformed and quantized residual data in the encoded bitstream. The compression achieved by the transformation and quantization processes can be irreversible, meaning that the transformation and quantization processes can introduce distortion into the decoded video data.

[0014] The video decoder decodes the residual data and adds it to the predicted blocks to produce reconstructed video blocks that better match the original video blocks than the predicted blocks alone. Due to losses resulting from the transformation and quantization of the residual data, the initial reconstructed blocks may have distortion or artifacts. One common type of artifact or distortion is called blockiness, which is the visibility of the boundaries of the blocks used to code the video data.

[0015] To further improve the quality of the decoded video, the video decoder can perform one or more filtering operations on the reconstructed video blocks. Examples of these filtering operations include deblocking filtering, sample-adaptive offset (SAO) filtering, and adaptive loop filtering (ALF). The parameters for these filtering operations may be determined by the video encoder and explicitly signaled in the encoded video bitstream, or they may be implicitly determined by the video decoder without the need for the parameters to be explicitly signaled in the encoded video bitstream.

[0016] This disclosure describes techniques relating to decoder-side motion vector derivation (DMVD) techniques, including template matching, bilateral matching, and decoder-side MV improvements. Although these techniques are referred to as being performed on the decoder side, the decoder-side motion vector derivation techniques described herein may also be performed by the video encoder, for example, to determine how to encode blocks of video data and to store the same reference data as the decoded reference data available to the video decoder. The techniques of this disclosure may be applied to any existing video codec such as High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and Essential Video Coding (EVC), or may be efficient tools in future video coding standards.

[0017] Various techniques of this disclosure can improve the performance of video codecs (encoders and decoders) and reduce the bitrate of coded video data. Generally, the techniques of this disclosure involve using DMVD to improve the initial motion vector for the current block of video data, which has integer precision. That is, the initial motion vector points to a fully integer pixel position rather than a fractional pixel position. Improvement of the initial motion vector may result in the improved motion vector having fractional pixel precision. The initial motion vector may be coded in the bitstream such that fewer bits are required to represent the initial motion vector in the bitstream (rather than requiring more bits for a motion vector with fractional pixel precision). Furthermore, encoding and decoding a fully integer precision motion vector may require fewer processing operations than encoding and decoding a motion vector with fractional pixel precision.

[0018] However, by refining the motion vector to potentially have fractional pixel precision, the improved motion vector becomes more accurate and therefore more precisely able to identify the predicted block for the current block. As a result, the residual for the current block can be reduced, thereby further reducing the bitrate of the bitstream, as well as the encoding and decoding operations required to encode or decode the residual data. Thus, these techniques can improve the operation of both the encoder and decoder and lower the bitrate of the bitstream containing the data representing the coded video sequence.

[0019] Furthermore, performing DMVD using integer pixel precision reduces the burden on the encoder and decoder compared to performing DMVD using fractional pixel precision. Specifically, to perform DMVD, the encoder or decoder performs a search in an area around the area identified by the initial motion vector. When the initial motion vector has integer pixel precision, the decoded pixels within that area can be used directly. In contrast, when the initial motion vector has sub-integer pixel precision (i.e., fractional pixel precision), the sub-integer pixel values ​​must be interpolated, thereby increasing the processing and memory consumption demands on the encoder and decoder.

[0020] Video coding standards include ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-T H.263, ISO / IEC MPEG-4 Visual, and ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC), which include their Scalable Video Coding (SVC) and Multi-view Video Coding (MVC) extensions.

[0021] In addition, High Efficiency Video Coding (HEVC), or ITU-T H.265, including its range extension, multiview extension (MV-HEVC), and scalable extension (SHVC), was developed by the Joint Collaboration Team on Video Coding (JCT-VC) and the Joint Collaboration Team on 3D Video Coding Extension Development (JCT-3V) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG). The HEVC draft specification, hereafter referred to as HEVC WD, is available from phenix.int-evry.fr / jct / doc_end_user / documents / 14_Vienna / wg11 / JCTVC-N1003-v1.zip.

[0022] The ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) are researching future video coding technologies that will have compression capabilities far exceeding those of the HEVC standard (including its current extensions and immediate extensions for screen content coding and high dynamic range coding). This group is working together on this research activity in a collaborative study known as the Joint Video Exploration Team (JVET) to evaluate the designs of compression technologies proposed by experts in this field. The latest version of the reference software, namely VVC Test Model 11.2 (VTM 11.2), is available from vcgit.hhi.fraunhofer.de / jvet / VVCSoftware_VTM. The Versatile Video Coding (VVC) draft specification is called Document JVET-T2001. The algorithm description for Versatile Video Coding and Test Model 11 (VTM11.0) is called Document JVET-T2002.

[0023] Figure 1 is a block diagram showing an exemplary video coding and decoding system 100 capable of performing the techniques of the present disclosure. The techniques of the present disclosure generally pertain to coding (encoding and / or decoding) video data. Generally, video data includes any data for processing video. Thus, video data may include raw, uncoded video, coded video, decoded (e.g., reconstructed) video, and video metadata such as signaling data.

[0024] As shown in Figure 1, in this example, system 100 includes a source device 102 that provides encoded video data to be decoded and displayed by a destination device 116. Specifically, the source device 102 provides video data to the destination device 116 via a computer-readable medium 110. The source device 102 and destination device 116 may comprise any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, mobile devices, tablet computers, set-top boxes, telephone handsets such as smartphones, televisions, cameras, display devices, digital media players, video gaming consoles, and video streaming devices. In some cases, the source device 102 and destination device 116 may be wireless communication devices and therefore may be called wireless communication devices.

[0025] In the example in Figure 1, the source device 102 includes a video source 104, memory 106, a video encoder 200, and an output interface 108. The destination device 116 includes an input interface 122, a video decoder 300, memory 120, and a display device 118. According to this disclosure, the video encoder 200 of the source device 102 and the video decoder 300 of the destination device 116 may be configured to apply techniques for model-based motion vector difference derivation and template matching prediction. Thus, the source device 102 represents an example of a video encoding device, while the destination device 116 represents an example of a video decoding device. In other examples, the source and destination devices may include other components or configurations. For example, the source device 102 may receive video data from an external video source, such as an external camera. Similarly, the destination device 116 may interface with an external display device rather than including an integrated display device.

[0026] System 100, as shown in Figure 1, is merely an example. In general, any digital video coding and / or decoding device may perform model-based motion vector difference derivation and template matching prediction techniques. Source device 102 and destination device 116 are merely examples of coding devices, such that source device 102 generates coded video data for transmission to destination device 116. This disclosure refers to a “coding” device as a device that performs coding (encoding and / or decoding) of data. Thus, video encoder 200 and video decoder 300 represent examples of coding devices, specifically video encoder and video decoder, respectively. In some examples, source device 102 and destination device 116 may operate substantially symmetrically, such that each of source device 102 and destination device 116 includes video coding and decoding components. Thus, system 100 may support one-way or two-way video transmission between source device 102 and destination device 116 for, for example, video streaming, video playback, video broadcasting, or video phone.

[0027] Generally, the video source 104 represents a source of video data (i.e., raw, uncoded video data), providing the video encoder 200 with a sequence of pictures (also called "frames") of video data, which then encodes the data for the pictures. The video source 104 of source device 102 may include a video feed interface for receiving video from a video capture device, such as a video camera, a video archive containing previously captured raw video, and / or a video content provider. As a further alternative, the video source 104 may generate computer graphics-based data as source video, or a combination of live video, archived video, and computer-generated video. In each case, the video encoder 200 encodes the captured, pre-captured, or computer-generated video data. The video encoder 200 can rearrange the pictures from the order in which they were received (sometimes called the "display order") to the coding order for encoding. The video encoder 200 may generate a bitstream containing the encoded video data. The source device 102 may then output the encoded video data to the computer-readable medium 110 via the output interface 108 for reception and / or retrieval, for example, via the input interface 122 of the destination device 116.

[0028] Memory 106 of source device 102 and memory 120 of destination device 116 represent general-purpose memory. In some examples, memories 106 and 120 may store raw video data, for example, raw video from video source 104 and raw decoded video data from video decoder 300. Additional or alternative, memories 106 and 120 may store software instructions that can be executed by, for example, video encoder 200 and video decoder 300, respectively. Although memories 106 and 120 are shown separately from video encoder 200 and video decoder 300 in this example, it should be understood that video encoder 200 and video decoder 300 may also include internal memory for functionally similar or equivalent purposes. Furthermore, memories 106 and 120 may store encoded video data, for example, output from video encoder 200 and input to video decoder 300. In some examples, portions of memory 106, 120 may be allocated as one or more video buffers to store, for example, raw, decoded, and / or encoded video data.

[0029] The computer-readable medium 110 may represent any type of medium or device capable of transferring encoded video data from the source device 102 to the destination device 116. For example, the computer-readable medium 110 may represent a communication medium that enables the source device 102 to directly transmit encoded video data to the destination device 116 in real time, for example, over a radio frequency network or a computer-based network. The output interface 108 can modulate the transmitted signal containing the encoded video data, and the input interface 122 can demodulate the received transmitted signal according to a communication standard such as a wireless communication protocol. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful to facilitate communication from the source device 102 to the destination device 116.

[0030] In some examples, source device 102 may output encoded data to storage device 112 via output interface 108. Similarly, destination device 116 may access encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, Blu-ray® disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0031] In some examples, the source device 102 may output the encoded video data to a file server 114 or another intermediate storage device capable of storing the encoded video data generated by the source device 102. The destination device 116 may access the stored video data from the file server 114 via streaming or download.

[0032] The file server 114 can be any type of server device capable of storing encoded video data and transmitting that encoded video data to the destination device 116. The file server 114 may represent a web server (for example, for a website), a server configured to provide a file transfer protocol service (such as the File Transfer Protocol (FTP) or File Delivery over Unidirectional Transport (FLUTE) protocol), a Content Delivery Network (CDN) device, a Hypertext Transfer Protocol (HTTP) server, a Multimedia Broadcast Multicast Service (MBMS) or Enhanced MBMS (eMBMS) server, and / or a Network Attached Storage (NAS) device. The file server 114 may, in addition or alternatively, implement one or more HTTP streaming protocols such as Dynamic Adaptive Streaming over HTTP (DASH), HTTP Live Streaming (HLS), Real Time Streaming Protocol (RTSP), or HTTP Dynamic Streaming.

[0033] The destination device 116 may access the encoded video data from the file server 114 through any standard data connection, including an internet connection. This may include wireless channels (e.g., Wi-Fi connection), wired connections (e.g., digital subscriber line (DSL), cable modem, etc.), or a combination of both suitable for accessing the encoded video data stored in the file server 114. The input interface 122 may be configured to operate according to one or more of the various protocols discussed above for retrieving or receiving media data from the file server 114 or other such protocols for retrieving media data.

[0034] The output interface 108 and input interface 122 may represent a wireless transmitter / receiver, a modem, a wired networking component (e.g., an Ethernet card), a wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In examples where the output interface 108 and input interface 122 include wireless components, they may be configured to transfer data such as encoded video data according to cellular communication standards such as 4G, 4G-LTE (Long-Term Evolution), LTE Advanced, or 5G. In some examples where the output interface 108 includes a wireless transmitter, the output interface 108 and input interface 122 may be configured to transfer data such as encoded video data according to other wireless standards such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee®), or the Bluetooth® standard. In some examples, the source device 102 and / or destination device 116 may include their respective system-on-chip (SoC) devices. For example, the source device 102 may include an SoC device for performing functions related to the video encoder 200 and / or the output interface 108, and the destination device 116 may include an SoC device for performing functions related to the video decoder 300 and / or the input interface 122.

[0035] The techniques of this disclosure can be applied to video coding that supports any of a variety of multimedia applications, such as television broadcasting by radio waves, cable television transmission, satellite television transmission, internet streaming video transmission such as dynamic adaptive streaming over HTTP (DASH), digital video encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications.

[0036] The input interface 122 of the destination device 116 receives an encoded video bitstream from a computer-readable medium 110 (e.g., a communication medium, a storage device 112, a file server 114, etc.). The encoded video bitstream may include signaling information defined by the video encoder 200, which is also used by the video decoder 300, such as syntax elements having values ​​that describe the characteristics and / or processing of video blocks or other coded units (e.g., slices, pictures, picture groups, sequences, etc.). The display device 118 displays the decoded picture of the decoded video data to the user. The display device 118 may represent any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0037] Although not shown in Figure 1, in some examples, the video encoder 200 and video decoder 300 may be integrated with an audio encoder and / or audio decoder, respectively, and may include an appropriate MUX-DEMUX unit or other hardware and / or software to handle multiplexed streams containing both audio and video in a common data stream. Where applicable, the MUX-DEMUX unit may comply with the ITU H.223 Multiplexer Protocol or other protocols such as the User Datagram Protocol (UDP).

[0038] The video encoder 200 and video decoder 300 can each be implemented as one or more suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the technique is partially implemented in software, the device may store instructions for the software in a suitable non-temporary computer-readable medium and execute those instructions in hardware using one or more processors to perform the technique of the Disclosure. Each of the video encoder 200 and video decoder 300 may be included in one or more encoders or decoders, any of which may be integrated as part of a composite encoder / decoder (codec) within their respective devices. A device including the video encoder 200 and / or video decoder 300 may comprise an integrated circuit, a microprocessor, and / or a wireless communication device such as a mobile phone.

[0039] The video encoder 200 and video decoder 300 may operate in accordance with video coding standards such as ITU-T H.265, also referred to as High Efficiency Video Coding (HEVC), or extensions thereof such as multiview and / or scalable video coding extensions. Alternatively, the video encoder 200 and video decoder 300 may operate in accordance with other proprietary or industry standards, such as Versatile Video Coding (VVC). The draft of the VVC standard is described in Bross et al., "Versatile Video Coding (Draft 9)," ITU-T SG 16 WP 3 and the Joint Video Experts Team (JVET) of ISO / IEC JTC 1 / SC 29 / WG 11, 18th meeting, April 15-24, JVET-R2001-v8 (hereinafter "VVC Draft 9"). However, the techniques of this disclosure are not limited to any particular coding standard.

[0040] Generally, the video encoder 200 and video decoder 300 may perform block-based coding of pictures. The term “block” generally refers to a structure containing data to be processed (e.g., to be coded, coded, or otherwise used in the coding and / or decoding process). For example, a block may contain a two-dimensional matrix of samples of luminance and / or chromaticity data. Generally, the video encoder 200 and video decoder 300 may code video data represented in YUV (e.g., Y, Cb, Cr) format. That is, rather than coding red, green, and blue (RGB) data for samples of a picture, the video encoder 200 and video decoder 300 may code luminance and chromaticity components, and the chromaticity component may include both red and blue chromaticity components. In some examples, the video encoder 200 converts the received RGB-formatted data to a YUV representation before coding, and the video decoder 300 converts the YUV representation to RGB format. Alternatively, pre-processing and post-processing units (not shown) may perform these conversions.

[0041] This disclosure may refer to coding a picture (e.g., encoding and decoding) in general, including the process of encoding or decoding the data of the picture. Similarly, this disclosure may refer to coding a block of a picture, including the process of encoding or decoding the data for the block, including predictive and / or residual coding. An encoded video bitstream generally contains a set of values ​​for syntax elements that represent the coding decision (e.g., coding mode) and the division of the picture into blocks. Thus, references to coding a picture or a block should generally be understood as coding values ​​for the syntax elements that make up the picture or block.

[0042] HEVC defines various blocks, including coding units (CUs), prediction units (PUs), and transformation units (TUs). According to HEVC, a video coder (such as video encoder 200) divides coding tree units (CTUs) into CUs according to a quadtree structure. That is, the video coder divides the CTUs and CUs into four equal, non-overlapping squares, and each node in the quadtree has either zero or four child nodes. Nodes without child nodes are sometimes called "leaf nodes," and the CU of such a leaf node may contain one or more PUs and / or one or more TUs. The video coder may further divide the PUs and TUs. For example, in HEVC, a residual quadtree (RQT) represents a division of TUs. In HEVC, PUs represent intra-predicted data, while TUs represent residual data. Intra-predicted CUs contain intra-predicted information, such as intra-mode indications.

[0043] As another example, a video encoder 200 and a video decoder 300 may be configured to operate according to VVC. According to VVC, a video coder (such as the video encoder 200) divides a picture into multiple coding tree units (CTUs). The video encoder 200 may divide the CTUs according to a tree structure such as a quadwood-binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure eliminates the concept of multiple division types, such as the separation of CUs, PUs, and TUs in HEVC. The QTBT structure includes two levels: a first level divided according to quadwood divisions and a second level divided according to binary tree divisions. The root node of the QTBT structure corresponds to a CTU. The leaf nodes of the binary tree correspond to coding units (CUs).

[0044] In MTT partitioning structures, blocks can be partitioned using quadtree (QT) partitions, binary tree (BT) partitions, and one or more types of triple tree (TT) (also called ternary tree (TT)) partitions. A triple tree or ternary tree partition is a partition in which a block is divided into three subblocks. In some examples, a triple tree or ternary tree partition divides a block into three subblocks without dividing the original block through a center. The partition types in MTT (e.g., QT, BT, and TT) can be symmetric or asymmetric.

[0045] In some examples, the video encoder 200 and video decoder 300 may use a single QTBT or MTT structure to represent the luminance component and the chromaticity component, respectively, while in other examples, the video encoder 200 and video decoder 300 may use two or more QTBT or MTT structures, such as one QTBT / MTT structure for the luminance component and another QTBT / MTT structure for both chromaticity components (or two QTBT / MTT structures for each chromaticity component).

[0046] The video encoder 200 and video decoder 300 may be configured to use a quadtree partition, QTBT partition, MTT partition, or other partition structure according to HEVC. For illustrative purposes, the description of the technique in this disclosure is presented in relation to the QTBT partition. However, it should be understood that the technique in this disclosure may also be applicable to video coders configured to use a quadtree partition or other types of partitions.

[0047] In some examples, a CTU includes a coding tree block (CTB) of a lumen sample, two corresponding CTBs of a chroma sample of a picture having three sample arrays, or a CTB of a sample of a picture coded using three separate color planes and syntax structures used to code a monochrome picture or sample. A CTB can be an N×N block of samples for some value N, such that the division of components into the CTB is a partition. Components can be an array from one of three arrays (lumen and two chroma) of a picture in a 4:2:0, 4:2:2, or 4:4:4 color format, or a single sample from that array, or an array of a picture in a monochrome format, or a single sample from that array. In some examples, a coding block is an M×N block of samples for some values ​​M and N, such that the division of the CTB into the coding block is a partition.

[0048] Blocks (e.g., CTUs or CUs) can be grouped in various ways within a picture. For example, a brick may refer to a rectangular area of ​​a row of CTUs within a particular tile in a picture. A tile can be a rectangular area of ​​CTUs within a particular tile column or row in a picture. A tile column refers to a rectangular area of ​​CTUs with a height equal to the height of the picture and a width specified by a syntax element (e.g., in a picture parameter set). A tile row refers to a rectangular area of ​​CTUs with a height specified by a syntax element (e.g., in a picture parameter set) and a width equal to the width of the picture.

[0049] In some examples, a tile may be divided into multiple bricks, each brick containing one or more CTU rows within the tile. A tile that is not divided into multiple bricks may also be called a brick. However, a brick that is a true subset of a tile may not be called a tile.

[0050] Bricks within a picture can also be arranged as slices. A slice can be an integer number of picture bricks that can exclusively reside in a single Network Abstraction Layer (NAL) unit. In some examples, a slice may consist of either a certain number of complete tiles or a continuous sequence of complete bricks of a single tile.

[0051] This disclosure may use "N×N" and "N to N" interchangeably to refer to the sample dimensions of a block (such as a CU or other video block) with respect to vertical and horizontal dimensions, for example, 16×16 samples or 16 to 16 samples. Generally, a 16×16 CU has 16 samples vertically (y=16) and 16 samples horizontally (x=16). Similarly, an N×N CU generally has N samples vertically and N samples horizontally, where N represents a non-negative integer. The samples in a CU can be arranged as rows and columns. Furthermore, a CU does not necessarily have to have the same number of samples horizontally as vertically. For example, a CU may have N×M samples, where M is not necessarily equal to N.

[0052] The video encoder 200 encodes video data for the CU, representing prediction information and / or residual information, as well as other information. The prediction information indicates how the CU should be predicted in order to form a prediction block for the CU. The residual information generally represents the sample-by-sample difference between the CU sample before encoding and the prediction block.

[0053] To predict a CU (Critical Unit), the video encoder 200 can generally form prediction blocks for the CU through inter-prediction or intra-prediction. Inter-prediction generally refers to predicting the CU from data of a previously coded picture, while intra-prediction generally refers to predicting the CU from previously coded data of the same picture. To perform inter-prediction, the video encoder 200 can generate prediction blocks using one or more motion vectors. The video encoder 200 can generally perform motion search to identify a reference block that closely matches the CU, for example, with respect to the difference between the CU and the reference block. The video encoder 200 can calculate a difference metric using absolute difference sum (SAD), squared difference sum (SSD), mean absolute difference (MAD), mean squared difference (MSD), or other such difference calculations to determine whether the reference block closely matches the current CU. In some examples, the video encoder 200 can predict the current CU using unidirectional or bidirectional prediction.

[0054] Some examples of VVC also offer an affine motion compensation mode, which can be considered an interpredictive mode. In affine motion compensation mode, the video encoder 200 may determine two or more motion vectors representing non-translational motion, such as zooming in or zooming out, rotation, projection motion, or other irregular motion types.

[0055] To perform intra-prediction, the video encoder 200 may select an intra-prediction mode to generate prediction blocks. Several examples of VVCs provide 67 intra-prediction modes, including various directional modes, as well as planar and DC modes. Generally, the video encoder 200 selects an intra-prediction mode that describes adjacent samples to the current block (e.g., a block of CUs) from which samples of the current block should be predicted. Such samples can generally be above, above and to the left of, or to the left of, the current block, in the same picture as the current block, assuming that the video encoder 200 codes CTUs and CUs in raster scanning order (left to right, top to bottom).

[0056] The video encoder 200 encodes data representing the prediction mode for the current block. For example, in interprediction mode, the video encoder 200 may encode data representing which of the various available interprediction modes is used, as well as motion information for the corresponding mode. For unidirectional or bidirectional interprediction, for example, the video encoder 200 may encode motion vectors using advanced motion vector prediction (AMVP) mode or merge mode. The video encoder 200 may encode motion vectors for affine motion compensation mode using similar modes.

[0057] According to predictions such as intra-prediction or inter-prediction of a block, the video encoder 200 may calculate residual data for the block. Residual data, such as a residual block, represents the sample-by-sample difference between the block and the predicted block for that block formed using the corresponding prediction mode. The video encoder 200 may apply one or more transformations to the residual block to generate transformed data in the transformation region instead of the sample region. For example, the video encoder 200 may apply a discrete cosine transform (DCT), integer transform, wavelet transform, or a conceptually similar transform to the residual video data. In addition, the video encoder 200 may apply secondary transformations such as a mode-dependent non-separable secondary transform (MDNSST), signal-dependent transform, or Carunen-Löwe ​​transform (KLT) following the initial transformation. The video encoder 200 generates transformation coefficients following the application of one or more transformations.

[0058] As described above, following any transformation to generate the transformation coefficients, the video encoder 200 may perform quantization of the transformation coefficients. Quantization generally refers to the process of quantizing the transformation coefficients to reduce the amount of data used to represent them as much as possible, thereby achieving further compression. By performing the quantization process, the video encoder 200 may reduce the bit depth associated with some or all of the transformation coefficients. For example, the video encoder 200 may round an n-bit value to an m-bit value during quantization, where n is greater than m. In some examples, to perform quantization, the video encoder 200 may perform a bitwise right shift of the value to be quantized.

[0059] Following quantization, the video encoder 200 may scan the transformation coefficients and generate a one-dimensional vector from a two-dimensional matrix containing the quantized transformation coefficients. The scan may be designed to place higher-energy (and therefore lower-frequency) transformation coefficients at the beginning of the vector and lower-energy (and therefore higher-frequency) transformation coefficients at the end. In some examples, the video encoder 200 may generate a serialized vector using a predetermined scan order for scanning the quantized transformation coefficients, and then entropy-encode the quantized transformation coefficients of the vector. In other examples, the video encoder 200 may perform adaptive scanning. After scanning the quantized transformation coefficients to form a one-dimensional vector, the video encoder 200 may entropy-encode the one-dimensional vector, for example, according to context-adaptive binary arithmetic coding (CABAC). The video encoder 200 may also entropy-encode values ​​for syntax elements that describe metadata associated with the encoded video data for use by the video decoder 300 when decoding the video data.

[0060] To perform CABAC, the video encoder 200 may assign a context within a context model to the symbols to be transmitted. The context may relate, for example, to whether the adjacent values ​​of the symbols are 0. Probability decisions may be based on the context assigned to the symbols.

[0061] The video encoder 200 may further generate syntax data for the video decoder 300, such as block-based syntax data, picture-based syntax data, and sequence-based syntax data, for example, in the form of a picture header, block header, slice header, or other syntax data such as a sequence parameter set (SPS), picture parameter set (PPS), or video parameter set (VPS). The video decoder 300 may similarly decode such syntax data to determine how to decode the corresponding video data.

[0062] In this way, the video encoder 200 can generate a bitstream containing encoded video data, for example, syntax elements describing the division of a picture into blocks (e.g., CUs) and prediction and / or residual information for those blocks. Finally, the video decoder 300 can receive the bitstream and decode the encoded video data.

[0063] Generally, the video decoder 300 performs the reverse process of the process performed by the video encoder 200 in order to decode the encoded video data of the bitstream. For example, the video decoder 300 may decode values ​​for syntax elements of the bitstream using CABAC in a substantially similar manner to, but reverse, the CABAC encoding process of the video encoder 200. The syntax elements may define partitioning information for partitioning a picture into CTUs and partitions for each CTU according to a corresponding partitioning structure such as a QTBT structure, in order to define the CUs of CTUs. The syntax elements may further define prediction and residual information for blocks of video data (e.g., CUs).

[0064] Residual information may be represented, for example, by quantized transformation coefficients. The video decoder 300 may reconstruct the residual block relative to the block by inverse quantizing and inverse transforming the quantized transformation coefficients of the block. The video decoder 300 uses the signaled prediction mode (intra-prediction or inter-prediction) and associated prediction information (e.g., motion information for inter-prediction) to form a predicted block relative to the block. The video decoder 300 may then combine the predicted block and the residual block (sample by sample) to reconstruct the original block. The video decoder 300 may perform additional processing, such as performing a deblocking process, to reduce visual artifacts along the block boundaries.

[0065] This disclosure may generally refer to “signaling” any information, such as syntax elements. The term “signaling” may generally refer to the communication of values ​​for syntax elements and / or other data used to decode the encoded video data. That is, the video encoder 200 may signal values ​​for syntax elements in the bitstream. In general, signaling refers to generating values ​​in the bitstream. As stated above, the source device 102 may transfer the bitstream to the destination device 116 substantially in real time or non-real time, which may occur, for example, when the destination device 116 stores the syntax elements in the storage device 112 for later retrieval.

[0066] Figures 2A and 2B are conceptual diagrams showing an exemplary quadtree-binary tree (QTBT) structure 130 and its corresponding coding tree unit (CTU) 132. Solid lines represent quadtree partitions, and dotted lines represent binary tree partitions. At each partition (i.e., non-leaf) node of the binary tree, one flag is signaled to indicate which partition type (i.e., horizontal or vertical) is used; in this example, 0 indicates a horizontal partition and 1 indicates a vertical partition. In a quadtree partition, the quadtree node divides a block horizontally and vertically into four subblocks of equal size, so there is no need to indicate the partition type. Thus, the video encoder 200 can encode syntax elements (such as partition information) for the domain tree level (i.e., solid lines) of the QTBT structure 130 and the video decoder 300 can decode syntax elements (such as partition information) for the predictive tree level (i.e., dashed lines) of the QTBT structure 130. The video encoder 200 can encode video data such as prediction data and transformation data for CUs represented by terminal leaf nodes of the QTBT structure 130, and the video decoder 300 can decode it.

[0067] In general, CTU132 in Figure 2B can be associated with parameters that define the size of the blocks corresponding to the nodes of the QTBT structure 130 at the first and second levels. These parameters may include the CTU size (representing the size of CTU132 in the example), the minimum quadtree size (MinQTSize, representing the smallest allowed quadtree leaf node size), the maximum binary tree size (MaxBTSize, representing the largest allowed binary tree root node size), the maximum binary tree depth (MaxBTDepth, representing the largest allowed binary tree depth), and the minimum binary tree size (MinBTSize, representing the smallest allowed binary tree leaf node size).

[0068] The root node of a QTBT structure corresponding to a CTU may have four child nodes at the first level of the QTBT structure, each of which may be partitioned according to a quadtree partition. That is, a node at the first level is either a leaf node (having no child nodes) or has four child nodes. An example of QTBT structure 130 represents such a node, including a parent node and child nodes with solid lines for branching. If a node at the first level is not larger than the maximum allowable binary tree root node size (MaxBTSize), the node may be further partitioned by its respective binary tree. Binary tree partitioning of a single node may be repeated until the resulting node reaches the minimum allowable binary tree leaf node size (MinBTSize) or the maximum allowable binary tree depth (MaxBTDepth). An example of QTBT structure 130 represents such a node, with dashed lines for branching. Binary tree leaf nodes are called coding units (CUs), which are used for prediction (e.g., in-picture or between-picture predictions) and transformations without further partitioning. As discussed above, CU can also be called “video block” or “block”.

[0069] In an example of a QTBT partitioned structure, the CTU size is set to 128x128 (a chroma sample and two corresponding 64x64 chroma samples), MinQTSize is set to 16x16, MaxBTSize is set to 64x64, MinBTSize (for both width and height) is set to 4, and MaxBTDepth is set to 4. To generate a quadtree leaf node, the quadtree partition is first applied to the CTU. The quadtree leaf node can have sizes ranging from 16x16 (i.e., MinQTSize) to 128x128 (i.e., CTU size). If the quadtree leaf node is 128x128, it is not further partitioned by a binary tree because its size exceeds MaxBTSize (i.e., 64x64 in this example). Otherwise, the quadtree leaf node can be further partitioned by a binary tree. Therefore, a quadtree leaf node is also the root node of a binary tree and has a binary tree depth of 0. When the binary tree depth reaches MaxBTDepth (4 in this example), no further partitioning is allowed. The fact that a binary tree node has a width equal to MinBTSize (4 in this example) suggests that no further vertical partitioning (i.e., partitioning by width) is allowed for that binary tree node. Similarly, the fact that a binary tree node has a height equal to MinBTSize suggests that no further horizontal partitioning (i.e., partitioning by height) is allowed for that binary tree node. As mentioned above, a leaf node of a binary tree is called a CU and is further processed according to prediction and transformation without further partitioning.

[0070] In HEVC, the largest coding unit within a slice is the CTB, also known as the CTU. A CTB contains a quadtree whose nodes are CUs. (Technically, an 8x8 CTB size can be supported.) CTB sizes can range from 16x16 to 64x64 in the HEVC main profile. CUs can be the same size as a CTB or as small as 8x8. Each CU is coded in one coding mode, for example, inter or intra. When a CU is intercoded, it may be further divided into two or four PUs, or it may be a single PU if no further division is applied. When there are two PUs within a single CU, the two PUs may be rectangles half the size of the CU, or two rectangles one-quarter or three-quarters the size of the CU.

[0071] Figures 3A and 3B are conceptual diagrams showing spatially adjacent motion vector candidates for merge mode and advanced motion vector prediction (AMVP). When CUs are intercoded, each PU has one set of motion information, which is derived using a unique inter-prediction mode. In the HEVC standard, there are two inter-prediction modes for a PU, named merge mode (skipping is considered a special case of merging) and advanced motion vector prediction (AMVP) mode, respectively.

[0072] In either AMVP mode or merge mode, the video encoder 200 and video decoder 300 may be configured to maintain a list of motion vector (MV) candidates for multiple motion vector predictors. The video encoder 200 and video decoder 300 may be configured to generate the motion vector of the current PU, as well as the reference index in merge mode, by taking one candidate from the MV candidate list.

[0073] In HEVC, the MV candidate list contains up to five candidates for merge mode and only two candidates for AMVP mode. A merge candidate may contain a set of motion information, such as motion vectors corresponding to both the reference picture list (lists 0 and 1) and the reference index. When a merge candidate is identified by the merge index, the video encoder 200 and video decoder 300 may be configured to determine the reference picture, as well as the associated motion vector, to be used for prediction of the current block. On the other hand, under AMVP mode for each potential prediction direction from either list 0 or list 1, the reference index is explicitly signaled along with the MV predictor (MVP) index to the MV candidate list, since the AMVP candidate contains only motion vectors. In AMVP mode, the predicted motion vector may be further refined. Candidates for both modes are similarly derived from the same spatially and temporally adjacent blocks.

[0074] Figures 3A and 3B are conceptual diagrams showing spatial adjacent motion vector candidates for merge mode and advanced motion vector prediction (AMVP). Spatial MV candidates are derived from adjacent blocks (shown in Figures 3A and 3B) for a given PU (PU0), but the process for generating candidates from blocks differs between merge mode and AMVP mode.

[0075] In merge mode, up to four spatial MV candidates can be derived for PU0 140 in the order shown in Figure 3A. The order is left (0), top (1), top right (2), bottom left (3), and top left (4).

[0076] As shown in Figure 3B, in AVMP mode, the adjacent blocks of block PU0 142 are divided into two groups: the left group containing blocks 0 and 1, and the upper group containing blocks 2, 3, and 4. For each group, potential candidates among the adjacent blocks that reference the same reference picture as the reference picture indicated by the signaled reference index have the highest priority when selected to form the final candidate for the group. It is possible that not all adjacent blocks contain motion vectors pointing to the same reference picture. Therefore, if no such candidate can be found, the difference in time distance can be compensated for by scaling the first available candidate to form the final candidate.

[0077] Figures 4A and 4B are conceptual diagrams showing time-motion vector predictor (TMVP) candidates and motion vector scaling for TMVP. The video encoder 200 and video decoder 300 may be configured to add time-motion vector predictor (TMVP) candidates to the MV candidate list after spatial motion vector candidates, if enabled and TMVP candidates are available. The process of deriving motion vectors for TMVP candidates is the same for both merge mode and AMVP mode. However, in HEVC, the target reference index for TMVP candidates in merge mode is set to 0.

[0078] Figure 4A shows an exemplary TMVP candidate for block PU0 156, and Figure 4B shows the motion vector scaling process. The primary block location for TMVP candidate derivation is the lower right block outside the collocated PU. This candidate is shown as block T150 in Figure 4A. The location of block T150 is used to compensate for bias to the blocks above and to the left used to generate spatial adjacency candidates. However, if block T150 is located outside the current CTB row, or if motion information is not available, block T152 is replaced by block T154 at the center of PU156, as shown with respect to block T152.

[0079] The motion vector for the TMVP candidate is derived from the co-position PU of the co-position picture, shown at the slice level. The motion vector for the co-position PU is called the co-position MV. Similar to the temporal direct mode in AVC, the co-position MV can be scaled to compensate for the time-distance difference in order to derive the TMVP candidate motion vector, as shown in Figure 4B.

[0080] The video encoder 200 and video decoder 300 may be configured to perform motion vector scaling. The value of the motion vector is assumed to be proportional to the distance between the pictures at presentation time. The motion vector associates two pictures, namely a reference picture and a picture containing the motion vector (i.e., a storage picture). When a motion vector is used to predict other motion vectors, the distance between the storage picture and the reference picture is calculated based on the picture order count (POC) value.

[0081] For a motion vector to be predicted, both its associated storage picture and reference picture may be different. Therefore, a new distance (based on POC) is calculated, and the motion vector is scaled based on these two POC distances. For spatially adjacent candidates, the storage pictures for two motion vectors may be the same, but the reference pictures may be different. In HEVC, motion vector scaling is applied to both TMVP and AMVP for both spatially and temporally adjacent candidates.

[0082] The video encoder 200 and video decoder 300 may be configured to perform artificial motion vector candidate generation. If the motion vector candidate list is incomplete (for example, less than a predetermined number of candidates), artificial motion vector candidates are generated and inserted at the end of the list until the list contains the specified number of candidates.

[0083] In merge mode, there are two types of artificial MV candidates: synthetic candidates derived solely for B slices, and zero-motion vector candidates used for AMVP when the first type does not yield sufficient artificial candidates.

[0084] For each pair of candidates already present in the candidate list and possessing the necessary motion information, a bidirectional composite motion vector candidate is derived by combining the motion vector of the first candidate referencing a picture in List 0 and the motion vector of the second candidate referencing a picture in List 1.

[0085] The video encoder 200 and video decoder 300 may be configured to perform a pruning process for candidate insertion. Candidates from different blocks may coincidentally be the same, which reduces the efficiency of the merge / AMVP candidate list. A pruning process is applied to solve this problem. The pruning process compares a candidate to other candidates in the current candidate list to some extent to avoid inserting identical candidates. To reduce complexity, only a limited number of pruning processes are applied instead of comparing each possible candidate to all other existing candidates.

[0086] Figure 5 is a conceptual diagram showing template matching (TM) performed on a search area around a block using an initial motion vector. The video encoder 200 and video decoder 300 may be configured to perform template matching TM prediction. TM prediction is a special merge mode based on the Frame-Rate Up Conversion (FRUC) technique. In this mode, block motion information is not signaled but derived on the decoder side. It applies to both AMVP mode and normal merge mode. In AMVP mode, MVP candidate selection is determined based on template matching to select the one that minimizes the difference between the current block template and a reference block template. In normal merge mode, a TM mode flag may be signaled to indicate the use of TM, and then TM is applied to merge candidates indicated by the merge index for MV improvement.

[0087] Figure 5 shows exemplary template matching being performed on a search area around the initial MV. As shown in Figure 5, the video encoder 200 and video decoder 300 may be configured to use template matching to derive motion information for the current CU by finding the best match between a template in the current picture (the adjacent block above and / or to the left of the current CU) and a block in the reference picture (the same size as the template). The MVP is refined by template matching using AMVP candidates selected based on the initial matching error. The merged MV corresponding to L0 and L1 is refined independently by template matching using merge candidates indicated by signaled merge indices, and less accurate candidates are further refined using better candidates as priors.

[0088] The video encoder 200 and video decoder 300 may be configured to implement a cost function. Motion-compensated interpolation is required when the motion vector points to a fractional sample position. To reduce complexity, bilinear interpolation may be used instead of normal 8-tap DCT-IF interpolation for both template matches to generate templates on the reference picture. The matching cost C for the template match is calculated as follows:

[0089]

number

[0090] Here, w is a weighting coefficient that is empirically set to 4, and MV and MV s These represent the MV currently being tested and the initial MV (for example, the MVP candidate in AMVP mode or the merged motion vector in merge mode), respectively. SAD is used as the matching cost for template matching.

[0091] When TM is used, motion is improved by using only lumen samples. The derived motion can be used for both lumen and chroma for MC interpretation. After MV is determined, the final MC is performed using an 8-tap interpolation filter for lumen and a 4-tap interpolation filter for chroma.

[0092] The video encoder 200 and video decoder 300 may be configured to perform a search process for motion vector (MV) improvement using template matching as discussed above. MV improvement is a pattern-based MV search with a template matching cost criterion and a hierarchical structure. Two search patterns are supported for MV improvement: diamond search and cross-search. The hierarchical structure specifies an iterative process for improving the MV, starting from coarse motion vector difference (MVD) precision (e.g., 1 / 4 pixel (pel)) and ending with high precision (e.g., 1 / 8-pel). The MV is searched directly with 1 / 4 luma-sample MVD precision using the diamond pattern, then searched with 1 / 4 luma-sample MVD precision using the cross-pattern, and then 1 / 8 luma-sample MVD improvement is performed using the cross-pattern. The search range for MV improvement is set to equal (-8, +8) luma-samples around the initial MV. When the current block is a biprediction block, both MVs are improved independently, and the best of them (in terms of matching cost) is set as the priority for further improving the other MV by biprediction using CU-level weight (BCW) weight values.

[0093] Figure 6 is a conceptual diagram illustrating an example of bilateral matching prediction where two motion vector difference (MVD) values ​​are proportional based on time distance. The video encoder 200 and video decoder 300 may be configured to perform bilateral matching prediction. Bilateral matching (also known as bilateral merging) (BM) prediction is another merging mode based on the FRUC technique. When a block is determined to apply the BM mode, two initial motion vectors MV0 and MV1 are derived by selecting merge candidates from a constructed merge list using signaled merge candidate indices. When performing bilateral matching, the video encoder 200 and video decoder 300 search around MV0 and MV1. The final MV0' and MV1' are derived based on the minimum bilateral matching cost.

[0094] The motion vector differences MVD0 (denoted by MV0'-MV0) and MVD1 (denoted by MV1'-MV1) pointing to two reference blocks may be proportional to the time distance (TD) between the current picture and the two reference pictures, for example, TD0 and TD1. Figure 6 shows an example of MVD0 and MVD1, where TD1 is four times TD0. Figure 6 demonstrates that the example of MVD0 and MVD1 is proportional based on time distance.

[0095] Figure 7 is a conceptual diagram illustrating an example of bilateral matching prediction where two MVDs are mirrored regardless of time distance. This is an arbitrarily chosen design where MVD0 and MVD1 are mirrored independently of the time distances TD0 and TD1. Figure 7 shows an example of mirrored MVD0 and MVD1, where TD1 is four times TD0.

[0096] Figure 8 is a conceptual diagram showing a 3x3 square search pattern in the search range. When performing bilateral matching, the video encoder 200 and video decoder 300 may be configured to perform a local search around the initial MV0 and MV1 to derive the final MV0' and MV1'. The local search loops through the search range [-8,8] by applying a 3x3 square search pattern. In each search iteration, the bilateral matching costs of the eight peripheral MVs in the search pattern are calculated and compared to the bilateral matching cost of the central MV. The MV with the smallest bilateral matching cost becomes the new central MV in the next search iteration. The local search terminates when the current central MV has the smallest cost within the 3x3 square search pattern, or when the local search reaches a predetermined maximum number of search iterations. Figure 8 shows an example of a 3x3 square search pattern in the search range [-8,8].

[0097] Figure 9 is a conceptual diagram illustrating decoder-side motion vector improvement (DMVR). The video encoder 200 and video decoder 300 may be configured to perform decoder-side motion vector improvement. Decoder-side motion vector improvement (DMVR) is applied in the VVC to improve the accuracy of the merge mode MV. In the biprediction operation, the improved MV is searched around the initial MV in the reference picture lists L0 and L1. The DMVR process calculates the distortion between two candidate blocks in the reference picture lists L0 and L1. As shown in Figure 9, the SAD between blocks is calculated based on each MV candidate around the initial MV. The MV candidate with the lowest SAD becomes the improved MV and is used to generate the bipredicted signal, for example, blocks 170 and 172 in Figure 9.

[0098] Figure 9 shows an example of decoder-side motion vector improvement. The improved MV derived by the DMVR process is used to generate interpredictive samples and is also used in time motion vector prediction for coding future pictures. Meanwhile, the original MV is used in the deblocking process and is also used in spatial motion vector prediction for coding future CUs. DMVR is a subblock-based merge mode with a predetermined maximum processing unit of 16 × 16 lumens. When the width and / or height of a CU is greater than 16 lumens, it can be further divided into subblocks with widths and / or heights equal to 16 lumens.

[0099] The video encoder 200 and video decoder 300 may be configured to implement a certain search scheme. In DVMR, the search points enclose the initial MV, and the MV offset follows the MV difference mirroring rule. In other words, every point checked by DMVR, denoted by a candidate MV pair (MV0, MV1), follows the following two equations: MV0' = MV0 + MV_offset MV1' = MV1 - MV_offset Here, MV_offset represents the improvement offset between the initial MV and the improved MV in one of the reference pictures. The improvement search range is two integer lumer samples from the initial MV. This search includes an integer sample offset search stage and a fractional sample improvement stage.

[0100] The video encoder 200 and video decoder 300 may be configured to apply a 25-point complete search for integer sample offset search. The SAD of the initial MV pair is calculated first. If the SAD of the initial MV pair is less than a threshold, the integer sample stage of the DMVR ends. Otherwise, the SADs of the remaining 24 points are calculated and checked in raster scan order. The point with the smallest SAD is selected as the output of the integer sample offset search stage. To reduce the disadvantage of uncertainty in DMVR refinement, it is proposed to prioritize the original MV during the DMVR process. The SAD between reference blocks referenced by the initial MV candidates is reduced by 1 / 4 of the SAD value.

[0101] Following the integer sample search, the video encoder 200 and video decoder 300 may be configured to perform fractional sample improvement. To reduce computational complexity, fractional sample improvement is derived by using a parametric error surface equation instead of an additional search using SAD comparison. Fractional sample improvement is conditionally invoked based on the output of the integer sample search stage. If the integer sample search stage terminates at a center with the smallest SAD in either the first or second iterative search, fractional sample improvement is further applied.

[0102] Sethuraman et al., "Decoder Side MV Refinement / Derivation with CTB-level concurrency and other normative complexity reduction techniques," ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 Joint Video Experts Team (JVET), Document JVET-K0041-v2, 11th Meeting, Ljubljana, Slovenia, July 10-18, 2018 (hereinafter "JVET-K0041"), available at phenix.it-sudparis.eu / jvet / doc_end_user / current_document.php?id=3542, describes a model-based fractional-pixel (fractional-pel) motion vector refinement process. JVET-K0041 describes using a quadratic parametric function to form a predictive error surface for each possible fractional-pel MV.

[0103] Essentially, a quadratic parametric function is an interpolation function that interpolates the value of the prediction error as an estimator. Based on the exact prediction error value from an integer-interval MV improvement, the parameters of the quantized parametric function can be derived, and thus the best motion sampling position for this error search can be found. The original MV is then adjusted to this exact motion sampling position instead of the codec actually performing a fractional-pel motion vector search. This parametric function interpolates the error surface of all possible sampling positions of the MV and takes the cost values ​​from five points as a criterion for finding the best position on this surface with the minimum cost value. These five points form an intersection shape, with a width of 1 pixel between each pair of adjacent points, and the center / left / right / top / bottom points are adjusted to (0,0), (-1,0), (1,0), (0,-1), and (0,1) relative to the point pointed to by the MV (found by the integer sample search). Specifically, this parametric error surface function is the 2D parabolic error surface equation. E x, y= A(x - Δx) 2 + B(y - Δy) 2 + C Here, E x, y represents the SAD related to the MV indicating (MV0'+x, MV1'+y), (Δx, Δy) are unknown terms corresponding to the position where the cost is minimized, and C corresponds to the minimum cost value.

[0104] By solving five equations for five unknowns, (Δx, Δy) are calculated as follows. Δx = floor((α(E -1,0 - E 1,0 )) / (2(E -1,0 + E 1,0 - 2E 0,0 ))) Δy = floor((α(E 0,-1 - E 0,1 )) / (2(E 0,-1 + E 0,1 - 2E 0,0 ))) Here, α is an integer scaling factor introduced to represent (Δx, Δy) with a certain fractional - pel precision. For example, it is 16 for 1 / 16 precision and 4 for 1 / 4 precision. In VVC, 16 is always used. To obtain the fractional - pel precision MV, the calculated fractional (Δx, Δy) is added to the MV discovered by integer sample search.

[0105] In VVC, (Δx, Δy) is not calculated and is set equal to (0, 0) when at least one of the following conditions is satisfied. The MV_offset as discussed above hits the horizontal boundary of the search range (e.g., in DMVR or bilateral interpolation, it is ±N pixels, where N can be 1, 2, 3, 4, 5, 6, 7, or 8 pixels or more). The MV_offset as discussed above hits the vertical boundary of the search range (e.g., in DMVR or bilateral interpolation, it is ±N pixels, where N can be 1, 2, 3, 4, 5, 6, 7, or 8 pixels or more). E 0,0 is equal to 0.

[0106] The video encoder 200 and video decoder 300 may be configured to perform bilinear interpolation and sample padding. In VVC, the resolution of the MV is 1 / 16 lumens of samples. Samples at fractional positions are interpolated using an 8-tap interpolation filter. In DMVR, the search points surround the initial fractional-pel MV with integer sample offsets, and therefore, samples at those fractional positions need to be interpolated for the DMVR search process. To reduce computational complexity, a bilinear interpolation filter is used to generate fractional samples for the search process in DMVR. Another important effect is that by using a bilinear filter with a search range of 2 samples, DMVR does not access as many reference samples as a normal motion compensation process. After the improved MV is obtained using the DMVR search process, a normal 8-tap interpolation filter is applied to generate the final prediction. To avoid accessing more reference samples than a normal MC process, samples that are not needed for the interpolation process based on the original MV but are needed for the interpolation process based on the improved MV may be padded from their available samples.

[0107] The video encoder 200 and video decoder 300 may be configured to implement one or more activation conditions. For example, the video encoder 200 and video decoder 300 may enable DMVR if all of the following conditions are met. • CU-level merge mode using bipredictive MV • For the current picture, one reference picture is in the past, and the other reference picture is in the future. • The distance from both reference pictures to the current picture (i.e., the POC difference) is the same. • CU has more than 64 luma samples • Both the height and width of the CU are 8 lumens or more. • BCW weight index indicates equal weights • Weighted prediction (WP) is not enabled for the current block. • Synthetic Intrainterface Prediction (CIIP) mode is not used for the current block.

[0108] Figure 10 is a conceptual diagram illustrating MMVD (Merge Mode with Motion Vector Difference). The video encoder 200 and video decoder 300 may be configured to implement MMVD (Merge Mode with Motion Vector Difference). MMVD is a merge mode that allows for the explicit signaling of additional motion vector differences to refine the motion field of the merge mode. MMVD flags may be signaled immediately after sending the skip flag, merge flag, and normal merge flag to indicate whether MMVD mode is used for CU. In MMVD, after a merge candidate is selected, it is further refined by signaled MVD information. This additional information includes merge candidate flags, an index for specifying the magnitude of the motion, and an index for indicating the direction of the motion. In MMVD mode, the information for the first two candidates in the merge list is selected to be used as the basis for the MV. Merge candidate flags are signaled to specify which candidate is used.

[0109] Figure 10 shows examples of MMVD search points in List 0 (L0) reference 180 and List 1 (L1) reference 182, respectively. The distance index specifies information about the magnitude of the motion and indicates a predefined offset from the starting point. As shown in Figure 10, the offset is added to either the horizontal or vertical component of the starting MV. The relationship between the distance index and the predetermined offset is 2. idx-2 Defined as such, where idx is an integer between 0 and 7. When picture header syntax (e.g., pic_fpel_mmvd_enabled_flag) is enabled, the index-offset mapping is 2 idx Please note that this may be changed.

[0110] The direction index represents the direction of the MVD relative to the starting point. The direction index can represent four directions, as shown in the table below, where 00, 01, 10, and 11 represent right, left, up, and down, respectively. Note that the meaning of the MVD sign can vary depending on the information of the starting MV. When the starting MV is a single-prediction MV, or a bi-prediction MV where both lists point to the same side of the current picture (i.e., the POCs of both references are both greater than the POC of the current picture, or both are less than the POC of the current picture), the sign in the table below specifies the sign of the MV offset added to the starting MV. When the starting MV is a bi-prediction MV where the two MVs point to different sides of the current picture (i.e., the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture), the sign in the table below specifies the sign of the MV offset added to the list0 MV component of the starting MV, and the sign for list1 MV has the opposite value.

[0111] [Table 1]

[0112] The processing steps of conventional TM techniques can impose an extra burden on hardware decoding design because conventional techniques require the execution of fractional-pel search. Therefore, conventional TM techniques inevitably generate samples in the search area during additional interpolation, resulting in extra decoding latency to calculate the TM cost. A model-based fractional-pel MV refinement method was introduced in VVC to avoid fractional-pel interpolation. This model-based fractional-pel MV refinement method can be combined with TM to reduce the need to perform interpolation for each level of MVD accuracy required during the TM's MV refinement process. Thus, a less complex decoder design can be expected.

[0113] This disclosure describes various techniques that can improve the performance of various decoder-side motion vector derivation / improvement techniques, including template matching, bilateral matching, decoder-side motion vector improvement, and model-based motion vector difference derivation. The video encoder 200 and video decoder 300 may be configured to perform the techniques of this disclosure, either individually or in any combination.

[0114] In one example, the video encoder 200 and video decoder 300 may be configured to perform model-based fractional pixel (fraction-pel) motion vector refinement against template matching (TM). Initially, the video encoder 200 and video decoder 300 perform initial MV(v x ,v y A predetermined search range around ) [±SR x ,±SR y The TM search process may be initiated with integer MVD precision for ], where SRx and SRy are integer values. Without loss of generality, this disclosure is (v x ,v y ) is represented with the highest MV memory precision (e.g., 16 for VVC, 4 for HEVC), SR x and SR y It is already scaled according to the best MV memory accuracy. When the TM search process is complete, the lowest template matching error within the search range (e.g., SAD, SSE, MRSAD, MRSSE, SSIM, MS-SSIM) is E 0,0 It is written as follows, and the template matching error for each adjacent point on the left / right / top / bottom (i.e., all of them located one pixel away from the center) is E -1,0 , E 1,0 , E 0,-1 , and E 0,1 It is written as follows.

[0115] To make it concise, E 0,0 The motion vector related to (v x ', v y It is written as ') and |(v x ',v y ')-(v x ,vy )|≦(SR x SR rating y ) is the result. Next, as discussed above, the video encoder 200 and video decoder 300 take the error value (E) as input to a model-based fractional-pel MV improvement method for deriving (Δx, Δy). 0,0 , E -1,0 , E 1,0 , E 0,-1 and E 0,1 ) is set and the motion vector obtained from TM is (v x '+Δx, v y It becomes '+Δy). Similarly, for example, if at least one of the conditions discussed above is met (which are again described below), then (Δx,Δy) can be the zero vector. |v x -v x | is SR x Equivalent to, |v y -v y | is SR y Equivalent to, E 0,0 It is equal to 0.

[0116] For simplicity of explanation, this disclosure states that the required scaling factor specified by the Adaptive Motion Vector Resolution (AMVR) mode is already (v) before being added to (Δx, Δy). x ',v y Assume that it is applied to ').

[0117] As an example, a model-based fractional -pel MV improvement method can be expressed as follows: MV(v x ',v y The aforementioned conditions can be modified so that they can be applied to improve only one direction of '). E 0,0 ≠0 and |v x -v x | <SR x If this is the case, then Δx is E 0,0 , E -1,0 , and E 1,0 It can be derived based on E. 0,0 =0 or |vx -v x |≧SR x If this is the case, Δx is set to be equal to 0. E 0,0 ≠0 and |v y -v y | <SR y If so, then Δy is E 0,0 , E 0,-1 , and E 0,1 It can be derived based on E. 0,0 =0 or |v y -v y |≧SR y If this is the case, Δy is set to be equal to 0.

[0118] In some examples, the video encoder 200 and video decoder 300 may be configured with additional enable conditions for model-based fractional-pel MV improvements. For example, the video encoder 200 and video decoder 300 may be configured to enable or disable model-based fractional-pel MV improvements with additional conditions in addition to those discussed above. The enable / disable conditions may be any of the following: • The initial MV of TM was improved by bilateral matching or DMVR. • The obtained MV from TM is further refined by bilateral matching or DMVR. • The conditions for enabling bilateral matching or DMVR are met, i.e., each of the following is true: ○CU-level merge mode using bipredictive MV ○For the current picture, one reference picture is in the past, and the other reference picture is in the future. ○ The distance from both reference pictures to the current picture (i.e., the POC difference) is the same. ○CU has more than 64 luma samples ○Both the height and width of the CU are 8 lumens or more. ○BCW weight index indicates equal weight ○Weighted prediction (WP) is not enabled for the current block. ○ Synthetic Intrainterface Prediction (CIIP) mode is not used for the current block.

[0119] Alternatively, the enable / disable conditions may be as follows: • The initial MV of TM is not improved by bilateral matching or DMVR. • The obtained MV of TM is not further improved by bilateral matching or DMVR. • The conditions for enabling bilateral matching or DMVR are not met, i.e., at least one of the following is not true: ○CU-level merge mode using bipredictive MV ○For the current picture, one reference picture is in the past, and the other reference picture is in the future. ○ The distance from both reference pictures to the current picture (i.e., the POC difference) is the same. ○CU has more than 64 luma samples ○Both the height and width of the CU are 8 lumens or more. ○BCW weight index indicates equal weight ○Weighted prediction (WP) is not enabled for the current block. ○ Synthetic Intrainterface Prediction (CIIP) mode is not used for the current block.

[0120] In some examples, the video encoder 200 and video decoder 300 may adjust the alpha term, which is adaptively defined according to the use of AMVR mode or a 1 / 2-pel interpolation filter (which is the same filter used when AMVR mode is 1 / 2-pel precision). For example: When the AMVR mode is full-pel, 2-pel, 4-pel, or less (i.e., greater than 4-pel), the model-based fractional-pel MV refinement method is not applied, or the alpha term is set to equal 0 or 1. When the AMVR mode is 1 / 2 - pel or when a 1 / 2 - pel interpolation filter is used, the α term is set equal to 2. When the AMVR mode is 1 / 4 - pel, the α term is set equal to 4. When the AMVR mode is 1 / 8 - pel, the α term is set equal to 8. When the AMVR mode is 1 / 16 - pel, the α term is set equal to 16. When AMVR is 1 / N - pel (where N is 32, 64, 128, or greater), the α term is set equal to N.

[0121] This disclosure assumes that (v x , v y ) is represented with the highest MV memory accuracy (e.g., 16 in VVC, 4 in HEVC), so the model output (Δx, Δy) needs to be adjusted according to the value of the α term. When α ≤ 16, the model output is as follows. Δx = s * floor((α(E -1,0 - E 1,0 )) / (2(E -1,0 + E 1,0 - 2E 0,0 ))) Δy = s * floor((α(E 0,-1 - E 0,1 )) / (2(E 0,-1 + E 0,1 - 2E 0,0 ))) Here, s = 16 / α. In other cases where α > 16, the MV memory accuracy must be higher than 16, and the scaling factor s of the model output is M / α, where M is the MV memory accuracy of (v x , v y ) and M ≥ N.

[0122] In another example, the α term is set equal to 16 (or the highest value according to the MV memory accuracy, e.g., 16 in VVC, 4 in HEVC) regardless of the use of the 1 / 2 - pel interpolation filter. Then, (v x ' + Δx, v yThe obtained MV of TM, which is (v'+Δx, v'+Δy), is rounded to the MV accuracy indicated by the AMVR mode.

[0123] In another example, the α term is set equal to 16 (or the highest value according to the MV memory accuracy, e.g., 16 in VVC and 4 in HEVC) regardless of the use of the AMVR mode. Then, the obtained MV of TM, which is (v'+Δx, v'+Δy), is rounded to the MV accuracy indicated by the AMVR mode. x '+Δx, v y '+Δy), is rounded to the MV accuracy indicated by the AMVR mode.

[0124] Some techniques of U.S. Provisional Application No. 63 / 131,676, filed on December 29, 2020, may be combined with the model-based fractional-pel MV improvement techniques of the present disclosure as discussed below. As disclosed herein, the TM search accuracy may vary according to the AMVR mode. For example, when the AMVR mode is 1 / 4-pel, the video encoder 200 and the video decoder 300 may start the TM search process with full-pel MVD accuracy, then perform it with 1 / 2-pel MVD accuracy, and stop with 1 / 4-pel MVD accuracy. In this example, TM does not search towards the highest MVD accuracy indicated by the AMVR mode for simplicity. In some examples, the video encoder 200 and the video decoder 300 may be configured to perform the following. · When the AMVR mode is 1 / 2-pel, TM starts and stops with full-pel MVD accuracy, and then, with α = 2, s = 8, and {E 0,0 , E -1,0 , E 1,0 , E 0,-1 , E 0,1}, the model-based method is applied. · When the AMVR mode is 1 / 4-pel, TM can start and stop with full-pel, and then, with α = 4, s = 4, and {E 0,0 , E -1,0 , E 1,0 , E 0,-1 , E 0,1}, the model-based method is applied. When the AMVR mode is 1 / 4-pel, the TM can start at full-pel and stop at 1 / 2-pel, then the model-based method is applied with α=2, s=4, and {E 0,0 , E -1,0 , E 1,0 , E 0,-1 , E 0,1} is {E 0,0 , E -0.5,0 , E 0.5,0 , E 0,-0.5 , E 0,0.5 It can be replaced by using}. When AMVR mode is 1 / 16-pel, TM can start and stop at full-pel, then α=16, s=1, and {E 0,0 , E -1,0 , E 1,0 , E 0,-1 , E 0,1 A model-based approach is applied. When the AMVR mode is 1 / 16-pel, the TM can start at full-pel and stop at 1 / 2-pel, then the model-based method is applied with α=8 and s=1, {E 0,0 , E -1,0 , E 1,0 , E 0,-1 , E 0,1} is {E 0,0 , E -0.5,0 , E 0.5,0 , E 0,-0.5 , E 0,0.5 It can be replaced by using}. When the AMVR mode is 1 / 16-pel, the TM can start at full-pel, then at 1 / 2-pel, and stop at 1 / 4-pel, then the model-based method is applied with α=4, s=1, {E 0,0 , E -1,0 , E 1,0 , E 0,-1 , E 0,1} is {E 0,0 , E -0.25,0 , E 0.25,0 , E 0,-0.25 , E 0,0.25It can be replaced by using}. When the AMVR mode is 1 / 16-pel, the TM can start at full-pel, then sequentially at 1 / 2-pel and 1 / 4-pel, and stop at 1 / 8-pel, then the model-based method is applied with α=2, s=1, {E 0,0 , E -1,0 , E 1,0 , E 0,-1 , E 0,1} is {E 0,0 , E -0.125,0 , E 0.125,0 , E 0,-0.125 , E 0,0.125 It can be replaced by using}.

[0125] In various examples, the four adjacent points are not always located one pixel apart from the center, because the video encoder 200 and video decoder 300 may be configured to perform TM search with higher MVD accuracy than full-pel MVD accuracy. The distance may be less than one pixel depending on the AMVR mode. For example, when the AMVR mode is 1 / 2-pel, {E 0,0 , E -1,0 , E 1,0 , E 0,-1 , E 0,1} is {E 0,0 , E -0.5,0 , E 0.5,0 , E 0,-0.5 , E 0,0.5 It can be replaced by using}.

[0126] In some examples, the video encoder 200 and video decoder 300 perform TM search with full-pel accuracy only, regardless of AMVR mode, with α=16, s=1, and {E 0,0 , E -1,0 , E 1,0 , E 0,-1 , E 0,1 A model-based method is applied as follows. The resulting MV is then rounded to the MV accuracy indicated by the AMVR mode.

[0127] In some examples, the video encoder 200 and video decoder 300 perform TM search with full-pel and 1 / 2-pel accuracy, regardless of AMVR mode, with α=8, s=1, and {E 0,0 , E -0.5,0 , E 0.5,0 , E 0,-0.5 , E 0,0.5 A model-based method is applied as follows. The resulting MV is then rounded to the MV accuracy indicated by the AMVR mode.

[0128] In some examples, the video encoder 200 and video decoder 300 perform TM search with full-pel accuracy, 1 / 2-pel accuracy, and 1 / 4-pel accuracy, regardless of AMVR mode, with α=4, s=1, and {E 0,0 , E -0.25,0 , E 0.25,0 , E 0,-0.25 , E 0,0.25 A model-based method is applied as follows. The resulting MV is then rounded to the MV accuracy indicated by the AMVR mode.

[0129] In some examples, the video encoder 200 and video decoder 300 perform TM search with full-pel accuracy, 1 / 2-pel accuracy, 1 / 4-pel accuracy, and 1 / 8-pel accuracy, regardless of AMVR mode, with α=2, s=1, and {E 0,0 , E -0.125,0 , E 0.125,0 , E 0,-0.125 , E 0,0.125 A model-based method is applied as follows. The resulting MV is then rounded to the MV accuracy indicated by the AMVR mode.

[0130] In some examples, the video encoder 200 and video decoder 300 perform TM search with full-pel and 1 / 2-pel accuracy, with α=8, s=2, and {E 0,0 , E -0.5,0 , E 0.5,0 , E 0,-0.5 , E0,0.5 A model-based method is applied as follows. The resulting MV is then rounded to the MV accuracy indicated by the AMVR mode.

[0131] The video encoder 200 and video decoder 300 may be configured to perform a simplified search process for a bipredicted TM merge mode. In some examples, the TM merge mode for a bipredicted block is a dual process in which (1) one of the MVs is selected for MV improvement performed by the TM, and (2) the improved MV is then set as a prior for further improvement of the other MV using BCW weight values. There are several rules that may be established for step 1 to select one MV from the bipredicted block, as follows: • MV related to the reference picture list Lx, x can be either 0 or 1. • When ph_mvd_l1_zero_flag=true, the MV associated with the reference picture list Lx is set to equal 1. (ph_mvd_l1_zero_flag may be named differently in various video standards, but its function remains the same, i.e., MVD L1 Set to 0 and MVD L1 (Note that this involves skipping syntax signaling.) • When ph_mvd_l1_zero_flag=false, x is set to equal to 1 in the MV associated with the reference picture list Lx. • Reference picture list Lx related MV, x is MV L0 and MV L1 It can be either 0 or 1 depending on the respective TM cost. MV L0 If the TM cost is lower, x is set to equal 0; otherwise, it is set to equal 1. • Reference picture list Lx related MV, x is MV L0 and MV L1 It can be either 0 or 1 depending on the respective TM cost. MVL0 If the TM cost is lower, x is set to equal 1; otherwise, it is set to equal 0.

[0132] In some examples, when the current block is bipredicted, one of the MVs is selected (according to one of the methods defined above) as a prior to further refine the other MV using BCW weights.

[0133] In some examples, the video encoder 200 and video decoder 300 may be configured to use a constrained search range for TM merge mode. For example, when performing bilateral matching (BM) mode before TM merge mode, the video encoder 200 and video decoder 300 may be configured to use TM to further refine the BM-refined MV within the same search range around the initial MV of BM. Thus, both TM and BM can share the same reference samples fetched from the reference picture. Similarly, when TM is performed before BM, BM may be used to further refine the TM-refined MV within the same search range around the initial MV of TM. Thus, both TM and BM can share the same reference samples fetched from the reference picture.

[0134] In some examples, the TM and BM may form a chained process such that the TM is executed before the BM, or the BM is executed before the TM. Regardless of which comes first, this search process may be restricted not only to the same search range but also to integer MVD precision only. The fractional-pel MV improvement may only be performed at the stage of whichever of the TM or BM is executed last in the chained process. For example, the video encoder 200 and video decoder 300 may consist of a chained process {BM -> TM -> subblock BM}, where the fractional-pel MV improvement is performed only at the subblock BM stage, but the others are performed only with integer MVD precision. In another example where the subblock BM may be disabled, the fractional-pel MV improvement is performed only at the TM stage, but the others are performed only with integer MVD precision.

[0135] In some cases, when the AMVR mode is 4-pel, the video encoder 200 and video decoder 300 may disable the TM AMVP mode. "Disabled" may mean that template matching is not applied to refine AMVP candidates and is not used to select AMVP candidates, so the AMVP mode runs the same way for VVC or HEVC without modification. In some cases, when the AMVR mode is 2-pel or less (for example, 4-pel or more pel), the video encoder 200 and video decoder 300 may disable the TM AMVP mode.

[0136] Figure 11 is a block diagram showing an exemplary video encoder 200 capable of performing the techniques of this disclosure. Figure 11 is provided for illustrative purposes and should not be considered a limitation of the techniques as broadly illustrated and described in this disclosure. For illustrative purposes, this disclosure describes the video encoder 200 in the context of video coding standards such as the ITU-T H.265 / HEVC video coding standard and the VVC video coding standard under development. However, the techniques of this disclosure are not limited to these video coding standards and are generally applicable to other video coding and decoding standards.

[0137] In the example shown in Figure 11, the video encoder 200 includes a video data memory 230, a mode selection unit 202, a residual generation unit 204, a conversion processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse conversion processing unit 212, a reconstruction unit 214, a filter unit 216, a decoded picture buffer (DPB) 218, and an entropy coding unit 220. Any or all of the video data memory 230, mode selection unit 202, residual generation unit 204, conversion processing unit 206, quantization unit 208, inverse quantization unit 210, an inverse conversion processing unit 212, a reconstruction unit 214, a filter unit 216, a DPB 218, and an entropy coding unit 220 may be implemented in one or more processors or processing circuits. For example, the units of the video encoder 200 may be implemented as one or more circuits or logic elements as part of a hardware circuit, or as part of a processor, ASIC, or FPGA. Furthermore, the video encoder 200 may include additional or alternative processors or processing circuits to perform these and other functions.

[0138] The video data memory 230 can store video data to be encoded by the components of the video encoder 200. The video encoder 200 can receive video data stored in the video data memory 230 from, for example, a video source 104 (Figure 1). The DPB 218 can act as a reference picture memory, storing reference video data for use by the video encoder 200 in predicting subsequent video data. The video data memory 230 and DPB 218 can be formed by any of various memory devices, such as dynamic random access memory (DRAM) including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The video data memory 230 and DPB 218 can be provided by the same memory device or separate memory devices. In various examples, the video data memory 230 may be on-chip with the other components of the video encoder 200, as shown, or off-chip relative to those components.

[0139] In this disclosure, references to the video data memory 230 should not be interpreted as being limited to memory inside the video encoder 200 unless otherwise stated, nor should they be interpreted as being limited to memory outside the video encoder 200 unless otherwise stated. Rather, references to the video data memory 230 should be understood as reference memory that stores video data received by the video encoder 200 for encoding (for example, video data for the current block to be encoded). Memory 106 in Figure 1 may also temporarily store outputs from various units of the video encoder 200.

[0140] The various units in Figure 11 are shown to help understand the operations performed by the video encoder 200. The units may be implemented as fixed-function circuits, programmable circuits, or a combination thereof. Fixed-function circuits refer to circuits that provide a specific function, with predefined operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks, offering flexibility in the operations they can perform. For example, a programmable circuit may execute software or firmware that operates the programmable circuit in a manner defined by software or firmware instructions. Fixed-function circuits may execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is generally immutable. In some examples, one or more units may be separate circuit blocks (fixed-function or programmable), and in some examples, one or more units may be integrated circuits.

[0141] The video encoder 200 may include a programmable core formed from an arithmetic logic unit (ALU), an EFU, digital circuits, analog circuits, and / or programmable circuits. In an example where the operation of the video encoder 200 is performed using software executed by the programmable circuits, memory 106 (Figure 1) may store software instructions (e.g., object code) that the video encoder 200 receives and executes, or another memory (not shown) within the video encoder 200 may store such instructions.

[0142] The video data memory 230 is configured to store the received video data. The video encoder 200 can retrieve a picture of the video data from the video data memory 230 and provide the video data to the residual generation unit 204 and the mode selection unit 202. The video data in the video data memory 230 may be raw video data to be encoded.

[0143] The mode selection unit 202 includes a motion estimation unit 222, a motion compensation unit 224, and an intra-prediction unit 226. The mode selection unit 202 may include additional functional units for performing video prediction according to other prediction modes. For example, the mode selection unit 202 may include a palette unit, an intra-block copy unit (which may be part of the motion estimation unit 222 and / or the motion compensation unit 224), an affine unit, a linear model (LM) unit, and so on.

[0144] The mode selection unit 202 generally coordinates multiple coding paths to test combinations of coding parameters and the rate distortion values ​​obtained for such combinations. The coding parameters may include the division of the CTU to the CU, the prediction mode for the CU, the transformation type for the residual data of the CU, and the quantization parameters for the residual data of the CU. The mode selection unit 202 can ultimately select a combination of coding parameters that has a better rate distortion value than other tested combinations.

[0145] The video encoder 200 divides the picture retrieved from the video data memory 230 into a series of CTUs, and may encapsulate one or more CTUs within a slice. The mode selection unit 202 may divide the picture's CTUs according to a tree structure, such as a QTBT structure or the HEVC quadtree structure described above. As described above, the video encoder 200 may form one or more CUs from dividing the CTUs according to a tree structure. Such CUs may also be commonly referred to as “video blocks” or “blocks”.

[0146] Generally, the mode selection unit 202 also controls its components (e.g., motion estimation unit 222, motion compensation unit 224, and intra-prediction unit 226) to generate a predicted block for the current block (e.g., the current CU, or, in HEVC, the overlapping portion of PU and TU). For intra-prediction of the current block, the motion estimation unit 222 may perform a motion search to identify one or more well-matching reference blocks among one or more reference pictures (e.g., one or more previously coded pictures stored in the DPB 218). Specifically, the motion estimation unit 222 may calculate a value representing how similar a potential reference block is to the current block, for example, according to the sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared difference (MSD), etc. The motion estimation unit 222 may generally perform these calculations using the sample-by-sample difference between the current block and the reference block under consideration. The motion estimation unit 222 may identify the reference block with the lowest value resulting from these calculations, indicating the reference block that best matches the current block.

[0147] The motion estimation unit 222 may form one or more motion vectors (MVs) that define the position of a reference block in a reference picture relative to the position of the current block in the current picture. The motion estimation unit 222 may then provide the motion vectors to the motion compensation unit 224. For example, for unidirectional interpretation, the motion estimation unit 222 may provide a single motion vector, while for bidirectional interpretation, the motion estimation unit 222 may provide two motion vectors. The motion compensation unit 224 may then use the motion vectors to generate predicted blocks. For example, the motion compensation unit 224 may use the motion vectors to extract data for the reference blocks. As another example, if the motion vectors have fractional sample precision, the motion compensation unit 224 may interpolate the values ​​for the predicted blocks according to one or more interpolation filters. Furthermore, for bidirectional interpretation, the motion compensation unit 224 may extract data for the two reference blocks identified by each motion vector and combine the extracted data, for example, through sample-wise averaging or weighted averaging.

[0148] Furthermore, the motion compensation unit 224 may be configured to perform decoder-side motion vector refinement / derivation techniques in any combination according to any or all of the techniques of this disclosure. For example, the motion compensation unit 224 may perform template matching and model-based fractional-pixel motion vector refinement to refine the full-pixel resolution motion vector initially received from the motion estimation unit 222 using these techniques.

[0149] For example, motion compensation unit 224 may receive an initial motion vector from motion estimation unit 222. The initial motion vector may have full integer pixel resolution. Motion compensation unit 224 and entropy coding unit 220 may encode the initial motion vector using advanced motion vector prediction (AMVP) with motion vector difference (MVD) values ​​having full integer pixel precision. Motion compensation unit 224 may further refine the initial motion vector, which may include, for example, determining a search range around a reference area identified by the initial motion vector in a reference picture. Motion compensation unit 224 may then perform a template matching search process within the search range to identify the best-matching region. The best-matching region may be the region that produces the lowest rate distortion optimization (RDO) value. The RDO value may be calculated using, for example, the sum of absolute difference values, the sum of squared difference values, the mean absolute difference value, the mean squared difference value, etc.

[0150] By determining the best-matching region, the motion compensation unit 224 may obtain an improved motion vector with integer precision. The motion compensation unit 224 may then further determine error values ​​for adjacent pixels within the best-matching region in order to perform model-based fractional-pixel motion vector improvement to derive motion vector difference values ​​(e.g., x (horizontal) and / or y (vertical) component motion vector difference values). The motion compensation unit 224 may further apply at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block. The improved motion vector may have fractional-pixel precision. The motion compensation unit 224 may then use the improved motion vector to generate a predicted block and provide the predicted block to the residual generation unit 204 and the reconstruction unit 214.

[0151] Specifically, for each of the motion vector difference components, the motion compensation unit 224 may determine whether the error value for the best matching region is equal to zero. The motion compensation unit 224 then further determines that the absolute value of the difference between a component of the initial motion vector and the corresponding motion vector difference value for that component is less than the corresponding search region value for that component, and then may apply the motion vector difference value for that component to that component of the initial motion vector.

[0152] As described above, the motion compensation unit 224 (or motion selection unit 202) may also determine whether to perform the motion vector improvement technique of the present disclosure. For example, the mode selection unit 202 or motion compensation unit 224 may determine that the initial motion vector is in coding unit (CU) level merge mode with bipredicted motion vectors, that the first reference picture for the current block has a display order value smaller than the display order value of the current picture containing the current block, that the second reference picture for the current block has a display order value larger than the display order value of the current picture, that the time distances between the current picture, the first reference picture and the second reference picture are equal, that the current block has more than 64 lumens, that the height of the current block is at least 8 lumens, that the width of the current block is at least 8 lumens, that the weight values ​​for biprediction with CU-level weights (BCW) are equal, that weighted prediction (WP) is not enabled for the current block, and that the synthetic intrainterpretation (CIIP) mode is not used for the current block, and in response, determine a motion vector improvement technique.

[0153] In some examples, if the current block is bipredicted, the motion compensation unit 224 may perform these motion vector improvement techniques for both motion vectors of the current block.

[0154] As another example, for intra-prediction or intra-prediction coding, the intra-prediction unit 226 may generate a prediction block from samples adjacent to the current block. For example, in directional mode, the intra-prediction unit 226 may generally mathematically combine the values ​​of adjacent samples and fill these calculated values ​​in a defined direction across the current block to produce a prediction block. As another example, in DC mode, the intra-prediction unit 226 may calculate the average of the samples adjacent to the current block and generate a prediction block that includes this obtained average for each sample in the prediction block.

[0155] The mode selection unit 202 provides the prediction block to the residual generation unit 204. The residual generation unit 204 receives raw, uncoded versions of the current block from the video data memory 230 and the prediction block from the mode selection unit 202. The residual generation unit 204 calculates the sample-by-sample difference between the current block and the prediction block. The resulting sample-by-sample difference defines the residual block relative to the current block. In some examples, the residual generation unit 204 may also determine the differences between sample values ​​in the residual block to generate the residual block using residual differential pulse code modulation (RDPCM). In some examples, the residual generation unit 204 may be formed using one or more subtractor circuits that perform binary subtraction.

[0156] In an example where the mode selection unit 202 divides CUs into PUs, each PU may be associated with a lumen prediction unit and a corresponding chroma prediction unit. The video encoder 200 and video decoder 300 may support PUs of various sizes. As shown above, the size of a CU may refer to the size of the lumen coding block of the CU, and the size of a PU may refer to the size of the lumen prediction unit of the PU. Assuming that the size of a particular CU is 2N×2N, the video encoder 200 may support PU sizes of 2N×2N or N×N for intra-prediction, and 2N×2N, 2N×N, N×2N, N×N, or similar symmetric PU sizes for inter-prediction. The video encoder 200 and video decoder 300 may also support asymmetric divisions for PU sizes of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-prediction.

[0157] In an example where the mode selection unit 202 does not further subdivide the CUs into PUs, each CU may be associated with a ruma coding block and a corresponding chroma coding block. As shown above, the size of a CU may refer to the size of the ruma coding block of the CU. The video encoder 200 and video decoder 300 may support CU sizes of 2N×2N, 2N×N, or N×2N.

[0158] As some examples, in other video coding techniques such as intra-block copy mode coding, affine mode coding, and linear model (LM) mode coding, the mode selection unit 202 generates a prediction block for the current block being coded via respective units related to the coding technique. In some examples such as palette mode coding, the mode selection unit 202 may not generate a prediction block and instead may generate a syntax element indicating a way to reconstruct the block based on the selected palette. In such a mode, the mode selection unit 202 may provide these syntax elements to the entropy coding unit 220 to be coded as such.

[0159] As described above, the residual generation unit 204 receives video data for the current block and the corresponding prediction block. The residual generation unit 204 then generates a residual block for the current block. To generate the residual block, the residual generation unit 204 calculates the sample-by-sample difference between the prediction block and the current block.

[0160] The transform processing unit 206 applies one or more transforms to the residual block to generate a block of transform coefficients (referred to herein as a "transform coefficient block"). The transform processing unit 206 may apply various transforms to the residual block to form the transform coefficient block. For example, the transform processing unit 206 may apply a discrete cosine transform (DCT), a directional transform, a Karhunen-Loeve transform (KLT), or a conceptually similar transform to the residual block. In some examples, the transform processing unit 206 may perform multiple transforms, such as a primary transform and a secondary transform, such as a rotation transform, on the residual block. In some examples, the transform processing unit 206 does not apply a transform to the residual block.

[0161] The quantization unit 208 can quantize the transformation coefficients in the transformation coefficient block to produce a quantized transformation coefficient block. The quantization unit 208 can quantize the transformation coefficients of the transformation coefficient block according to the quantization parameter (QP) value associated with the current block. The video encoder 200 can adjust the degree of quantization applied to the transformation coefficient block associated with the current block by adjusting the QP value associated with the CU (for example, via the mode selection unit 202). Quantization can result in information loss, and therefore the quantized transformation coefficients may be less precise than the original transformation coefficients generated by the transformation processing unit 206.

[0162] The inverse quantization unit 210 and the inverse transformation processing unit 212 can reconstruct residual blocks from transformation coefficient blocks by applying inverse quantization and inverse transformation, respectively, to the quantized transformation coefficient blocks. The reconstruction unit 214 can generate reconstructed blocks corresponding to the current blocks (which may be with some distortion) based on the reconstructed residual blocks and prediction blocks generated by the mode selection unit 202. For example, the reconstruction unit 214 can generate a reconstructed block by adding a sample from the reconstructed residual block to a corresponding sample from the prediction block generated by the mode selection unit 202.

[0163] The filter unit 216 may perform one or more filtering operations on the reconstructed block. For example, the filter unit 216 may perform a deblocking operation to reduce blockingness artifacts along the edges of the CU. In some examples, the operation of the filter unit 216 may be skipped.

[0164] The video encoder 200 stores the reconstructed blocks in the DPB 218. For example, in cases where the operation of the filter unit 216 is not required, the reconstruction unit 214 may store the reconstructed blocks in the DPB 218. In cases where the operation of the filter unit 216 is required, the filter unit 216 may store the filtered reconstructed blocks in the DPB 218. The motion estimation unit 222 and the motion compensation unit 224 may retrieve a reference picture from the DPB 218, formed from the reconstructed (and possibly filtered) blocks, to interpret blocks of the picture to be encoded later. In addition, the intraprediction unit 226 may use the reconstructed blocks in the DPB 218 of the current picture to intrapret other blocks in the current picture.

[0165] In general, the entropy coding unit 220 can entropy code syntax elements received from other functional components of the video encoder 200. For example, the entropy coding unit 220 can entropy code quantized transformation coefficient blocks from the quantization unit 208. As another example, the entropy coding unit 220 can entropy code prediction syntax elements from the mode selection unit 202 (e.g., motion information for inter-prediction or intra-mode information for intra-prediction). The entropy coding unit 220 can perform one or more entropy coding operations on syntax elements, which are another example of video data, to generate entropy coded data. For example, the entropy coding unit 220 can perform context-adaptive variable-length coding (CAVLC) operation, CABAC operation, variable-length to variable-length (V2V) coding operation, syntax-based context-adaptive binary arithmetic coding (SBAC) operation, probability interval partitioned entropy (PIPE) coding operation, exponential Golomb coding operation, or another type of entropy coding operation on the data. In some examples, the entropy coding unit 220 may operate in a bypass mode in which syntax elements are not entropically coded.

[0166] The video encoder 200 may output a bitstream containing entropy-encoded syntax elements required to reconstruct a slice or block of a picture. Specifically, the entropy encoding unit 220 may output a bitstream.

[0167] The behavior described above is described in relation to blocks. Such a description should be understood as the behavior for rumacoding blocks and / or chromacoding blocks. As described above, in some examples, the rumacoding block and chromacoding block are the ruma and chroma components of the CU. In some examples, the rumacoding block and chromacoding block are the ruma and chroma components of the PU.

[0168] In some cases, actions performed for a rumacoding block do not need to be repeated for a chromacoding block. For example, actions to identify the motion vector (MV) and reference picture for a rumacoding block do not need to be repeated to identify the MV and reference picture for a chromablock. Rather, the MV for the rumacoding block may be scaled to determine the MV for the chromablock, and the reference picture may be the same. In another example, the intra-prediction process may be the same for both rumacoding and chromacoding blocks.

[0169] Thus, the video encoder 200 in Figure 11 represents an example of a device for encoding and decoding video data, comprising a memory configured to store video data and one or more processors implemented in the circuit, the one or more processors being configured to determine an initial motion vector for the current block of video data, the initial motion vector having integer motion vector difference (MVD) precision, determine a search range around a reference area identified by the initial motion vector in a reference picture, perform a template matching search process within the search range to identify the best matching region, determine an error value of adjacent pixels for the best matching region, use the error value of adjacent pixels to perform a model-based fractional pixel motion vector improvement to derive a motion vector difference value, apply at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block, and decode the current block using the improved motion vector.

[0170] Figure 12 is a block diagram showing an exemplary video decoder 300 capable of performing the techniques of this disclosure. Figure 12 is provided for illustrative purposes and is not intended to limit the techniques that are more broadly illustrated and described in this disclosure. For illustrative purposes, this disclosure describes a video decoder 300 using VVC and HEVC (ITU-T H.265) techniques. However, the techniques of this disclosure may be performed by video coding devices configured according to other video coding standards.

[0171] In the example shown in Figure 12, the video decoder 300 includes a coding picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transformation processing unit 308, a reconstruction unit 310, a filter unit 312, and a decoding picture buffer (DPB) 314. Any or all of the CPB memory 320, entropy decoding unit 302, prediction processing unit 304, inverse quantization unit 306, inverse transformation processing unit 308, reconstruction unit 310, filter unit 312, and DPB 314 may be implemented in one or more processors or processing circuits. For example, the units of the video decoder 300 may be implemented as one or more circuits or logic elements as part of a hardware circuit, or as part of a processor, ASIC, or FPGA. Furthermore, the video decoder 300 may include additional or alternative processors or processing circuits to perform these and other functions.

[0172] The prediction processing unit 304 includes a motion compensation unit 316 and an intra-prediction unit 318. The prediction processing unit 304 may include additional units for performing predictions according to other prediction modes. For example, the prediction processing unit 304 may include a pallet unit, an intra-block copy unit (which may form part of the motion compensation unit 316), an affine unit, a linear model (LM) unit, and so on. In other examples, the video decoder 300 may include more, fewer, or different functional components.

[0173] The CPB memory 320 can store video data, such as an encoded video bitstream, to be decoded by the components of the video decoder 300. Video data stored in the CPU memory 320 may be obtained, for example, from a computer-readable medium 110 (Figure 1). The CPU memory 320 may include a CPB that stores encoded video data (e.g., syntax elements) from the encoded video bitstream. The CPB memory 320 may also store video data other than syntax elements of the coded picture, such as temporary data representing outputs from various units of the video decoder 300. The DPB 314 generally stores the decoded picture, and the video decoder 300 may output and / or use this decoded picture as reference video data when decoding subsequent data or pictures from the encoded video bitstream. The CPB memory 320 and DPB 314 may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM) including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The CPB memory 320 and DPB 314 may be provided by the same memory device or by separate memory devices. In various examples, the CPB memory 320 may be on-chip with the other components of the video decoder 300, or it may be off-chip relative to those components.

[0174] As an addition or alternative, in some examples, the video decoder 300 may retrieve coded video data from memory 120 (Figure 1). That is, memory 120 may store data such as that discussed above for the CPB memory 320. Similarly, memory 120 may store instructions to be executed by the video decoder 300 when some or all of the functions of the video decoder 300 are implemented in software to be performed by the processing circuit of the video decoder 300.

[0175] The various units shown in Figure 12 are presented to help understand the operations performed by the video decoder 300. The units may be implemented as fixed-function circuits, programmable circuits, or a combination thereof. Similar to Figure 11, fixed-function circuits refer to circuits that provide a specific function, with predefined operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functionality in their operations. For example, a programmable circuit may execute software or firmware that operates it, in a manner defined by software or firmware instructions. Fixed-function circuits may execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is generally immutable. In some examples, one or more units may be separate circuit blocks (fixed-function or programmable), and in some examples, one or more units may be integrated circuits.

[0176] The video decoder 300 may include a programmable core formed from an ALU, EFU, digital circuitry, analog circuitry, and / or programmable circuitry. In an example where the operation of the video decoder 300 is performed by software running on the programmable circuitry, on-chip memory or off-chip memory may store software instructions (e.g., object code) that the video decoder 300 receives and executes.

[0177] The entropy decoding unit 302 can receive video data encoded from the CPB and reconstruct the syntax elements by entropy decoding the video data. The prediction processing unit 304, the inverse quantization unit 306, the inverse transformation processing unit 308, the reconstruction unit 310, and the filter unit 312 can generate the decoded video data based on the syntax elements extracted from the bitstream.

[0178] Generally, the video decoder 300 reconstructs the picture block by block. The video decoder 300 may perform the reconstruction operation for each block individually (where the block currently being reconstructed, i.e., decoded, may be called the "current block").

[0179] The entropy decoding unit 302 can entropy decode the syntax elements that define the quantized transformation coefficients of the quantized transformation coefficient block, as well as transformation information such as quantization parameters (QP) and / or transformation mode indications. The inverse quantization unit 306 may use the QP associated with the quantized transformation coefficient block to determine the degree of quantization and, similarly, the degree of inverse quantization that the inverse quantization unit 306 should apply. The inverse quantization unit 306 may, for example, perform a bitwise left shift operation to inverse quantize the quantized transformation coefficients. In this way, the inverse quantization unit 306 may form a transformation coefficient block containing the transformation coefficients.

[0180] After the inverse quantization unit 306 has formed a transformation coefficient block, the inverse transformation processing unit 308 may apply one or more inverse transformations to the transformation coefficient block to generate a residual block associated with the current block. For example, the inverse transformation processing unit 308 may apply an inverse DCT, an inverse integer transformation, an inverse Carunenlebe transformation (KLT), an inverse rotation transformation, an inverse direction transformation, or another inverse transformation to the transformation coefficient block.

[0181] Furthermore, the prediction processing unit 304 generates a prediction block according to the prediction information syntax element entropy-decoded by the entropy decoding unit 302. For example, if the prediction information syntax element indicates that the current block is inter-predicted, the motion compensation unit 316 may generate a prediction block. In this case, the prediction information syntax element may indicate a reference picture in the DPB 314 from which the reference block should be fetched, as well as a motion vector that specifies the position of the reference block in the reference picture relative to the position of the current block in the current picture. The motion compensation unit 316 may generally perform inter-prediction processing in substantially the same manner as described with respect to the motion compensation unit 224 (FIG. 11).

[0182] Furthermore, according to the techniques of the present disclosure, the motion compensation unit 316 may improve the decoded motion vector received from the entropy decoding unit 302, for example, using template matching and model-based fractional pixel motion vector refinement. The motion compensation unit 316 may apply any or all of the techniques of the present disclosure for decoder-side motion vector improvement / derivation, either alone or in any combination.

[0183] Moreover, the motion compensation unit 316 may be configured to perform decoder-side motion vector improvement / derivation techniques according to any or all of the techniques of the present disclosure in any combination. For example, the motion compensation unit 316 may perform template matching and model-based fractional pixel motion vector refinement to improve the full pixel resolution motion vector first received from the entropy decoding unit 302 using these techniques.

[0184] For example, motion compensation unit 316 may receive data representing an initial motion vector from entropy decoding unit 302. The initial motion vector may have full integer pixel resolution. Motion compensation unit 316 and entropy decoding unit 302 may decode the initial motion vector using advanced motion vector prediction (AMVP) with motion vector difference (MVD) values ​​having full integer pixel precision. Motion compensation unit 316 may further refine the initial motion vector, which may include, for example, determining a search range around a reference area identified by the initial motion vector in a reference picture. Motion compensation unit 316 may then perform a template matching search process within the search range to identify the best matching region. The best matching region may be the region that produces the lowest rate distortion optimization (RDO) value. The RDO value may be calculated using, for example, the sum of absolute difference values, the sum of squared difference values, the mean absolute difference value, the mean squared difference value, etc.

[0185] By determining the best-matching region, the motion compensation unit 316 may obtain an improved motion vector with integer precision. The motion compensation unit 316 may then further determine error values ​​for adjacent pixels within the best-matching region in order to perform model-based fractional-pixel motion vector improvement to derive motion vector difference values ​​(e.g., x (horizontal) and / or y (vertical) component motion vector difference values). The motion compensation unit 316 may further apply at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block. The improved motion vector may have fractional-pixel precision. The motion compensation unit 316 may then use the improved motion vector to generate a predicted block and provide the predicted block to the reconstruction unit 310.

[0186] Specifically, for each of the motion vector difference components, the motion compensation unit 316 may determine whether the error value for the best matching region is equal to zero. The motion compensation unit 316 then further determines that the absolute value of the difference between a component of the initial motion vector and the corresponding motion vector difference value for that component is less than the corresponding search region value for that component, and then may apply the motion vector difference value for that component to that component of the initial motion vector.

[0187] As stated above, the motion compensation unit 316 may also decide whether to perform motion vector improvement techniques of the present disclosure. For example, the motion compensation unit 316 may decide that the initial motion vector is in coding unit (CU) level merge mode with bipredicted motion vectors, that the first reference picture for the current block has a display order value smaller than the display order value of the current picture containing the current block, that the second reference picture for the current block has a display order value larger than the display order value for the current picture, that the time distances between the current picture, the first reference picture and the second reference picture are equal, that the current block has more than 64 lumens, that the height of the current block is at least 8 lumens, that the width of the current block is at least 8 lumens, that the weight values ​​for biprediction with CU level weights (BCW) are equal, that weighted prediction (WP) is not enabled for the current block, and that synthetic intrainterpretation (CIIP) mode is not used for the current block, and in response, decide on a motion vector improvement technique.

[0188] In some examples, if the current block is bipredicted, the motion compensation unit 316 may perform these motion vector improvement techniques for both motion vectors for the current block.

[0189] As another example, if the prediction information syntax element indicates that the current block is intra-predicted, the intra-prediction unit 318 may generate a predicted block according to the intra-prediction mode indicated by the prediction information syntax element. Again, the intra-prediction unit 318 may generally perform intra-prediction processing in substantially the same manner as described with respect to the intra-prediction unit 226 (Figure 11). The intra-prediction unit 318 may retrieve data from the DPB 314 for samples adjacent to the current block.

[0190] The reconstruction unit 310 may reconstruct the current block using the predicted block and the residual block. For example, the reconstruction unit 310 may reconstruct the current block by adding samples from the residual block to the corresponding samples from the predicted block.

[0191] The filter unit 312 may perform one or more filtering operations on the reconstructed block. For example, the filter unit 312 may perform a deblocking operation to reduce blockingness artifacts along the edges of the reconstructed block. In not all examples, the filter unit 312 will perform any operations.

[0192] The video decoder 300 may store the reconstructed blocks in the DPB 314. For example, in cases where the operation of the filter unit 312 is not required, the reconstruction unit 310 may store the reconstructed blocks in the DPB 314. In cases where the operation of the filter unit 312 is performed, the filter unit 312 may store the filtered reconstructed blocks in the DPB 314. As discussed above, the DPB 314 may provide the prediction processing unit 304 with reference information such as the current picture for intra-prediction and samples of previously decoded pictures for subsequent motion compensation. Furthermore, the video decoder 300 may output the decoded pictures from the DPB 314 for later presentation on a display device such as the display device 118 in Figure 1.

[0193] Thus, the video decoder 300 in Figure 12 represents an example of a device for decoding video data, comprising a memory configured to store video data and one or more processors implemented in the circuit, the one or more processors being configured to determine an initial motion vector for the current block of video data, the initial motion vector having integer motion vector difference (MVD) precision, determine a search range around a reference area identified by the initial motion vector in a reference picture, perform a template matching search process within the search range to identify the best matching region, determine an error value of adjacent pixels for the best matching region, use the error value of adjacent pixels to perform a model-based fractional pixel motion vector improvement to derive a motion vector difference value, apply at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block, and decode the current block using the improved motion vector.

[0194] Figure 13 is a flowchart illustrating an exemplary method for encoding a current block using the technique of the present disclosure. The current block may comprise a current CU. While the video encoder 200 (Figures 1 and 11) is described, it should be understood that other devices may be configured to perform a similar method to that shown in Figure 13.

[0195] In this example, the video encoder 200 first predicts the current block (350). For example, the video encoder 200 may form a predicted block for the current block. Specifically, the video encoder 200 may apply any or all of the techniques of the present disclosure, either alone or in any combination, to perform decoder-side motion vector derivation / improvement, for example, using model-based motion vector difference derivation and template matching prediction, to form an improved motion vector, and then use the improved motion vector to generate a predicted block. The video encoder 200 may then compute a residual block for the current block (352). To compute the residual block, the video encoder 200 may compute the difference between the original uncoded block and the predicted block for the current block. The video encoder 200 may then transform and quantize the coefficients of the residual block (354). The video encoder 200 may then scan the quantized transformed coefficients of the residual block (356). During or following the scan, the video encoder 200 may entropy encode the coefficients (358). For example, the video encoder 200 may encode the coefficients using CAVLC or CABAC. The video encoder 200 may then output the entropy-encoded data of the block (360).

[0196] The video encoder 200 may also encode the current block and then decode the current block to use the decoded version of the current block as reference data for data to be coded later (for example, in inter-prediction mode or intra-prediction mode). Thus, the video encoder 200 may reconstruct the residual block by inverse quantization and inverse transformation of the coefficients (362). The video encoder 200 may combine the residual block with the prediction block to form a decoded block (364). The video encoder 200 may then store the decoded block in the DPB 218 (366).

[0197] Thus, the method in Figure 13 represents an example of a method for decoding video data, comprising the steps of: determining an initial motion vector for the current block of video data, wherein the initial motion vector has integer motion vector difference (MVD) precision; determining a search range around a reference area identified by the initial motion vector in a reference picture; performing a template matching search process within the search range to identify the best matching region; determining the error value of adjacent pixels for the best matching region; using the error value of adjacent pixels to perform a model-based fractional pixel motion vector improvement to derive a motion vector difference value; applying at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block; and decoding the current block using the improved motion vector. The method in Figure 13 also represents such a method for encoding video data.

[0198] Figure 14 is a flowchart illustrating an exemplary method for decoding the current block using the technique of the present disclosure. The current block may comprise the current CU. While the video decoder 300 (Figures 1 and 12) is described, it should be understood that other devices may be configured to perform a method similar to that of Figure 14.

[0199] The video decoder 300 may receive entropy-encoded data for the current block, such as entropy-encoded prediction information and entropy-encoded data for the coefficients of the residual block corresponding to the current block (370). The video decoder 300 may entropy-decode the entropy-encoded data to determine the prediction information for the current block and to reconstruct the coefficients of the residual block (372). The video decoder 300 may predict the current block, for example, using an inter-prediction mode as indicated by the prediction information for the current block, in order to compute a prediction block for the current block (374). Specifically, the video decoder 300 may apply any or all of the techniques of the present disclosure, individually or in any combination, to perform decoder-side motion vector derivation / improvement, for example, using model-based motion vector difference derivation and template matching prediction, to form an improved motion vector, and then generate a prediction block using the improved motion vector. The video decoder 300 may then backscan the reconstructed coefficients to produce a block of quantized transformation coefficients (376). Next, the video decoder 300 can inversely quantize and inversely transform the quantized transformation coefficients to generate residual blocks (378). Finally, the video decoder 300 can decode the current block by combining the predicted block and the residual block (380).

[0200] Thus, the method in Figure 14 represents an example of a method for decoding video data, comprising the steps of: determining an initial motion vector for the current block of video data, wherein the initial motion vector has integer motion vector difference (MVD) precision; determining a search range around a reference area identified by the initial motion vector in a reference picture; performing a template matching search process within the search range to identify the best matching region; determining the error value of adjacent pixels for the best matching region; using the error value of adjacent pixels to derive a motion vector difference value by performing a model-based fractional pixel error vector improvement; applying at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block; and decoding the current block using the improved motion vector.

[0201] Figure 15 is a flowchart illustrating an exemplary method for encoding or decoding video data using the technique of the present disclosure. The method of Figure 15 is described with respect to the video decoder 300 of Figures 1 and 12. Other devices, such as the video encoder 200 of Figures 1 and 11, may also be configured to perform this method or a similar method.

[0202] First, the video decoder 300 may obtain an initial motion vector having an integer-precision motion vector difference (MVD) value for the current block (400). The video decoder 300 may decode data representing the initial motion vector. In some examples, the video decoder 300 may use bilateral matching to refine the decoded data representing the motion vector, and the resulting motion vector may be the initial motion vector. In examples where the video encoder 200 performs the method, the video encoder 200 may obtain the initial motion vector through a search process performed by, for example, the motion estimation unit 222 as discussed above.

[0203] The video decoder 300 may then determine a search range (402), which is around a reference area identified by an initial motion vector in the reference picture. The video decoder 300 may then perform a template matching search process within the search range to identify the best matching region (404).

[0204] The video decoder 300 may then determine the error value of adjacent pixels for the best matching region (406). The video decoder 300 may use the error value to derive a motion vector difference value by performing a model-based fractional pixel motion vector improvement (408). The video decoder 300 may then apply the motion vector difference value to the initial motion vector to obtain the improved motion vector (410). The video decoder 300 may then decode the current block using the improved motion vector (412). In an example where the video encoder 200 performs the method, the video encoder 200 may both encode and decode the current block using the improved motion vector.

[0205] Thus, the method in Figure 15 represents an example of a method for decoding video data, comprising the steps of: determining an initial motion vector for the current block of video data, wherein the initial motion vector has integer motion vector difference (MVD) precision; determining a search range around a reference area identified by the initial motion vector in a reference picture; performing a template matching search process within the search range to identify the best matching region; determining the error value of adjacent pixels for the best matching region; using the error value of adjacent pixels to derive a motion vector difference value by performing a model-based fractional pixel error vector improvement; applying at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block; and decoding the current block using the improved motion vector.

[0206] Various examples of the techniques used in this disclosure are summarized in the following clauses.

[0207] Clause 1: A method for decoding video data, comprising the steps of: decoding data representing an initial motion vector for a current block of video data, wherein the initial motion vector has integer motion vector difference (MVD) precision; determining a search range around a reference area identified by the initial motion vector in a reference picture; performing a template matching search process within the search range to identify the best matching region; determining an error value for adjacent pixels to the best matching region; using the error value for adjacent pixels to perform a model-based fractional pixel motion vector improvement to derive a motion vector difference value; applying at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block; and decoding the current block using the improved motion vector.

[0208] Clause 2: The method of Clause 1, wherein the motion vector difference value includes a horizontal motion vector difference value and a vertical motion vector difference value.

[0209] Clause 3: Any method of Clauses 1 and 2, wherein the step of applying at least one of the motion vector difference values ​​comprises the step of applying all of the motion vector difference values.

[0210] Clause 4: Any method of Clauses 1 and 2, wherein the step of applying at least one motion vector difference value comprises determining that the error value of the best-matching region is not equal to zero, and applying the motion vector difference value for a component to that component in response to determining that the absolute value of the difference between the component of the initial motion vector and the corresponding motion vector difference value for that component is less than the corresponding search region value for that component.

[0211] Clause 5: The method of either Clause 1 or 2, wherein the step of applying at least one of the motion vector difference values ​​comprises a step of preventing the application of the motion vector difference value for a component to that component in response to determining at least one of the following: that the error value of the best matching region is equal to zero, or that the absolute value of the difference between a component of the initial motion vector and the corresponding motion vector difference value for that component is greater than or equal to the corresponding search region value for that component.

[0212] Clause 6: The step of performing model-based fractional pixel motion vector improvement is performed if 1) the initial motion vector is improved using bilateral matching or decoder-side motion vector (DMVR) improvement, 2) the improved motion vector is further improved using bilateral matching or DMVR, or 3) the initial motion vector is in coding unit (CU) level merge mode with bilateral predictive motion vectors, the first reference picture for the current block has a display order value smaller than the display order value of the current picture containing the current block, the second reference picture for the current block has a display order value larger than the display order value of the current picture, the current pixel Any method of clauses 1 to 5, comprising the step of performing a model-based fractional pixel motion vector improvement in response to determining one or more of the following: the time distance between the Cha and the first and second reference pictures is equal; the current block has more than 64 lumens; the height of the current block is at least 8 lumens; the width of the current block is at least 8 lumens; the weight values ​​for biprediction with CU level weights (BCW) are equal; weighted prediction (WP) is not enabled for the current block; and the synthetic intra-interpretation (CIIP) mode is not used for the current block.

[0213] Clause 7: The step of performing model-based fractional-pixel motion vector improvement is performed if 1) the initial motion vector is not improved using bilateral matching or decoder-side motion vector (DMVR) improvement, 2) the improved motion vector is not further improved using bilateral matching or DMVR, or 3) the initial motion vector is in coding unit (CU) level merge mode with bilateral predictive motion vectors, the first reference picture for the current block has a display order value smaller than the display order value of the current picture containing the current block, the second reference picture for the current block has a display order value larger than the display order value of the current picture, and the current picture and the first Any method of clauses 1 to 5, comprising the step of performing a model-based fractional-pixel motion vector improvement in response to determining that at least one of the following is not true: the time distance between a reference picture and a second reference picture is equal; the current block has more than 64 luma pixels; the height of the current block is at least 8 luma pixels; the width of the current block is at least 8 luma pixels; the weight values ​​for biprediction with CU-level weights (BCW) are equal; weighted prediction (WP) is not enabled for the current block; and the synthetic intra-interpretation (CIIP) mode is not used for the current block.

[0214] Clause 8: The motion vector difference value comprises a horizontal motion vector difference (Δx) and a vertical motion vector difference (Δy), and the error value of adjacent pixels is E x,y The step of determining the error value is E x, y =A(x -Δx) 2 +B(y -Δy) 2 The steps include calculating +C and performing model-based fractional pixel motion vector improvements, s*floor((α(E -1,0 -E 1,0 )) / (2(E -1,0 +E 1,0 -2E 0,0The steps are to calculate Δx according to ))) and s*floor((α(E 0,-1 -E 0,1 )) / (2(E 0,-1 +E 0,1 -2E 0,0 The method comprises the step of calculating Δy according to one of the methods of clauses 1 to 7, wherein α is an integer scaling factor introduced to represent (Δx, Δy) with decimal-pel precision, and s is the scaling value.

[0215] Clause 9: The method of Clause 8, further comprising the step of determining the value of α according to the Adaptive Motion Vector Resolution (AMVR) mode for the current block.

[0216] Clause 10: Any method of Clauses 8 and 9, further comprising the step of determining the value of α depending on whether a 1 / 2 pixel interpolation filter is used.

[0217] Clause 11: Any method of Clauses 9 and 10, wherein the step of determining the value of α is to determine that the value of α is equal to 0 or 1 when the AMVR mode is one of full pixels, 2 pixels, 4 pixels, or greater than 4 pixels.

[0218] Clause 12: Any method of Clauses 9 to 11, wherein the step of determining the value of α comprises the step of determining that the value of α is equal to 2 when the AMVR mode is 1 / 2 pixel or when a 1 / 2 pixel interpolation filter is used.

[0219] Clause 13: Any method of Clauses 9 to 12, wherein the step of determining the value of α is to determine that the value of α is equal to 4 when the AMVR mode is 1 / 4 pixel.

[0220] Clause 14: Any method of Clauses 9 to 13, wherein the step of determining the value of α is to determine that the value of α is equal to 8 when the AMVR mode is 1 / 8 pixel.

[0221] Clause 15: Any method of Clauses 9 to 14, wherein the step of determining the value of α is to determine that the value of α is equal to 16 when the AMVR mode is 1 / 16 pixel.

[0222] Clause 16: Any method of Clauses 9 to 15, wherein the step of determining the value of α comprises the step of determining that the value of α is equal to N when the AMVR mode is 1 / N pixels.

[0223] Any method of clauses 9 through 16, further comprising the step of calculating the value of s as equal to 16 / α.

[0224] Clause 18: further comprising the step of calculating the value of s equal to M / α, where M is the motion vector memory precision, using any method from Clauses 9 to 16.

[0225] Clause 19: Any method from Clauses 9 to 16, wherein when the AMVR mode is 1 / 2 pixel, the step of performing a template matching search process comprises the step of performing a template matching search process with full pixel motion vector difference accuracy, the step of determining the value of α comprises the step of determining that the value of α is equal to 2, and the method further comprises the step of determining that the value of s is equal to 8.

[0226] Clause 20: Any method of Clauses 9 to 16, wherein when the AMVR mode is 1 / 4 pixel, the step of performing a template matching search process comprises the step of performing a template matching search process with full pixel motion vector difference accuracy, the step of determining the value of α comprises the step of determining that the value of α is equal to 4, and the method further comprises the step of determining that the value of s is equal to 4.

[0227] Clause 21: Any method from Clauses 9 to 16, wherein when the AMVR mode is 1 / 4 pixel, the step of performing a template matching search process comprises the step of performing a template matching search process with full pixel motion vector difference accuracy, the step of determining the value of α comprises the step of determining that the value of α is equal to 2, and the method further comprises the step of determining that the value of s is equal to 4.

[0228] Clause 22: Any method of Clauses 9 to 16, wherein when the AMVR mode is 1 / 16 pixels, the step of performing a template matching search process comprises the step of performing a template matching search process with full pixel motion vector difference accuracy, the step of determining the value of α comprises the step of determining that the value of α is equal to 2, and the method further comprises the step of determining that the value of s is equal to 1.

[0229] Clause 23: Any method of Clauses 9 to 16, wherein when the AMVR mode is 1 / 16 pixels, the step of performing a template matching search process comprises the step of performing a template matching search process with full pixel motion vector difference accuracy, the step of determining the value of α comprises the step of determining that the value of α is equal to 8, and the method further comprises the step of determining that the value of s is equal to 1.

[0230] Clause 24: Any method of Clauses 9 to 16, wherein when the AMVR mode is 1 / 16 pixel, the step of performing a template matching search process comprises the steps of performing a template matching search process with full pixel motion vector difference accuracy and with 1 / 2 pixel motion vector difference accuracy, the step of determining the value of α comprises the step of determining that the value of α is equal to 4, and the method further comprises the step of determining that the value of s is equal to 1.

[0231] Clause 25: Any method of Clauses 9 to 16, wherein when the AMVR mode is 1 / 16 pixel, the step of performing a template matching search process comprises the steps of performing a template matching search process with full pixel motion vector difference accuracy, with 1 / 2 pixel motion vector difference accuracy, and with 1 / 4 pixel motion vector difference accuracy, and the step of determining the value of α comprises the step of determining that the value of α is equal to 2, and the method further comprises the step of determining that the value of s is equal to 1.

[0232] Clause 26: Any method of Clauses 1 to 25, wherein the current block is bipredicted, the initial motion vector comprises a first initial motion vector, the improved motion vector comprises a first improved motion vector, and the method further comprises the step of improving a second initial motion vector for the current block using the first improved motion vector.

[0233] Clause 27: The method of Clause 26, wherein the first initial motion vector is associated with reference picture list 0.

[0234] Clause 28: The method of Clause 26, wherein a first initial motion vector is associated with reference picture list 1.

[0235] Clause 29: The method of Clause 26, wherein a first initial motion vector is associated with a reference picture list x, and the method further comprises the step of determining that x is equal to 1 when the data indicates that the motion vector difference value for reference picture list 1 is forced to be equal to 0, and the data of the syntax element for the motion vector difference value for reference picture list 1 is not decoded.

[0236] Clause 30: The method of Clause 26, wherein a first initial motion vector is associated with a reference picture list x, and the method further comprises the step of determining that x is equal to 1 when the data indicates that the motion vector difference value for reference picture list 1 is not forced to be equal to 0, and the data of the syntax element for the motion vector difference value for reference picture list 1 is decoded.

[0237] Clause 31: The method of Clause 26, wherein a first initial motion vector is associated with reference picture list x, and the method further comprises the step of determining the value of x as either 0 or 1 according to the TM cost of the motion vectors for reference picture list 0 and reference picture list 1.

[0238] The method of Clause 31, wherein the step of determining the value of x is to determine that x is equal to 0 when the TM cost of the motion vector for reference picture list 0 is less than the TM cost of the motion vector for reference picture list 1.

[0239] The method of Clause 31, wherein the step of determining the value of x is to determine that x is equal to 0 when the TM cost of the motion vector for reference picture list 1 is less than the TM cost of the motion vector for reference picture list 0.

[0240] Clause 34: Any method of Clauses 1 through 33, further comprising the step of refining the decoded data representing the motion vectors using bilateral matching to form the initial motion vectors.

[0241] Clause 35: Any method of Clauses 1 to 33, further comprising the step of refining the improved motion vector using bilateral matching to form a further improved motion vector, wherein the step of decoding the current block comprises the step of decoding the current block using the further improved motion vector.

[0242] Clause 36: Bilateral matching and template matching share the same search scope, in any manner described in Clauses 34 and 35.

[0243] Clause 37: Any method of Clauses 1 to 36, wherein the step of performing a template matching search process comprises the step of performing a template matching search process after determining that the adaptive motion vector resolution (AMVR) mode is not a 4-pixel mode.

[0244] Clause 38: Any method of Clauses 1 to 36, wherein the step of performing a template matching search process comprises the step of performing a template matching search process after determining that the adaptive motion vector resolution (AMVR) mode is greater than the 2-pixel mode.

[0245] Clause 39: Any method of Clauses 1 through 38, further comprising the step of encoding the current block before decoding the current block.

[0246] Clause 40: A device for decoding video data, comprising one or more means for performing any method of Clauses 1 to 39.

[0247] Clause 41: The device of Clause 40, comprising one or more means, one or more processors implemented in a circuit.

[0248] Clause 42: Any device under Clauses 40 and 41 further comprising a display configured to display decoded video data.

[0249] Clause 43: Any device under Clauses 40 to 42, comprising one or more of the following: a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

[0250] Clause 44: A device of Clauses 40 to 43 further comprising memory configured to store video data.

[0251] Clause 45: A computer-readable storage medium that, when executed, stores instructions causing the processor of a device to perform any of the methods described in Clauses 1 through 39 for decoding video data.

[0252] Clause 46: A device for decoding video data, comprising: means for decoding data representing an initial motion vector for a current block of video data, wherein the initial motion vector has integer motion vector difference (MVD) precision; means for determining a search range around a reference area identified by the initial motion vector in a reference picture; means for performing a template matching search process within the search range to identify the best matching region; means for determining an error value of an adjacent pixel for the best matching region; means for using the error value of an adjacent pixel to perform a model-based fractional pixel motion vector improvement to derive a motion vector difference value; means for applying at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block; and means for decoding the current block using the improved motion vector.

[0253] Clause 47: A method for decoding video data, comprising the steps of: decoding data representing an initial motion vector for a current block of video data, wherein the initial motion vector has integer motion vector difference (MVD) precision; determining a search range around a reference area identified by the initial motion vector in a reference picture; performing a template matching search process within the search range to identify the best matching region; determining an error value for an adjacent pixel to the best matching region; using the error value for an adjacent pixel to derive a motion vector difference value by performing a model-based fractional pixel error vector improvement; applying at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block; and decoding the current block using the improved motion vector.

[0254] Clause 48: The method of Clause 47, wherein the motion vector difference value includes a horizontal motion vector difference value and a vertical motion vector difference value.

[0255] Clause 49: The method of Clause 47, wherein the step of applying at least one of the motion vector difference values ​​comprises the step of applying all of the motion vector difference values.

[0256] The method of Clause 50: The method of Clause 47, wherein the step of applying at least one motion vector difference value comprises determining that the error value of the best matching region is not equal to zero, and applying the motion vector difference value for a component to a component in response to determining that the absolute value of the difference between the component of the initial motion vector and the corresponding motion vector difference value for that component is less than the corresponding search region value for that component.

[0257] The method of Clause 51: The method of Clause 47, wherein the step of applying at least one of the motion vector difference values ​​comprises the step of preventing the application of the motion vector difference value for a component to that component in response to determining at least one of the following: that the error value of the best matching region is equal to zero, or that the absolute value of the difference between a component of the initial motion vector and the corresponding motion vector difference value for that component is greater than or equal to the corresponding search region value for that component.

[0258] Clause 52: The method of Clause 47, wherein the step of performing model-based fractional-pixel motion vector refinement is performed in response to determining that the initial motion vector is in coding unit (CU) level merge mode with bipredictive motion vectors, that a first reference picture for the current block has a display order value smaller than the display order value of the current picture containing the current block, that a second reference picture for the current block has a display order value larger than the display order value of the current picture, that the time distances between the current picture, the first reference picture and the second reference picture are equal, that the current block has more than 64 luma pixels, that the height of the current block is at least 8 luma pixels, that the width of the current block is at least 8 luma pixels, that the weight values ​​for bipredictive (BCW) with CU level weights are equal, that weighted prediction (WP) is not enabled for the current block, and that synthetic intrainterpredictive (CIIP) mode is not used for the current block.

[0259] Clause 53: The method of Clause 47, wherein the step of performing model-based fractional-pixel motion vector improvement is a step of performing model-based fractional-pixel motion vector improvement in response to determining that 1) an initial motion vector has been improved using bilateral matching or decoder-side motion vector (DMVR) improvement, or 2) an improved motion vector has been further improved using bilateral matching or DMVR.

[0260] Clause 54: The step of performing model-based fractional pixel motion vector improvement is performed if 1) the initial motion vector is not improved using bilateral matching or decoder-side motion vector (DMVR) improvement, 2) the improved motion vector is not further improved using bilateral matching or DMVR, or 3) the initial motion vector is in coding unit (CU) level merge mode with bilateral predictive motion vectors, the first reference picture for the current block has a display order value smaller than the display order value of the current picture containing the current block, the second reference picture for the current block has a display order value larger than the display order value of the current picture, the current picture The method of Clause 47, comprising the step of performing a model-based fractional pixel motion vector improvement in response to determining that at least one of the following is not true: the time distance between the first reference picture and the second reference picture is equal; the current block has more than 64 lumens; the height of the current block is at least 8 lumens; the width of the current block is at least 8 lumens; the weight values ​​for biprediction with CU level weights (BCW) are equal; weighted prediction (WP) is not enabled for the current block; and the synthetic intra-interprediction (CIIP) mode is not used for the current block.

[0261] Clause 55: The motion vector difference value comprises a horizontal motion vector difference (Δx) and a vertical motion vector difference (Δy), and the error value of adjacent pixels is E x,y The step of determining the error value is E x, y =A(x -Δx) 2 +B(y -Δy) 2 The steps include calculating +C and performing model-based fractional pixel motion vector improvements, s*floor((α(E -1,0 -E 1,0 )) / (2(E -1,0 +E 1,0 -2E 0,0The steps are to calculate Δx according to ))) and s*floor((α(E 0,-1 -E 0,1 )) / (2(E 0,-1 +E 0,1 -2E 0,0 The method of Clause 47, comprising the step of calculating Δy according to ))), wherein α is an integer scaling factor introduced to represent (Δx, Δy) with decimal-pel precision, and s is the scaling value.

[0262] Clause 56: The method of Clause 55, further comprising the step of determining the value of α to be equal to 16, regardless of whether a 1 / 2 pixel interpolation filter is used.

[0263] Clause 57: The method of Clause 55, further comprising the step of determining the value of α according to the Adaptive Motion Vector Resolution (AMVR) mode for the current block.

[0264] The method of Clause 55, further comprising the step of determining the value of α depending on whether a 1 / 2 pixel interpolation filter is used.

[0265] Clause 59: The method of Clause 47, wherein the current block is bipredicted, the initial motion vector comprises a first initial motion vector, the improved motion vector comprises a first improved motion vector, and the method further comprises the step of improving a second initial motion vector for the current block using the first improved motion vector.

[0266] Clause 60: The method of Clause 47, further comprising the step of improving decoded data representing a motion vector using bilateral matching to form an initial motion vector.

[0267] Clause 61: The method of Clause 47, further comprising the step of refining the improved motion vector using bilateral matching to form a further improved motion vector, wherein the step of decoding the current block comprises the step of decoding the current block using the further improved motion vector.

[0268] Clause 62: The method of Clause 47, wherein the step of performing a template matching search process comprises the step of performing a template matching search process after determining that the adaptive motion vector resolution (AMVR) mode is not a 4-pixel mode.

[0269] Clause 63: The method of Clause 47, further comprising the step of encoding the current block before decrypting the current block.

[0270] Clause 64: A device for decoding video data, comprising a memory configured to store video data and one or more processors implemented in circuitry, wherein one or more processors are configured to decode data representing an initial motion vector for a current block of video data, the initial motion vector having integer motion vector difference (MVD) precision, determine a search range around a reference area identified by the initial motion vector in a reference picture, perform a template matching search process within the search range to identify the best matching region, determine an error value of adjacent pixels for the best matching region, use the error value of adjacent pixels to perform a model-based fractional pixel motion vector improvement to derive a motion vector difference value, apply at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block, and decode the current block using the improved motion vector.

[0271] Clause 65: The device of Clause 64, wherein, in order to apply at least one motion vector difference value, one or more processors are configured to apply a motion vector difference value for a component to that component in response to determining that the error value of the best matching region is not equal to zero and that the absolute value of the difference between the component of the initial motion vector and the corresponding motion vector difference value for that component is less than the corresponding search region value for that component.

[0272] Clause 66: A device of Clause 64, configured to perform model-based fractional-pixel motion vector refinement in response to one or more processors determining that the initial motion vector is in coding unit (CU) level merge mode with bipredictive motion vectors, that a first reference picture for the current block has a display order value smaller than the display order value of the current picture containing the current block, that a second reference picture for the current block has a display order value larger than the display order value of the current picture, that the time distances between the current picture, the first reference picture and the second reference picture are equal, that the current block has more than 64 lumens, that the height of the current block is at least 8 lumens, that the width of the current block is at least 8 lumens, that the weight values ​​for bipredictive (BCW) with CU level weights are equal, that weighted prediction (WP) is not enabled for the current block, and that synthetic intrainterpredictive (CIIP) mode is not used for the current block.

[0273] Clause 67: The motion vector difference value comprises a horizontal motion vector difference (Δx) and a vertical motion vector difference (Δy), and the error value of adjacent pixels is E x,y To obtain the value and determine the error value, one or more processors, E x, y =A(x -Δx) 2 +B(y -Δy) 2 Configured to calculate +C, one or more processors perform model-based fractional pixel motion vector improvements, s*floor((α(E -1,0 -E 1,0 )) / (2(E -1,0 +E 1,0 -2E 0,0 Calculate Δx according to ))) and s*floor((α(E 0,-1 -E 0,1 )) / (2(E 0,-1 +E 0,1 -2E 0,0A device of clause 64, configured to calculate Δy according to ))) and determine the value of α to be equal to 16 regardless of whether a 1 / 2 pixel interpolation filter is used, where α is an integer scaling factor introduced to represent (Δx, Δy) with some decimal-pel precision, and s is the scaling value.

[0274] Clause 68: The device of Clause 64, wherein one or more processors are further configured to use bilateral matching to refine the decoded data representing the motion vectors in order to form the initial motion vectors.

[0275] Clause 69: The device of Clause 64, wherein one or more processors are configured to further refine the refined motion vectors using bilateral matching to form a further refined motion vector, and one or more processors are configured to decode the current block using the further refined motion vector.

[0276] Clause 70: The device of Clause 64, wherein one or more processors are configured to perform a template matching search process after determining that the Adaptive Motion Vector Resolution (AMVR) mode is not 4-pixel mode.

[0277] Clause 71: The device of Clause 64, further comprising a display configured to display decoded video data.

[0278] Clause 72: A device under Clause 64 comprising one or more of the following: a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

[0279] Clause 73: A computer-readable storage medium storing instructions, wherein, when executed, the instructions cause a processor of a device for decoding video data to decode data representing an initial motion vector for a current block of video data, the initial motion vector having integer motion vector difference (MVD) precision, to determine a search range around a reference area identified by the initial motion vector in a reference picture, to perform a template matching search process within the search range to identify the best matching region, to determine an error value of adjacent pixels for the best matching region, to perform a model-based fractional pixel motion vector improvement using the error value of adjacent pixels to derive a motion vector difference value, to apply at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block, and to decode the current block using the improved motion vector.

[0280] A computer-readable storage medium according to Clause 73, comprising instructions that cause a processor to apply at least one motion vector difference value, causing the processor to determine that the error value of the best matching region is not equal to zero, and that the absolute value of the difference between the components of an initial motion vector and the corresponding motion vector difference value for that component is less than the corresponding search region value for that component, and then causing the processor to apply the motion vector difference value for that component to that component.

[0281] A computer-readable storage medium of Clause 73, comprising instructions causing a processor to perform model-based fractional-pixel motion vector refinement, in response to the processor determining that the initial motion vector is in coding unit (CU) level merge mode with bipredictive motion vectors; that a first reference picture for the current block has a display order value smaller than the display order value of the current picture containing the current block; that a second reference picture for the current block has a display order value greater than the display order value of the current picture; that the time distances between the current picture, the first reference picture, and the second reference picture are equal; that the current block has more than 64 lumens; that the height of the current block is at least 8 lumens; that the width of the current block is at least 8 lumens; that the weight values ​​for bipredictive (BCW) with CU level weights are equal; that weighted prediction (WP) is not enabled for the current block; and that synthetic intrainterpredictive (CIIP) mode is not used for the current block.

[0282] Clause 76: The motion vector difference value comprises a horizontal motion vector difference (Δx) and a vertical motion vector difference (Δy), and the error value of adjacent pixels is E x,y An instruction that contains a value and causes the processor to determine the error value, E x, y =A(x -Δx) 2 +B(y -Δy) 2 The instructions include an instruction to calculate +C, and an instruction to cause the processor to perform model-based fractional pixel motion vector improvement, which causes the processor to perform s*floor((α(E -1,0 -E 1,0 )) / (2(E -1,0 +E 1,0 -2E 0,0 Calculate Δx according to ))) and s*floor((α(E 0,-1 -E 0,1 )) / (2(E 0,-1 +E 0,1 -2E 0,0A computer-readable storage medium of Clause 73, comprising instructions to calculate Δy according to ))) and to determine the value of α to be equal to 16 regardless of whether a 1 / 2 pixel interpolation filter is used, wherein α is an integer scaling factor introduced to represent (Δx, Δy) with some decimal-pel precision, and s is the scaling value.

[0283] Clause 77: A computer-readable storage medium of Clause 73, further comprising instructions to cause the processor to improve decoded data representing motion vectors using bilateral matching to form initial motion vectors.

[0284] A computer-readable storage medium of Clause 73, further comprising instructions to cause the processor to decode the current block, the instructions to cause the processor to decode the current block using the further improved motion vector, and further comprising instructions to cause the processor to decode the current block using the further improved motion vector.

[0285] Clause 79: A computer-readable storage medium of Clause 73, wherein the instructions causing the processor to perform a template matching search process include instructions causing the processor to perform a template matching search process after determining that the adaptive motion vector resolution (AMVR) mode is not 4-pixel mode.

[0286] Clause 80: A device for decoding video data, comprising: means for decoding data representing an initial motion vector for a current block of video data, wherein the initial motion vector has integer motion vector difference (MVD) precision; means for determining a search range around a reference area identified by the initial motion vector in a reference picture; means for performing a template matching search process within the search range to identify the best matching region; means for determining an error value of an adjacent pixel for the best matching region; means for using the error value of an adjacent pixel to perform a model-based fractional pixel motion vector improvement to derive a motion vector difference value; means for applying at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block; and means for decoding the current block using the improved motion vector.

[0287] Clause 81: A method for decoding video data, comprising the steps of: decoding data representing an initial motion vector for a current block of video data, wherein the initial motion vector has integer motion vector difference (MVD) precision; determining a search range around a reference area identified by the initial motion vector in a reference picture; performing a template matching search process within the search range to identify the best matching region; determining an error value for adjacent pixels to the best matching region; using the error value for adjacent pixels to derive a motion vector difference value by performing a model-based fractional pixel error vector improvement; applying at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block; and decoding the current block using the improved motion vector.

[0288] Clause 82: The method of Clause 81, wherein the motion vector difference value includes a horizontal motion vector difference value and a vertical motion vector difference value.

[0289] Clause 83: Any method of Clauses 81 and 82, wherein the step of applying at least one of the motion vector difference values ​​comprises the step of applying all of the motion vector difference values.

[0290] Clause 84: Any method of Clauses 81 and 82, wherein the step of applying at least one motion vector difference value comprises determining that the error value of the best matching region is not equal to zero, and applying the motion vector difference value for a component to that component in response to determining that the absolute value of the difference between the component of the initial motion vector and the corresponding motion vector difference value for that component is less than the corresponding search region value for that component.

[0291] Clause 85: Any method of Clauses 81 and 82, wherein the step of applying at least one of the motion vector difference values ​​comprises the step of preventing the application of the motion vector difference value for a component to that component in response to determining at least one of the following: that the error value of the best matching region is equal to zero, or that the absolute value of the difference between a component of the initial motion vector and the corresponding motion vector difference value for that component is greater than or equal to the corresponding search region value for that component.

[0292] Clause 86: Any method of Clauses 81 to 85, wherein the step of performing model-based fractional-pixel motion vector refinement comprises the step of performing model-based fractional-pixel motion vector refinement in response to determining that the initial motion vector is in coding unit (CU) level merge mode with bipredictive motion vectors, that a first reference picture for the current block has a display order value smaller than the display order value of the current picture containing the current block, that a second reference picture for the current block has a display order value larger than the display order value of the current picture, that the time distances between the current picture, the first reference picture and the second reference picture are equal, that the current block has more than 64 luma pixels, that the height of the current block is at least 8 luma pixels, that the width of the current block is at least 8 luma pixels, that the weight values ​​for bipredictive (BCW) with CU level weights are equal, that weighted prediction (WP) is not enabled for the current block, and that synthetic intrainterpredictive (CIIP) mode is not used for the current block.

[0293] Clause 87: Any method of Clauses 81 to 85, wherein the step of performing model-based fractional-pixel motion vector improvement comprises the step of performing model-based fractional-pixel motion vector improvement in response to determining that 1) the initial motion vector has been improved using bilateral matching or decoder-side motion vector (DMVR) improvement, or 2) the improved motion vector has been further improved using bilateral matching or DMVR.

[0294] Clause 88: The step of performing model-based fractional-pixel motion vector improvement is performed if 1) the initial motion vector was not improved using bilateral matching or decoder-side motion vector (DMVR) improvement, 2) the improved motion vector was not further improved using bilateral matching or DMVR, or 3) the initial motion vector is in coding unit (CU) level merge mode with bilateral predictive motion vectors, the first reference picture for the current block has a display order value smaller than the display order value of the current picture containing the current block, the second reference picture for the current block has a display order value larger than the display order value of the current picture, and the current picture and the first reference Any method of clauses 81 to 85, comprising the step of performing a model-based fractional pixel motion vector improvement in response to determining that at least one of the following is not true: the time distance between the illuminated picture and the second reference picture is equal; the current block has more than 64 lumens; the height of the current block is at least 8 lumens; the width of the current block is at least 8 lumens; the weight values ​​for biprediction with CU level weights (BCW) are equal; weighted prediction (WP) is not enabled for the current block; and the synthetic intra-interpretation (CIIP) mode is not used for the current block.

[0295] Clause 89: The motion vector difference value comprises a horizontal motion vector difference (Δx) and a vertical motion vector difference (Δy), and the error value of adjacent pixels is E x,y The step of determining the error value is E x, y =A(x -Δx) 2 +B(y -Δy) 2 The steps include calculating +C and performing model-based fractional pixel motion vector improvements, s*floor((α(E -1,0 -E 1,0 )) / (2(E -1,0 +E 1,0 -2E 0,0The steps are to calculate Δx according to ))) and s*floor((α(E 0,-1 -E 0,1 )) / (2(E 0,-1 +E 0,1 -2E 0,0 The method comprises the step of calculating Δy according to one of the methods of clauses 81 to 88, wherein α is an integer scaling factor introduced to represent (Δx, Δy) with decimal-pel precision, and s is the scaling value.

[0296] The method of Clause 89, further comprising the step of determining the value of α to be equal to 16, regardless of whether a 1 / 2 pixel interpolation filter is used.

[0297] Clause 91: Any method of Clauses 89 and 90, further comprising the step of determining the value of α according to the Adaptive Motion Vector Resolution (AMVR) mode for the current block.

[0298] Clause 92: Any method of Clauses 89 and 91, further comprising the step of determining the value of α depending on whether a 1 / 2 pixel interpolation filter is used.

[0299] Clause 93: Any method of Clauses 81 to 92, wherein the current block is bipredicted, the initial motion vector comprises a first initial motion vector, the improved motion vector comprises a first improved motion vector, and the method further comprises the step of improving a second initial motion vector for the current block using the first improved motion vector.

[0300] Clause 94: Any method of Clauses 81 to 93, further comprising the step of improving the decoded data representing the motion vectors using bilateral matching to form the initial motion vectors.

[0301] Clause 95: Any method of Clauses 81 to 93, further comprising the step of refining the improved motion vector using bilateral matching to form a further improved motion vector, wherein the step of decoding the current block comprises the step of decoding the current block using the further improved motion vector.

[0302] Clause 96: Any method of Clauses 81 to 95, wherein the step of performing a template matching search process comprises the step of performing a template matching search process after determining that the adaptive motion vector resolution (AMVR) mode is not a 4-pixel mode.

[0303] Clause 97: Any method of Clauses 81 to 96, further comprising the step of encoding the current block before decrypting the current block.

[0304] Clause 98: A device for decoding video data, comprising a memory configured to store video data and one or more processors implemented in circuitry, wherein one or more processors are configured to decode data representing an initial motion vector for a current block of video data, the initial motion vector having integer motion vector difference (MVD) precision, determine a search range around a reference area identified by the initial motion vector in a reference picture, perform a template matching search process within the search range to identify the best matching region, determine an error value of adjacent pixels for the best matching region, use the error value of adjacent pixels to perform a model-based fractional pixel motion vector improvement to derive a motion vector difference value, apply at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block, and decode the current block using the improved motion vector.

[0305] Clause 99: A device of Clause 98, wherein, in order to apply at least one motion vector difference value, one or more processors are configured to apply a motion vector difference value for a component to that component in response to determining that the error value of the best matching region is not equal to zero and that the absolute value of the difference between the component of the initial motion vector and the corresponding motion vector difference value for that component is less than the corresponding search region value for that component.

[0306] Clause 100: A device according to any of Clauses 98 and 99, configured to perform model-based fractional-pixel motion vector refinement in response to one or more processors determining that the initial motion vector is in coding unit (CU) level merge mode with bipredictive motion vectors, that a first reference picture for the current block has a display order value smaller than the display order value of the current picture containing the current block, that a second reference picture for the current block has a display order value larger than the display order value of the current picture, that the time distances between the current picture, the first reference picture and the second reference picture are equal, that the current block has more than 64 lumens, that the height of the current block is at least 8 lumens, that the width of the current block is at least 8 lumens, that the weight values ​​for bipredictive (BCW) with CU level weights are equal, that weighted prediction (WP) is not enabled for the current block, and that synthetic intrainterpredictive (CIIP) mode is not used for the current block.

[0307] Clause 101: The motion vector difference value comprises a horizontal motion vector difference (Δx) and a vertical motion vector difference (Δy), and the error value of adjacent pixels is E x,y To obtain the value and determine the error value, one or more processors, E x, y =A(x -Δx) 2 +B(y -Δy) 2Configured to calculate +C, one or more processors perform model-based fractional pixel motion vector improvements, s*floor((α(E -1,0 -E 1,0 )) / (2(E -1,0 +E 1,0 -2E 0,0 Calculate Δx according to ))) and s*floor((α(E 0,-1 -E 0,1 )) / (2(E 0,-1 +E 0,1 -2E 0,0 The device is configured to calculate Δy according to ))) and determine the value of α to be equal to 16 regardless of whether a 1 / 2 pixel interpolation filter is used, where α is an integer scaling factor introduced to represent (Δx, Δy) with some decimal-pel precision, and s is the scaling value, with clauses 98 to 100.

[0308] Clause 102: Any device according to any of Clauses 98 to 101, wherein one or more processors are further configured to use bilateral matching to refine the decoded data representing the motion vectors in order to form the initial motion vectors.

[0309] Clause 103: Any device of Clauses 98 to 102, wherein one or more processors are configured to further refine the refined motion vectors using bilateral matching to form further refined motion vectors, and one or more processors are configured to decode the current block using the further refined motion vectors.

[0310] Clause 104: A device in any of Clauses 98 to 103, in which one or more processors are configured to perform a template matching search process after determining that the Adaptive Motion Vector Resolution (AMVR) mode is not 4-pixel mode.

[0311] Clause 105: Any device from Clauses 98 to 104 further comprising a display configured to display decoded video data.

[0312] Clause 106: A device comprising one or more of the following: a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box, as described in any of Clauses 98 to 105.

[0313] Clause 107: A computer-readable storage medium storing instructions, wherein, when executed, the instructions cause a processor of a device for decoding video data to decode data representing an initial motion vector for a current block of video data, the initial motion vector having integer motion vector difference (MVD) precision, to determine a search range around a reference area identified by the initial motion vector in a reference picture, to perform a template matching search process within the search range to identify the best matching region, to determine an error value of adjacent pixels for the best matching region, to perform a model-based fractional pixel motion vector improvement using the error value of adjacent pixels to derive a motion vector difference value, to apply at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block, and to decode the current block using the improved motion vector.

[0314] A computer-readable storage medium according to Clause 107, comprising instructions that cause a processor to apply at least one motion vector difference value, causing the processor to determine that the error value of the best-match region is not equal to zero, and that the absolute value of the difference between a component of an initial motion vector and the corresponding motion vector difference value for that component is less than the corresponding search region value for that component, and to apply the motion vector difference value for that component to that component.

[0315] A computer-readable storage medium of either of Clauses 107 or 108, comprising instructions that cause the processor to perform model-based fractional-pixel motion vector refinement in response to the processor determining that the initial motion vector is in coding unit (CU) level merge mode with bipredictive motion vectors, that a first reference picture for the current block has a display order value smaller than the display order value of the current picture containing the current block, that a second reference picture for the current block has a display order value larger than the display order value of the current picture, that the time distances between the current picture, the first reference picture and the second reference picture are equal, that the current block has more than 64 luma pixels, that the height of the current block is at least 8 luma pixels, that the width of the current block is at least 8 luma pixels, that the weight values ​​for bipredictive (BCW) with CU level weights are equal, that weighted prediction (WP) is not enabled for the current block, and that synthetic intrainterpredictive (CIIP) mode is not used for the current block.

[0316] Clause 110: The motion vector difference value comprises a horizontal motion vector difference (Δx) and a vertical motion vector difference (Δy), and the error value of adjacent pixels is E x,y An instruction that contains a value and causes the processor to determine the error value, E x, y =A(x -Δx) 2 +B(y -Δy) 2 The instructions include an instruction to calculate +C, and an instruction to cause the processor to perform model-based fractional pixel motion vector improvement, which causes the processor to perform s*floor((α(E -1,0 -E 1,0 )) / (2(E -1,0 +E 1,0 -2E 0,0 Calculate Δx according to ))) and s*floor((α(E 0,-1 -E 0,1 )) / (2(E 0,-1 +E 0,1 -2E 0,0A computer-readable storage medium of any of clauses 107 to 109, comprising instructions to calculate Δy according to ))) and to determine the value of α to be equal to 16 regardless of whether a 1 / 2 pixel interpolation filter is used, wherein α is an integer scaling factor introduced to represent (Δx, Δy) with some decimal-pel precision, and s is the scaling value.

[0317] Clause 111: A computer-readable storage medium according to any of Clauses 107 to 110, further comprising instructions causing a processor to improve decoded data representing motion vectors using bilateral matching to form initial motion vectors.

[0318] A computer-readable storage medium according to any of the clauses 107 to 111, further comprising instructions to cause the processor to decode the current block, the instructions to cause the processor to decode the current block using the further improved motion vector, and further comprising instructions to cause the processor to decode the current block using the further improved motion vector.

[0319] Clause 113: A computer-readable storage medium of any of Clauses 107 to 112, wherein the instructions causing the processor to perform a template matching search process include instructions causing the processor to perform a template matching search process after determining that the adaptive motion vector resolution (AMVR) mode is not 4-pixel mode.

[0320] Clause 114: A device for decoding video data, comprising: means for decoding data representing an initial motion vector for a current block of video data, wherein the initial motion vector has integer motion vector difference (MVD) precision; means for determining a search range around a reference area identified by the initial motion vector in a reference picture; means for performing a template matching search process within the search range to identify the best matching region; means for determining an error value of an adjacent pixel for the best matching region; means for using the error value of an adjacent pixel to perform a model-based fractional pixel motion vector improvement to derive a motion vector difference value; means for applying at least one of the motion vector difference values ​​to the initial motion vector to determine an improved motion vector for the current block; and means for decoding the current block using the improved motion vector.

[0321] It should be noted that, in some examples, some actions or events of any of the techniques described herein may be performed in a different order, added, merged, or omitted entirely (for example, not all actions or events described may be necessary for the practice of the technique). Furthermore, in some examples, actions or events may not be performed sequentially, but rather simultaneously, for example, through multithreading, interrupt handling, or across multiple processors.

[0322] In one or more examples, the described functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a computer-readable medium, transmitted through a computer-readable medium, or executed by a hardware-based processing unit. The computer-readable medium may include computer-readable storage media corresponding to tangible media such as data storage media, or communication media including any medium that facilitates the transfer of computer programs from one place to another according to a communication protocol, for example. Thus, the computer-readable medium may generally correspond to (1) non-transient tangible computer-readable storage media, or (2) communication media such as signals or carrier waves. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described herein. A computer program product may include computer-readable media.

[0323] As an example, and not an limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is appropriately called computer-readable media. For example, if instructions are transmitted from a website, server or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but instead refer to non-temporary tangible storage media. As used herein, the terms "disk" and "disc" include compact discs (CDs), laser discs, optical discs, digital multipurpose discs (DVDs), floppy disks, and Blu-ray discs. A disk typically reproduces data magnetically, while a disc reproduces data optically using a laser. Combinations of the above should also be included within the scope of computer-readable media.

[0324] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the terms “processor” and “processing circuit” as used herein may refer to any of the above-described structures or any other structure suitable for implementing the techniques described herein. In addition, in some embodiments, the functions described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a composite codec. Furthermore, the techniques may be fully implemented in one or more circuits or logic elements.

[0325] The techniques of this disclosure may be implemented in a wide variety of devices or apparatus, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chipsets). Various components, modules, or units have been described in this disclosure to highlight the functional aspects of devices configured to perform the disclosed techniques, but these do not necessarily require implementation by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit, or provided by a set of interoperable hardware units, including one or more processors as described above, along with appropriate software and / or firmware.

[0326] Various examples were described. These and other examples fall within the scope of the following claims. [Explanation of Symbols]

[0327] 102 Source Device 104 Video Sources 106 memory 108 Output Interfaces 110 Computer-readable media 112 Storage Devices 114 File Server 116 Destination device 118 Display Devices 120 memory 122 Input Interfaces 130 QTBT structure 132 Coding Tree Units 150 blocks 152 blocks 154 Central Block 170 blocks 172 blocks 180 See L0 182 See L1 200 video encoders 202 Mode Selection Unit 204 Residual Generation Unit 206 Conversion Processing Unit 208 Quantization Units 210 Inverse Quantization Unit 212 Inverse Transform Processing Unit 214 Reconstruction Unit 216 Filter Unit 218 Decoded picture buffer 220 Entropy Coding Units 222 Motion Estimation Unit 224 Motion Compensation Unit 226 Intra Prediction Units 230 video data memory 300 video decoders 302 Entropy Decoding Unit 304 Predictive Processing Unit 306 Inverse Quantization Unit 308 Inverse Transform Processing Unit 310 Reconstruction Unit 312 Filter Unit 314 DPB 316 Motion Compensation Unit 318 Intra Prediction Units 320 CPB memory

Claims

1. A method for decoding video data, A step of determining an initial motion vector for the current block of video data, wherein the initial motion vector has integer motion vector difference (MVD) precision; The steps include determining the search range around the reference area around the initial motion vector in the reference picture, The steps include: running the template matching search process using integer-precision samples in the search range in order to identify the best matching region in the search range without using fractional samples during the template matching search process; The steps include determining the error value of adjacent pixels within the best matching region, A step of performing a model-based fractional pixel motion vector improvement and using the error value of the adjacent pixel to derive a motion vector difference value, wherein at least one of the motion vector difference values ​​includes a non-zero fractional value, the motion vector difference value comprises a horizontal motion vector difference (Δx) and a vertical motion vector difference (Δy), the error value of the adjacent pixel comprises an E x,y value, the step of determining the error value comprises a step of calculating E x,y = A(x-Δx)² + B(y-Δy)² + C, and performing the model-based fractional pixel motion vector improvement, The steps involve calculating Δx according to s*floor((α(E -1,0 -E 1,0 )) / (2(E -1,0 +E 1,0 -2E 0,0 ))), The method comprises the step of calculating Δy according to s*floor((α(E 0,-1 -E 0,1 )) / (2(E 0,-1 +E 0,1 -2E 0,0 ))), α is an integer scaling factor introduced to represent (Δx, Δy) with some decimal -pel precision, where s is the scaling value, and the step is... A step of applying at least one of the motion vector difference values ​​to the initial motion vector in order to determine an improved motion vector for the current block, wherein the improved motion vector identifies a fractional sample position in the reference picture. A method comprising the step of decoding the current block using the improved motion vector.

2. The method according to claim 1, wherein the motion vector difference value includes a horizontal motion vector difference value and a vertical motion vector difference value.

3. The method according to claim 1, wherein the step of applying at least one of the motion vector difference values ​​comprises the step of applying all of the motion vector difference values.

4. The step of applying at least one of the motion vector difference values ​​is, The step of determining that the error value of the best matching region is not equal to 0, The method according to claim 1, comprising the step of applying the motion vector difference value for a component to the component in response to the determination that the absolute value of the difference between a component of the initial motion vector and a corresponding motion vector difference value for the component is less than the corresponding search region value for the component.

5. The method according to claim 1, wherein the step of applying the at least one of the motion vector difference values ​​comprises the step of preventing the application of the motion vector difference value for the component to the component in response to determining at least one of the following: that the error value of the best matching region is equal to zero, or that the absolute value of the difference between the component of the initial motion vector and the corresponding motion vector difference value for the component is greater than or equal to the corresponding search region value for the component.

6. The method according to claim 1, wherein performing the model-based fractional pixel motion vector refinement is performed in response to determining that the initial motion vector is in coding unit (CU) level merge mode with bipredictive motion vectors, that a first reference picture for the current block has a display order value smaller than the display order value of the current picture containing the current block, that a second reference picture for the current block has a display order value larger than the display order value of the current picture, that the time distances between the current picture, the first reference picture, and the second reference picture are equal, that the current block has more than 64 luma pixels, that the height of the current block is at least 8 luma pixels, that the width of the current block is at least 8 luma pixels, that the weight values ​​for bipredictive (BCW) with CU level weights are equal, that weighted prediction (WP) is not enabled for the current block, and that synthetic intrainterpretation (CIIP) mode is not used for the current block.

7. The method according to claim 1, wherein the current block is bipredicted, the initial motion vector comprises a first initial motion vector, the improved motion vector comprises a first improved motion vector, and the method further comprises the step of improving a second initial motion vector for the current block using the first improved motion vector.

8. A device for decoding video data, A memory configured to store video data, The circuit comprises one or more processors implemented in the circuit, and the one or more processors Determine the initial motion vector for the current block of the video data, and ensure that the initial motion vector has integer motion vector difference (MVD) precision. Determine the search range around the reference area around the initial motion vector in the reference picture, In order to identify the best matching region within the search range without using fractional samples during the template matching search process, the template matching search process is performed using integer-precision samples within the search range. Determine the error value of adjacent pixels within the aforementioned best matching region. Performing a model-based fractional pixel motion vector improvement is performed using the error values ​​of the adjacent pixels to derive motion vector difference values, wherein at least one of the motion vector difference values ​​contains a non-zero fractional value, the motion vector difference value comprises a horizontal motion vector difference (Δx) and a vertical motion vector difference (Δy), the error values ​​of the adjacent pixels comprise the values ​​E x,y, determining the error values ​​comprises calculating E x,y = A(x-Δx)² + B(y-Δy)² + C, and performing the model-based fractional pixel motion vector improvement is performed, Calculate Δx according to s*floor((α(E -1,0 -E 1,0 )) / (2(E -1,0 +E 1,0 -2E 0,0 ))), The method includes calculating Δy according to s*floor((α(E 0,-1 -E 0,1 )) / (2(E 0,-1 +E 0,1 -2E 0,0 ))), α is an integer scaling factor introduced to represent (Δx, Δy) with some decimal -pel precision, where s is the scaling value. To determine an improved motion vector for the current block, at least one of the motion vector difference values ​​is applied to the initial motion vector, and the improved motion vector identifies a fractional sample position in the reference picture. The current block is decoded using the improved motion vector. A device configured in such a way.

9. In order to apply at least one of the motion vector difference values, one or more processors It is determined that the error value of the best matching region is not equal to 0. The device according to claim 8, configured to apply the motion vector difference value for a component to the component in response to the determination that the absolute value of the difference between a component of the initial motion vector and a corresponding motion vector difference value for the component is less than the corresponding search region value for the component.

10. The device according to claim 8, configured to perform the model-based fractional-pixel motion vector refinement in response to one or more processors determining that the initial motion vector is in coding unit (CU) level merge mode with bipredictive motion vectors, that a first reference picture for the current block has a display order value smaller than the display order value of the current picture containing the current block, that a second reference picture for the current block has a display order value larger than the display order value of the current picture, that the time distances between the current picture and the first and second reference pictures are equal, that the current block has more than 64 lumens, that the height of the current block is at least 8 lumens, that the width of the current block is at least 8 lumens, that the weight values ​​for bipredictive (BCW) with CU level weights are equal, that weighted prediction (WP) is not enabled for the current block, and that synthetic intrainterpredictive (CIIP) mode is not used for the current block.

11. The device according to claim 8, wherein one or more processors are further configured to use bilateral matching to refine the decoded data representing the motion vector in order to form the initial motion vector.

12. The device according to claim 8, wherein one or more processors are further configured to use bilateral matching to refine the refined motion vector to form a further refined motion vector, and the one or more processors are configured to decode the current block using the further refined motion vector.

13. The device according to claim 8, further comprising a display configured to display the decoded video data.

14. The device according to claim 8, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

15. A computer-readable storage medium storing instructions, wherein, when the instructions are executed, the instructions cause a processor of a device for decoding video data to perform the method according to any one of claims 1 to 7.

Images