Template Matching-Based Affine Prediction for Video Coding

Abstract

The video decoder may be configured to determine that a current block in a current picture of video data is coded in an affine prediction mode, determine one or more control point motion vectors (CPMVs) for the current block, identify an initial predictive block for the current block in a reference picture using the one or more CPMVs, determine a current template for the current block in the current picture, determine an initial reference template for the first predictive block in the reference picture, and perform a motion vector refinement process to determine a modified predictive block based on a comparison of the initial reference template and the current template.

Description

Claiming priority

[0001]

[0001] This application claims priority to U.S. Patent Application No. 17 / 715,571, filed April 7, 2022, U.S. Provisional Application No. 63 / 173,861, filed April 12, 2021, and U.S. Provisional Application No. 63 / 173,949, filed April 12, 2021, the entire contents of each of which are incorporated herein by reference. U.S. Patent Application No. 17 / 715,571, filed April 7, 2022, claims the benefit of U.S. Provisional Application No. 63 / 173,861, filed April 12, 2021, and U.S. Provisional Application No. 63 / 173,949, filed April 12, 2021. [Technical field]

[0002]

[0002] This disclosure relates to video encoding and decoding. [Background technology]

[0003] Digital video capabilities may be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite radio telephones, so-called "smartphones," video teleconferencing devices, video streaming devices, and the like. Digital video devices implement video coding techniques, such as those described in standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265 / High Efficiency Video Coding (HEVC), and extensions to such standards. By implementing such video coding techniques, video devices may more efficiently transmit, receive, encode, decode, and / or store digital video information.

[0004]

[0004] Video coding techniques include spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences. In block-based video coding, a video slice (e.g., a video picture or a portion of a video picture) may be partitioned into video blocks, which may also be referred to as coding tree units (CTUs), coding units (CUs) and / or coding nodes. Video blocks in an intra-coded (I) slice of a picture are encoded using spatial prediction with respect to reference samples in neighboring blocks in the same picture. Video blocks in an inter-coded (P or B) slice of a picture may use spatial prediction with respect to reference samples in neighboring blocks in the same picture or temporal prediction with respect to reference samples in other reference pictures. A picture may be referred to as a frame, and a reference picture may be referred to as a reference frame. Summary of the Invention

[0005]

[0005] This disclosure describes techniques related to affine prediction modes, which are a type of inter-prediction mode that potentially accounts for object rotations that may occur in a sequence of pictures. An affine motion model of a block may be determined based on a motion vector of a control point of the block, sometimes referred to as a control-point motion vector (CPMV). In some implementations, the control points of the block are the top-left and top-right corners of the block. In some implementations, the control points of the block further include a bottom-left corner of the block. A video coder (i.e., a video encoder or a video decoder) may calculate motion vectors of sub-blocks of the block based on the CPMV of the block to identify the location of the predictive sub-blocks in a reference picture. The predictive sub-blocks may form a predictive block.

[0006]

[0006] This disclosure describes decoder-side techniques that may improve the prediction sub-block, and therefore the prediction block. That is, the techniques of this disclosure may cause a video decoder to form a prediction block using a sub-block different from the sub-block originally determined or located using CPMV. By implementing a motion vector refinement process to determine a modified prediction block for an affine coded block in the manner described in this disclosure, the video decoder may determine a more accurate prediction block compared to conventional affine prediction. Utilizing the techniques of this disclosure to determine a more accurate prediction block may improve overall coding quality without increasing signaling overhead.

[0007]

[0007] According to one example of the present disclosure, a method for decoding video data includes determining that a current block in a current picture of the video data is coded in an affine prediction mode, determining one or more control point motion vectors (CPMVs) for the current block, identifying an initial predictive block for the current block in a reference picture using the one or more CPMVs, determining a current template for the current block in the current picture, determining an initial reference template for the initial predictive block in the reference picture, and performing a motion vector refinement process to determine a modified predictive block based on a comparison between the initial reference template and the current template.

[0008]

[0008] According to another example of the present disclosure, a device for decoding video data includes a memory and one or more processors implemented in a circuit coupled to the memory, where the one or more processors are configured to determine that a current block in a current picture of the video data is coded in an affine prediction mode, determine one or more control point motion vectors (CPMVs) for the current block, identify an initial predictive block for the current block in a reference picture using the one or more CPMVs, determine a current template for the current block in the current picture, determine an initial reference template for the first predictive block in the reference picture, and perform a motion vector refinement process to determine a modified predictive block based on a comparison between the initial reference template and the current template.

[0009]

[0009] According to another example of the present disclosure, a computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to determine that a current block in a current picture of video data is coded in an affine prediction mode, determine one or more control point motion vectors (CPMVs) for the current block, identify an initial predictive block for the current block in a reference picture using the one or more CPMVs, determine a current template for the current block in the current picture, determine an initial reference template for the first predictive block in the reference picture, and perform a motion vector refinement process to determine a modified predictive block based on a comparison of the initial reference template and the current template.

[0010]

[0010] According to another example of the present disclosure, an apparatus for decoding video data includes means for determining that a current block in a current picture of the video data is coded in an affine prediction mode, means for determining one or more control point motion vectors (CPMVs) for the current block, means for identifying an initial predictive block for the current block in a reference picture using the one or more CPMVs, means for determining a current template for the current block in the current picture, means for determining an initial reference template for the first predictive block in the reference picture, and means for performing a motion vector refinement process to determine a modified predictive block based on a comparison between the initial reference template and the current template.

[0011]

[0011] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0012] [Figure 1]

[0012] A block diagram illustrating an example video encoding and decoding system in which the techniques of this disclosure may be implemented. [Figure 2A]

[0013] 1 is a conceptual diagram illustrating an exemplary Quad Tree Binary Tree (QTBT) structure. [Figure 2B] A conceptual diagram showing a corresponding coding tree unit (CTU). [Figure 3A]

[0014] FIG. 13 is a conceptual diagram illustrating spatially adjacent motion vector candidates for merge mode. [Figure 3B]

[0015] 1 is a conceptual diagram illustrating spatially neighboring motion vector candidates for Advanced Motion Vector Prediction (AMVP) mode. [Figure 4A]

[0016] 1 is a conceptual diagram showing temporal motion vector candidates; [Figure 4B]

[0017] FIG. 1 is a conceptual diagram showing motion vector scaling. [Diagram 5]

[0018] FIG. 13 illustrates an example of template matching performed on a search area around an initial motion vector. [Figure 6A]

[0019] A conceptual diagram showing a control-point-based six-parameter affine motion model. [Figure 6B]

[0020] A conceptual diagram showing a control-point-based four-parameter affine motion model. [Figure 7]

[0021] FIG. 2 illustrates an example of an affine motion vector field for each sub-block. [Figure 8]

[0022] FIG. 13 is a diagram showing an example of a sub-block motion vector. [Figure 9A]

[0023] FIG. 2 illustrates a current template block and a reference template block. [Figure 9B] FIG. 2 illustrates a current template block and a reference template block. [Figure 9C] FIG. 2 illustrates a current template block and a reference template block. [Figure 10]

[0024] 1 is a conceptual diagram illustrating an example of per-sample weights that may be assigned to samples of neighboring blocks for calculating template matching costs. [Figure 11]

[0025] 1 is a block diagram illustrating an example video encoder that may implement the techniques of this disclosure. [Figure 12]

[0026] 1 is a block diagram illustrating an example video decoder that may implement the techniques of this disclosure. [Figure 13]

[0027] 11 is a flowchart illustrating an example process for encoding a current block in accordance with techniques of this disclosure. [Figure 14]

[0028] 6 is a flowchart illustrating an example process for decoding a current block in accordance with techniques of this disclosure. [Figure 15]

[0029] 6 is a flowchart illustrating an example process for decoding a current block in accordance with techniques of this disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013]

[0030] Video coding (e.g., video encoding and / or video decoding) generally involves either predicting a block of video data from already coded blocks of video data in the same picture (e.g., intra prediction) or predicting a block of video data from already coded blocks of video data in a different picture (e.g., inter prediction). In some instances, the video encoder also calculates residual data by comparing the predicted block to the original block. The residual data thus represents the difference between the predicted block and the original block. To reduce the number of bits required to signal the residual data, the video encoder transforms and quantizes the residual data and signals the transformed and quantized residual data in the encoded bitstream. The compression achieved by the transform and quantization process may be lossy, meaning that the transform and quantization process may introduce distortions into the decoded video data.

[0014]

[0031] A video decoder decodes the residual data and adds it to the predictive block to generate a reconstructed video block that matches the original video block more closely than the predictive block alone. Due to losses introduced by transforming and quantizing the residual data, the initial reconstructed block may have distortions or artifacts. One common type of artifacts or distortions is called blockiness, where boundaries of blocks used to code the video data are visible.

[0015]

[0032] To further improve the quality of the decoded video, the video decoder may perform one or more filtering operations on the reconstructed video blocks. Examples of these filtering operations include deblocking filtering, sample adaptive offset (SAO) filtering, and adaptive loop filtering (ALF). Parameters for these filtering operations may be either determined by the video encoder and explicitly signaled in the encoded video bitstream, or implicitly determined by the video decoder without the parameters having to be explicitly signaled in the encoded video bitstream.

[0016]

[0033] This disclosure describes techniques related to affine prediction modes, which are a type of inter prediction mode that potentially accounts for object rotations that may occur in a sequence of pictures. An affine motion model of a block may be determined based on a motion vector of a control point of the block, sometimes referred to as a control point motion vector (CPMV). In some implementations, the control points of the block are the top-left and top-right corners of the block. In some implementations, the control points of the block further include a bottom-left corner of the block. A video coder (i.e., a video encoder or a video decoder) may calculate motion vectors of sub-blocks of the block based on the CPMV of the block to identify the location of the predictive sub-blocks in a reference picture. The predictive sub-blocks may form a predictive block.

[0017]

[0034] This disclosure describes decoder-side techniques that may improve the prediction sub-block, and therefore the prediction block. That is, the techniques of this disclosure may cause the video decoder to form a prediction block using a sub-block different from the sub-block originally determined or located using CPMV. By implementing a motion vector refinement process to determine a modified prediction block for an affine coded block in the manner described in this disclosure, the video decoder may determine a more accurate prediction block compared to conventional affine prediction. Utilizing the techniques of this disclosure to determine a more accurate prediction block may improve the overall coding quality without increasing signaling overhead.

[0018]

[0035] Although the techniques of this disclosure have been generally described as being performed by a video decoder, it should be understood that the techniques described herein may also be performed by a video encoder. For example, the techniques of this disclosure may be performed by a video encoder as part of a process for determining how to encode video blocks and generating reference pictures that may be used to encode subsequent pictures of the video.

[0019]

[0036] 1 is a block diagram illustrating an example video encoding and decoding system 100 that may implement the techniques of this disclosure. The techniques of this disclosure are generally directed to coding (encoding and / or decoding) video data. In general, video data includes any data for processing video. Thus, video data may include raw uncoded video, coded video, decoded (e.g., reconstructed) video, and video metadata, such as signaling data.

[0020]

[0037] 1, system 100 includes a source device 102 that provides encoded video data to be decoded and displayed by a destination device 116, in this example. In particular, source device 102 provides the video data to destination device 116 via a computer-readable medium 110. Source device 102 and destination device 116 may comprise any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, mobile devices, tablet computers, set-top boxes, telephone handsets such as smartphones, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, broadcast receiver devices, and the like. In some cases, source device 102 and destination device 116 may be equipped for wireless communication and thus may be referred to as wireless communication devices.

[0021]

[0038] In the example of FIG. 1, source device 102 includes a video source 104, memory 106, video encoder 200, and output interface 108. Destination device 116 includes an input interface 122, a video decoder 300, memory 120, and a display device 118. According to this disclosure, the video encoder 200 of source device 102 and the video decoder 300 of destination device 116 may be configured to apply techniques for performing template-based affine prediction. Thus, source device 102 represents an example of a video encoding device, and destination device 116 represents an example of a video decoding device. In other examples, the source and destination devices may include other components or arrangements. For example, source device 102 may receive video data from an external video source, such as an external camera. Similarly, destination device 116 may interface with an external display device rather than including an integrated display device.

[0022]

[0039] The system 100 shown in FIG. 1 is only an example. In general, any digital video encoding and / or decoding device may implement techniques for performing template-based affine prediction. The source device 102 and the destination device 116 are only examples of coding devices, such that the source device 102 generates coded video data for transmission to the destination device 116. This disclosure refers to a "coding" device as a device that performs coding (encoding and / or decoding) of data. Thus, the video encoder 200 and the video decoder 300 represent examples of coding devices, in particular video encoders and video decoders, respectively. In some examples, the source device 102 and the destination device 116 may operate substantially symmetrically, such that each of the source device 102 and the destination device 116 includes video encoding and video decoding components. Thus, the system 100 may support one-way or two-way video transmission between the source device 102 and the destination device 116, for example, video streaming, video playback, video broadcasting, or video telephony.

[0023]

[0040] Generally, video source 104 represents a source of video data (i.e., raw, unencoded video data) and provides a continuous series of pictures (also called “frames”) of the video data to video encoder 200, which encodes the data for the pictures. Video source 104 of source device 102 may include a video capture device, such as a video camera, a video archive containing previously captured raw video, and / or a video feed interface for receiving video from a video content provider. As a further alternative, video source 104 may generate computer graphics-based data as the source video, or a combination of live, archival, and computer-generated video. In each case, video encoder 200 encodes the captured, pre-captured, or computer-generated video data. Video encoder 200 may rearrange the pictures from a receiving order (sometimes called a “display order”) to a coding order for coding. Video encoder 200 may generate a bitstream including the encoded video data. The source device 102 may then output the encoded video data via the output interface 108 onto a computer-readable medium 110 for receipt and / or retrieval by, for example, an input interface 122 of the destination device 116 .

[0024]

[0041] The memory 106 of the source device 102 and the memory 120 of the destination device 116 represent general-purpose memories. In some examples, the memories 106, 120 may store raw video data, e.g., raw video from the video source 104 and raw decoded video data from the video decoder 300. Additionally or alternatively, the memories 106, 120 may store software instructions executable by, e.g., the video encoder 200 and the video decoder 300, respectively. Although the memories 106 and 120 are shown in this example separately from the video encoder 200 and the video decoder 300, it should be understood that the video encoder 200 and the video decoder 300 may also include internal memories for functionally similar or equivalent purposes. Additionally, the memories 106, 120 may store encoded video data, e.g., the output from the video encoder 200 and the input to the video decoder 300. In some examples, portions of the memory 106, 120 may be allocated as one or more video buffers, for example, to store raw decoded and / or encoded video data.

[0025]

[0042] The computer-readable medium 110 may represent any type of medium or device capable of transporting encoded video data from the source device 102 to the destination device 116. In one example, the computer-readable medium 110 represents a communication medium for enabling the source device 102 to transmit the encoded video data directly to the destination device 116 in real time, for example, via a radio frequency network or a computer-based network. The output interface 108 may modulate a transmission signal including the encoded video data, and the input interface 122 may demodulate a received transmission signal according to a communication standard such as a wireless communication protocol. The communication medium may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful in facilitating communication from the source device 102 to the destination device 116.

[0026]

[0043] In some examples, source device 102 may output the encoded data from output interface 108 to storage device 112. Similarly, destination device 116 may access the encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0027]

[0044] In some examples, source device 102 may output the encoded video data to a file server 114 or another intermediate storage device, which may store the encoded video data generated by source device 102. Destination device 116 may access the stored video data from file server 114 via streaming or download.

[0028]

[0045] The file server 114 may be any type of server device capable of storing encoded video data and transmitting the encoded video data to a destination device 116. The file server 114 may represent a web server (e.g., for a website), a server configured to provide file transfer protocol services (such as File Transfer Protocol (FTP) or File Delivery over Unidirectional Transport (FLUTE) protocol), a content delivery network (CDN) device, a HyperText Transfer Protocol (HTTP) server, a Multimedia Broadcast Multicast Service (MBMS) or enhanced MBMS (eMBMS) server, and / or a network attached storage (NAS) device. The file server 114 may additionally or alternatively implement one or more HTTP streaming protocols, such as Dynamic Adaptive Streaming over HTTP (DASH), HTTP Live Streaming (HLS), Real Time Streaming Protocol (RTSP), HTTP Dynamic Streaming, etc.

[0029]

[0046] The destination device 116 may access the encoded video data from the file server 114 through any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a Digital Subscriber Line (DSL), a cable modem, etc.), or a combination of both, that is suitable for accessing the encoded data stored on the file server 114. The input interface 122 may be configured to operate according to any one or more of the various protocols described above for retrieving or receiving media data from the file server 114, or other such protocols for retrieving media data.

[0030]

[0047] The output interface 108 and the input interface 122 may represent a wireless transmitter / receiver, a modem, a wired networking component (e.g., an Ethernet card), a wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In examples in which the output interface 108 and the input interface 122 comprise wireless components, the output interface 108 and the input interface 122 may be configured to transfer data, such as encoded video data, according to a cellular communication standard, such as 4G, 4G-LTE (Long Term Evolution), LTE Advanced, 5G, etc. In some examples in which the output interface 108 comprises a wireless transmitter, the output interface 108 and the input interface 122 may be configured to transfer data, such as encoded video data, according to other wireless standards, such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee), the Bluetooth standard, etc. In some examples, the source device 102 and / or the destination device 116 may include respective system-on-chip (SoC) devices. For example, the source device 102 may include a SoC device for performing functions attributable to the video encoder 200 and / or the output interface 108, and the destination device 116 may include a SoC device for performing functions attributable to the video decoder 300 and / or the input interface 122.

[0031]

[0048] The techniques of this disclosure may be applied to video coding supporting any of a variety of multimedia applications, such as over-the-air television broadcast, cable television transmission, satellite television transmission, Internet streaming video transmission such as Dynamic Adaptive Streaming over HTTP (DASH), digital video encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications.

[0032]

[0049] The input interface 122 of the destination device 116 receives the encoded video bitstream from the computer-readable medium 110 (e.g., a communication medium, a storage device 112, a file server 114, etc.). The encoded video bitstream may include signaling information defined by the video encoder 200 that is also used by the video decoder 300, such as syntax elements having values ​​that describe characteristics and / or processing of video blocks or other coded units (e.g., slices, pictures, groups of pictures, sequences, etc.). The display device 118 displays decoded pictures of the decoded video data to a user. The display device 118 may represent any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

[0033]

[0050] 1, in some examples, the video encoder 200 and the video decoder 300 may each be integrated with an audio encoder and / or an audio decoder and may include an appropriate MUX-DEMUX unit, or other hardware and / or software, to handle multiplexed streams that include both audio and video in a common data stream. Where applicable, the MUX-DEMUX unit may conform to the ITU H.223 multiplexer protocol, or other protocols such as the User Datagram Protocol (UDP).

[0034]

[0051] The video encoder 200 and the video decoder 300 may each be implemented as any of a variety of suitable encoder and / or decoder circuits, or any combination thereof, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, etc. When the techniques are implemented partially in software, the device may store the software's instructions on a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to implement the techniques of this disclosure. Each of the video encoder 200 and the video decoder 300 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder / decoder (CODEC) in the respective device. The device including the video encoder 200 and / or the video decoder 300 may comprise an integrated circuit, a microprocessor, and / or a wireless communication device such as a cellular phone.

[0035]

[0052] Video encoder 200 and video decoder 300 may operate according to a video coding standard, such as ITU-T H.265, also known as High Efficiency Video Coding (HEVC), or extensions thereof, such as multiview and / or scalable video coding extensions. Alternatively or additionally, video encoder 200 and video decoder 300 may operate according to other proprietary or industry standards, such as ITU-T H.266, also known as Generic Video Coding (VVC), and extensions thereof, such as extensions for screen content or high dynamic range. A draft of the VVC standard is set forth in Bross et al., "Versatile Video Coding (Draft 10)," Joint Video Experts Team (JVET) of ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11, 18th Meeting, Teleconference, June 22-July 1, 2020, JVET-S2001-v17 (hereinafter, "VVC Draft 10"). However, the techniques of this disclosure are not limited to any particular coding standard.

[0036]

[0053] Generally, the video encoder 200 and the video decoder 300 may perform block-based coding of pictures. The term “block” generally refers to a structure that includes data to be processed (e.g., encoded, decoded, or otherwise used in an encoding and / or decoding process). For example, a block may include a two-dimensional matrix of samples of luminance and / or chrominance data. Generally, the video encoder 200 and the video decoder 300 may code video data represented in a YUV (e.g., Y, Cb, Cr) format. That is, rather than coding red, green, and blue (RGB) data for a sample of a picture, the video encoder 200 and the video decoder 300 may code a luminance component and a chrominance component, where the chrominance component may include both red and blue hues of chrominance components. In some examples, the video encoder 200 converts received RGB format data to a YUV representation prior to encoding, and the video decoder 300 converts the YUV representation to the RGB format. Alternatively, pre-processing and post-processing units (not shown) may perform these conversions.

[0037]

[0054] This disclosure may generally refer to coding (e.g., encoding and decoding) a picture to include the process of encoding or decoding data for a picture. Similarly, this disclosure may refer to coding a block of a picture to include the process of encoding or decoding data for the block, e.g., predictive and / or residual coding. A coded video bitstream generally includes a set of values ​​for syntax elements that represent coding decisions (e.g., coding modes) and partitioning of a picture into blocks. Thus, references to coding a picture or a block should generally be understood as coding values ​​of syntax elements that form the picture or block. In this disclosure, a current block or current picture generally refers to a block or picture currently being coded or decoded, as opposed to a block or picture that has been decoded or a block or picture that has not yet been decoded.

[0038]

[0055] HEVC defines various blocks, including coding units (CUs), prediction units (PUs), and transform units (TUs). According to HEVC, a video coder (such as video encoder 200) partitions coding tree units (CTUs) into CUs according to a quadtree structure. That is, the video coder partitions CTUs and CUs into four equal non-overlapping squares, and each node of the quadtree has either zero or four child nodes. A node without children may be called a "leaf node", and a CU of such a leaf node may include one or more PUs and / or one or more TUs. A video coder may further partition PUs and TUs. For example, in HEVC, a residual quadtree (RQT) represents the partition of TUs. In HEVC, a PU represents inter-predicted data, while a TU represents residual data. An intra-predicted CU includes intra-prediction information, such as an intra-mode indication.

[0039]

[0056] As another example, the video encoder 200 and the video decoder 300 may be configured to operate according to VVC. According to VVC, a video coder (such as the video encoder 200) partitions a picture into multiple CTUs. The video encoder 200 may partition the CTUs according to a tree structure, such as a quad-tree binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure eliminates the concept of multiple partition types, such as the separation between CUs, PUs, and TUs in HEVC. The QTBT structure includes two levels, a first level partitioned according to a quad-tree partition and a second level partitioned according to a binary tree partition. The root node of the QTBT structure corresponds to a CTU. The leaf nodes of the binary tree correspond to CUs.

[0040]

[0057] In the MTT partitioning structure, blocks may be partitioned using quad tree (QT) partitioning, binary tree (BT) partitioning, and one or more types of triple tree (TT) (also called ternary tree (TT)) partitioning. A triple tree or ternary tree partitioning is a partitioning in which a block is split into three sub-blocks. In some examples, a triple tree or ternary tree partitioning splits a block into three sub-blocks without splitting the original block through the center. The partitioning types in MTT (e.g., QT, BT, and TT) may be symmetric or asymmetric.

[0041]

[0058] In some examples, video encoder 200 and video decoder 300 may use a single QTBT or MTT structure to represent each of the luminance and chrominance components, and in other examples, video encoder 200 and video decoder 300 may use two or more QTBT or MTT structures, such as one QTBT / MTT structure for the luminance component and another QTBT / MTT structure for both chrominance components (or two QTBT / MTT structures for each chrominance component).

[0042]

[0059] Video encoder 200 and video decoder 300 may be configured to use quadtree partitioning, QTBT partitioning, MTT partitioning, or other partition structures in accordance with HEVC. For purposes of explanation, the description of the techniques of this disclosure is presented with respect to QTBT partitioning. However, it should be understood that the techniques of this disclosure may also be applied to video coders configured to use quadtree partitioning, or other types of partitioning as well.

[0043]

[0060] In some examples, the CTU includes a coding tree block (CTB) of luma samples, two corresponding CTBs of chroma samples for a picture with three sample arrays, or a CTB of samples for a monochrome picture, or a picture coded using three separate color planes and syntax structures used to code the samples. The CTB may be an N×N block of samples for some value of N such that the division of the components into the CTB is partitioned. The components are arrays or single samples from one of the three arrays (luma and two chromas) that configure the picture in 4:2:0, 4:2:2, or 4:4:4 color format, or an array or single sample of an array that configures the picture in monochrome format. In some examples, the coding block is an M×N block of samples for some value of M and N such that the division of the CTB into coding blocks is partitioned.

[0044]

[0061] Blocks (e.g., CTUs or CUs) may be grouped in various ways in a picture. As an example, a brick may refer to a rectangular region of a CTU row in a particular tile in a picture. A tile may be a rectangular region of a CTU in a particular tile column and a particular tile row in a picture. A tile column refers to a rectangular region of a CTU having a height equal to the height of the picture and a width specified by a syntax element (e.g., in a picture parameter set). A tile row refers to a rectangular region of a CTU having a height specified by a syntax element (e.g., in a picture parameter set) and a width equal to the width of the picture.

[0045]

[0062] In some examples, a tile may be partitioned into multiple bricks, each of which may include one or more CTU rows within the tile. A tile that is not partitioned into multiple bricks may also be referred to as a brick. However, a brick that is a true subset of a tile may not be referred to as a tile.

[0046]

[0063] The bricks in a picture may also be arranged into slices. A slice may be an integer number of bricks of a picture that may be contained entirely in a single Network Abstraction Layer (NAL) unit. In some examples, a slice includes either several complete tiles or only a continuous sequence of complete bricks of one tile.

[0047]

[0064] This disclosure may use "NxN" and "N by N" interchangeably to refer to sample dimensions of a block (such as a CU or other video block) in terms of vertical and horizontal dimensions, e.g., 16x16 samples or 16 by 16 samples. Generally, a 16x16 CU has 16 samples in the vertical direction (y=16) and 16 samples in the horizontal direction (x=16). Similarly, an NxN CU generally has N samples in the vertical direction and N samples in the horizontal direction, where N represents a non-negative integer value. Samples in a CU may be arranged in rows and columns. Moreover, a CU does not necessarily have to have the same number of samples in the horizontal direction as in the vertical direction. For example, a CU may comprise NxM samples, where M is not necessarily equal to N.

[0048]

[0065] The video encoder 200 encodes video data for a CU that represents prediction and / or residual information, as well as other information. The prediction information indicates how the CU should be predicted to form a predictive block for the CU. The residual information generally represents sample-by-sample differences between samples of the CU prior to encoding and the predictive block.

[0049]

[0066] To predict a CU, the video encoder 200 may generally form a predictive block for the CU through inter prediction or intra prediction. Inter prediction generally refers to predicting a CU from data of a previously coded picture, while intra prediction generally refers to predicting a CU from previously coded data of the same picture. To perform inter prediction, the video encoder 200 may generate a predictive block using one or more motion vectors. The video encoder 200 may generally perform a motion search to identify a reference block that closely matches the CU, for example, with respect to a difference between the CU and the reference block. The video encoder 200 may calculate a difference metric using a sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared difference (MSD), or other such difference calculation to determine whether the reference block closely matches the current CU. In some examples, the video encoder 200 may predict the current CU using unidirectional prediction or bidirectional prediction.

[0050]

[0067] Some examples of VVC also provide an affine motion compensation mode, which may be considered an inter-prediction mode. In an affine motion compensation mode, video encoder 200 may determine two or more motion vectors that represent non-translational motion, such as zooming in or out, rotation, perspective motion, or other irregular motion types.

[0051]

[0068] To perform intra prediction, the video encoder 200 may select an intra prediction mode to generate a predictive block. Some examples of VVC provide 67 intra prediction modes, including various directional modes, as well as planar and DC modes. In general, the video encoder 200 selects an intra prediction mode that describes neighboring samples relative to a current block (e.g., a block of a CU) from which samples of the current block should be predicted. Such samples may generally be above, to the left of, or to the left of the current block in the same picture as the current block, assuming that the video encoder 200 codes the CTUs and CUs in raster scan order (left to right, top to bottom).

[0052]

[0069] Video encoder 200 encodes data representing a prediction mode of the current block. For example, in an inter prediction mode, video encoder 200 may encode data representing which of various available inter prediction modes is used, as well as motion information for the corresponding mode. For example, in unidirectional or bidirectional inter prediction, video encoder 200 may encode motion vectors using advanced motion vector prediction (AMVP) or merge mode. Video encoder 200 may use similar modes to encode motion vectors for affine motion compensation modes.

[0053]

[0070] Following prediction, such as intra- or inter-prediction, of a block, the video encoder 200 may calculate residual data for the block. The residual data, such as a residual block, represents sample-by-sample differences between the block and a predictive block for the block formed using a corresponding prediction mode. The video encoder 200 may apply one or more transforms to the residual block to produce transformed data in a transform domain rather than the sample domain. For example, the video encoder 200 may apply a discrete cosine transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform to the residual video data. Additionally, the video encoder 200 may apply a secondary transform following a first transform, such as a mode-dependent non-separable quadratic transform (MDNSST), a signal-dependent transform, a Karhunen-Loeve transform (KLT), etc. The video encoder 200 produces transform coefficients following application of the one or more transforms.

[0054]

[0071] As mentioned above, following any transformation to produce transform coefficients, the video encoder 200 may perform quantization of the transform coefficients. Quantization generally refers to a process in which transform coefficients are quantized to possibly reduce the amount of data used to represent the transform coefficients, providing further compression. By performing a quantization process, the video encoder 200 may reduce the bit depth associated with some or all of the transform coefficients. For example, the video encoder 200 may truncate an n-bit value to an m-bit value during quantization, where n is greater than m. In some examples, to perform quantization, the video encoder 200 may perform a bitwise right shift of the value to be quantized.

[0055]

[0072] Following quantization, the video encoder 200 may scan the transform coefficients to produce a one-dimensional vector from a two-dimensional matrix including the quantized transform coefficients. The scan may be designed to place higher energy (and therefore lower frequency) transform coefficients at the front of the vector and lower energy (and therefore higher frequency) transform coefficients at the rear of the vector. In some examples, the video encoder 200 may utilize a predefined scan order to scan the quantized transform coefficients to produce a serialized vector and then entropy encode the quantized transform coefficients of the vector. In other examples, the video encoder 200 may perform an adaptive scan. After scanning the quantized transform coefficients to form a one-dimensional vector, the video encoder 200 may entropy encode the one-dimensional vector, e.g., according to context-adaptive binary arithmetic coding (CABAC). The video encoder 200 may also entropy encode values ​​of syntax elements describing metadata associated with the encoded video data for use by the video decoder 300 in decoding the video data.

[0056]

[0073] To implement CABAC, the video encoder 200 may assign a context in a context model to a symbol to be transmitted. The context may relate, for example, to whether neighboring values ​​of the symbol are zero-valued or not. The probability determination may be based on the context assigned to the symbol.

[0057]

[0074] Video encoder 200 may further generate syntax data, such as block-based syntax data, picture-based syntax data, and sequence-based syntax data, for video decoder 300, e.g., in a picture header, block header, slice header, or other syntax data, such as a sequence parameter set (SPS), a picture parameter set (PPS), or a video parameter set (VPS). Video decoder 300 may similarly decode such syntax data to determine how to decode corresponding video data.

[0058]

[0075] In this manner, video encoder 200 may generate a bitstream including syntax elements that describe encoded video data, e.g., partitions of a picture into blocks (e.g., CUs) and predictive and / or residual information for the blocks. Finally, video decoder 300 may receive the bitstream and decode the encoded video data.

[0059]

[0076] In general, the video decoder 300 performs an inverse process of that performed by the video encoder 200 to decode the encoded video data of the bitstream. For example, the video decoder 300 may decode values ​​for syntax elements of the bitstream using CABAC in a manner that is inverse to, but substantially similar to, the CABAC encoding process of the video encoder 200. The syntax elements may define partition information for partitioning a picture into CTUs and partitions of each CTU according to a corresponding partition structure, such as a QTBT structure, to define CUs of the CTU. The syntax elements may further define prediction and residual information for blocks (e.g., CUs) of video data.

[0060]

[0077] The residual information may be represented, for example, by quantized transform coefficients. The video decoder 300 may dequantize and inverse transform the quantized transform coefficients of the block to reconstruct a residual block for the block. The video decoder 300 uses the signaled prediction mode (intra- or inter-prediction) and associated prediction information (e.g., motion information for inter-prediction) to form a predictive block for the block. The video decoder 300 may then combine (sample-by-sample) the predictive block and the residual block to reconstruct the original block. The video decoder 300 may perform additional processing, such as performing a deblocking process to reduce visual artifacts along block boundaries.

[0061]

[0078] This disclosure may generally refer to "signaling" certain information, such as syntax elements. The term "signaling" may generally refer to communication of values ​​for syntax elements and / or other data used to decode encoded video data. That is, video encoder 200 may signal values ​​for syntax elements in a bitstream. Generally, signaling refers to generating values ​​in a bitstream. As mentioned above, source device 102 may transport a bitstream to destination device 116 in substantially real-time or non-real-time, which may occur when storing syntax elements in storage device 112 for later retrieval by destination device 116.

[0062]

[0079] 2A and 2B are conceptual diagrams illustrating an exemplary quad-tree binary tree (QTBT) structure 130 and corresponding CTU 132. Solid lines represent quad-tree splitting, and dotted lines indicate binary tree splitting. At each split (i.e., non-leaf) node of the binary tree, one flag is signaled to indicate which splitting type (i.e., horizontal or vertical) is used, where in this example, 0 indicates horizontal splitting and 1 indicates vertical splitting. In quad-tree splitting, there is no need to indicate the splitting type, since the quad-tree node splits the block horizontally and vertically into four sub-blocks with equal size. Thus, video encoder 200 may encode, and video decoder 300 may decode, syntax elements (such as splitting information) for the region tree level of QTBT structure 130 (i.e., solid lines) and syntax elements (such as splitting information) for the prediction tree level of QTBT structure 130 (i.e., dashed lines). Video encoder 200 may encode, and video decoder 300 may decode, video data, such as prediction and transform data, for CUs represented by terminal leaf nodes of QTBT structure 130.

[0063]

[0080] 2B may be associated with parameters that define the sizes of blocks corresponding to nodes of the QTBT structure 130 at the first and second levels. These parameters may include a CTU size (representing the size of the CTU 132 in a sample), a minimum quad tree size (MinQTSize representing the minimum allowable quad tree leaf node size), a maximum binary tree size (MaxBTSize representing the maximum allowable binary tree root node size), a maximum binary tree depth (MaxBTDepth representing the maximum allowable binary tree depth), and a minimum binary tree size (MinBTSize representing the minimum allowable binary tree leaf node size).

[0064]

[0081] The root node of the QTBT structure corresponding to the CTU may have four child nodes at the first level of the QTBT structure, each of which may be partitioned according to a quadtree partition. That is, the nodes at the first level are either leaf nodes (without child nodes) or have four child nodes. The example QTBT structure 130 represents such a node including a parent node and a child node with solid lines for branching. If the nodes at the first level are not larger than the maximum allowed binary tree root node size (MaxBTSize), the nodes may be further partitioned by their respective binary trees. The binary tree splitting of one node may be repeated until the nodes resulting from the split reach the minimum allowed binary tree leaf node size (MinBTSize) or the maximum allowed binary tree depth (MaxBTDepth). The example QTBT structure 130 represents such a node with dashed lines for branching. The binary tree leaf nodes are called CUs, and the CUs are used for prediction (e.g., intra-picture or inter-picture prediction) and transformation without further partitioning. As explained above, a CU may also be referred to as a "video block" or a "block."

[0065]

[0082] In one example of a QTBT partitioning structure, the CTU size is set as 128×128 (luma samples and two corresponding 64×64 chroma samples), MinQTSize is set as 16×16, MaxBTSize is set as 64×64, MinBTSize is set as 4 (for both width and height), and MaxBTDepth is set as 4. Quad tree partitioning is first applied to the CTU to generate a quad tree leaf node. The quad tree leaf node may have a size from 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If the quad tree leaf node is 128×128, the leaf quad tree node may not be further split by the binary tree since the size exceeds MaxBTSize (i.e., 64×64 in this example). In other cases, the quad tree leaf node may be further partitioned by the binary tree. Thus, the quad tree leaf node is also the root node for the binary tree and has the binary tree depth as 0. When the binary tree depth reaches MaxBTDepth (4 in this example), no further splitting is allowed. A binary tree node having a width equal to MinBTSize (4 in this example) implies that no further vertical splitting (i.e., width division) is allowed for that binary tree node. Similarly, a binary tree node having a height equal to MinBTSize implies that no further horizontal splitting (i.e., height division) is allowed for that binary tree node. As mentioned above, the leaf node of the binary tree is called a CU and is further processed according to the prediction and transformation without further partitioning.

[0066]

[0083] As introduced above, the video encoder 200 and the video decoder 300 may be configured to perform motion vector prediction. In HEVC, there are two inter-prediction modes for a prediction unit (PU), referred to as merge mode (skip is considered as a special case of merge) and AMVP mode, respectively. In the AMVP mode and the merge mode, the video encoder 200 and the video decoder 300 maintain a motion vector (MV) candidate list for multiple motion vector predictors. The motion vector of the current PU, as well as the reference index in the merge mode, are generated by selecting one candidate from the MV candidate list.

[0067]

[0084] In an HEVC implementation, the MV candidate list includes up to five candidates for merge mode and two candidates for AMVP mode. The merge candidate may include a set of motion information, e.g., motion vectors corresponding to both reference picture lists (list 0 and list 1), and a reference index. By receiving the merge candidate identified by the merge index, the video decoder 300 determines the reference picture used for prediction of the current block as well as the associated motion vector. On the other hand, under AMVP mode, for each possible prediction direction from either list 0 or list 1, the video decoder 300 receives an MV predictor (MVP) index into the MV candidate list, since the AMVP candidate includes only a motion vector. The video decoder 300 further receives a motion vector difference (MVD) and a reference index that explicitly identifies the reference picture. In AMVP mode, the predicted motion vector may be further refined.

[0068]

[0085] Candidates for both modes are derived similarly from the same spatial and temporal neighboring blocks. In HEVC, video encoder 200 and video decoder 300 may derive spatial MV candidates from neighboring blocks as shown in Figures 3A and 3B for a particular PU (PU0), but the techniques for generating candidates from blocks differ in merge mode and AMVP mode.

[0069]

[0086] 3A is a conceptual diagram illustrating spatial neighbor candidates of block 140 for merge mode. FIG 3B is a conceptual diagram illustrating spatial neighbor candidates of block 142 for AMVP mode. In merge mode, video encoder 200 and video decoder 300 may derive up to four spatial MV candidates in the order shown in FIG 3A. The order is left block (0, A1), top block (1, B1), top right block (2, B0), bottom left block (3, A0), and top left block (4, B2).

[0070]

[0087] In the AMVP mode, the video encoder 200 and the video decoder 300 may divide the neighboring blocks into two groups, a left group including blocks 0 and 1, and a top group including blocks 2, 3, and 4, as shown in FIG. 3B. For each group, a potential candidate in the neighboring blocks that references the same reference picture as the reference picture indicated by the signaled reference index has the highest priority to be selected to form the final candidate of the group. It is possible that not all the neighboring blocks contain motion vectors that point to the same reference picture. Therefore, if such a candidate cannot be found, the video encoder 200 and the video decoder 300 may scale the first available candidates to form the final candidate. Thus, the temporal distance difference may be compensated.

[0071]

[0088] Temporal motion vector prediction in HEVC is discussed here. Video encoder 200 and video decoder 300 may be configured to add a temporal motion vector predictor (TMVP) candidate to the MV candidate list after the spatial motion vector candidate if the candidate is valid and available. The process of motion vector derivation for a TMVP candidate is the same for both merge mode and AMVP mode. However, in HEVC, the target reference index for a TMVP candidate in merge mode is set to 0.

[0072]

[0089] FIG. 4A shows an example TMVP candidate for block 154 (PU0), and FIG. 4B shows the motion vector scaling process 156. The primary block location for TMVP candidate derivation is the bottom right block outside the collocated PU. This candidate is shown as block "T" in FIG. 4A. Block T's location is used to compensate for the bias towards the top left block used to generate spatial neighbor candidates. However, if the block is located outside the current CTB row or motion information is not available, the block is replaced with the center block of the PU.

[0073]

[0090] Video encoder 200 and video decoder 300 may derive motion vectors for TMVP candidates from the co-located PUs of the co-located picture, shown at the slice level. The motion vectors for the co-located PUs are called co-located MVs. Similar to the temporal direct mode in AVC, to derive TMVP motion vector candidates, the co-located MVs may be scaled to compensate for the temporal distance difference, as shown in FIG. 4B.

[0074]

[0091] Next, other aspects of motion prediction in HEVC related to the techniques described herein are described. Video encoder 200 and video decoder 300 may be configured to perform motion vector scaling. It is assumed that the value of a motion vector is proportional to the distance of pictures in presentation time. A motion vector relates two pictures, a reference picture and a picture that contains the motion vector (i.e., a containing picture). When a motion vector is utilized to predict another motion vector, the distance between the containing picture and the reference picture is calculated based on a Picture Order Count (POC) value.

[0075]

[0092] For a motion vector to be predicted, the associated containing picture may be different from the reference picture. Therefore, the video encoder 200 and the video decoder 300 may calculate a new distance based on the POC. The video encoder 200 and the video decoder 300 may scale the motion vector based on these two POC distances. For spatial neighbor candidates, the containing pictures for the two motion vectors are the same, but the reference pictures are different. In HEVC, motion vector scaling is applied to both TMVP and AMVP for spatial and temporal neighbor candidates.

[0076]

[0093] Video encoder 200 and video decoder 300 may be configured to perform artificial motion vector candidate generation: if the motion vector candidate list is not complete, artificial motion vector candidates are generated and inserted at the end of the list until the list is filled.

[0077]

[0094] In merge mode, there are two types of pseudo MV candidates: composite candidates derived only for B slices, and zero candidates that are used only for AMVP if the first type does not provide enough pseudo candidates. For each pair of candidates that are already in the candidate list and have the necessary motion information, a bidirectional composite motion vector candidate is derived by combining the motion vector of the first candidate that references a picture in list 0 and the motion vector of the second candidate that references a picture in list 1.

[0078]

[0095] The video encoder 200 and the video decoder 300 may be configured to implement a pruning process for candidate insertion. Candidates from different blocks may coincidentally be the same, which reduces the efficiency of the merge / AMVP candidate list. To solve this problem, a pruning process is applied. When implementing a pruning process, the video encoder 200 or the video decoder 300 compares a candidate with other candidates in the current candidate list to some extent to avoid inserting the same candidate. To reduce complexity, instead of comparing each potential candidate with all other existing candidates, only a limited number of pruning processes are applied.

[0079]

[0096] The video encoder 200 and the video decoder 300 may be configured to implement template matching prediction. Template matching prediction is a special merge mode based on the frame rate up-conversion (FRUC) technique. In this mode, partial motion information of a block is not signaled but is derived at the decoder side. Template matching may be applied to both the AMVP mode and the normal merge mode. In the AMVP mode, the selection of the MVP candidate is determined based on template matching to pick up the template that reaches the minimum difference between the template of the current block and the template of the reference block. In the normal merge mode, a template matching mode flag is signaled to indicate the use of template matching. Then, the video encoder 200 and the video decoder 300 may apply template matching to the merge candidate indicated by the merge index for MV refinement.

[0080]

[0097] As shown in FIG. 5, template matching is used to derive motion information of the current CU 160 by finding the closest match between the current template 162 in the current picture and the reference template 164 (same size as the template) in the reference picture. If the AMVP candidate is selected based on the initial matching error, the video encoder 200 and the video decoder 300 may refine the MVP using template matching. If the merge candidate is indicated by the signaled merge index, the video encoder 200 and the video decoder 300 may be configured to refine the MVs corresponding to L0 and L1 independently by template matching, and then further refine the less accurate MVs based on the more accurate MVs.

[0081]

[0098] The video encoder 200 and the video decoder 300 may be configured to determine a cost function. When the motion vector points to a fractional sample position, motion compensated interpolation is needed. To reduce complexity, bilinear interpolation may be used for template matching to generate a template on the reference picture instead of the usual 8-tap DCT-IF interpolation. The matching cost C of template matching is calculated as follows:

[0082]

number

[0083] where w is a weighting factor that can be set to an integer such as 0, 1, 2, 3, or 4, and MV and MV s and denote the currently tested MV and the first MV (i.e., the MVP candidate in AMVP mode or the merged motion in merge mode), respectively. The SAD is used as the matching cost for template matching.

[0084]

[0099] When template matching is used, the video encoder 200 and the video decoder 300 may be configured to refine the motion using only luma samples. The derived motion may be used for both luma and chroma for motion-compensated (MC) inter prediction. After the MV is determined, the final MC is performed using an 8-tap interpolation filter for luma and a 4-tap interpolation filter for chroma.

[0085]

[0100] The video encoder 200 and the video decoder 300 may be configured to determine and implement a search process. The MV refinement is a pattern-based MV search with a criterion of template matching cost and a hierarchical structure. Two search patterns are supported for MV refinement: diamond search and cross search. The hierarchical structure specifies an iterative process for refinement of MVs starting at a coarse MVD precision (e.g., quarter-pel) and ending at a fine MVD precision (e.g., 1 / 8-pel). The MVs may be directly searched at 1 / 4 luma sample MVD precision with diamond pattern, followed by 1 / 4 luma sample MVD precision with cross pattern, and then 1 / 8 luma sample MVD refinement with cross pattern. The search range of MV refinement is set equal to (-8, +8) luma samples around the first MV. When the current block is bi-predictive, both MVs are refined independently, and then the best MV among them (in terms of matching cost) is set as the preferred one for further refinement of the other MV with BCW weight values.

[0086]

[0101] The video encoder 200 and the video decoder 300 may be configured to perform affine prediction. In HEVC, only a transform motion model is applied for motion compensated prediction (MCP). While in the real world, there are many kinds of motions, such as zoom in / out, rotation, perspective motion and other irregular motions. In VTM-6, a block-based affine transform motion compensated prediction is applied. As shown in FIG. 6A, the affine motion field of a block is described by the motion information of two control points (170A and 170B), also called a four-parameter model. As shown in FIG. 6B, the affine motion field of a block is described by the motion information of three control points (172A-172C) and three control point motion vectors, also called a six-parameter model.

[0087]

[0102] For the four-parameter affine motion model, the motion vector at a sample location (x,y) in a block is derived as follows:

[0088]

number

[0089]

[0103] For a six-parameter affine motion model, the motion vector at a sample location (x,y) in a block is derived as follows:

[0090]

number

[0091]

[0104] In the above formula, (mv 0x ,mv 0y ) represents the CPMV in the upper left corner, and (mv 1x ,mv 1y ) and (mv 2x ,mv 2y ) represent the CPMVs in the upper right and lower left corners, respectively.

[0092]

[0105] To simplify the motion compensation prediction, the video encoder 200 and the video decoder 300 may be configured to apply block-based affine transform prediction. FIG. 7 shows a block 170, which is a 16×16 luma block including 16 4×4 luma sub-blocks. To derive a motion vector for each 4×4 luma sub-block, the video encoder 200 and the video decoder 300 calculate the motion vector of the center sample of each sub-block as shown in FIG. 7 according to the above formula and rounded to 1 / 16 fractional accuracy. Arrows 172A and 172B identify two of the 16 motion vectors for the sub-block. The other 14 arrows also correspond to motion vectors but are not labeled in FIG. 7. A motion compensation interpolation filter is applied to generate a prediction of each sub-block with the derived motion vector. The sub-block size of the chroma components is also set to be 4×4. The MV of a 4×4 chroma sub-block is calculated as the average of the MVs of the four corresponding 4×4 luma sub-blocks.

[0093]

[0106] The video encoder 200 and the video decoder 300 may be configured to perform prediction refinement using optical flow for affine mode. Prediction refinement with optical flow (PROF) is used to improve sub-block-based affine motion compensation prediction without increasing memory access bandwidth for motion compensation. In VVC, after sub-block-based affine motion compensation is performed, the luma prediction samples are refined by adding the difference derived by the optical flow equation.

[0094]

[0107] In one example implementation of PROF, the video decoder 300 may be configured to perform the following four steps.

[0095] Step 1) Sub-block based affine MC is performed to generate the sub-block prediction I(i,j).

[0096] Step 2) Spatial gradients of subblock predictions g x (i,j) and g y (i,j) is computed at each sample location using a 3-tap filter [-1,0,1]. The gradient computation is exactly the same as the gradient computation in BDOF.

[0097]

number

[0098] Here, shift1 is used to control the accuracy of the gradient. Sub-block (e.g., 4×4) predictions are extended by one sample on each side for gradient calculation. To avoid additional memory bandwidth and additional interpolation calculations, those extended samples on the extended boundary are copied from the nearest integer pixel location in the reference picture.

[0099] Step 3) The luma prediction refinement is calculated according to the optical flow formula:

[0100]

number

[0101] Here, Δv(i,j) is the difference between the sample MV calculated for sample location (i,j) denoted by v(i,j) and the sub-block MV of the sub-block to which sample (i,j) belongs, as shown in FIG. 8. Δv(i,j) is quantized in units of 1 / 32 luma sample precision. FIG. 8 shows the sub-block MV V SB and pixel Δv(i,j) (arrow 190).

[0102]

[0108] Since the affine model parameters and the sample location relative to the center of the sub-block do not change for each sub-block, Δv(i,j) can be calculated for the first sub-block and reused for other sub-blocks in the same CU. dx(i,j) and dy(i,j) are calculated from sample location (i,j) to the sub-block (x SB,y SB ), Δv(i,j) may be derived by the following equation:

[0103]

number

[0104]

[0109] To maintain accuracy, we use subblocks (x SB ,y SB The center of ((W SB -1) / 2,(H SB -1) / 2), where W SB and H SB are the width and height of the sub-block, respectively.

[0105]

[0110] For the four-parameter affine model, we have:

[0106]

number

[0107]

[0111] For the six-parameter affine model, we have:

[0108]

number

[0109] Here, (v 0x ,v 0y ), (v 1x ,v 1y ) and (v 2x ,v 2y ) are the top-left, top-right, and bottom-left control point motion vectors, and w and h are the width and height of the CU.

[0110] Step 4) Finally, the luma prediction refinement ΔI(i,j) is added to the sub-block prediction I(i,j). The final prediction I′ is generated as:

[0111]

number

[0112]

[0112] PROF is not applied in two cases for affine coded CUs: 1) All control points MV are the same, which indicates that the CU has only translational motion, and 2) the affine motion parameters are larger than the specified limit because sub-block based affine MC is degraded to CU based MC to avoid large memory access bandwidth requirements.

[0113]

[0113] The video encoder 200 and the video decoder 300 may be configured to apply a fast encoding process to reduce the encoding complexity of affine motion estimation using PROF. PROF is not applied in the affine motion estimation stage in the following two situations: a) if this CU is not a root block and its parent block is not coded with affine mode as its best mode, PROF is not applied because the current CU is unlikely to use affine mode as its best mode; if the magnitudes of the four affine parameters (C, D, E, F) are all smaller than a predefined threshold and the current picture is not a low-latency picture, PROF is not applied because the improvement introduced by PROF is small in this case. In this way, affine motion estimation using PROF may be accelerated.

[0114]

[0114] Existing techniques have several potential problems. The signaling overhead of CPMV can increase significantly for a block when compared to that of the translational model of inter prediction. Therefore, decoder-side improvements for CPMV can improve the accuracy of CPMV and reduce the signaling overhead. This disclosure describes techniques that can address some of these problems.

[0115]

[0115] Template matching based affine prediction (hereinafter abbreviated as AffTM) is a decoder-side inter-prediction mode for improving the CPMV of an affine coded block. Similar to template matching, as described above, the video decoder 300 may determine an initial reference template block based on an initially determined CPMV, and then search within a search area for other reference templates with reduced matching costs. The video decoder 300 may then determine the best set of CPMVs to replace the initial CPMV.

[0116]

[0116] The video encoder 200 and the video decoder 300 may be configured to determine a reference template block. The samples of the reference template block are generated on a sub-block basis based on a motion field derived using CPMV. Under the assumption that the current block and the corresponding current template block 192 are located in the same affine motion field, the video encoder 200 and the video decoder 300 may use equation (2-1) or (2-2) to determine the MV of the sub-block (e.g., A0, A1, ..., A1 in FIG. 9A on the current template block 192). n-1 and L0,L 1, ...L n-1), where the sample location (x,y) is the centroid of the respective subblock. Video encoder 200 and video decoder 300 then fetch or interpolate the subblock samples for the reference template block based on the respective subblock MV. As illustrated by reference template 194A in the example shown in FIG. 9B, the reference template subblock does not need to be immediately adjacent to any boundary subblock of the predictive block. Furthermore, the interpolation filter used to generate the subblock samples on the reference template block may be any one or more of: no filter (so that the subblock MV is clipped or rounded to integer precision before fetching the reference samples), a 2-tap bilinear filter, a 6-tap DCTIF as in AVC, an 8-tap DCTIF as in HEVC or VVC, or a switchable filter as in VVC.

[0117]

[0117] In another example, as illustrated by reference template 194B in Figure 9C, a reference template sub-block can be immediately adjacent to a boundary sub-block of the corresponding prediction block. Thus, the MV(A0,..., n-1 and L 0,...,n-1 ) is identical to the corresponding immediately adjacent sub-block located on the boundary of the current block.

[0118]

[0118] In another example, the MV of a sub-block other than A0 and L0 on the current template block may be calculated through equation (2-1) or (2-2) at a sample location (x,y) that is on the centroid between the sub-block itself and its immediate neighboring sub-block on the boundary of the current block. For A0 and L0, the sample location (x,y) may be (0,0) if both A0 and L0 exist, the centroid between A0 and the first sub-block on the current block if only A0 exists, or the centroid between L0 and the first sub-block on the current block if only L0 exists.

[0119]

[0119] In another example, the video encoder 200 and the video decoder 300 may be configured to apply PROF to a reference template block.

[0120]

[0120] In another example, when all CPMVs are identical to each other, the video encoder 200 and the video decoder 300 may replace the prediction process of AffTM with that of normal block-based template matching, as described above. One of the CPMVs may be regarded as the first MV and used for block-based template matching.

[0121]

[0121] In another example, when all the initial CPMVs are identical to each other, the video encoder 200 and the video decoder 300 may be configured to perform regular block-based template matching, as described above, before AffTM to refine the initial CPMVs. One of the CPMVs may be considered as the initial MV for the regular template matching process. This example may be further extended to searching for a translation model, as described above.

[0122]

[0122] The video encoder 200 and the video decoder 300 may be configured to perform a search process. This section presents some search processes for AffTM. Without loss of generality, all algorithms are presented using a 6-parameter affine model. These algorithms can be easily converted for the 4-parameter affine mode by simply removing the bottom left CPMV from the description. The search range, e.g., ±2, ±4, ±6, ±8 pixels, may be predefined or signaled. The initial search point of the CPMV may be either the CPMV corresponding to the reference picture list of the AMVP candidate, the merge candidate, or the CPMV corresponding to the reference picture list of the block.

[0123]

[0123] The video encoder 200 and the video decoder 300 may be configured to perform a rectangular search. The rectangular search applies a rectangular pattern to refine the CPMV of a CU in order, one vector at a time, starting with the top-left CPMV, then the top-right one, and ending with the bottom-left one. Note that the bottom-left CPMV is searched only if the CU is of a six-parameter model to have this third CPMV. The rectangular search pattern may be specified as a sequence of delta motion vectors, dMv={(0,0),(-1,1),(0,1),(1,1),(1,0),(1,-1),(0,-1),(-1,-1),(-1,0)}, or any other order based on these nine delta motion vectors. Without loss of generality, this section refers to the above-mentioned dMv as an initial search step size s0 and a minimum search step size s2, which are determined according to the AMVR index or 1 / 16 indication when the CU is coded by the affine AMVP mode and the affine merge mode, respectively. min As an example, we use the function s0 for all i∈{0,1,...,min}. min may be set to greater than or equal to s i+1 s i The rectangle search process is a seven-step process specified as follows: 1. A set of search step sizes {s0,s1,...,s min}, dMv, and mv0, which represent the top-left, top-right, and bottom-left CPMVs, respectively. (0) , mv1 (0) and mv2 (0) Assuming that, the search process starts at iteration i=0. 2.s i , dMv, and {mv0 (i) ,mv1 (i) ,mv2 (i)}, the search subprocess starts with mv0 (i) , then mv1 (i) , and finally mv1 (i) (The order is, in some cases, {mv2 (i),mv1 (i) ,mv0 (i)}.) 3.s i , dMv and mv0 (i) Using the above, the search subprocess calculates the respective template matching costs for all of these sets of CPMVs individually. (i) +d*s i ,mv1 (i) ,mv2 (i) , for all d∈dMv}. This search subprocess is (i)* =mv0 (i) +argmin d {cost(S0),cost(S1),...,cost(S8)}*s i It can be shown as: 4. As in step 3, the search subprocess continues with S={mv0 (i)* ,mv1 (i) +d*s i ,mv2 (i) , for all d∈dMv}, calculate the template matching cost for each, and the best result is mv1 (i)* It is shown as: 5. As in step 3, the search subprocess continues with S={mv0 (i)* ,mv1 (i)* ,mv2 (i) +d*s i , for all d∈dMv}, calculate the template matching cost for each, and the best result is mv2 (i)* It is shown as: 6. The output of the search subprocess is {mv0 (i)* ,mv1 (i)* ,mv2 (i)*}.

[0124] The search process is performed with a search step size of s i If step 6 is visited more than a predefined threshold number of times when (i+1) ,mv1 (i+1),mv2 (i+1)} equal to the subprocess output and go to step 7.

[0125] Otherwise, the subprocess output is {mv0 (i) ,mv1 (i) ,mv2 (i)}, the search process is (i+1) ,mv1 (i+1) ,mv2 (i+1)} equal to the subprocess output and go to step 7.

[0126] Otherwise (if the subprocess output is {mv0 (i) ,mv1 (i) ,mv2 (i)}), {mv0 (i) ,mv1 (i) ,mv2 (i)} is {mv0 (i)* ,mv1 (i)* ,mv2 (i)*} and the search process continues at step 2. 7.s i s min , the search process sets i equal to i+1 and returns to step 2. Otherwise, the search process returns to output {mv0 (i+1) ,mv1 (i+1) ,mv2 (i+1)} at the end.

[0127]

[0124] The video encoder 200 and the video decoder 300 may be configured to perform a cross search, which uses a cross pattern to improve the CPMV. Its search process is identical to the rectangular search, except that the delta motion vector is defined differently. The delta motion vector of this search pattern is defined as dMv={(0,0),(-1,0),(0,-1)(0,1),(1,0)}.

[0128]

[0125] The video encoder 200 and the video decoder 300 may be configured to perform a diagonal search, which uses a diagonal pattern to improve CPMV. Its search process is identical to the rectangular search, except that the delta motion vector is defined differently, as dMv={(0,0),(-1,-1),(-1,1),(1,1),(1,-1)}.

[0129]

[0126] The video encoder 200 and the video decoder 300 may be configured to implement a diamond search, which uses a diagonal pattern to improve CPMV. Its search process is identical to the rectangular search, except that the delta motion vector is defined differently, as dMv={(0,0),(0,2),(1,1),(2,0),(1,-1),(0,-2),(-1,-1),(-2,0),(-1,1)}.

[0130]

[0127] In another example, the output of the diamond search may be used as the input for a cross search, with the output of the cross search being treated as the final output of the combined search process.

[0131]

[0128] The video encoder 200 and the video decoder 300 may be configured to perform a two-pass eight-point search. The two-pass eight-point search is a search process with two search patterns (i.e., a cross pattern and a diagonal pattern) that are conditionally used during the search process. The search process is identical to the rectangular search except for steps 3-5. In the two-pass eight-point search, dMV comprises two sets of delta motion vectors, dMv0={(0,0),(-1,0),(0,-1)(0,1),(1,0)} and dMv1={(-1,-1),(-1,1),(1,1),(1,-1)}). The following shows the difference with respect to the rectangular search. 1-2. These steps are the same as those for square search. 3.s i , dMv and mv0 (i)Using the above, the search subprocess calculates the respective template matching costs for all of these sets of CPMVs individually. (i) +d*s i ,mv1 (i) ,mv2 (i) , for all d∈dMv0}. This search subprocess continues until d0 * =argmin d It can be denoted as {cost(S0),cost(S1),...,cost(S5)}. Next, d0 * If is equal to (0,0), mv0 (i)* mv0 (i) is set equal to Otherwise, the subprocess uses S={mv0 (i) +d*s i ,mv1 (i) ,mv2 (i) , for all d∈dMv1U d0 * Calculate the template matching cost for each of the {for} and find the best delta motion vector for them, d1 * The result is shown as mv0 (i)* =mv0 (i) +d1 * *s i It is. 4. As in step 3, the search subprocess continues with S={mv0 (i)* ,mv1 (i) +d*s i , mv2 (i) , for all d∈dMv0}, and when necessary, calculate the template matching cost for other S={mv0 (i)* ,mv1 (i) +d*s i , mv2 (i) , for all d∈dMv1U d0 * Calculate the template matching cost for each of the}. The best search result is mv1 (i)* = mv1 (i) +d1 * *s i (d0 * ≠(0,0)) or mv1(i) (d0 * =(0,0)). 5. As in step 3, the search subprocess continues with S={mv0 (i)* ,mv1 (i)* ,mv2 (i) +d*s i , for all d∈dMv0}, and when necessary, calculate the template matching cost for other S={mv0 (i)* ,mv1 (i) ,mv2 (i) +d*s i , for all d∈dMv1U d0 * Calculate the template matching cost for each of the}. The best search result is mv2 (i)* =mv2 (i) +d1 * *s i (d0 * ≠(0,0)) or mv2 (i) (d0 * =(0,0)). 6-7. These steps are the same as those for square search.

[0132]

[0129] The video encoder 200 and the video decoder 300 may be configured to perform a gradient-based search to simultaneously update all CPMVs. (0) ,mv1 (0) ,mv2 (0)}, the CPMV is used to generate a reference template block that is used to compute sample domain gradient values ​​horizontally and vertically and to compute the prediction residual (i.e., the delta between the current template block and the reference template block). These values ​​are then used in a gradient-based search to update the given CPMV. (1) ,mv1 (1) ,mv2 (1)The new CPMV, denoted as {}, then serves as input to another iteration of the gradient-based search. The iterative process may terminate when a condition is met. The condition may be, for example, that the number of iterations exceeds a predefined (or signaled) threshold or that the CPMV does not change between two iterations.

[0133]

[0130] The video encoder 200 and the video decoder 300 may be configured to perform a translation model search. When all CPMVs happen to be the same before, during, or after the above-mentioned search process is applied, all search processes of the AffTM are terminated, and one of the best CPMVs (e.g., a random one since all CPMVs are the same) is used in normal block-based template matching, as described in template matching prediction above, as its initial search point for further motion vector refinement.

[0134]

[0131] The video encoder 200 and the video decoder 300 may be configured to calculate a template matching cost. The template matching cost may be defined (or signaled) as one of the following metrics: SAD, sum of absolute transformed difference (SATD), sum of squared error (SSE), mean removed absolute transformed difference (MRSAD), mean removed absolute transformed difference (MRSATD). MRSAD may be used conditionally if illumination compensation is used for the currently processed block.

[0135] In another example, the video encoder 200 and the video decoder 300 may assign a per-sample weight value to each sample on the template block. For example, for a template block of W×H, the per-sample weight value may be N*w x,y and the respective samples c of the current block template and the reference block template are denoted as x,y and p x,ywhere N may be a positive integer (e.g., 1, 2, 3, 4, 5, etc.). The template matching cost may be defined for simplicity as:

[0136]

number

[0137] or

[0138]

number

[0139] When local illumination compensation (LIC) or MRSAD is used, the formula may be as follows for simplicity:

[0140]

number

[0141] or

[0142]

number

[0143] In these equations, Δ x,y p x,y from the average of c x,y (In other words, mean(p x,y )-mean(c x,y ). Because the weight value assignments of the left template are the transpose of the weight values ​​of the top template, only the weight value assignments of the top template need to be determined.

[0144]

[0134] In another example, the weight values ​​per sample may be region-based, where the template block is equally divided into 16 regions and the template samples within a region share a single weight value.

[0145]

[0135] Figure 10 is a conceptual diagram illustrating an example of per-sample weights that may be assigned to samples of neighboring blocks to calculate template matching costs. In some examples, the video encoder 200 and the video decoder 300 may assign larger weight values ​​to regions closer to the current block, and / or smaller weight values ​​are assigned to regions closer to the top-left corner of the current block. Two examples are shown in Figure 10. In both examples for current CUs 198A and 198B, the video encoder 200 and the video decoder 300 may assign larger weight values ​​to regions closer to the current block, while in the example of 198A, the video encoder 200 and the video decoder 300 further adjust the weight value for the region closer to the top-left corner of the current block down.

[0146]

[0136] In another example, the above metrics may be summed in a weighted manner, as described above for template matching prediction, with all CPMV delta motion vectors derived by AffTM.

[0147]

[0137] The video encoder 200 and the video decoder 300 may be configured to perform a bi-predictive search process. In some examples, the video encoder 200 and the video decoder 300 may be configured to individually refine the CPMV corresponding to each reference picture list of a bi-predictive block by AffTM.

[0148]

[0138] In some examples, the video encoder 200 and the video decoder 300 may be configured to first individually refine the CPMV corresponding to each reference picture list of the bi-predictive block using AffTM. Then, the video encoder 200 and the video decoder 300 may further refine the CPMV corresponding to the reference picture list, which may correspond to other reference picture lists as preferred. For example, the video encoder 200 and the video decoder 300 may select the bi-prediction weight w for L1 and the CPMV of L1 to be further refined. First, the current template block used during the refinement becomes the weighted delta between the original current template block C and the reference template block R0 corresponding to L0.

[0149] C' = (C-(1-w)R0) / w

[0139] This subtraction process is also called high frequency elimination, and C' is used in the same manner as the current template block used during the search process of L1 CPMV. Note that this high frequency elimination can be implemented inversely as C'=(C-wR0) / (1-w).

[0150]

[0140] In some examples, the video encoder 200 and the video decoder 300 may be configured to apply high frequency removal when the CPMV of a reference picture list Lx is to be refined, where x may be 0 or 1. In some examples, the video encoder 200 and the video decoder 300 may be configured to apply high frequency removal when the CPMV of a reference picture list Lx is to be refined, where x may be 0 or 1. After AffTM is performed on the CPMV of Lx, high frequency removal is applied based on the CPMV of Lx, and then AffTM can be performed on the CPMV of other reference picture lists. This iterative process ends when none of the CPMVs are changed during the search process of AffTM.

[0151]

[0141] In some examples, the video encoder 200 and the video decoder 300 may be configured to first apply high frequency removal to the CPMV of Lx according to the BWC weight value. A rule may be applied. The rule may be, for example, that the CPMV of L0 is improved first when the BCW weight of L0 is larger or that the CPMV of L0 is improved first when the BCW weight of L0 is smaller.

[0152]

[0142] In some examples, the video encoder 200 and the video decoder 300 may be configured to apply high frequency removal to the CPMV of Lx first depending on a ph_mvd_l1_zero_flag flag (which indicates that the MVD of L1 CPMV is always 0 and may be named differently across video coding standards). A rule may be applied. The rule may be, for example, that the CPMV of L0 is refined first when the flag is true or that the CPMV of L0 is refined first when the flag is false.

[0153]

[0143] In some examples, the video encoder 200 and the video decoder 300 may be configured to first refine the CPMV of a reference picture list Lx when the template matching cost is higher than that of other reference picture lists, as described in the examples above.

[0154]

[0144] In some examples, the video encoder 200 and the video decoder 300 may be configured to first refine the CPMVs of reference picture lists Lx, which achieve a higher template matching cost than those of other reference picture lists after costs based on the initial CPMV are calculated for each reference picture list, respectively.

[0155]

[0145] In some examples, the video encoder 200 and the video decoder 300 may be configured to convert bi-predictive CPMV to uni-predictive CPMV. After performing AffTM, it is considered that there are two template matching cost values: cost0 for CPMV corresponding to L0 before high frequency elimination is applied, and cost1 for CPMV corresponding to L1. The third cost value is from that generated by AffTM after high frequency elimination is applied. If the third one is higher than one of the other two, then either the CPMV corresponds to the reference L0 or L1 is dropped depending on which of cost0 and cost1 is larger.

[0156]

[0146] The video encoder 200 and the video decoder 300 may be configured to perform a model transformation from a four-parameter model to a six-parameter model. The affine model may be transformed from a four-parameter one to a six-parameter one. At the coordinate position of the bottom-left corner on the current block (i.e., (0, height of block)), the CPMV of the bottom-left corner may be calculated based on equation (2-1). Then, the motion model of the current block is treated as a six-parameter affine model for AffTM.

[0157]

[0147] Video encoder 200 and video decoder 300 may be configured to perform a model transformation from a translational model to an affine model. In some examples, the AffTM process described previously may be applied on top of regular block-based template matching in template matching merge mode, with all initial CPMVs set equal to the translational MVs generated by the template matching process. If the template matching cost of applying the additional AffTM process is less than the cost of regular template matching, the CPMVs of the AffTM process are used for affine motion compensation for the current block instead of the translational motion model from the original template matching process.

[0158]

[0148] In some examples, the transform is applied only if neither bilateral matching (or decoder-side motion refinement (DMVR) as in VVC) nor bidirectional optical flow (BDOF) is applied to the current block.

[0159] In some examples, the video decoder 300 may be configured to always use a four-parameter affine model as the target transformation model. In some examples, the video decoder 300 may be configured to use a six-parameter affine model as the target transformation model. In some examples, the video decoder 300 may determine the final motion model by minimizing a template matching cost.

[0160]

[0150] Figure 11 is a block diagram illustrating an example video encoder 200 that may implement the techniques of this disclosure. Figure 11 is provided for illustrative purposes and should not be considered as limiting the techniques broadly illustrated and described in this disclosure. For purposes of explanation, this disclosure describes video encoder 200 in accordance with VVC (ITU-T H.266 under development) and HEVC (ITU-T H.265) techniques. However, the techniques of this disclosure may be implemented by video encoding devices configured for other video coding standards.

[0161] 11, the video encoder 200 includes a video data memory 230, a mode selection unit 202, a residual generation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a filter unit 216, a decoded picture buffer (DPB) 218, and an entropy coding unit 220. Any or all of the video data memory 230, the mode selection unit 202, the residual generation unit 204, the transform processing unit 206, the quantization unit 208, the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the filter unit 216, the DPB 218, and the entropy coding unit 220 may be implemented in one or more processors or processing circuits. For example, the units of video encoder 200 may be implemented as one or more circuits or logic elements, as part of hardware circuitry, or as part of a processor, ASIC, or FPGA. Moreover, video encoder 200 may include additional or alternative processors or processing circuits for performing these and other functions.

[0162]

[0152] The video data memory 230 may store video data to be encoded by components of the video encoder 200. The video encoder 200 may receive video data stored in the video data memory 230, for example, from the video source 104 (FIG. 1). The DPB 218 may act as a reference picture memory that stores reference video data for use in predicting subsequent video data by the video encoder 200. The video data memory 230 and the DPB 218 may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The video data memory 230 and the DPB 218 may be provided by the same memory device or separate memory devices. In various examples, the video data memory 230 may be on-chip with other components of the video encoder 200, as shown, or off-chip relative to those components.

[0163] In this disclosure, references to video data memory 230 should not be construed as limited to memory internal to video encoder 200, unless specifically so described, or to memory external to video encoder 200, unless specifically so described. Instead, references to video data memory 230 should be understood as a reference memory that stores video data that video encoder 200 receives for encoding (e.g., video data for a current block to be encoded). Memory 106 of FIG. 1 may also provide temporary storage of outputs from various units of video encoder 200.

[0164]

[0154] The various units in FIG. 11 are shown to aid in understanding the operations performed by the video encoder 200. The units may be implemented as fixed-function circuits, programmable circuits, or a combination thereof. A fixed-function circuit refers to a circuit that provides a specific function and is pre-configured with respect to the operations that may be performed. A programmable circuit refers to a circuit that may be programmed to perform various tasks and to provide flexible functionality in the operations that may be performed. For example, a programmable circuit may execute software or firmware that causes the programmable circuit to operate in a manner defined by the software or firmware instructions. A fixed-function circuit may execute software instructions (e.g., to receive parameters or output parameters), but the type of operations that the fixed-function circuit performs is generally unchanging. In some examples, one or more of the units may be separate circuit blocks (fixed function or programmable), and in some examples, one or more of the units may be an integrated circuit.

[0165]

[0155] Video encoder 200 may include an arithmetic logic unit (ALU), a basic functional unit (EFU), a programmable core formed of digital circuits, analog circuits, and / or programmable circuits. In examples in which the operations of video encoder 200 are implemented using software executed by programmable circuits, memory 106 (FIG. 1) may store instructions (e.g., object code) of the software that video encoder 200 receives and executes, or another memory (not shown) within video encoder 200 may store such instructions.

[0166]

[0156] The video data memory 230 is configured to store the received video data. The video encoder 200 may retrieve pictures of the video data from the video data memory 230 and provide the video data to the residual generation unit 204 and the mode selection unit 202. The video data in the video data memory 230 may be raw video data to be encoded.

[0167]

[0157] The mode selection unit 202 includes a motion estimation unit 222, a motion compensation unit 224, and an intra prediction unit 226. The mode selection unit 202 may include additional functional units for performing video prediction according to other prediction modes. By way of example, the mode selection unit 202 may include a palette unit, an intra block copy unit (which may be part of the motion estimation unit 222 and / or the motion compensation unit 224), an affine unit, a linear model (LM) unit, etc.

[0168]

[0158] The mode selection unit 202 generally coordinates multiple encoding passes to test combinations of encoding parameters and produces rate-distortion values ​​for such combinations. The encoding parameters may include partitioning of the CTU into CUs, prediction modes for the CUs, transform types for residual data of the CUs, quantization parameters for residual data of the CUs, etc. The mode selection unit 202 may finally select a combination of encoding parameters that has a rate-distortion value that is better than other tested combinations.

[0169]

[0159] The video encoder 200 may partition a picture retrieved from the video data memory 230 into a series of CTUs, encapsulating one or more CTUs in a slice. The mode selection unit 202 may partition the CTUs of the picture according to a tree structure, such as the QTBT structure or quadtree structure of HEVC described above. As described above, the video encoder 200 may form one or more CUs from partitioning the CTUs according to the tree structure. Such a CTU may also be generally referred to as a "video block" or "block."

[0170]

[0160] In general, the mode selection unit 202 also controls its components (e.g., the motion estimation unit 222, the motion compensation unit 224, and the intra prediction unit 226) to generate a prediction block for a current block (e.g., the current CU, or in HEVC, the overlapping portion of the PU and TU). For inter prediction of the current block, the motion estimation unit 222 may perform motion search to identify one or more closely matching reference blocks in one or more reference pictures (e.g., one or more previous coded pictures stored in the DPB 218). In particular, the motion estimation unit 222 may calculate a value representing how similar a potential reference block is to the current block according to, for example, a sum of absolute differences (SAD), a sum of squared differences (SSD), a mean absolute difference (MAD), a mean squared difference (MSD), etc. The motion estimation unit 222 may generally perform these calculations using a sample-by-sample difference between the current block and the reference block under consideration. Motion estimation unit 222 may identify the reference block having the lowest value resulting from these calculations, which indicates the reference block that most closely matches the current block.

[0171]

[0161] The motion estimation unit 222 may form one or more motion vectors (MVs) that define the position of a reference block in a reference picture relative to the position of a current block in a current picture. The motion estimation unit 222 may then provide the motion vectors to the motion compensation unit 224. For example, in unidirectional inter prediction, the motion estimation unit 222 may provide a single motion vector, while in bidirectional inter prediction, the motion estimation unit 222 may provide two motion vectors. The motion compensation unit 224 may then generate a predictive block using the motion vectors. For example, the motion compensation unit 224 may use the motion vectors to retrieve data of the reference block. As another example, if the motion vectors have fractional sample accuracy, the motion compensation unit 224 may interpolate values ​​for the predictive block according to one or more interpolation filters. Furthermore, in bidirectional inter prediction, the motion compensation unit 224 may retrieve data for the two reference blocks identified by the respective motion vectors and combine the retrieved data through, for example, sample-wise averaging or weighted averaging.

[0172]

[0162] In accordance with the techniques described herein, the motion estimation unit 222 and the motion compensation unit 224 may be configured to encode and decode blocks of video data using an affine prediction mode. Furthermore, the motion estimation unit 222 and the motion compensation unit 224 may be configured to perform a motion vector refinement process described herein.

[0173]

[0163] As another example, in intra prediction, or intra predictive coding, the intra prediction unit 226 may generate a predictive block from samples neighboring a current block. For example, in a directional mode, the intra prediction unit 226 may generally mathematically combine values ​​of neighboring samples and populate these calculated values ​​in a defined direction across the current block to produce a predictive block. As another example, in a DC mode, the intra prediction unit 226 may calculate an average of the neighboring samples for the current block and generate a predictive block to include this resulting average for each sample of the predictive block.

[0174]

[0164] The mode selection unit 202 provides the prediction block to the residual generation unit 204. The residual generation unit 204 receives a raw, uncoded version of the current block from the video data memory 230 and receives the prediction block from the mode selection unit 202. The residual generation unit 204 calculates sample-by-sample differences between the current block and the prediction block. The resulting sample-by-sample differences define a residual block for the current block. In some examples, the residual generation unit 204 may also determine differences between sample values ​​in the residual block to generate the residual block using residual differential pulse code modulation (RDPCM). In some examples, the residual generation unit 204 may be formed using one or more subtractor circuits that perform binary subtraction.

[0175]

[0165] In an example where the mode selection unit 202 partitions a CU into PUs, each PU may be associated with a luma prediction unit and a corresponding chroma prediction unit. The video encoder 200 and the video decoder 300 may support PUs having various sizes. As indicated above, the size of a CU may refer to the size of the luma coding block of the CU, and the size of a PU may refer to the size of the luma prediction unit of the PU. Assuming that the size of a particular CU is 2N×2N, the video encoder 200 may support a PU size of 2N×2N or N×N for intra prediction, and a symmetric PU size of 2N×2N, 2N×N, N×2N, N×N, or the like for inter prediction. The video encoder 200 and the video decoder 300 may also support asymmetric partitioning for PU sizes of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter prediction.

[0176]

[0166] In examples where the mode selection unit 202 does not further partition the CUs into PUs, each CU may be associated with a luma coding block and a corresponding chroma coding block. As described above, the size of a CU may refer to the size of the luma coding block of the CU. The video encoder 200 and the video decoder 300 may support CU sizes of 2Nx2N, 2NxN, or Nx2N.

[0177]

[0167] In other video coding techniques, such as intra block copy mode coding, affine mode coding, and linear model (LM) mode coding, as some examples, the mode selection unit 202 generates a predictive block for the current block being coded via a respective unit associated with the coding technique. In some examples, such as palette mode coding, the mode selection unit 202 may not generate a predictive block, but instead generate syntax elements that indicate how the block should be reconstructed based on a selected palette. In such modes, the mode selection unit 202 may provide these syntax elements to be coded to the entropy coding unit 220.

[0178]

[0168] As described above, the residual generation unit 204 receives video data for a current block and a corresponding predictive block. The residual generation unit 204 then generates a residual block for the current block. To generate the residual block, the residual generation unit 204 calculates sample-by-sample differences between the predictive block and the current block.

[0179]

[0169] Transform processing unit 206 applies one or more transforms to the residual block to generate a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 may apply various transforms to the residual block to form the transform coefficient block. For example, transform processing unit 206 may apply a discrete cosine transform (DCT), a directional transform, a Karhunen-Loeve transform (KLT), or a conceptually similar transform to the residual block. In some examples, transform processing unit 206 may perform multiple transforms on the residual block, e.g., a linear transform and a secondary transform, such as a rotation transform. In some examples, transform processing unit 206 does not apply a transform to the residual block.

[0180]

[0170] The quantization unit 208 may quantize the transform coefficients in the transform coefficient block to generate a quantized transform coefficient block. The quantization unit 208 may quantize the transform coefficients of the transform coefficient block according to a quantization parameter (QP) value associated with the current block. The video encoder 200 (e.g., via the mode selection unit 202) may adjust the degree of quantization applied to the transform coefficient block associated with the current block by adjusting the QP value associated with the CU. Quantization may result in loss of information, and thus the quantized transform coefficients may have less precision than the original transform coefficients generated by the transform processing unit 206.

[0181]

[0171] The inverse quantization unit 210 and the inverse transform processing unit 212 may apply inverse quantization and inverse transform to the quantized transform coefficient block, respectively, to reconstruct a residual block from the transform coefficient block. The reconstruction unit 214 may generate a reconstructed block corresponding to the current block (potentially with some distortion) based on the reconstructed residual block and the predictive block generated by the mode selection unit 202. For example, the reconstruction unit 214 may add samples of the reconstructed residual block to corresponding samples from the predictive block generated by the mode selection unit 202 to produce a reconstructed block.

[0182]

[0172] Filter unit 216 may perform one or more filter operations on the reconstructed block. For example, filter unit 216 may perform a deblocking operation to reduce blockiness artifacts along the edges of a CU. In some examples, the operations of filter unit 216 may be skipped.

[0183]

[0173] The video encoder 200 stores the reconstructed blocks in the DPB 218. For example, in an example where the operation of the filter unit 216 is not performed, the reconstruction unit 214 may store the reconstructed blocks in the DPB 218. In an example where the operation of the filter unit 216 is performed, the filter unit 216 may store the filtered reconstructed blocks in the DPB 218. The motion estimation unit 222 and the motion compensation unit 224 may retrieve reference pictures formed from the reconstructed (and potentially filtered) blocks from the DPB 218 to inter predict blocks of a later-encoded picture. Furthermore, the intra prediction unit 226 may use the reconstructed blocks in the DPB 218 of the current picture to intra predict other blocks in the current picture.

[0184]

[0174] In general, the entropy encoding unit 220 may entropy encode syntax elements received from other functional components of the video encoder 200. For example, the entropy encoding unit 220 may entropy encode quantized transform coefficient blocks from the quantization unit 208. As another example, the entropy encoding unit 220 may entropy encode predictive syntax elements (e.g., motion information for inter prediction, or intra mode information for intra prediction) from the mode selection unit 202. The entropy encoding unit 220 may perform one or more entropy encoding operations on syntax elements, which are another example of video data, to generate entropy encoded data. For example, the entropy encoding unit 220 may perform a context-adaptive variable length coding (CAVLC) operation, a CABAC operation, a variable-to-variable (V2V) length coding operation, a syntax-based context-adaptive binary arithmetic coding (SBAC) operation, a probability interval partitioned entropy (PIPE) coding operation, an exponential-Golomb coding operation, or another type of entropy coding operation on the data. In some examples, the entropy encoding unit 220 may operate in a bypass mode in which syntax elements are not entropy coded.

[0185]

[0175] Video encoder 200 may output a bitstream that includes entropy coding syntax elements needed to reconstruct blocks of a slice or picture. In particular, entropy coding unit 220 may output the bitstream.

[0186]

[0176] The operations described above are described with respect to blocks. Such descriptions should be understood as being operations for luma coding blocks and / or chroma coding blocks. As described above, in some examples, the luma coding block and the chroma coding block are the luma component and the chroma component of a CU. In some examples, the luma coding block and the chroma coding block are the luma component and the chroma component of a PU.

[0187]

[0177] In some examples, operations performed with respect to luma coding blocks do not need to be repeated for chroma coding blocks. As an example, operations for identifying motion vectors (MVs) and reference pictures for luma coding blocks do not need to be repeated to identify MVs and reference pictures for chroma blocks. Rather, the MVs for luma coding blocks may be scaled to determine the MVs for chroma blocks, and the reference pictures may be the same. As another example, the intra prediction process may be the same for luma coding blocks and chroma coding blocks.

[0188]

[0178] Figure 12 is a block diagram illustrating an example video decoder 300 that may perform the techniques of this disclosure. Figure 12 is provided for purposes of explanation and not to limit the techniques broadly illustrated and described in this disclosure. For purposes of explanation, this disclosure describes the video decoder 300 in accordance with the techniques of VVC (ITU-T H.266) and HEVC (ITU-T H.265 under development). However, the techniques of this disclosure may be performed by video coding devices configured for other video coding standards.

[0189]

[0179] In the example of FIG. 12, the video decoder 300 includes a coded picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transform processing unit 308, a reconstruction unit 310, a filter unit 312, and a decoded picture buffer (DPB) 314. Any or all of the CPB memory 320, the entropy decoding unit 302, the prediction processing unit 304, the inverse quantization unit 306, the inverse transform processing unit 308, the reconstruction unit 310, the filter unit 312, and the DPB 314 may be implemented in one or more processors or processing circuits. For example, the units of the video decoder 300 may be implemented as one or more circuits or logic elements, as part of a hardware circuit, or as part of a processor, an ASIC, or an FPGA. Moreover, the video decoder 300 may include additional or alternative processors or processing circuits for performing these and other functions.

[0190]

[0180] Prediction processing unit 304 includes a motion compensation unit 316 and an intra prediction unit 318. Prediction processing unit 304 may include additional units for performing prediction according to other prediction modes. By way of example, prediction processing unit 304 may include a palette unit, an intra block copy unit (which may form part of motion compensation unit 316), an affine unit, a linear model (LM) unit, etc. In other examples, video decoder 300 may include more, fewer, or different functional components.

[0191]

[0181] The CPB memory 320 may store video data, such as an encoded video bitstream, to be decoded by components of the video decoder 300. The video data stored in the CPB memory 320 may be obtained, for example, from the computer-readable medium 110 (FIG. 1). The CPB memory 320 may include a CPB that stores encoded video data (e.g., syntax elements) from the encoded video bitstream. The CPB memory 320 may also store video data other than syntax elements of a coded picture, such as temporary data representing output from various units of the video decoder 300. The DPB 314 generally stores decoded pictures that the video decoder 300 may output and / or use as reference video data when decoding subsequent data or pictures of the encoded video bitstream. The CPB memory 320 and the DPB 314 may be formed by any of a variety of memory devices, such as DRAM, including SDRAM, MRAM, RRAM, or other types of memory devices. The CPB memory 320 and the DPB 314 may be provided by the same memory device or separate memory devices. In various examples, the CPB memory 320 may be on-chip with other components of the video decoder 300 or off-chip relative to those components.

[0192] Additionally or alternatively, in some examples, video decoder 300 may retrieve coded video data from memory 120 (FIG. 1). That is, memory 120 may store data as described above in conjunction with CPB memory 320. Similarly, memory 120 may store instructions to be executed by video decoder 300 when some or all of the functionality of video decoder 300 is implemented in software to be executed by processing circuitry of video decoder 300.

[0193]

[0183] The various units shown in FIG. 12 are illustrated to aid in understanding the operations performed by the video decoder 300. The units may be implemented as fixed-function circuits, programmable circuits, or a combination thereof. As in FIG. 11, a fixed-function circuit refers to a circuit that provides a specific function and is pre-configured with respect to the operations that may be performed. A programmable circuit refers to a circuit that may be programmed to perform various tasks and to provide flexible functionality in the operations that may be performed. For example, a programmable circuit may execute software or firmware that causes the programmable circuit to operate in a manner defined by the software or firmware instructions. Although a fixed-function circuit may execute software instructions (e.g., to receive parameters or output parameters), the type of operations that the fixed-function circuit performs is generally immutable. In some examples, one or more of the units may be separate circuit blocks (fixed function or programmable), and in some examples, one or more of the units may be an integrated circuit.

[0194]

[0184] The video decoder 300 may include a programmable core formed from ALUs, EFUs, digital circuits, analog circuits, and / or programmable circuits. In examples in which the operations of the video decoder 300 are performed by software executing on programmable circuits, on-chip or off-chip memory may store instructions (e.g., object code) of the software that the video decoder 300 receives and executes.

[0195]

[0185] The entropy decoding unit 302 may receive the encoded video data from the CPB and entropy decode the video data to recover the syntax elements. The prediction processing unit 304, the inverse quantization unit 306, the inverse transform processing unit 308, the reconstruction unit 310, and the filter unit 312 may generate decoded video data based on the syntax elements extracted from the bitstream.

[0196]

[0186] Generally, the video decoder 300 reconstructs a picture on a block-by-block basis. The video decoder 300 may perform a reconstruction operation on each block individually (wherein the block currently being reconstructed, i.e., decoded, may be referred to as the "current block").

[0197]

[0187] The entropy decoding unit 302 may entropy decode syntax elements defining the quantized transform coefficients of a quantized transform coefficient block, as well as transform information such as a quantization parameter (QP) and / or a transform mode indication(s). The inverse quantization unit 306 may use the QP associated with the quantized transform coefficient block to determine the degree of quantization and, similarly, the degree of inverse quantization that the inverse quantization unit 306 should apply. The inverse quantization unit 306 may perform, for example, a bitwise left shift operation to inverse quantize the quantized transform coefficients. The inverse quantization unit 306 may thereby form a transform coefficient block including the transform coefficients.

[0198]

[0188] After the inverse quantization unit 306 forms the transform coefficient block, the inverse transform processing unit 308 may apply one or more inverse transforms to the transform coefficient block to generate a residual block associated with the current block. For example, the inverse transform processing unit 308 may apply an inverse DCT, an inverse integer transform, an inverse Karhunen-Loeve transform (KLT), an inverse rotational transform, an inverse transform, or another inverse transform to the transform coefficient block.

[0199]

[0189] Furthermore, prediction processing unit 304 generates a prediction block according to the prediction information syntax element entropy decoded by entropy decoding unit 302. For example, if the prediction information syntax element indicates that the current block is inter predicted, motion compensation unit 316 may generate a prediction block. In this case, the prediction information syntax element may indicate a reference picture in DPB 314 from which a reference block should be retrieved, as well as a motion vector that identifies the location of the reference block in the reference picture relative to the location of the current block in the current picture. Motion compensation unit 316 may generally perform the inter prediction process in a manner substantially similar to that described with respect to motion compensation unit 224 (FIG. 11). In accordance with techniques described herein, motion compensation unit 316 may be configured to decode blocks of video data using an affine prediction mode and may be configured to perform the motion vector refinement process described herein.

[0200]

[0190] As another example, if the prediction information syntax element indicates that the current block is intra predicted, the intra prediction unit 318 may generate a prediction block according to the intra prediction mode indicated by the prediction information syntax element. Again, the intra prediction unit 318 may generally perform an intra prediction process in a manner substantially similar to that described with respect to the intra prediction unit 226 (FIG. 11). The intra prediction unit 318 may retrieve data of neighboring samples for the current block from the DPB 314.

[0201]

[0191] The reconstruction unit 310 may reconstruct the current block using the predictive block and the residual block. For example, the reconstruction unit 310 may add samples of the residual block to corresponding samples of the predictive block to reconstruct the current block.

[0202]

[0192] Filter unit 312 may perform one or more filter operations on the reconstructed block. For example, filter unit 312 may perform a deblocking operation to reduce blockiness artifacts along edges of the reconstructed block. The operations of filter unit 312 are not necessarily performed in all instances.

[0203]

[0193] The video decoder 300 may store the reconstructed blocks in the DPB 314. For example, in examples where the operations of the filter unit 312 are not performed, the reconstruction unit 310 may store the reconstructed blocks in the DPB 314. In examples where the operations of the filter unit 312 are performed, the filter unit 312 may store the filtered reconstructed blocks in the DPB 314. As described above, the DPB 314 may provide reference information to the prediction processing unit 304, such as samples of the current picture for intra prediction and previously decoded pictures for subsequent motion compensation. Moreover, the video decoder 300 may output decoded pictures (e.g., decoded video) from the DPB 314 for subsequent presentation on a display device, such as the display device 118 of FIG. 1.

[0204]

[0194] Figure 13 is a flowchart illustrating an example process for encoding a current block according to the techniques of this disclosure. The current block may comprise a current CU. Although described with respect to video encoder 200 (Figures 1 and 11), it should be understood that other devices may be configured to perform a process similar to that of Figure 13.

[0205]

[0195] In this example, the video encoder 200 first predicts the current block (350). For example, the video encoder 200 may form a predictive block for the current block using template-based affine prediction, as described in this disclosure. The video encoder 200 may then calculate a residual block for the current block (352). To calculate the residual block, the video encoder 200 may calculate a difference between an original uncoded block and a predictive block for the current block. The video encoder 200 may then transform the residual block and quantize transform coefficients of the residual block (354). The video encoder 200 may then scan the quantized transform coefficients of the residual block (356). During or following the scan, the video encoder 200 may entropy code the transform coefficients (358). For example, the video encoder 200 may code the transform coefficients using CAVLC or CABAC. The video encoder 200 may then output entropy coded data for the block (360).

[0206]

[0196] Figure 14 is a flowchart illustrating an example process for decoding a current block of video data in accordance with the techniques of this disclosure. The current block may comprise a current CU. Although described with respect to video decoder 300 (Figures 1 and 12), it should be understood that other devices may be configured to perform a process similar to that of Figure 14.

[0207]

[0197] The video decoder 300 may receive entropy coded data for the current block, such as entropy coded prediction information and entropy coded data for transform coefficients of a residual block that corresponds to the current block (370). The video decoder 300 may entropy decode the entropy coded data to determine prediction information for the current block and to reconstruct the transform coefficients of the residual block (372). The video decoder 300 may predict the current block, e.g., using an intra or inter prediction mode indicated by the prediction information of the current block to compute a predictive block of the current block (374). The video decoder 300 may predict the current block, e.g., using template-based affine prediction as described in this disclosure. The video decoder 300 may then inverse scan the reconstructed transform coefficients to create a block of quantized transform coefficients (376). The video decoder 300 may then inverse quantize the transform coefficients and apply an inverse transform to the transform coefficients to produce the residual block (378). Video decoder 300 may finally decode the current block by combining the predictive block and the residual block (380).

[0208]

[0198] Figure 15 is a flowchart illustrating an example process for decoding a current block of video data in accordance with the techniques of this disclosure. The current block may comprise a current CU. Although described with respect to video decoder 300 (Figures 1 and 12), it should be understood that other devices, including video encoders, may be configured to perform a process similar to that of Figure 15.

[0209]

[0199] The video decoder 300 may determine that a current block in a current picture of the video data is coded in an affine prediction mode (400). The affine prediction mode may be, for example, a four-parameter affine prediction mode, a six-parameter affine prediction mode, or some other such affine prediction mode.

[0210]

[0200] The video decoder 300 may determine one or more control point motion vectors (CPMVs) for the current block (402). The video decoder 300 may identify an initial predictive block for the current block in a reference picture using the one or more CPMVs (404). To identify the initial predictive block for the current block, the video decoder 300 may, for example, locate multiple sub-blocks in a reference frame using the CPMVs.

[0211]

[0201] The video decoder 300 may determine a current template for a current block in the current picture (406). The current template may include multiple sub-blocks located above or to the left of the current block, e.g., as shown in Figure 9A.

[0212]

[0202] The video decoder 300 may determine an initial reference template for a first predictive block in the reference picture (408). The initial reference template may include multiple sub-blocks located above or to the left of the initial predictive block, for example, as shown in Figures 9B and 9C.

[0213]

[0203] The video decoder 300 may perform a motion vector refinement process to determine a modified predictive block based on a comparison between the initial reference template and the current template (410). To perform a motion vector refinement process to determine a modified predictive block, the video decoder 300 may further search within a search area around the initial reference template for a subsequent reference template that matches the current template more closely than the initial reference template. The comparison between the initial reference template and the current template may be, for example, a template matching cost, and the video decoder 300 may determine the template matching cost based on a weighted sample-by-sample comparison of samples in the initial reference template and samples in the current template.

[0214]

[0204] The video decoder 300 may determine a predictive block based on the modified predictive block, add the predictive block to the residual block to determine a reconstructed block, apply one or more filtering operations to the reconstructed block, and output a picture of decoded video data including the filtered reconstructed block.

[0215]

[0205] The following numbered clauses describe one or more aspects of the devices and techniques described in this disclosure.

[0216]

[0206] Clause 1A. A method for decoding video data, comprising determining one or more control point motion vectors (CPMVs) for a current block, wherein the one or more CPMVs correspond to an initial predictive block for the current block, and performing a motion vector refinement process to determine a modified predictive block.

[0217]

[0207] Clause 2A. The method of clause 1A, wherein the motion vector refinement process comprises performing template matching.

[0218]

[0208] Clause 3A. The method of clause 2A, wherein the one or more CPMVs comprise a set of initial CPMVs, and the template matching process comprises determining a set of refined CPMVs.

[0219]

[0209] Clause 4A. The method of clause 3A, wherein determining the set of refined CPMVs comprises adding one or more delta motion vector values ​​to one or more CPMVs to determine the set of refined CPMVs.

[0220]

[0210] Clause 5A. The method of clause 3A or 4A, further comprising determining a search area based on one or more CPMVs, wherein determining a set of improved CPMVs comprises limiting the improved CPMVs to be within the search area.

[0221]

[0211] Clause 6A. The method of any of clauses 3A to 5A, further comprising determining a search pattern; and determining a set of refined CPMVs based on the search pattern.

[0222]

[0212] Clause 7A. The method of any of clauses 1A-6A, wherein performing a motion vector refinement process to determine a revised prediction block comprises performing one or more template matching cost calculations.

[0223]

[0213] Clause 8A. The method of any one of clauses 1A to 7A, wherein the decoding method is performed as part of the encoding process.

[0224]

[0214] Clause 9A. A device for decoding video data, comprising one or more means for carrying out the method according to any of clauses 1A to 8A.

[0225]

[0215] Clause 10A. A device as described in clause 9A, wherein the one or more means comprise one or more processors implemented in circuitry.

[0226]

[0216] Clause 11A. The device of any of clauses 9A and 10A, further comprising a memory for storing video data.

[0227]

[0217] Clause 12A. The device of any of clauses 9A to 11A, further comprising a display configured to display the decoded video data.

[0228]

[0218] Clause 13A. A device according to any of clauses 9A to 12A, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

[0229]

[0219] Clause 14A. A device according to any one of clauses 9A to 13A, wherein the device comprises a video decoder.

[0230]

[0220] Clause 15A. The device of any one of clauses 9A to 14A, wherein the device comprises a video encoder.

[0231]

[0221] Clause 16A. A computer-readable storage medium having stored thereon instructions which, when executed, cause one or more processors to perform a method according to any one of clauses 1A to 8A.

[0232]

[0222] Clause 1B. A method for decoding video data, comprising: determining that a current block in a current picture of the video data is coded in an affine prediction mode; determining one or more control point motion vectors (CPMVs) for the current block; identifying an initial predictive block for the current block in a reference picture using the one or more CPMVs; determining a current template for the current block in the current picture; determining an initial reference template for the first predictive block in the reference picture; and performing a motion vector refinement process to determine a modified predictive block based on a comparison between the initial reference template and the current template.

[0233]

[0223] Clause 2B. The method described in clause 1B, wherein performing a motion vector refinement process to determine a modified prediction block further comprises searching within a search area around the initial reference template for a subsequent reference template that more closely matches the current template than the initial reference template.

[0234]

[0224] Clause 3B. The method of clause 1B, wherein the comparison of the initial reference template and the current template comprises a template matching cost.

[0235]

[0225] Clause 4B. The method of clause 3B, further comprising determining a template matching cost based on a weighted sample-by-sample comparison of samples in the initial reference template and samples in the current template.

[0236]

[0226] Clause 5B. The method of clause 1B, wherein the initial reference template comprises a plurality of sub-blocks located above or to the left of the initial predicted block.

[0237]

[0227] Clause 6B. The method of clause 1B, wherein the affine prediction mode comprises a four-parameter affine prediction mode.

[0238]

[0228] Clause 7B. The method of clause 1B, wherein the affine prediction mode comprises a six-parameter affine prediction mode.

[0239]

[0229] Clause 8B. The method described in clause 1B, further comprising determining a prediction block based on the modified prediction block, adding the prediction block to the residual block to determine a reconstructed block, applying one or more filtering operations to the reconstructed block, and outputting a picture of decoded video data including the filtered reconstructed block.

[0240]

[0230] Clause 9B. The method of clause 1B, wherein the method of decoding is performed as part of a video encoding process.

[0241]

[0231] Clause 10B. A device for decoding video data, comprising: a memory; and one or more processors implemented in circuitry coupled to the memory, wherein the one or more processors are configured to: determine that a current block in a current picture of the video data is coded in an affine prediction mode; determine one or more control point motion vectors (CPMVs) for the current block; identify an initial predictive block for the current block in a reference picture using the one or more CPMVs; determine a current template for the current block in the current picture; determine an initial reference template for the first predictive block in the reference picture; and perform a motion vector refinement process to determine a modified predictive block based on a comparison between the initial reference template and the current template.

[0242]

[0232] Clause 11B. The device described in Clause 10B, wherein to perform a motion vector refinement process to determine a modified prediction block, the one or more processors are further configured to search within a search area around the initial reference template for a subsequent reference template that matches the current template more closely than the initial reference template.

[0243]

[0233] Clause 12B. The device of clause 10B, wherein the comparison between the initial reference template and the current template comprises a template matching cost.

[0244]

[0234] Clause 13B. The device of clause 12B, wherein the one or more processors are further configured to determine a template matching cost based on a weighted sample-by-sample comparison of samples in the initial reference template and samples in the current template.

[0245]

[0235] Clause 14B. The device of clause 10B, wherein the initial reference template comprises a plurality of sub-blocks located above or to the left of the initial prediction block.

[0246]

[0236] Clause 15B. The device of clause 10B, wherein the affine prediction mode comprises a four-parameter affine prediction mode.

[0247]

[0237] Clause 16B. The device of clause 10B, wherein the affine prediction mode comprises a six-parameter affine prediction mode.

[0248]

[0238] Clause 17B. The device of clause 10B, wherein the one or more processors are further configured to determine a prediction block based on the modified prediction block, add the prediction block to the residual block to determine a reconstructed block, apply one or more filtering operations to the reconstructed block, and output a picture of decoded video data including the filtered reconstructed block.

[0249]

[0239] Clause 18B. The device of clause 10B, wherein the device comprises a wireless communication device further comprising a receiver configured to receive encoded video data.

[0250]

[0240] Clause 19B. The device of clause 18B, wherein the wireless communications device comprises a telephone handset, wherein the receiver is configured to demodulate a signal comprising encoded video data in accordance with a wireless communications standard.

[0251]

[0241] Clause 20B. The device of clause 10B, further comprising a display configured to display the video data.

[0252]

[0242] Clause 21B. The device of clause 10B, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

[0253]

[0243] Clause 22B. The device of clause 10B, wherein the device comprises a video encoding device.

[0254]

[0244] Clause 23B. A computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to determine that a current block in a current picture of video data is coded in an affine prediction mode, determine one or more control point motion vectors (CPMVs) for the current block, identify an initial predictive block for the current block in a reference picture using the one or more CPMVs, determine a current template for the current block in the current picture, determine an initial reference template for the first predictive block in the reference picture, and perform a motion vector refinement process to determine a modified predictive block based on a comparison between the initial reference template and the current template.

[0255]

[0245] Clause 24B. A computer-readable storage medium as described in clause 23B, wherein to perform a motion vector refinement process to determine a modified prediction block, the instructions further cause the one or more processors to search within a search area around the initial reference template for a subsequent reference template that matches the current template more closely than the initial reference template.

[0256]

[0246] Clause 25B. The computer-readable storage medium of clause 23B, wherein the comparison between the initial reference template and the current template comprises a template matching cost.

[0257]

[0247] Clause 26B. A computer-readable storage medium as described in clause 25B, wherein the instructions cause one or more processors to determine a template matching cost based on a weighted sample-by-sample comparison of samples in an initial reference template to samples in a current template.

[0258]

[0248] Clause 27B. The computer-readable storage medium of clause 23B, wherein the initial reference template comprises a plurality of sub-blocks located above or to the left of the initial predicted block.

[0259]

[0249] Clause 28B. A computer-readable storage medium as described in clause 23B, wherein the instructions cause one or more processors to determine a prediction block based on the modified prediction block, add the prediction block to a residual block to determine a reconstructed block, apply one or more filtering operations to the reconstructed block, and output a picture of decoded video data including the filtered reconstructed block.

[0260]

[0250] Clause 29B. An apparatus for decoding video data, comprising: means for determining that a current block in a current picture of the video data is coded in an affine prediction mode; means for determining one or more control point motion vectors (CPMVs) for the current block; means for identifying an initial predictive block for the current block in a reference picture using the one or more CPMVs; means for determining a current template for the current block in the current picture; means for determining an initial reference template for the first predictive block in the reference picture; and means for performing a motion vector refinement process to determine a modified predictive block based on a comparison between the initial reference template and the current template.

[0261]

[0251] Clause 30B. The apparatus described in clause 29B, wherein the comparison between the initial reference template and the current template comprises a template matching cost, and the apparatus further comprises means for determining the template matching cost based on a weighted sample-by-sample comparison of samples in the initial reference template and samples in the current template.

[0262]

[0252] Clause 1C. A method for decoding video data, comprising: determining that a current block in a current picture of the video data is coded in an affine prediction mode; determining one or more control point motion vectors (CPMVs) for the current block; identifying an initial predictive block for the current block in a reference picture using the one or more CPMVs; determining a current template for the current block in the current picture; determining an initial reference template for the first predictive block in the reference picture; and performing a motion vector refinement process to determine a modified predictive block based on a comparison between the initial reference template and the current template.

[0263]

[0253] Clause 2C. The method described in clause 1C, wherein performing a motion vector refinement process to determine a modified prediction block further comprises searching within a search area around the initial reference template for a subsequent reference template that more closely matches the current template than the initial reference template.

[0264]

[0254] Clause 3C. The method of clause 1C or 2C, wherein the comparison of the initial reference template and the current template comprises a template matching cost.

[0265]

[0255] Clause 4C. The method of clause 3C, further comprising determining a template matching cost based on a weighted sample-by-sample comparison of samples in the initial reference template and samples in the current template.

[0266]

[0256] Clause 5C. The method of any of clauses 1C to 4C, wherein the initial reference template comprises a plurality of sub-blocks located above or to the left of the initial predicted block.

[0267]

[0257] Clause 6C. The method of any of clauses 1C to 5C, wherein the affine prediction mode comprises a four parameter affine prediction mode.

[0268]

[0258] Clause 7C. The method of any of clauses 1C to 5C, wherein the affine prediction mode comprises a six-parameter affine prediction mode.

[0269]

[0259] Clause 8C. A method according to any of clauses 1C to 7C, further comprising determining a prediction block based on the modified prediction block, adding the prediction block to the residual block to determine a reconstructed block, applying one or more filtering operations to the reconstructed block, and outputting a picture of decoded video data including the filtered reconstructed block.

[0270]

[0260] Clause 9C. The method of any of clauses 1C to 8C, wherein the decoding method is implemented as part of a video encoding process.

[0271]

[0261] Clause 10C. A device for decoding video data, comprising: a memory; and one or more processors implemented in circuitry coupled to the memory, the one or more processors configured to: determine that a current block in a current picture of the video data is coded in an affine prediction mode; determine one or more control point motion vectors (CPMVs) for the current block; identify an initial predictive block for the current block in a reference picture using the one or more CPMVs; determine a current template for the current block in the current picture; determine an initial reference template for the first predictive block in the reference picture; and perform a motion vector refinement process to determine a modified predictive block based on a comparison between the initial reference template and the current template.

[0272]

[0262] Clause 11C. The device described in Clause 10C, wherein to perform a motion vector refinement process to determine a modified prediction block, the one or more processors are further configured to search within a search area around the initial reference template for a subsequent reference template that more closely matches the current template than the initial reference template.

[0273]

[0263] Clause 12C. The device of clause 10C or 11C, wherein the comparison between the initial reference template and the current template comprises a template matching cost.

[0274]

[0264] Clause 13C. The device of clause 12C, wherein the one or more processors are further configured to determine a template matching cost based on a weighted sample-by-sample comparison of samples in the initial reference template and samples in the current template.

[0275]

[0265] Clause 14C. The device of any of clauses 10C to 13C, wherein the initial reference template comprises a plurality of sub-blocks located above or to the left of the initial predicted block.

[0276]

[0266] Clause 15C. The device of any of clauses 10C to 14C, wherein the affine prediction mode comprises a four-parameter affine prediction mode.

[0277]

[0267] Clause 16C. The device of any of clauses 10C to 14C, wherein the affine prediction mode comprises a six-parameter affine prediction mode.

[0278]

[0268] Clause 17C. A device as described in any of Clauses 10C to 16C, wherein the one or more processors are further configured to determine a prediction block based on the modified prediction block, add the prediction block to the residual block to determine a reconstructed block, apply one or more filtering operations to the reconstructed block, and output a picture of decoded video data including the filtered reconstructed block.

[0279]

[0269] Clause 18C. A device according to any one of clauses 10C to 17C, comprising a wireless communications device further comprising a receiver configured to receive encoded video data.

[0280]

[0270] Clause 19C. The device of clause 18C, wherein the wireless communications device comprises a telephone handset, wherein the receiver is configured to demodulate a signal comprising encoded video data in accordance with a wireless communications standard.

[0281]

[0271] Clause 20C. The device of any of clauses 10C to 19C, further comprising a display configured to display the decoded video data.

[0282]

[0272] Clause 21C. A device according to any of clauses 10C to 20C, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

[0283]

[0273] Clause 22C. The device of any one of clauses 10C to 21C, wherein the device comprises a video encoding device.

[0284]

[0274] Clause 23C. A computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to determine that a current block in a current picture of video data is coded in an affine prediction mode, determine one or more control point motion vectors (CPMVs) for the current block, identify an initial predictive block for the current block in a reference picture using the one or more CPMVs, determine a current template for the current block in the current picture, determine an initial reference template for the first predictive block in the reference picture, and perform a motion vector refinement process to determine a modified predictive block based on a comparison between the initial reference template and the current template.

[0285]

[0275] Clause 24C. A computer-readable storage medium as described in clause 23C, wherein the instructions further cause the one or more processors to search within a search area around the initial reference template for a subsequent reference template that matches the current template more closely than the initial reference template to perform a motion vector refinement process to determine a modified prediction block.

[0286]

[0276] Clause 25C. The computer-readable storage medium of clause 23C, wherein the comparison between the initial reference template and the current template comprises a template matching cost.

[0287]

[0277] Clause 26C. A computer-readable storage medium as described in clause 25C, wherein the instructions cause one or more processors to determine a template matching cost based on a weighted sample-by-sample comparison of samples in an initial reference template to samples in a current template.

[0288]

[0278] Clause 27C. The computer-readable storage medium of clause 23C, wherein the initial reference template comprises a plurality of sub-blocks located above or to the left of the initial predicted block.

[0289]

[0279] Clause 28C. A computer-readable storage medium as described in clause 23C, wherein the instructions cause one or more processors to determine a prediction block based on the modified prediction block, add the prediction block to a residual block to determine a reconstructed block, apply one or more filtering operations to the reconstructed block, and output a picture of decoded video data including the filtered reconstructed block.

[0290]

[0280] Clause 29C. An apparatus for decoding video data, comprising: means for determining that a current block in a current picture of the video data is coded in an affine prediction mode; means for determining one or more control point motion vectors (CPMVs) for the current block; means for identifying an initial predictive block for the current block in a reference picture using the one or more CPMVs; means for determining a current template for the current block in the current picture; means for determining an initial reference template for the first predictive block in the reference picture; and means for performing a motion vector refinement process to determine a modified predictive block based on a comparison between the initial reference template and the current template.

[0291]

[0281] Clause 30C. The apparatus described in clause 29C, wherein the comparison between the initial reference template and the current template comprises a template matching cost, and the apparatus further comprises means for determining the template matching cost based on a weighted sample-by-sample comparison of samples in the initial reference template and samples in the current template.

[0292]

[0282] In accordance with the above examples, it should be recognized that some acts or events of any of the techniques described herein may be performed in a different sequence, added, merged, or omitted entirely (e.g., not all described acts or events may be necessary to the practice of the techniques). Moreover, in some examples, acts or events may be performed simultaneously rather than sequentially, for example, through multi-threaded processing, interrupt processing, or multiple processors.

[0293]

[0283] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium, which corresponds to a tangible medium, such as a data storage medium, or may include a communication medium, which includes any medium that facilitates transfer of a computer program from one place to another, for example according to a communication protocol. In this manner, a computer-readable medium may generally correspond to (1) a tangible computer-readable storage medium that is non-transitory, or (2) a communication medium, such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code and / or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0294]

[0284] By way of example and not limitation, such computer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer. Also, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but instead cover non-transitory tangible storage media. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks typically reproduce data magnetically and discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer readable media.

[0295]

[0285] The instructions may be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated circuits or discrete logic circuits. Thus, the terms "processor" and "processing circuitry" as used herein may refer to any of the above structures, or any other structures suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided in dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a composite codec. Also, the techniques may be fully implemented in one or more circuits or logic elements.

[0296]

[0286] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs) or sets of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to highlight functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit or provided by a collection of interoperable hardware units, including one or more processors described above, along with suitable software and / or firmware.

[0297]

[0287] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. 1. A method for decoding video data, comprising the steps of: determining that a current block in a current picture of the video data is coded in an affine prediction mode; determining one or more control point motion vectors (CPMVs) for the current block; identifying an initial prediction block for the current block in a reference picture using the one or more CPMVs; determining a current template for the current block in the current picture; determining an initial reference template for the initial predictive block in the reference picture, and samples of reference template blocks of the initial reference template are generated on a sub-block basis based on a motion field derived by using the one or more CPMVs. performing a motion vector refinement process to determine a modified prediction block based on a comparison of the initial reference template and the current template; A method for providing the above.

2. performing the motion vector refinement process to determine the modified prediction block, searching within a search area about the initial reference template for a subsequent reference template that matches the current template more closely than the initial reference template; The method of claim 1 further comprising:

3. The comparison of the initial reference template to the current template comprises a template matching cost, and optionally determining the template matching cost based on a weighted sample-by-sample comparison of samples in the initial reference template to samples in the current template; The method of claim 1 further comprising:

4. The method of claim 1 , wherein the initial reference template comprises a plurality of sub-blocks located above or to the left of the initial predicted block.

5. The method of claim 1 , wherein the affine prediction mode comprises a four-parameter affine prediction mode.

6. The method of claim 1 , wherein the affine prediction mode comprises a six-parameter affine prediction mode.

7. determining a prediction block based on the modified prediction block; and adding the prediction block to a residual block to determine a reconstructed block; applying one or more filtering operations to the reconstructed block; and outputting a picture of the decoded video data including the filtered reconstructed block; and The method of claim 1 further comprising:

8. The method of claim 1 , wherein the method of decoding is performed as part of a video encoding process.

9. 1. A device for decoding video data, comprising: Memory, one or more processors implemented in circuitry coupled to the memory; wherein the one or more processors: determining that a current block in a current picture of the video data is coded in an affine prediction mode; determining one or more control point motion vectors (CPMVs) for the current block; identifying an initial prediction block for the current block in a reference picture using the one or more CPMVs; determining a current template for the current block in the current picture; determining an initial reference template for the initial predictive block in the reference picture, and samples of reference template blocks of the initial reference template are generated on a sub-block basis based on a motion field derived by using the one or more CPMVs. performing a motion vector refinement process to determine a modified prediction block based on a comparison of the initial reference template and the current template; A device configured to:

10. The device of claim 9, further configured to carry out a method according to any one of claims 2 to 8.

11. The device comprises a wireless communication device further comprising a receiver configured to receive encoded video data, and optionally:

10. The device of claim 9, wherein the wireless communication device comprises a telephone handset, and the receiver is configured to demodulate a signal comprising the encoded video data in accordance with a wireless communication standard.

12. a display configured to display the decoded video data; The device of claim 9 further comprising:

13. The device of claim 9 , wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

14. The device of claim 9 , wherein the device comprises a video encoding device.

15. A computer readable storage medium having stored thereon instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 8.

Images