Regression-based decoder-side affine motion derivation for video coding

Abstract

A video codec may be configured to partition a codec block into sub-blocks, and generate an initial sub-block motion vector for the plurality of sub-blocks. The video codec may also refine the initial sub-block motion vector of the plurality of sub-blocks using decoder-side motion vector refinement to produce a refined sub-block motion vector of the plurality of sub-blocks, and perform linear regression on the refined sub-block motion vector and coordinates of the plurality of sub-blocks to derive an affine motion model. The video codec may then codec a codec using the derived affine motion model.

Description

[0001] This application claims priority to U.S. Patent Application Serial No. 18 / 471,131, filed on September 20, 2023, and U.S. Provisional Patent Application Serial No. 63 / 379,033, filed on October 11, 2022, the entire contents of which are incorporated herein by reference. U.S. Patent Application Serial No. 18 / 471,131, filed on September 20, 2023, claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 379,033, filed on October 11, 2022. Technical Field

[0002] The present disclosure relates to video encoding and video decoding. Background Art

[0003] Digital video capabilities may be incorporated into a wide range of devices, including digital televisions, digital live broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radio telephones, so-called "smart phones", video teleconferencing devices, video streaming devices, etc. Digital video devices implement video codec technologies, such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 Part 10 Advanced Video Codec (AVC), ITU-T H.265 / High Efficiency Video Codec (HEVC), ITU-T H.266 / Versatile Video Codec (VVC), and extensions of such standards, as well as proprietary video codecs / formats such as AOMedia Video 1 (AV1) developed by the Alliance for Open Media. Video devices may more efficiently send, receive, encode, decode, and / or store digital video information by implementing such video codec technologies.

[0004] Video coding techniques include spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences. For block-based video coding, a video slice (e.g., a video picture or a portion of a video picture) may be partitioned into video blocks, which may also be referred to as codec tree units (CTUs), codec units (CUs), and / or codec nodes. Video blocks in a slice of an intra-frame codec (I) of a picture are encoded using spatial prediction relative to reference samples in neighboring blocks in the same picture. Video blocks in a slice of an inter-frame codec (P or B) of a picture may use spatial prediction relative to reference samples in neighboring blocks in the same picture or temporal prediction relative to reference samples in other reference pictures. Pictures may be referred to as frames, and reference pictures may be referred to as reference frames. Summary of the invention

[0005] In general, the present disclosure describes techniques for inter prediction in a video codec. More specifically, the present disclosure describes methods related to affine motion derivation at a decoder side. The techniques of the present disclosure may include partitioning a block into a plurality of sub-blocks and determining an initial sub-block motion vector for each of the sub-blocks. The initial sub-block motion vector may be based on affine merge candidates from neighboring blocks and may represent an initial affine motion model.

[0006] The video codec may then perform a decoder-side motion vector refinement (DMVR) process for each of the sub-blocks. The DMVR process may include performing bilateral matching for each of the sub-blocks. The DMVR process produces a refined sub-block motion vector for each of the sub-blocks. For example, the bilateral matching process may return a motion vector offset value to add to the initial motion vector for each of the sub-blocks. Note that the initial motion vector and the determined motion vector offset may be different for each of the sub-blocks.

[0007] The video codec can then perform a linear regression process on the refined sub-block motion vectors to derive an updated affine motion model. The video codec can determine the control point motion vectors from the updated affine motion model, and then use the control point motion vectors to encode and decode the block. By applying the linear regression process to the refined sub-block motion vectors, a more accurate affine motion model can be obtained. In this way, the encoding and decoding efficiency can be improved and distortion can be reduced.

[0008] In one example, the present disclosure describes a method including: partitioning a codec block into sub-blocks, generating initial sub-block motion vectors for a plurality of sub-blocks, refining the initial sub-block motion vectors for the plurality of sub-blocks using decoder-side motion vector refinement to produce refined sub-block motion vectors for the plurality of sub-blocks, and performing linear regression on the refined sub-block motion vectors and coordinates of the plurality of sub-blocks to derive an affine motion model.

[0009] In another example, the present disclosure describes an apparatus configured to decode video data, the apparatus comprising: a memory, and one or more processors in communication with the memory, the one or more processors configured to partition a codec block into sub-blocks, generate initial sub-block motion vectors for a plurality of sub-blocks, refine the initial sub-block motion vectors for the plurality of sub-blocks using decoder-side motion vector refinement to produce refined sub-block motion vectors for the plurality of sub-blocks, and perform linear regression on the refined sub-block motion vectors and coordinates of the plurality of sub-blocks to derive an affine motion model.

[0010] In another example, the present disclosure describes an apparatus configured to decode video data, the apparatus including means for partitioning a codec block into sub-blocks, means for generating initial sub-block motion vectors for a plurality of sub-blocks, means for refining the initial sub-block motion vectors for the plurality of sub-blocks using decoder-side motion vector refinement to produce refined sub-block motion vectors for the plurality of sub-blocks, and means for performing linear regression on the refined sub-block motion vectors and coordinates of the plurality of sub-blocks to derive an affine motion model.

[0011] In another example, the disclosure describes a non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to partition a codec block into sub-blocks, generate initial sub-block motion vectors for a plurality of sub-blocks, refine the initial sub-block motion vectors for the plurality of sub-blocks using decoder-side motion vector refinement to produce refined sub-block motion vectors for the plurality of sub-blocks, and perform linear regression on the refined sub-block motion vectors and coordinates of the plurality of sub-blocks to derive an affine motion model.

[0012] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a block diagram illustrating an example video encoding and decoding system that may perform the techniques of this disclosure.

[0014] Figure 2 is a conceptual diagram showing an example of two-sided matching.

[0015] Figure 3 A conceptual diagram illustrating an example of regression-based decoder-side affine pattern derivation.

[0016] Figure 4 A conceptual diagram illustrating another example of regression-based decoder-side affine pattern derivation.

[0017] Figure 5 is a block diagram illustrating an example video encoder that may perform the techniques of this disclosure.

[0018] Figure 6 is a block diagram illustrating an example video decoder that may perform the techniques of this disclosure.

[0019] Figure 7 is a flowchart illustrating an example method for encoding a current block according to the techniques of this disclosure.

[0020] Figure 8 is a flowchart illustrating an example method for decoding a current block according to the techniques of this disclosure.

[0021] Fig. 9is a flowchart illustrating another example method for encoding a current block according to the techniques of this disclosure.

[0022] Fig.10 is a flowchart illustrating another example method for decoding a current block according to the techniques of this disclosure. DETAILED DESCRIPTION

[0023] In general, the present disclosure describes techniques for inter prediction in a video codec. More specifically, the present disclosure describes methods related to affine motion derivation at a decoder side. The techniques of the present disclosure may include partitioning a block into a plurality of sub-blocks and determining an initial sub-block motion vector for each of the sub-blocks. The initial sub-block motion vector may be based on affine merge candidates from neighboring blocks and may represent an initial affine motion model.

[0024] The video codec may then perform a decoder-side motion vector refinement (DMVR) process for each of the sub-blocks. The DMVR process may include performing bilateral matching for each of the sub-blocks. The DMVR process produces a refined sub-block motion vector for each of the sub-blocks. For example, the bilateral matching process may return a motion vector offset value to add to the initial motion vector for each of the sub-blocks. Note that the initial motion vector and the determined motion vector offset may be different for each of the sub-blocks.

[0025] The video codec can then perform a linear regression process on the refined sub-block motion vectors to derive an updated affine motion model. The video codec can determine the control point motion vectors from the updated affine motion model, and then use the control point motion vectors to encode and decode the block. By applying the linear regression process to the refined sub-block motion vectors, a more accurate affine motion model can be obtained. In this way, the encoding and decoding efficiency can be improved and distortion can be reduced.

[0026] Figure 1 A block diagram of an example video encoding and decoding system 100 is shown that can perform the disclosed techniques for regression-based decoder-side affine motion derivation. The disclosed techniques generally relate to encoding and decoding (encoding and / or decoding) video data. In general, video data includes any data used to process video. Thus, video data can include original, unencoded video, encoded video, decoded (e.g., reconstructed) video, and video metadata, such as signaling data.

[0027] like Figure 1, in this example, system 100 includes a source device 102 that provides encoded video data to be decoded and displayed by a target device 116. Specifically, source device 102 provides video data to target device 116 via computer-readable medium 110. Source device 102 and target device 116 may be or may include any of a wide range of devices, such as desktop computers, notebook (i.e., laptop) computers, mobile devices, tablet computers, set-top boxes, telephone handsets such as smart phones, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, broadcast receiver devices, etc. In some cases, source device 102 and target device 116 may be equipped for wireless communication, and thus may be referred to as wireless communication devices.

[0028] exist Figure 1 In the example of, source device 102 comprises video source 104, memory 106, video encoder 200 and output interface 108. Target device 116 comprises input interface 122, video decoder 300, memory 120 and display device 118. According to the present disclosure, the video encoder 200 of source device 102 and the video decoder 300 of target device 116 can be configured to apply the technology for deriving based on the decoder side affine motion of regression. Therefore, source device 102 represents an example of video encoding device, and target device 116 represents an example of video decoding device. In other examples, source device and target device may include other components or arrangements. For example, source device 102 may receive video data from an external video source (such as an external camera). Similarly, target device 116 may interface with an external display device, rather than including an integrated display device.

[0029] like Figure 1 The system 100 shown is only an example. In general, any digital video encoding and / or decoding device can perform the technology for affine motion derivation based on regression of the decoder side. The source device 102 and the target device 116 are only examples of such a codec device that the source device 102 generates the video data of the codec for sending to the target device 116. The present disclosure refers to the "codec" device as a device that performs the codec (coding and / or decoding) of the data. Therefore, the video encoder 200 and the video decoder 300 represent examples of codec devices, and particularly represent video encoders and video decoders, respectively. In some examples, the source device 102 and the target device 116 can operate in a substantially symmetrical manner, so that each of the source device 102 and the target device 116 includes a video encoding and decoding component. Therefore, the system 100 can support one-way or two-way video transmission between the source device 102 and the target device 116, for example, for video streaming, video playback, video broadcasting or video telephony.

[0030] Typically, video source 104 represents a source of video data (i.e., raw, unencoded video data), and provides a series of consecutive pictures (also referred to as "frames") of video data to video encoder 200, which encodes the data of the pictures. Video source 104 of source device 102 may include a video capture device, such as a camera, a video archive containing previously captured raw video, and / or a video feed interface for receiving video from a video content provider. As another alternative, video source 104 may generate computer graphics-based data as source video, or a combination of live video, archived video, and computer-generated video. In each case, video encoder 200 encodes captured, pre-captured, or computer-generated video data. Video encoder 200 may rearrange pictures from the order in which they are received (sometimes referred to as "display order") into a codec order for encoding and decoding. Video encoder 200 may generate a bitstream comprising encoded video data. Source device 102 may then output the encoded video data onto computer-readable medium 110 via output interface 108 for receipt and / or retrieval by, for example, input interface 122 of target device 116 .

[0031] The memory 106 of the source device 102 and the memory 120 of the target device 116 represent general purpose memory. In some examples, the memories 106, 120 can store raw video data, for example, raw video from the video source 104 and raw decoded video data from the video decoder 300. Additionally or alternatively, the memories 106, 120 can store software instructions that can be executed by, for example, the video encoder 200 and the video decoder 300, respectively. Although the memory 106 and the memory 120 are shown separately from the video encoder 200 and the video decoder 300 in this example, it should be understood that the video encoder 200 and the video decoder 300 can also include internal memory for functionally similar or equivalent purposes. In addition, the memories 106, 120 can store encoded video data, for example, video data output from the video encoder 200 and input to the video decoder 300. In some examples, portions of the memories 106, 120 can be allocated as one or more video buffers, for example to store raw, decoded and / or encoded video data.

[0032] The computer-readable medium 110 may represent any type of medium or device capable of transmitting the encoded video data from the source device 102 to the target device 116. In one example, the computer-readable medium 110 represents a communication medium that enables the source device 102 to transmit the encoded video data directly to the target device 116 in real time, for example, via a radio frequency network or a computer-based network. According to a communication standard (such as a wireless communication protocol), the output interface 108 can modulate a transmission signal including the encoded video data, and the input interface 122 can demodulate the received transmission signal. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network (such as a local area network, a wide area network, or a global network such as the Internet). The communication medium may include a router, a switch, a base station, or any other equipment that can be used to facilitate communication from the source device 102 to the target device 116.

[0033] In some examples, source device 102 may output the encoded data from output interface 108 to storage device 112. Similarly, target device 116 may access the encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0034] In some examples, source device 102 may output the encoded video data to file server 114 or another intermediate storage device that may store the encoded video data generated by source device 102. Target device 116 may access the stored video data from file server 114 via streaming or downloading.

[0035] The file server 114 may be any type of server device capable of storing encoded video data and sending the encoded video data to the target device 116. The file server 114 may represent a web server (e.g., for a website), a server configured to provide a file transfer protocol service (e.g., File Transfer Protocol (FTP) or File Delivery over Unidirectional Transport (FLUTE) protocol), a content delivery network (CDN) device, a hypertext transfer protocol (HTTP) server, a Multimedia Broadcast Multicast Service (MBMS) or an enhanced MBMS (eMBMS) server, and / or a network attached storage (NAS) device. The file server 114 may additionally or alternatively implement one or more HTTP streaming protocols, such as HTTP Dynamic Adaptive Streaming (DASH), HTTP Live Streaming (HLS), Real-Time Streaming Protocol (RTSP), HTTP Dynamic Streaming, etc.

[0036] Target device 116 may access the encoded video data from file server 114 through any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of both suitable for accessing the encoded video data stored on file server 114. Input interface 122 may be configured to operate according to any one or more of the various protocols discussed above for retrieving or receiving media data from file server 114, or other such protocols for retrieving media data.

[0037] Output interface 108 and input interface 122 may represent wireless transmitters / receivers, modems, wired networking components (e.g., Ethernet cards), wireless communication components that operate according to any of the various IEEE 802.11 standards, or other physical components. In examples where output interface 108 and input interface 122 include wireless components, output interface 108 and input interface 122 may be configured to transmit data (e.g., encoded video data) according to a cellular communication standard (e.g., 4G, 4G-LTE (Long Term Evolution), LTE Advanced, 5G, etc.). In some examples where output interface 108 includes a wireless transmitter, output interface 108 and input interface 122 may be configured to transmit data (e.g., encoded video data) according to other wireless standards (e.g., IEEE 802.11 specifications, IEEE 802.15 specifications (e.g., ZigBee TM ), Bluetooth TMStandards, etc.) to transmit data (such as encoded video data). In some examples, source device 102 and / or target device 116 may include corresponding system-on-chip (SoC) devices. For example, source device 102 may include a SoC device for performing functions attributed to video encoder 200 and / or output interface 108, and target device 116 may include a SoC device for performing functions attributed to video decoder 300 and / or input interface 122.

[0038] The techniques disclosed herein can be applied to video encoding and decoding to support any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, Internet streaming video transmission (such as HTTP Dynamic Adaptive Streaming (DASH)), digital video encoded onto a data storage medium, decoding of digital video stored on a data storage medium, or other applications.

[0039] The input interface 122 of the target device 116 receives the encoded video bitstream from the computer-readable medium 110 (e.g., a communication medium, a storage device 112, a file server 114, etc.). The encoded video bitstream may include signaling information defined by the video encoder 200, which is also used by the video decoder 300, such as syntax elements with values ​​describing characteristics and / or processing of video blocks or other codec units (e.g., slices, pictures, groups of pictures, sequences, etc.). The display device 118 displays decoded pictures of the decoded video data to a user. The display device 118 may represent any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

[0040] although Figure 1 Not shown, but in some examples, the video encoder 200 and the video decoder 300 may each be integrated with an audio encoder and / or an audio decoder, and may include appropriate MUX-DEMUX units or other hardware and / or software to process a multiplexed stream including both audio and video in a common data stream.

[0041] The video encoder 200 and the video decoder 300 can each be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the technology is implemented partially in software, the device can store instructions for the software in a suitable non-transitory computer-readable medium, and use one or more processors to execute the instructions in hardware to perform the technology of the present disclosure. Each of the video encoder 200 and the video decoder 300 can be included in one or more encoders or decoders, and any of the encoders or decoders can be integrated as part of a combined encoder / decoder (CODEC) in the corresponding device. The device including the video encoder 200 and / or the video decoder 300 can implement the video encoder 200 and / or the video decoder 300 in a processing circuit such as an integrated circuit and / or a microprocessor. Such a device can be a wireless communication device, such as a cellular phone or any other type of device described herein.

[0042] The video encoder 200 and the video decoder 300 may operate according to a video codec standard such as ITU-T H.265, also known as High Efficiency Video Codec (HEVC) or an extension thereof, such as a multi-view and / or scalable video codec extension. Alternatively, the video encoder 200 and the video decoder 300 may operate according to other proprietary or industry standards such as ITU-T H.266, also known as Universal Video Codec (VVC). In other examples, the video encoder 200 and the video decoder 300 may operate according to a proprietary video codec / format such as AOMedia Video 1 (AV1), an extension of AV1, and / or a successor version of AV1 (e.g., AV2). In other examples, the video encoder 200 and the video decoder 300 may operate according to other proprietary formats or industry standards. However, the technology of the present disclosure is not limited to any particular codec standard or format. In general, the video encoder 200 and the video decoder 300 may be configured to perform the technology of the present disclosure in conjunction with any video codec technology using an affine mode.

[0043] Typically, the video encoder 200 and the video decoder 300 can perform block-based encoding and decoding of pictures. The term "block" generally refers to a structure that includes data to be processed (e.g., encoded, decoded, or otherwise used in an encoding and / or decoding process). For example, a block may include a two-dimensional matrix of samples of luminance and / or chrominance data. Typically, the video encoder 200 and the video decoder 300 can encode and decode video data represented in a YUV (e.g., Y, Cb, Cr) format. That is, the video encoder 200 and the video decoder 300 can encode and decode luminance components and chrominance components instead of encoding and decoding red, green, and blue (RGB) data of samples of a picture, wherein the chrominance component may include both a red hue chrominance component and a blue hue chrominance component. In some examples, the video encoder 200 converts the received RGB formatted data into a YUV representation before encoding, and the video decoder 300 converts the YUV representation into an RGB format. Alternatively, a pre-processing and post-processing unit (not shown) may perform these conversions.

[0044] The present disclosure may generally refer to the encoding and decoding of a picture (e.g., encoding and decoding) to include the process of encoding or decoding the data of a picture. Similarly, the present disclosure may refer to the encoding and decoding of a block of a picture to include the process of encoding or decoding the data of the block, such as prediction and / or residual encoding and decoding. The encoded video bitstream typically includes a series of values ​​for syntax elements that represent codec decisions (e.g., codec modes) and partitioning of the picture into blocks. Therefore, references to encoding and decoding a picture or a block should generally be understood as encoding and decoding the values ​​of the syntax elements that form the picture or block.

[0045] HEVC defines various blocks, including codec units (CUs), prediction units (PUs), and transform units (TUs). According to HEVC, a video codec (such as the video encoder 200) partitions a codec tree unit (CTU) into CUs according to a quadtree structure. That is, the video codec partitions the CTU and the CU into four equal, non-overlapping squares, and each node of the quadtree has zero or four child nodes. A node without child nodes may be referred to as a "leaf node", and the CU of such a leaf node may include one or more PUs and / or one or more TUs. The video codec may also partition the PU and the TU. For example, in HEVC, the residual quadtree (RQT) represents the partitioning of the TU. In HEVC, the PU represents inter-frame prediction data, and the TU represents residual data. The intra-predicted CU includes intra-frame prediction information, such as an intra-frame mode indication.

[0046] As another example, the video encoder 200 and the video decoder 300 may be configured to operate according to VVC. According to VVC, a video codec (e.g., the video encoder 200) partitions a picture into multiple CTUs. The video encoder 200 may partition the CTU according to a tree structure, such as a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure removes the concept of multiple partition types, such as the separation between CU, PU, ​​and TU of HEVC. The QTBT structure includes two levels: a first level partitioned according to quadtree partitioning and a second level partitioned according to binary tree partitioning. The root node of the QTBT structure corresponds to the CTU. The leaf nodes of the binary tree correspond to the CU.

[0047] In the MTT partitioning structure, quadtree (QT) partitioning, binary tree (BT) partitioning, and one or more types of triple tree (TT) (also known as ternary tree, TT) partitioning can be used to partition blocks. Triple tree or ternary tree partitioning is a partitioning in which a block is split into three sub-blocks. In some examples, the ternary tree or ternary tree partitioning splits the block into three sub-blocks without dividing the original block through the center. The partitioning types in MTT (e.g., QT, BT, and TT) can be symmetric or asymmetric.

[0048] When operating according to the AV1 codec, the video encoder 200 and the video decoder 300 can be configured to encode and decode video data in blocks. In AV1, the largest codec block that can be processed is called a super block. In AV1, a super block can be 128×128 luminance samples or 64×64 luminance samples. However, in subsequent video codec formats (e.g., AV2), super blocks can be defined by different (e.g., larger) luminance sample sizes. In some examples, the super block is the highest level of the block quadtree. The video encoder 200 can also divide the super block into smaller codec blocks. The video encoder 200 can divide the super block and other codec blocks into smaller blocks using square or non-square partitioning. Non-square blocks can include N / 2×N, N×N / 2, N / 4×N, and N×N / 4 blocks. The video encoder 200 and the video decoder 300 can perform separate prediction and transform processing on each codec block.

[0049] AV1 also defines tiles of video data. A tile is a rectangular array of super blocks that can be encoded and decoded independently of other tiles. That is, the video encoder 200 and the video decoder 300 can encode and decode the codec blocks within the tile, respectively, without using video data from other tiles. However, the video encoder 200 and the video decoder 300 can perform filtering across tile boundaries. The size of the tile can be uniform or non-uniform. Tile-based encoding and decoding can achieve parallel processing and / or multi-threading for encoder and decoder implementations.

[0050] In some examples, the video encoder 200 and the video decoder 300 may use a single QTBT or MTT structure to represent each of the luma component and the chroma components, while in other examples, the video encoder 200 and the video decoder 300 may use two or more QTBT or MTT structures, such as one QTBT / MTT structure for the luma component and another QTBT / MTT structure for the two chroma components (or two QTBT / MTT structures for the corresponding chroma components).

[0051] The video encoder 200 and the video decoder 300 may be configured to use quadtree partitioning, QTBT partitioning, MTT partitioning, super block partitioning, or other partitioning structures.

[0052] In some examples, a CTU includes a codec tree block (CTB) of luma samples, two corresponding CTBs of chroma samples of a picture having three sample arrays, or a CTB of samples of a monochrome picture or a picture encoded using three separate color planes and syntax structures for encoding samples. A CTB can be an N×N block of samples for some value of N, such that the division of components into CTBs is a partition. A component is an array or a single sample from one of the three arrays (luminance and two chroma) that make up a picture in 4:2:0, 4:2:2, or 4:4:4 color format, or an array or a single sample of an array that makes up a picture in monochrome format. In some examples, a codec block is an M×N block of samples for some values ​​of M and N, such that the division of a CTB into a codec block is a partition.

[0053] Blocks (e.g., CTUs or CUs) can be grouped in a picture in various ways. As an example, a brick may refer to a rectangular area of ​​a CTU row within a particular tile in a picture. A tile may be a rectangular area of ​​a CTU within a particular tile column and a particular tile row in a picture. A tile column refers to a rectangular area of ​​a CTU having a height equal to the height of the picture and a width specified by a syntax element (e.g., such as in a picture parameter set). A tile row refers to a rectangular area of ​​a CTU having a height specified by a syntax element (e.g., such as in a picture parameter set) and a width equal to the width of the picture.

[0054] In some examples, a tile may be partitioned into multiple bricks, each of which may include one or more CTU rows within the tile. A tile that is not partitioned into multiple bricks may also be referred to as a brick. However, a brick that is a proper subset of a tile may not be referred to as a tile. Bricks in a picture may also be arranged in slices. A slice may be an integer number of bricks of a picture that may be exclusively contained in a single network abstraction layer (NAL) unit. In some examples, a slice includes a continuous sequence of several complete tiles or only complete bricks of one tile.

[0055] The present disclosure may use "N×N" and "N by N" interchangeably to refer to the sample size of a block (such as a CU or other video block) in terms of vertical and horizontal dimensions, for example, 16×16 samples or 16 by 16 samples. In general, a 16×16 CU will have 16 samples in the vertical direction (y=16) and 16 samples in the horizontal direction (x=16). Likewise, an N×N CU typically has N samples in the vertical direction and N samples in the horizontal direction, where N represents a non-negative integer value. The samples in a CU may be arranged in rows and columns. Furthermore, a CU does not necessarily need to have the same number of samples in the horizontal direction as in the vertical direction. For example, a CU may include N×M samples, where M is not necessarily equal to N.

[0056] The video encoder 200 encodes the video data of the CU representing the prediction and / or residual information and other information. The prediction information indicates how to predict the CU in order to form the prediction block of the CU. The residual information generally indicates the sample-by-sample difference between the samples of the CU before encoding and the prediction block.

[0057] In order to predict a CU, the video encoder 200 can generally form a prediction block of the CU by inter-frame prediction or intra-frame prediction. Inter-frame prediction generally refers to predicting a CU from data of a previously coded picture, while intra-frame prediction generally refers to predicting a CU from previously coded data of the same picture. In order to perform inter-frame prediction, the video encoder 200 can use one or more motion vectors to generate a prediction block. The video encoder 200 can generally perform a motion search to identify a reference block that closely matches the CU, for example, in terms of the difference between the CU and the reference block. The video encoder 200 can calculate a difference metric using the sum of absolute differences (SAD), the sum of squared differences (SSD), the mean absolute difference (MAD), the mean square difference (MSD), or other such difference calculations to determine whether the reference block closely matches the current CU. In some examples, the video encoder 200 can use unidirectional prediction or bidirectional prediction to predict the current CU.

[0058] Some examples of VVC also provide an affine motion compensation mode, which can be considered an inter-frame prediction mode. In the affine motion compensation mode, the video encoder 200 can determine two or more motion vectors representing non-translational motion (such as zooming in or out, rotation, perspective motion, or other irregular motion types).

[0059] To perform intra prediction, the video encoder 200 can select an intra prediction mode to generate a prediction block. Some examples of VVC provide sixty-seven intra prediction modes, including various directional modes, as well as planar modes and DC modes. Typically, the video encoder 200 selects an intra prediction mode that describes neighboring samples of a current block (e.g., a block of a CU), and predicts samples of the current block from the intra prediction mode. Assuming that the video encoder 200 encodes and decodes CTUs and CUs in a raster scan order (from left to right, from top to bottom), such samples can typically be above, above left, or to the left of the current block in the same picture as the current block.

[0060] The video encoder 200 encodes data representing the prediction mode of the current block. For example, for inter-frame prediction mode, the video encoder 200 can encode data indicating which of the various available inter-frame prediction modes is used and motion information of the corresponding mode. For example, for unidirectional or bidirectional inter-frame prediction, the video encoder 200 can encode motion vectors using advanced motion vector prediction (AMVP) or merge mode. The video encoder 200 can use a similar mode to encode motion vectors for affine motion compensation mode.

[0061] AV1 includes two general techniques for encoding and decoding codec blocks of video data. The two general techniques are intra prediction (e.g., intra prediction or spatial prediction) and inter prediction (e.g., inter prediction or temporal prediction). In the context of AV1, when predicting a block of a current video data frame using an intra prediction mode, the video encoder 200 and the video decoder 300 do not use video data from other video data frames. For most intra prediction modes, the video encoder 200 encodes a block of the current frame based on the difference between the sample values ​​in the current block and the prediction values ​​generated from the reference samples in the same frame. The video encoder 200 determines the prediction values ​​generated from the reference samples based on the intra prediction mode.

[0062] After prediction (such as intra-frame prediction or inter-frame prediction of a block), the video encoder 200 can calculate residual data for the block. The residual data (e.g., a residual block) represents the sample-by-sample difference between the block and the predicted block of the block formed using the corresponding prediction mode. The video encoder 200 can apply one or more transforms to the residual block to generate transform data in a transform domain rather than a sample domain. For example, the video encoder 200 can apply a discrete cosine transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform to the residual video data. In addition, the video encoder 200 can apply a secondary transform after the first transform, such as a mode-dependent non-separable secondary transform (MDNSST), a signal-dependent transform, a Karhunen-Loeve transform (KLT), etc. The video encoder 200 generates transform coefficients after applying one or more transforms.

[0063] As described above, after any transforms that produce transform coefficients, the video encoder 200 may perform quantization of the transform coefficients. Quantization generally refers to the process of quantizing the transform coefficients to potentially reduce the amount of data used to represent the transform coefficients, thereby providing further compression. By performing the quantization process, the video encoder 200 may reduce the bit depth associated with some or all of the transform coefficients. For example, the video encoder 200 may round down an n-bit value to an m-bit value during quantization, where n is greater than m. In some examples, to perform quantization, the video encoder 200 may perform a bit-by-bit right shift of the value to be quantized.

[0064] After quantization, the video encoder 200 can scan the transform coefficients to generate a one-dimensional vector from a two-dimensional matrix including the quantized transform coefficients. The scan can be designed to place higher energy (and therefore lower frequency) transform coefficients in front of the vector and lower energy (and therefore higher frequency) transform coefficients in the back of the vector. In some examples, the video encoder 200 can scan the quantized transform coefficients using a predefined scan order to generate a serialized vector, and then entropy encode the quantized transform coefficients of the vector. In other examples, the video encoder 200 can perform adaptive scanning. After scanning the quantized transform coefficients to form a one-dimensional vector, the video encoder 200 can entropy encode the one-dimensional vector, for example, according to context adaptive binary arithmetic coding (CABAC). The video encoder 200 can also entropy encode the values ​​of the syntax elements describing the metadata associated with the encoded video data for use by the video decoder 300 when decoding the video data.

[0065] To perform CABAC, video encoder 200 may assign context within a context model to a symbol to be sent. The context may relate to, for example, whether neighboring values ​​of the symbol are zero values. The probability determination may be based on the context assigned to the symbol.

[0066] The video encoder 200 may also generate syntax data, such as block-based syntax data, picture-based syntax data, and sequence-based syntax data, such as in a picture header, a block header, a slice header, or other syntax data, such as a sequence parameter set (SPS), a picture parameter set (PPS), or a video parameter set (VPS), to the video decoder 300. The video decoder 300 may also decode such syntax data to determine how to decode the corresponding video data.

[0067] In this way, the video encoder 200 can generate a bitstream including the encoded video data, for example, a syntax element describing the partitioning of a picture into blocks (e.g., CUs) and prediction and / or residual information of the blocks. Finally, the video decoder 300 can receive the bitstream and decode the encoded video data.

[0068] In general, the video decoder 300 performs a process that is inverse to the process performed by the video encoder 200 to decode the encoded video data of the bitstream. For example, the video decoder 300 may decode the values ​​of the syntax elements of the bitstream using CABAC in a manner substantially similar to but inverse to the CABAC encoding process of the video encoder 200. The syntax elements may define partitioning information for partitioning a picture into CTUs and partitioning each CTU according to a corresponding partitioning structure (such as a QTBT structure) to define CUs of the CTUs. The syntax elements may also define prediction and residual information for a block (e.g., a CU) of video data.

[0069] The residual information may be represented by, for example, quantized transform coefficients. The video decoder 300 may inverse quantize and inverse transform the quantized transform coefficients of the block to reproduce a residual block for the block. The video decoder 300 uses the signaled prediction mode (intra-frame or inter-frame prediction) and related prediction information (e.g., motion information for inter-frame prediction) to form a prediction block for the block. The video decoder 300 may then combine the prediction block and the residual block (on a sample-by-sample basis) to reproduce the original block. The video decoder 300 may perform additional processing, such as performing a deblocking process to reduce visual artifacts along the boundaries of the blocks.

[0070] The present disclosure may generally refer to "signaling" certain information, such as syntax elements. The term "signaling" may generally refer to the communication of values ​​of syntax elements and / or other data used to decode encoded video data. That is, the video encoder 200 may signal the values ​​of syntax elements in a bitstream. Generally speaking, signaling refers to generating values ​​in a bitstream. As mentioned above, the source device 102 may transmit the bitstream to the target device 116 in substantially real time or non-real time, such as may occur when storing syntax elements to the storage device 112 for later retrieval by the target device 116.

[0071] According to the technology of the present disclosure, as will be explained in more detail below, the video encoder 200 and the video decoder 300 can be configured to determine an affine motion model using a regression-based technique. In one example, the video encoder 200 and the video decoder 300 can be configured to segment a codec block into sub-blocks, generate initial sub-block motion vectors for multiple sub-blocks, refine the initial sub-block motion vectors of the multiple sub-blocks using decoder-side motion vector refinement to produce refined sub-block motion vectors for the multiple sub-blocks, and perform linear regression on the refined sub-block motion vectors and the coordinates of the multiple sub-blocks to derive an affine motion model. The video encoder 200 and the video decoder 300 can then encode and decode the codec block using the affine motion model. For example, the video encoder 200 and the video decoder 300 can determine the control point motion vectors from the derived affine motion model, and encode and decode the codec block using the determined control point motion vectors.

[0072] Affine mode

[0073] In some examples, the affine motion model may be described as follows:

[0074]

[0075] Where (v x ,v y ) is the motion vector at the coordinate (x, y), and a, b, c, d, e and f are six affine parameters. The above affine motion mode affine motion model may be referred to as a 6-parameter affine motion model. In some examples, the video encoder 200 and the video decoder 300 may be configured to divide the picture into blocks for block-based encoding and decoding. The affine motion model of the block may also be composed of three motion vectors (MVs) at three different positions that are not in the same row. as well as These three locations are usually called control points, and the three motion vectors are called control point motion vectors (CPMV). In the case where the three control points are located at the three corners of the block, the affine motion can be described as follows:

[0076]

[0077] where blkW and blkH are the width and height of the block.

[0078] In affine mode, video encoder 200 and video decoder 300 may be configured to derive different motion vectors for each pixel in a block according to an associated affine motion model. Thus, motion compensation may be performed on a per-pixel basis. However, to reduce implementation complexity, sub-block-based motion compensation may be performed, where the block is divided into multiple sub-blocks (having a smaller block size) and each sub-block is associated with one motion vector for block-based motion compensation.

[0079] Video encoder 200 and video decoder 300 may be configured to derive the motion vector for each sub-block using the representative coordinates of the sub-block. In some examples, the center position of the sub-block is used as the representative coordinate. In one example, video encoder 200 and video decoder 300 may divide the block into non-overlapping sub-blocks. The block width is blkW, the block height is blkH, the sub-block width is sbW, and the sub-block height is sbH. In this example, there are blkH / sbH rows of sub-blocks and blkW / sbW sub-blocks in each row.

[0080] For a six-parameter affine motion model, the motion vector of the sub-block (referred to as sub-block MV) at the i-th row (0 <= i < blkW / sbW) and the j-th column (0 <= j < blkH / sbH) is derived as follows:

[0081]

[0082] Video encoder 200 and video decoder 300 may be configured to round the sub-block MV to a predefined precision and store the motion vector in a motion buffer for motion compensation and motion vector prediction.

[0083] The simplified four-parameter affine model (for scaling and rotation motion) is described as follows:

[0084]

[0085] Similar to the six-parameter affine model, the four-parameter affine model for a block may be represented by two CPMVs at two corners (usually the upper left and upper right) of the block: And The motion field is then described as follows:

[0086]

[0087] The sub-block MV for the i-th row and the j-th column is derived as follows:

[0088]

[0089] Prediction refinement of affine models

[0090] After performing sub-block based affine motion compensation, the video encoder 200 and the video decoder 300 may be configured to refine the prediction signal. Refining the prediction signal may include adding an offset derived based on pixel-by-pixel motion and a gradient of the prediction signal. The offset ΔI at position (m, n) may be calculated as:

[0091] ΔI(m,n)=g x (m,n)*Δv x (m,n)+g y (m,n)*Δv y (m,n),

[0092] Among them, respectively, g x (m,n) is the horizontal gradient of the prediction signal, g y (m,n) is the vertical gradient of the prediction signal. Δv x (m,n) and Δv y (m,n) is the difference in x and y components between the motion vector calculated at pixel position (m,n) and the sub-block MV. The coordinates of the top left sample of the sub-block may be (0,0). In this case, the center of the sub-block is Given the affine motion parameters a, b, c, and d, Δv x (m,n) and Δv y (m,n) can be derived as follows:

[0093]

[0094] In the control point based affine motion model, the video encoder 200 and the video decoder 300 may calculate the affine motion parameters a, b, c, and d from the CPMV as follows:

[0095]

[0096] Regression-based affine pattern derivation

[0097] In the enhanced compression model version 6.0 (ECM6.0), the linear regression-based affine merge candidate derivation method proposed in "Regressionbased affine derivation" by Y. Zhang et al. (Joint Video Experts Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29, 27th meeting, by teleconference, July 13-22, 2022) (hereinafter referred to as "JVET-AA0107") is adopted. In this proposal, a linear regression process is applied to derive an affine motion model. The input of the linear regression is the motion information and corresponding coordinates from a set of previously encoded and decoded sub-blocks (e.g., adjacent sub-blocks).

[0098] The purpose of linear regression is to predict the value of a single unknown dependent variable using the independent variables whose values ​​are known. Each predicted value is weighted, where the weights represent their relative contribution to the overall prediction, and they are added together to form the prediction. The general model of multiple linear regression is given by Y = a + b1X1 + b2X2 + … + b n X n Given, where Y is the dependent variable, and X1, ..., X n There are n independent variables. When calculating weights a, b1, ..., b n When , the least squares method is applied to minimize the mean square error between the statistical observation sample and the estimated value.

[0099] The affine motion model can be equivalently expressed by two linear equations in matrix form:

[0100]

[0101] An affine motion model is essentially a linear equation of coordinates with certain coefficients. Given a corresponding motion vector (v x ,v y ) and a set of N sub-blocks with coordinates (x, y), the linear coefficients can be solved by linear regression Examples of linear regression processes can be found in co-pending U.S. patent application Ser. No. 18 / 188,908, filed Mar. 23, 2023, and in R. Ghaznavi-Youvalarim, A. Aminlou, and J. Lainema, “Regression-Based Motion Vector Field for Video Coding,” IEEE Transactions on Circuits and Systems for Video Technology, Vol. 31, No. 5, May 2021.

[0102] As a general example, the video encoder 200 and the video decoder 300 may be configured to receive a current video data block to be encoded and decoded using an affine merge mode. The video encoder 200 and the video decoder 300 may use the base motion vector field and the guided motion vector field as inputs to a multiple linear regression process to determine a refined affine model of the current video data block from the multiple linear regression process. In one example, the base motion vector field may be determined from an affine model of an adjacent block encoded and decoded using an affine mode. The guided motion vector field may include motion information from an adjacent sub-block (e.g., a sub-block adjacent to the current block). The video encoder 200 and the video decoder 300 may determine one or more affine merge candidates for the current block using the refined affine model, and encode and decode the current video data block using the affine merge candidates. By refining the affine model of the adjacent block using motion information from the adjacent sub-block, a more accurate affine model of the current block may be determined, thereby generating a more accurate affine merge candidate for the current block.

[0103] Decoder-side motion vector refinement

[0104] In some example video codecs, such as the Universal Video Codec (VVC) standard, the video encoder 200 and the video decoder 300 may be configured to apply a decoder-side motion vector refinement (DMVR) based on bilateral matching (BM) to increase the accuracy of the MV of the bi-prediction merge candidate. When performing the example BM method, the video encoder 200 and the video decoder 300 may be configured to calculate the sum of absolute differences (SAD) between two candidate blocks in the reference picture list L0 (e.g., the first prediction direction) and the reference picture list L1 (e.g., the second prediction direction).

[0105] like Figure 2 As shown, the video encoder 200 and the video decoder 300 can be configured to calculate the SAD between block 500 and block 502 based on each MV candidate around the initial MV. SAD can be called distortion cost calculation or bilateral matching cost calculation. The MV candidate with the lowest SAD becomes the refined MV and is used to generate the dual prediction signal. In some examples, the SAD of the initial MV is subtracted by 1 / 4 of the SAD value to serve as a regularization term. Figure 2 In the example of , the temporal distances (eg, picture order count (POC) differences) from the two reference pictures to the current picture should be the same, so the motion vector difference (MVD0) of MV0 is simply the motion vector difference (MVD1) of MV1 with the opposite sign.

[0106] The refinement search range is two integer brightness samples from the initial MV. The search includes an integer sample offset search stage and a fractional sample refinement stage. A 25-point full search is applied to the integer sample offset search. The SAD of the initial MV pair is first calculated. If the SAD of the initial MV pair is less than a threshold, the integer sample stage of DMVR is terminated. Otherwise, the SAD of the remaining 24 points is calculated and checked in raster scan order. The point with the smallest SAD is selected as the output of the integer sample offset search stage.

[0107] The integer sample search is followed by fractional sample refinement. To reduce computational complexity, fractional sample refinement can be derived by using a parameter error surface equation instead of an additional search using SAD comparison. In one example, fractional sample refinement is conditionally called based on the output of the integer sample search stage. When the integer sample search stage terminates with a center with a minimum SAD in the first or second iteration search, fractional sample refinement is further applied.

[0108] In the parametric error surface based sub-pixel offset estimation, the center position cost and the costs at four neighboring positions from the center are used to fit a 2-D parabolic error surface equation of the following form:

[0109] E(x,y)=A(x min ) 2 +B(yy min ) 2 +C, (1)

[0110] Where (x min ,y min ) corresponds to the fractional position with the minimum cost, and C corresponds to the minimum cost value. By solving the above equation using the cost values ​​of the five search points, (x min ,y min ) will be calculated as:

[0111] x min =(E(-1,0)-E(1,0)) / (2(E(-1,0)+E(1,0)-2E(0,0))) (2)

[0112] y min =(E(0,-1)-E(0,1)) / (2((E(0,-1)+E(0,1)-2E(0,0))) (3)

[0113] x min and min The value of is automatically constrained to be between -8 and 8, since all cost values ​​are positive and the minimum is positive E(0,0). This corresponds to a half-pixel shift with 1 / 16 pixel MV accuracy in VVC. min ,ymin ) is added to the integer distance refinement MV to obtain the sub-pixel accurate refinement difference MV.

[0114] In VVC, the resolution of MV is 1 / 16 luma sample. Samples at fractional positions are interpolated using an 8-tap interpolation filter. In DMVR mode, the search points are around the initial fractional pixel MV with an integer sample offset. Therefore, samples at those fractional positions are interpolated for the DMVR search process. In order to reduce computational complexity, a bilinear interpolation filter is used to generate fractional samples for the search process in DMVR. Another effect is that by using a bilinear filter as a search range of 2 samples, the DVMR process does not access more reference samples than the normal motion compensation process. After obtaining the refined MV through the DMVR search process, a normal 8-tap interpolation filter is applied to generate the final prediction. In order not to access more reference samples for the normal motion compensation process, samples that are not required by the interpolation process based on the original MV but are required by the interpolation process based on the refined MV will be filled from those available samples.

[0115] When the width and / or height of a CU is greater than 16 luma samples, the CU will also be split into sub-blocks with a height and / or width equal to 16 samples for the DMVR process.

[0116] In VVC, the DMVR process can be applied to CUs encoded and decoded with the following modes and features:

[0117] CU-level merge mode with bi-predicted MV

[0118] –Relative to the current picture, one reference picture is in the past and the other reference picture is in the future.

[0119] – The distances from the two reference images to the current image (i.e., POC difference) are the same – Both reference images are short-term reference images

[0120] –CU has more than 64 luma samples

[0121] – CU height and CU width are both greater than or equal to 8 luma samples

[0122] – Bi-prediction with CU level weights (BCW) weight index indicates equal weights

[0123] – Weighted Prediction (WP) is not enabled for this block

[0124] – Combined Inter-Intra Prediction (CIIP) mode is not used for the current block

[0125] Decoder-side motion vector refinement for affine merge mode

[0126] In "Non-EE2: DMVR for affine merge coded blocks" by J. Chen et al. (Joint Video Experts Team (JVET) of ITU-T SG16WP 3 and ISO / IEC JTC 1 / SC 29, 27th Meeting, Teleconference, July 13-22, 2022) (hereinafter referred to as "JVET-AA0144"), decoder-side motion vector refinement (DMVR) is proposed for bi-directional predictive affine merge candidates. In one example, the video encoder 200 and the video decoder 300 are configured to add a translational MV offset (e.g., a final offset or a CPMV offset) to all CPMVs of the candidate in the affine merge list if the candidate satisfies the DMVR condition. The video encoder 200 and the video decoder 300 can derive the MV offset (e.g., the final offset) by minimizing the cost of bilateral matching, which is similar to a conventional DMVR process. The difference is that affine motion compensation is performed to generate predictors in both directions. The MV offset search process is the same as the first pass of multi-pass DMVR (prediction unit level) in Enhanced Compression Mode (ECM). A 3x3 square search pattern is used to cycle through the search range of possible candidate offsets (e.g., [-8, +8] in the horizontal direction and [-8, +8] in the vertical direction) to find the best integer MV offset. Then, a half-pixel search is performed around the best integer position, and finally an error table estimation is performed to determine the MV offset with 1 / 16 precision.

[0127] To compute the bilateral matching cost (or more generally, the distortion cost) given a bidirectional affine motion, the following procedure can be applied:

[0128] 1) Add the candidate offset to each of the CPMVs in both directions to update the CPMVs.

[0129] 2) Apply affine motion compensation according to the updated CPMV to generate predictors in two directions.

[0130] 3) Calculate the distortion between the generated predictors using a predefined cost criterion.

[0131] The candidate offset with the lowest bilateral matching cost is selected as the final offset to be added to the CPMV. Typically, affine motion compensation is sub-block based, where the current block is split into non-overlapping sub-blocks. Conventional motion compensation is performed on each sub-block, where all samples in the sub-block share the same motion. In VVC, prediction refinement with optical flow (PROF) can be applied after sub-block based motion compensation to further improve the predictor.

[0132] In co-pending U.S. patent application Ser. No. 18 / 351,342 filed on July 12, 2023 (hereinafter, the '342 application), a less complex and more practical DMVR design for affine motion models is described. The DMVR design can be summarized as follows:

[0133] 1) Divide the current block into sub-blocks.

[0134] 2) Generate the initial motion vector (in two prediction directions) of each sub-block (sub-block motion field) according to the initial affine motion model.

[0135] 3) Loop through each sub-block and calculate the bilateral matching cost of all possible offset sub-blocks.

[0136] 4) For each possible offset, accumulate the sub-block bilateral matching costs to generate a bilateral matching cost corresponding to the entire block.

[0137] 5) Determine the best offset by selecting the offset with the minimum bilateral matching cost corresponding to the entire block.

[0138] Note that in this way the sub-block motion field is generated only once, instead of for each candidate offset.

[0139] The sub-block size in the above process can be the same as the sub-block size used for affine motion compensation, which is usually 4×4. The sub-blocks can also be of different sizes. In some examples, the sub-block size can be a larger sub-block (e.g., 8×8, 16×16), that is, a smaller number of sub-blocks, to reduce the complexity of the DMVR process.

[0140] In step 3) above, given the offset and the initial motion vector (generated in step 2), a candidate motion vector can be derived (in the case of mirrored bilateral matching, the offset is added to the motion vector in one direction and subtracted from the motion vector in the other direction), and then motion compensation is performed to generate a predictor for the corresponding subblock. For simplicity, prediction refinement can be skipped. In addition, note that pre-interpolation can be applied to generate predictors for all possible offsets in one step, which significantly reduces complexity. Bilinear interpolation can be used instead of the 8-tap (6-tap or 12-tap) interpolation filter typically used for final motion compensation.

[0141] A sub-pixel offset estimation based on a parametric error surface may also be applied after step 5) to generate a sub-pixel offset.

[0142] Certain size constraints may be applied. For example, DMVR is only applied to affine blocks larger than N×N, where N is set equal to 8, 16, 32, etc. In another example, the size constraint depends on the sub-block size used in the DMVR process, where N is set equal to the sub-block size.

[0143] High-level syntax may be used to control whether DMVR for affine blocks is applied and signaled in the bitstream, e.g., in a sequence parameter set, a picture header, a slice header, etc.

[0144] In a co-pending U.S. provisional patent application with serial number 63,377,659 filed on September 29, 2022, sub-block skipping or subsampling is disclosed to further reduce complexity, where a subset of sub-blocks can be used during an affine DMVR process.

[0145] For example, the video encoder 200 and the video decoder 300 may receive a current block of video data to be decoded using the affine DMVR mode. The video encoder 200 and the video decoder 300 may divide the current block into a plurality of sub-blocks. In one example, to divide the block into the plurality of sub-blocks, the video encoder 200 and the video decoder 300 may sub-sample the plurality of sub-blocks to determine a first subset of the plurality of sub-blocks on which the affine DMVR process is performed.

[0146] In one example, the first subset of the plurality of sub-blocks is half of the plurality of sub-blocks. For example, the first subset of the plurality of sub-blocks may include the plurality of sub-blocks in every other row of the block or the sub-blocks in every other column of the block. In another example, the first subset of the plurality of sub-blocks is a checkerboard (or plum blossom) pattern in the block. In other examples, the first subset of the plurality of sub-blocks is a quarter of the sub-blocks of the block, for example, including sub-blocks in every other column and every other row of the sub-blocks in the block.

[0147] The video encoder 200 and the video decoder 300 may determine the final offset of the affine DMVR mode using the first subset of the plurality of sub-blocks. For example, the video encoder 200 and the video decoder 300 may determine the final offset of the affine DMVR mode using only the first plurality of sub-blocks, and may skip bilateral matching cost calculation for a second subset of the plurality of sub-blocks that is different from the first subset of the plurality of sub-blocks.

[0148] To determine the final offset, the video encoder 200 and the video decoder 300 may determine a corresponding initial motion vector (for both the first prediction direction and the second prediction direction) for each sub-block that is not skipped (e.g., for a first subset of sub-blocks) according to the initial affine motion model. The video encoder 200 and the video decoder 300 may determine the initial affine motion model from the candidate affine motion models (e.g., candidate CPMVs or motion vectors) in the affine candidate merge list.

[0149] The video encoder 200 and the video decoder 300 may generate a plurality of candidate motion vectors for each of the first plurality of sub-blocks. Generating the plurality of candidate motion vectors may add each corresponding candidate offset in the plurality of candidate offsets to the corresponding initial motion vector. For example, as described above, the candidate offsets may be within a search range of [-8, +8] in the horizontal direction and within a search range of [-8, +8] in the vertical direction. In other examples, other search ranges may be used.

[0150] The video encoder 200 and the video decoder 300 may loop over each sub-block that is not skipped (e.g., a first subset of sub-blocks) and perform an affine motion compensation process to generate a predictor for each prediction direction. Based on these predictors, the video encoder 200 and the video decoder 300 may calculate the sub-block bilateral matching costs for all possible candidate offsets. For example, the video encoder 200 and the video decoder 300 may calculate the corresponding bilateral matching costs for each sub-block in the first plurality of sub-blocks for each candidate motion vector in the plurality of candidate motion vectors.

[0151] For each possible offset, the video encoder 200 and the video decoder 300 may accumulate the corresponding sub-block bilateral matching costs to generate bilateral matching costs corresponding to all sub-blocks that are not skipped (e.g., the first subset of sub-blocks). For example, the video encoder 200 and the video decoder 300 may accumulate the corresponding bilateral matching costs of all sub-blocks in the first plurality of sub-blocks for each of the plurality of candidate motion vectors to generate the corresponding accumulated bilateral matching costs.

[0152] Then, the video encoder 200 and the video decoder 300 can determine the best offset by selecting the candidate offset with the minimum cumulative bilateral matching cost corresponding to the entire block. For example, the video encoder 200 and the video decoder 300 can determine the final offset as the corresponding candidate offset associated with the lowest corresponding cumulative bilateral matching cost.

[0153] The video encoder 200 and the video decoder 300 may encode or decode (e.g., respectively encode or decode) the video data block using the final offset to generate a coded or decoded video data block. In one example, to encode or decode the video data block using the final offset, the video encoder 200 and the video decoder 300 may determine a control point motion vector of the video data block, add the final offset to the control point motion vector to create a refined control point motion vector, and encode or decode the video data block using the refined control point motion vector.

[0154] Example

[0155] This disclosure describes techniques for regression-based decoder-side affine motion derivation. Figure 3As shown, the exemplary method of the present disclosure can be summarized as follows:

[0156] The codec block is divided into sub-blocks, and the corresponding initial sub-block MVs are usually generated from the initial motion information of the codec block. In one example, the initial motion information may be an initial affine motion model from a candidate in the affine merge list. In this example, the initial affine motion model may be represented by a set of control point motion vectors (e.g., 2 or 3 CPMVs).

[0157] • Perform a decoder-side motion vector refinement (DMVR) technique (eg, bilateral matching) for each of the sub-blocks to generate a set of refined MVs.

[0158] • The refined MVs and the corresponding coordinates of those sub-blocks are used as input to the linear regression process.

[0159] The output of linear regression is the derived affine motion model.

[0160] It should be noted that, depending on the application, the output affine model can be the affine parameters But it can also be a control point motion vector derived from such affine parameters. It should also be noted that the initial motion information of the codec block can be a bidirectional affine motion and thus the initial sub-block MV is generated according to the initial affine motion. In some examples, the initial motion information can also be a regular bidirectional motion and thus the generated initial sub-block MVs are all equal to the initial bidirectional motion. However, after bilateral matching, the refined sub-block MVs may be different for different sub-blocks.

[0161] refer to Figure 3 , the video encoder 200 and the video decoder 300 may be configured to partition the codec block 508 into a plurality of sub-blocks. In some examples, the video encoder 200 and the video decoder 300 may be configured to determine the size of the sub-blocks. That is, the size of the sub-blocks may vary by slice, picture, sequence, etc. In other examples, the video encoder 200 and the video decoder 300 may be configured to partition the codec block 508 into fixed sub-blocks (e.g., 4×4 sub-blocks).

[0162] Then, the video encoder 200 and the video decoder 300 may generate an initial sub-block motion vector 510 for each sub-block. In one example, the initial sub-block motion vector 510 may be based on a control point motion vector representing an input affine motion model. For example, the video encoder 200 and the video decoder 300 may determine the control point motion vector from a candidate in an affine merge list. In this example, each of the sub-blocks may have a different initial sub-block motion vector based on the coordinates of the sub-block. In other examples, when the initial sub-block motion vector is based on a bi-predictive motion (e.g., a bi-predictive motion of an adjacent block), each of the initial sub-block motion vectors 510 may be the same.

[0163] In some examples of the present disclosure, the video encoder 200 and the video decoder 300 may generate initial sub-block motion vectors 510 and then refine those motion vectors for all sub-blocks of the codec block 508. In other examples, the video encoder 200 and the video decoder 300 may generate initial sub-block motion vectors 510 and then refine those motion vectors for a subset of sub-blocks of the codec block 508. For example, the video encoder 200 and the video decoder 300 may perform the techniques of the present disclosure using half, quarter, or any other subset of sub-blocks in any mode.

[0164] The video encoder 200 and the video decoder 300 may then use a bilateral matching unit 520 to refine the initial sub-block motion vectors 510 for the plurality of sub-blocks to generate refined sub-block motion vectors 522. More generally, the video encoder 200 and the video decoder 300 may use DMVR techniques to generate refined motion vectors. In this example, the video encoder 200 and the video decoder 300 may independently refine each initial sub-block motion vector 510 using decoder-side motion vector refinement (e.g., bilateral matching). For example, the bilateral matching unit 520 may independently determine a bilateral matching cost for each of a plurality of candidate offsets for each of the sub-blocks (e.g., without considering costs determined for other sub-blocks). The candidate offset that results in the lowest bilateral matching cost for a particular subset is selected as the offset used to generate the refined sub-block motion vector for the sub-block.

[0165] Thus, because each of the initial sub-block motion vectors 510 is individually refined, the resulting refined sub-block motion vectors may not necessarily fit the affine model because they may not have a linear correlation with each other. The linear regression unit 530 may perform a linear regression process on the refined sub-block motion vectors 522 to generate an affine motion model 532. The linear regression process performed by the linear regression unit 530 is described in more detail below. This linear regression is sometimes referred to as multivariate linear regression or surface fitting. As described above, the output of the linear regression unit 530 may be affine parameters (e.g., ), but may also be control point motion vectors derived from such affine parameters. For example, the video encoder 200 and the video decoder 300 may be configured to derive control point motion vectors from the affine parameters output by the linear regression unit 530. The video encoder 200 and the video decoder 300 may then perform encoding and decoding (e.g., encoding and decoding, respectively) based on the control point motion vectors determined from the affine motion model 532. By applying the linear regression process to the refined sub-block motion vectors, a more accurate affine motion model may be obtained. In this way, encoding and decoding efficiency may be improved and distortion may be reduced.

[0166] In some examples, the video encoder 200 and the video decoder 300 may not necessarily use the affine motion model 532 to encode the codec block 508. Instead, the control point motion vector determined from the affine motion model 532 may be a set of possible motion information from two or more sets of motion information that the video encoder 200 and the video decoder 300 may select between when determining how to encode the codec block.

[0167] Figure 4 A conceptual diagram showing another example of regression-based decoder-side affine pattern derivation according to this example. Figure 4 The bottom track of Figure 3 That is, the video encoder 200 and the video decoder 300 are configured to refine the initial sub-block motion vector 510 and perform a linear regression process on the refined sub-block motion vector 522 to generate an affine motion model 532. Substantially in parallel with this process, the video encoder 200 and the video decoder 300 may perform a different DMVR process on the initial sub-block motion vector 510 using a bilateral matching unit 550. The bilateral matching unit 550 determines a motion vector offset from a plurality of candidate motion vector offsets that achieves a minimum bilateral matching (BM) cost for all sub-blocks in the codec block 508 (e.g., a BM cost accumulated for all sub-blocks), rather than refining each of the initial sub-block motion vectors independently (e.g., per sub-block).

[0168] For example, the double-sided matching unit 550 may perform the DMVR process described above, which may be summarized as follows:

[0169] – Divide the codec block 508 into sub-blocks.

[0170] - Generate an initial sub-block motion vector 510 for each sub-block according to the initial affine motion model.

[0171] – Loop over each sub-block and compute the sub-block bilateral matching costs for all possible motion vector offsets.

[0172] - For each possible motion vector offset, the sub-block bilateral matching costs are accumulated to generate a bilateral matching cost corresponding to the entire block.

[0173] – Determine the best motion vector offset by selecting the motion vector offset corresponding to the entire block with the minimum bilateral matching cost.

[0174] The output of the bilateral matching unit 550 is a motion vector offset and a BM cost 552. The video encoder 200 and the video decoder 300 may then use the cost calculation unit 540 to determine the bilateral matching cost (e.g., regression cost 542) of the control point motion vector derived from the affine motion model 532. The cost comparison unit 560 may then determine whether the BM cost 552 or the regression cost 542 is the lowest. If the regression cost 542 is the lowest, the video encoder 200 and the video decoder 300 will use the control point motion vector derived from the regression process (e.g., affine motion model 532) to encode and decode the codec block 508. If the BM cost 552 is the lowest, the video encoder 200 and the video decoder 300 will use the motion vector offset derived from the bilateral matching unit 550 to encode and decode the codec block 508. The motion vector offset is added to the input motion information (e.g., the input control point motion vector) to derive a refined control point motion vector. The video encoder 200 and the video decoder 300 then use the refined control point motion vectors to encode and decode the codec block 508. By using regression-based refinement and DMVR simultaneously, more options for refined control point motion vectors are generated, thereby increasing the chances of improving codec efficiency and / or reducing distortion.

[0175] Applying to Affine Merge Candidates - Example 1

[0176] The following describes the application of decoder-side motion refinement for bi-predictive affine merge candidates. The input to this process is:

[0177] – luma position (xCb, yCb), which specifies the top left sample of the current codec block relative to the top left luma sample of the current picture,

[0178] – Two variables cbWidth and cbHeight specifying the width and height of the current brightness codec block,

[0179] – Luma affine control point motion vector cpMvLX[cpIdx], where cpIdx=0..numCpMv-1 and X is 0 or 1,

[0180] – The number of control point motion vectors numCpMv.

[0181] The output of this process is the refined luminance affine control point motion vector refinedCpMvLX[cpIdx], where cpIdx=0..numCpMv-1 and X is 0 or 1.

[0182] 1) Determine the width sbWidth and height sbHeight of the sub-block. It should be noted that in some examples, the sub-block size may be fixed.

[0183] 2) Determine the step size stepX in the x direction and stepY in the y direction.

[0184] Note that in some examples, the use and determination of the step size can be skipped. Additionally, in some examples, the sub-blocks can be sub-sampled. For example, if the sub-block width is X (e.g., 4 samples wide), but stepX is 8, it means that the first left X coordinate of the sub-block is 0, the sub-block coordinate at X coordinate 4 is skipped, and the left X coordinate of the second sub-block is 8, and so on.

[0185] 3) Determine the sub-block list, the luma position (xSb[sbIdx], ySb[sbIdx]) specifies the upper left sample of the sub-block relative to the upper left luma sample of the current picture, where sbIdx=0…numSubBlk-1, numSubBlk specifies the number of sub-blocks.

[0186] 4) Determine the sub-block motion vector sbMvLX[sbIdx], where sbIdx=0…numSubBlk-1 and X is 0 or 1. The sub-block motion vector is based on the luminance affine control point motion vector cpMvLX[cpIdx]

[0187] 5) For sbIdx=0...numSubBlk-1, the following applies:

[0188] a) Derive the bilateral matching cost array of the sub-block

[0189] bmCost[sbIdx][dX+srchRng][dY+srchRng], where dX=-

[0190] srchRng..srchRng and dY=-srchRng..srchRng, where srchRng represents the search range.

[0191] b) Determine the integer differential motion vector dSbMv[sbIdx]=(dXBest, dYBest) such that bmCost[sbIdx][dXBest+srchRng][dYBest+srchRng] corresponds to the minimum value in the 2-D array bmCost[sbIdx]. For X being 0 or 1, the refined sub-block motion vector is refinedSbMvLX[sbIdx].

[0192] refinedsbMvL0[sbIdx]=sbMvL0[sbIdx]+dSbMv[sbIdx], refinedsbMvL1[sbIdx]=sbMvL1[sbIdx]-dSbMv[sbIdx].

[0193] c) Determine the sub-pixel difference motion vector dSbMvSubPel[sbIdx] of the sub-block.

[0194] refinedSbMvL0[sbIdx]=refinedSbMvL0[sbIdx]+

[0195] dSbMvSubPel[sbIdx], refinedSbMvL1[sbIdx]=

[0196] refinedSbMvL1[sbIdx]–dSbMvSubPel[sbIdx]. Sub-pixel bilateral matching search or error surface based derivation (with bmCost[sbIdx] as input) can be used for this process.

[0197] 6) Derive the bilateral matching cost array bmCostTotal[dX+2][dY+2] of the codec block, where dX=-srchRng..srchRng and dY=-srchRng..srchRng, by summing bmCost over sbIdx=0…numSubBlk-1.

[0198] 7) Determine the integer differential motion vector dMv = (dMvXIntBest, dMvYIntBest) such that minCost = bmCostTotal[dMvXIntBest+srchRng][dMvYIntBest+srchRng] corresponds to the minimum value in the 2-D array bmCostTotal. Set refinedCpMvL0[cpIdx] equal to cpMvL0[cpIdx]+dMv and set refinedCpMvL1[cpIdx] equal to cpMvL1[cpIdx]-dMv, where cpIdx = 0..numCPMV-1.

[0199] 8) Determine the sub-pixel differential motion vector dMvSubPel of the codec block. Sub-pixel bilateral matching search or error surface-based derivation (with bmCostTotal as input)

[0200] can be used in this process. If a sub-pixel bilateral matching search is applied, then minCost can be updated accordingly.

[0201] 9) Set refinedCpMvL0[cpIdx] equal to refinedCpMvL0[cpIdx]

[0202] +dMvSubPel, set refinedCpMvL1[cpIdx] equal to refinedCpMvL1[cpIdx] – dMvSubPel, where cpIdx = 0..numCpMv

[0203] –1.

[0204] 10) Call the linear regression process with (xCb, yCb), cbWidth, cbHeight, refinedSbMvLX[sbIdx], and (xSb[sbIdx]+sbWidth / 2, ySb[sbIdx]+sbHeight / 2) as input. The output is the derived control point motion vector cpMvRegressionLX[cpIdx], where cpIdx = 0..numCpMv–1.

[0205] 11) Derive the bilateral matching cost bmCostRegression corresponding to cpMvRegressionLX[cpIdx], where cpIdx = 0..numCpMV-1. If bmCostRegression is less than minCost, then refineCpMvLX[cpIdx]

[0206] Set equal to cpMvRegressionLX[cpIdx], where cpIdx = 0..numCpMv–1.

[0207] In some examples, step 5).c may not be applied. In another example, step 8). In yet another example, an early skip process may be applied after step 7), where the process stops when minCost is less than a predefined threshold (eg, the area of ​​the codec block).

[0208] Different cost criteria may be applied, such as the sum of absolute differences, the sum of absolute Hamdard transform differences, or other cost criteria. Some cost regularization term that depends on the differential motion vector may also be added to the cost. In one example, the regularization term is s*(abs(dMvX)+abs(dmvY)), where s is the scaling factor and (dMvX, dmvY) is the differential motion vector.

[0209] In another example, the cost is adjusted or scaled depending on the differential motion vector, where the larger the magnitude of the differential motion vector, the larger the scaling factor. For example:

[0210] If both dMvX and dMvY are equal to 0, then set bmCostTotal[dX+2][dY+2] equal to bmCostTotal[dX+2][dY+2] – (bmCostTotal[dX+2][dY+2] >> 2),

[0211] Otherwise, if abs(dMvX)+abs(dmvY)>=4 and abs(dMvX)+abs(dmvY)<7, then set bmCostTotal[dX+2][dY+2] equal to bmCostTotal[dX+2][dY+2]+(bmCostTotal[dX+2][dY+2]>>2)

[0212] Different cost criteria can also be applied to different steps.

[0213] Applying to Affine Merge Candidates - Example 2

[0214] Another example method for determining an affine motion mode that does not require linear regression is described below. In this example, the refined sub-block motion vectors from the corner sub-blocks (e.g., 3 corners) of the block are used to determine the affine motion model. For example, the refined MVs of the numCpMv corner sub-blocks (typically at positions (xCb, yCb), (xCb+cbWidth-sbWidth, yCb), and (xCb, yCb+cbHeight-sbHeight)) are used as the derived control point motion vectors.

[0215] The inputs to this process are:

[0216] – luma position (xCb, yCb), which specifies the top left sample of the current luma codec block relative to the top left luma sample of the current picture,

[0217] – Two variables cbWidth and cbHeight specifying the width and height of the current brightness codec block,

[0218] – Luminance affine control point motion vector cpMvLX[cpIdx], where cpIdx =

[0219] 0..numCpMv–1 and X is 0 or 1,

[0220] – The number of control point motion vectors numCpMv.

[0221] The output of this process is the refined luminance affine control point motion vector refinedCpMvLX[cpIdx], where cpIdx=0..numCpMv-1 and X is 0 or 1.

[0222] Determine the width sbWidth of the sub-block and the height sbHeight of the sub-block.

[0223] Determine the step size stepX in the x direction and stepY in the y direction.

[0224] Determine the sub-block list, the luma position (xSb[sbIdx], ySb[sbIdx]) specifies the top left sample of the sub-block relative to the top left luma sample of the current picture, where sbIdx=0…numSubBlk-1, numSubBlk specifies the number of sub-blocks.

[0225] Determine the sub-block motion vector sbMvLX[sbIdx], where sbIdx=0…numSubBlk-1 and X is 0 or 1.

[0226] For sbIdx=0...numSubBlk-1, the following applies:

[0227] a. Derive the bilateral matching cost array bmCost[sbIdx][dX+srchRng][dY+srchRng] of the sub-block, where dX = -srchRng..srchRng and dY = -srchRng..srchRng, where srchRng represents the search range.

[0228] b. Determine the integer differential motion vector dSbMv[sbIdx] = (dXBest, dYBest) such that bmCost[sbIdx][dXBest + srchRng][dYBest + srchRng] corresponds to the minimum value in the 2-D array bmCost[sbIdx]. For X being 0 or 1, the refined sub-block motion vector is refinedSbMvLX[sbIdx]. refinedSbMvL0[sbIdx] = sbMvL0[sbIdx] + dSbMv[sbIdx], refinedSbMvL1[sbIdx] = sbMvL1[sbIdx] – dSbMv[sbIdx].

[0229] c. Determine the sub-pixel differential motion vector dSbMvSubPel[sbIdx] of the sub-block. refinedSbMvL0[sbIdx] = refinedSbMvL0[sbIdx] + dSbMvSubPel[sbIdx], refinedSbMvL1[sbIdx] = refinedSbMvL1[sbIdx] - dSbMvSubPel[sbIdx]. Sub-pixel bilateral matching search or error surface based derivation (with bmCost[sbIdx] as input) can be used for this process.

[0230] The bilateral matching cost array bmCostTotal[dX+2][dY+2] of the codec block is derived, where dX=-srchRng..srchRng and dY=-srchRng..srchRng, by summing bmCost over sbIdx=0...numSubBlk-1.

[0231] Determine the integer differential motion vector dMv = (dMvXIntBest, dMvYIntBest) such that minCost = bmCostTotal[dMvXIntBest + srchRng][dMvYIntBest + srchRng] corresponds to the minimum value in the 2-D array bmCostTotal. Set refinedCpMvL0[cpIdx] equal to cpMvL0[cpIdx] + dMv and set refinedCpMvL1[cpIdx] equal to cpMvL1[cpIdx] - dMv, where cpIdx = 0..numCpMv - 1.

[0232] Determine the sub-pixel differential motion vector dMvSubPel of the coded block. Sub-pixel bilateral matching search or error surface based derivation (with bmCostTotal as input) can be used for this process. If sub-pixel bilateral matching search is applied, minCost can be updated accordingly.

[0233] Set refinedCpMvL0[cpIdx] equal to refinedCpMvL0[cpIdx]+dMvSubPel, and set refinedCpMvL1[cpIdx] equal to refinedCpMvL1[cpIdx]–dMvSubPel, where cpIdx=0..numCpMv–1.

[0234] The refined MVs of numCpMv corner sub-blocks (typically at positions (xCb, yCb), (xCb+cbWidth-sbWidth, yCb), and (xCb, yCb+cbHeight-sbHeight)) are selected as the derived control point motion vectors cpMvCornerLX[cpIdx], where cpIdx = 0..numCpMv–1.

[0235] Derive the bilateral matching cost bmCostDerived corresponding to cpMvCornerLX[cpIdx], where cpIdx = 0..numCpMv - 1. If bmCostDerived is less than minCost, then set refinedCpMvLX[cpIdx] equal to cpMvCornerLX[cpIdx], where cpIdx = 0..numCpMv - 1.

[0236] Linear Regression

[0237] An example of linear regression is described below to derive an updated affine motion model (and refined control point motion vectors generated from the updated affine motion model) using the initial affine motion model and the refined sub-block motion vectors, as described above. However, note that other linear regression methods may also be applied.

[0238] Multiple linear regression, also known as multiple regression, is a statistical technique that can be used to analyze the relationship between a single dependent variable and several independent variables. The purpose of multiple regression analysis is to predict the value of a single unknown dependent variable using the independent variables whose values ​​are known. Each predictor value is weighted, where the weight represents their relative contribution to the overall prediction, and is summed to form the prediction. The general model of multiple linear regression is given by equation (1):

[0239] Y=a+b1X1+b2X2+…+b n X n (1)

[0240] In equation (1), Y is the dependent variable, and X1, ..., X n are n independent variables. In one example, the weights a, b1, ..., b can be calculated using the least squares method. n , where the mean squared error between the statistical observations and the estimated values ​​is minimized.

[0241] In VVC, block-based affine transformation motion compensation prediction is applied to better represent magnification / reduction, rotation, perspective motion and other irregular motions, where the motion vector of each sub-block is derived based on a linear model. The motion vector at the sample position (x, y) at the center of the sub-block is derived as:

[0242]

[0243] In equation (2), mv x As an example, mv 0x ,mv 1x and mv 2xare the horizontal motion vector components of the top left, top right, and bottom left control point motion vectors (CPMVs), which are known for a given affine model. W and H are the width and height of the current CU, which are also known. In this case, to derive mv x The equation can be written as a linear regression formula. y The same technique can be applied. By this observation, the affine motion model can be equivalently represented by two linear equations in matrix form:

[0244]

[0245] Among them, respectively, and b x2 =mv 0x ; and b y2 =mv 0y .

[0246] Using the above equations, the derivation process of the affine motion model can essentially be regarded as the derivation process of certain coefficients in the linear equation. The estimation of the linear coefficients can be solved by the multivariate linear regression process.

[0247] To apply the multivariate linear regression process to affine model derivation, the video codec can obtain the motion vector information and its corresponding sub-block coordinates. The coordinates are the independent variables, while the motion vector components are the dependent variables. Given N motion vectors and their corresponding sub-block center coordinates, the affine model parameters can be derived using the following equations:

[0248]

[0249] Among them, det(B c,d ) and det(A) are matrices B c,d and the determinant of the matrix A, and B c,d and A are both 3×3 matrices. For a 3×3 matrix, the determinant can be calculated by the following equation:

[0250] det(M)=(M 0,0 ×M 1,1 ×M 2,2 +M 1,0 ×M 2,1 ×M 0,2 +M 2,0 ×M 0,1 ×M 1,2 )-(M 0,0 ×M 2,1 ×M 1,2 +M 1,0 ×M 0,1 ×M 2,2 +M 2,0 ×M1,1 ×M 0,2 ), (5)

[0251] Its i,j where is the element at position (i,j) in the matrix, and i,j∈{0,1,2}.

[0252] To construct the matrix B c,d and A, using the following equation:

[0253]

[0254] Using this approach, a more optimized affine model can be derived to better describe the input motion vectors considering their corresponding coordinates in the sense of minimum mean square error.

[0255] For elaboration, an example is given here to illustrate how a video codec can determine an output linear model from an input motion vector field. In this example, the video codec has obtained motion information from some motion vector field, and N motion vectors can be represented by {(mv x0 ,mv y0 ),(mv x1 ,mv y1 ),…,(mv xN-1 ,mv yN-1 )}, and the corresponding center coordinates are expressed as {(x0,y0),(x1,y1),…,(x N-1 ,y N-1 )}.

[0256] In the first step, the video codec can determine the anchor coordinates (which are the upper left coordinates of the entire picture) that serve as the origin of the output linear model instead of using (0, 0) to avoid large values ​​in the calculation. This is also to easily derive affine merge candidates after deriving the linear model parameters from the linear regression process. Typically, the video codec can be configured to use the upper left coordinates of the current CU as the origin. Using (x tl ,y tl ) represents the origin, and is obtained by subtracting (x tl ,y tl ) to calculate the relative coordinates. By doing this, the video codec can obtain {(x0-x tl ,y0-y tl ),(x1-x tl ,y1-y tl ),…,(x N-1 -x tl ,y N-1 -y tl )}. Given the following:

[0257] x i'=x i -x tl

[0258] y i '=y i -y tl ,

[0259] where i∈{0,1,…,N-1}, and then the video codec obtains {(x0',y0'),(x1',y1'),…,(x N-1 ',y N-1 ')}.

[0260] For motion vectors, the video codec can also choose an initial motion vector to subtract from to avoid large values ​​in the calculation, but this is not necessary since motion vector values ​​are relatively small compared to the coordinate values. Furthermore, motion vectors can have negative values, and subtracting a negative value from a positive value only ends up with an even larger positive value.

[0261] For N available adjacent MVs, the output linear model parameters can be estimated using the mean square error (MSE) minimization method, where the MSE is calculated as the estimated MV {(mv x0 ',mv y0 '),(mv x1 ',mv y1 '),…,(mv xN-1 ',mv yN-1 ')} and the actual value of the input MV {(mv x0 ,mv y0 ),(mv x1 ,mv y1 ),…,(mv xN-1 ,mv yN-1 )}, as shown in the following equation:

[0262]

[0263] Where c∈{x,y} represents the horizontal or vertical component of the motion vector.

[0264] Mathematically, the solution to minimizing MSE is given by equation (4).

[0265] With parameter b x,0 As an example, the matrix A is derived from A i,j =suml′l′ i,j Derived from , where i,j∈{0,1,2}, where l′0 is the x′ of each sub-block, l′1 is the y′ of each sub-block and l′2=1 for all sub-blocks. By The element of matrix A at position (0, 0) is represented by . Similarly, other elements in matrix A can be derived.

[0266] For matrix B, when deriving b x,0 When , we can also derive the matrix B x,0 From equation (9), except and (because here j=d=0) are equal to the element A from the matrix A i,j For the element Bundle As an example, akin, and Can be exported.

[0267] Using matrices A and B x,0 , the coefficients of the 3×3 matrix The and determinants can be calculated using equation (5).

[0268] Applying a similar process to the other coefficients, all linear model parameters can be solved

[0269]

[0270] Using the estimated parameters, the affine merge candidates of the current CU can be derived using equation (2), where it is expressed as (mv tl ,mv tl ),(mv tr ,mv tr ),(mv bl ,mv bl )'s top-left, top-right, and bottom-left CPMVs are equal to:

[0271] (mv tl ,mv tl )=(b x2 ,b y2 )

[0272] (mv tr ,mv tr )=(b x0 W+b x2 ,b y0 W+b y2 )

[0273] (mvbl,mvbl)=(b x1 H+b x2 ,b y1 H+b y2 ),

[0274] Where W and H are the width and height of the current CU. The variables x and y in equation (2) are replaced by the relative distance from the upper left position of the current CU to the position of the corresponding CPMV.

[0275] Figure 5 is a block diagram illustrating an example video encoder 200 that may perform the techniques of this disclosure. Figure 5 The above description is provided for the purpose of explanation and should not be considered as limiting the techniques as broadly illustrated and described in this disclosure. For the purpose of explanation, this disclosure describes a video encoder 200 according to the techniques of VVC and HEVC. However, the techniques of this disclosure may be performed by video encoding devices configured for other video coding standards and video coding formats (e.g., AV1 and successors of the AV1 video coding format).

[0276] exist Figure 5 In the example of , the video encoder 200 includes a video data memory 230, a mode selection unit 202, a residual generation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a filter unit 216, a decoded picture buffer (DPB) 218, and an entropy coding unit 220. Any one or all of the video data memory 230, the mode selection unit 202, the residual generation unit 204, the transform processing unit 206, the quantization unit 208, the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the filter unit 216, the DPB 218, and the entropy coding unit 220 can be implemented in one or more processors or in processing circuits. For example, the units of the video encoder 200 can be implemented as one or more circuits or logic elements, as part of a hardware circuit, or as part of a processor, ASIC, or FPGA. In addition, the video encoder 200 may include additional or alternative processors or processing circuits to perform these and other functions.

[0277] Video data memory 230 may store video data to be encoded by components of video encoder 200. Video encoder 200 may receive video data from, for example, video source 104 ( Figure 1) receives video data stored in video data memory 230. DPB218 can act as a reference picture memory, which stores reference video data for predicting subsequent video data by video encoder 200. Video data memory 230 and DPB 218 can be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM)), magnetoresistive RAM (MRAM), resistive RAM (RRAM) or other types of memory devices. Video data memory 230 and DPB 218 can be provided by the same memory device or a separate memory device. In various examples, as shown, video data memory 230 can be on-chip with other components of video encoder 200, or off-chip relative to those components.

[0278] In the present disclosure, references to the video data memory 230 should not be interpreted as limited to memory inside the video encoder 200, unless specifically described as such, or should not be interpreted as limited to memory outside the video encoder 200, unless specifically described as such. Instead, references to the video data memory 230 should be understood as references to memory storing video data received by the video encoder 200 for encoding (e.g., video data of a current block to be encoded). Figure 1 The memory 106 may also provide temporary storage for outputs from the various units of the video encoder 200 .

[0279] Figure 5 Various units are shown to help understand the operations performed by the video encoder 200. The units may be implemented as fixed-function circuits, programmable circuits, or a combination thereof. Fixed-function circuits refer to circuits that provide specific functionality and are preset in the operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functions in the operations that can be performed. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by instructions of the software or firmware. Fixed-function circuits can execute software instructions (e.g., to receive parameters or output parameters), but the type of operation performed by the fixed-function circuit is generally immutable. In some examples, one or more of the units may be different circuit blocks (fixed-function or programmable), and in some examples, one or more of the units may be integrated circuits.

[0280] The video encoder 200 may include an arithmetic logic unit (ALU), an elementary function unit (EFU), a digital circuit, an analog circuit, and / or a programmable core formed by a programmable circuit. In an example where the operation of the video encoder 200 is performed using software executed by a programmable circuit, the memory 106 ( Figure 1) may store instructions (eg, object code) for software received and executed by the video encoder 200, or another memory (not shown) within the video encoder 200 may store such instructions.

[0281] The video data memory 230 is configured to store received video data. The video encoder 200 may retrieve pictures of the video data from the video data memory 230 and provide the video data to the residual generation unit 204 and the mode selection unit 202. The video data in the video data memory 230 may be original video data to be encoded.

[0282] Mode selection unit 202 includes motion estimation unit 222, motion compensation unit 224, and intra prediction unit 226. Mode selection unit 202 may include additional functional units for performing video prediction according to other prediction modes. As an example, mode selection unit 202 may include a palette unit, an intra-block copy unit (which may be part of motion estimation unit 222 and / or motion compensation unit 224), an affine unit, a linear model (LM) unit, or the like.

[0283] The mode selection unit 202 typically coordinates multiple encoding passes to test combinations of encoding parameters and the resulting rate-distortion values ​​of such combinations. The encoding parameters may include the partitioning of CTUs into CUs, the prediction mode of the CU, the transform type of the residual data of the CU, the quantization parameter of the residual data of the CU, etc. The mode selection unit 202 may ultimately select a combination of encoding parameters that has a better rate-distortion value than other tested combinations.

[0284] The video encoder 200 may partition a picture retrieved from the video data memory 230 into a series of CTUs and encapsulate one or more CTUs in a slice. The mode selection unit 202 may partition the CTUs of the picture according to a tree structure, such as an MTT structure, a QTBT structure, a super block structure, or a quadtree structure described above. As described above, the video encoder 200 may form one or more CUs by partitioning the CTUs according to the tree structure. This CU may also be generally referred to as a "video block" or "block".

[0285] In general, the mode selection unit 202 also controls its components (e.g., the motion estimation unit 222, the motion compensation unit 224, and the intra prediction unit 226) to generate a prediction block for the current block (e.g., the current CU, or in HEVC, the overlapping portion of the PU and TU). For inter prediction of the current block, the motion estimation unit 222 can perform a motion search to identify one or more closely matching reference blocks in one or more reference pictures (e.g., one or more previously coded pictures stored in the DPB 218). Specifically, the motion estimation unit 222 can calculate a value indicating how similar the potential reference block is to the current block, for example, based on the sum of absolute differences (SAD), the sum of squared differences (SSD), the mean absolute difference (MAD), the mean squared difference (MSD), etc. The motion estimation unit 222 can generally perform these calculations using the sample-by-sample difference between the current block and the reference block under consideration. The motion estimation unit 222 can identify the reference block with the lowest value produced by these calculations, thereby indicating the reference block that most closely matches the current block.

[0286] The motion estimation unit 222 may form one or more motion vectors (MVs) that define the position of a reference block in a reference picture relative to the position of a current block in a current picture. The motion estimation unit 222 may then provide the motion vectors to the motion compensation unit 224. For example, for unidirectional inter prediction, the motion estimation unit 222 may provide a single motion vector, while for bidirectional inter prediction, the motion estimation unit 222 may provide two motion vectors. The motion compensation unit 224 may then use the motion vectors to generate a prediction block. For example, the motion compensation unit 224 may use the motion vectors to retrieve data for the reference block. As another example, if the motion vector has fractional sample precision, the motion compensation unit 224 may interpolate the values ​​of the prediction block according to one or more interpolation filters. In addition, for bidirectional inter prediction, the motion compensation unit 224 may retrieve data for two reference blocks identified by the corresponding motion vectors and combine the retrieved data, for example, by sample-by-sample averaging or weighted averaging.

[0287] When operating according to the AV1 video codec format, the motion estimation unit 222 and the motion compensation unit 224 may be configured to encode codec blocks (e.g., both luma and chroma codec blocks) of the video data using translational motion compensation, affine motion compensation, overlapped block motion compensation (OBMC), and / or composite inter-intra prediction.

[0288] According to the technology of the present disclosure, as explained above, the motion estimation unit 222 can be configured to determine an affine motion model using a regression-based technique. In one example, the video encoder 200 can be configured to segment the codec block into sub-blocks. The motion estimation unit 222 can generate initial sub-block motion vectors for multiple sub-blocks, and use decoder-side motion vector refinement to refine the initial sub-block motion vectors for the multiple sub-blocks to produce refined sub-block motion vectors for the multiple sub-blocks. The motion estimation unit 222 can perform linear regression on the refined sub-block motion vectors and the coordinates of the multiple sub-blocks to derive an affine motion model. The motion compensation unit 224 can then encode the codec block using the affine motion model. For example, the motion compensation unit 224 can determine the control point motion vector from the derived affine motion model and encode the codec block using the determined control point motion vector.

[0289] As another example, for intra prediction or intra prediction codec, the intra prediction unit 226 can generate a prediction block from samples adjacent to the current block. For example, for directional mode, the intra prediction unit 226 can generally mathematically combine the values ​​of adjacent samples and pad these calculated values ​​across the current block in a defined direction to produce a prediction block. As another example, for DC mode, the intra prediction unit 226 can calculate the average of adjacent samples of the current block and generate a prediction block to include the resulting average for each sample of the prediction block.

[0290] When operating according to the AV1 video coding format, the intra prediction unit 226 may be configured to encode codec blocks (e.g., luma and chroma codec blocks) of video data using directional intra prediction, non-directional intra prediction, recursive filter intra prediction, luma to chroma (CFL) prediction, intra block copy (IBC), and / or palette mode. The mode selection unit 202 may include additional functional units for performing video prediction according to other prediction modes.

[0291] The mode selection unit 202 provides the prediction block to the residual generation unit 204. The residual generation unit 204 receives the original uncoded version of the current block from the video data memory 230 and receives the prediction block from the mode selection unit 202. The residual generation unit 204 calculates the sample-by-sample difference between the current block and the prediction block. The resulting sample-by-sample difference defines a residual block for the current block. In some examples, the residual generation unit 204 may also determine the difference between the sample values ​​in the residual block to generate the residual block using residual differential pulse code modulation (RDPCM). In some examples, the residual generation unit 204 may be formed using one or more subtractor circuits that perform binary subtraction.

[0292] In the example where the mode selection unit 202 partitions the CU into PUs, each PU may be associated with a luma prediction unit and a corresponding chroma prediction unit. The video encoder 200 and the video decoder 300 may support PUs of various sizes. As indicated above, the size of the CU may refer to the size of the luma codec block of the CU, and the size of the PU may refer to the size of the luma prediction unit of the PU. Assuming that the size of a particular CU is 2N×2N, the video encoder 200 may support PU sizes of 2N×2N or N×N for intra-frame prediction, and symmetric PU sizes of 2N×2N, 2N×N, N×2N, N×N, or similar sizes for inter-frame prediction. The video encoder 200 and the video decoder 300 may also support asymmetric partitioning of PU sizes of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-frame prediction.

[0293] In an example where the mode selection unit 202 does not further partition the CU into PUs, each CU may be associated with a luma codec block and a corresponding chroma codec block. As above, the size of a CU may refer to the size of the luma codec block of the CU. The video encoder 200 and the video decoder 300 may support CU sizes of 2N×2N, 2N×N, or N×2N.

[0294] For other video codec techniques, such as intra-block copy mode codec, affine mode codec, and linear model (LM) mode codec, as some examples, the mode selection unit 202 generates a prediction block for the current block being encoded via a corresponding unit associated with the codec technique. In some examples, such as palette mode codec, the mode selection unit 202 may not generate a prediction block, but instead generate syntax elements indicating how to reconstruct the block based on the selected palette. In such a mode, the mode selection unit 202 may provide these syntax elements to the entropy coding unit 220 for encoding.

[0295] As described above, the residual generation unit 204 receives video data of a current block and a corresponding prediction block. The residual generation unit 204 then generates a residual block for the current block. To generate the residual block, the residual generation unit 204 calculates the sample-by-sample difference between the prediction block and the current block.

[0296] The transform processing unit 206 applies one or more transforms to the residual block to generate a block of transform coefficients (referred to herein as a "transform coefficient block"). The transform processing unit 206 may apply various transforms to the residual block to form a transform coefficient block. For example, the transform processing unit 206 may apply a discrete cosine transform (DCT), a directional transform, a Karhunen-Loeve transform (KLT), or a conceptually similar transform to the residual block. In some examples, the transform processing unit 206 may perform multiple transforms on the residual block, for example, a primary transform and a secondary transform, such as a rotation transform. In some examples, the transform processing unit 206 does not apply a transform to the residual block.

[0297] When operating according to AV1, the transform processing unit 206 may apply one or more transforms to the residual block to generate a block of transform coefficients (referred to herein as a "transform coefficient block"). The transform processing unit 206 may apply various transforms to the residual block to form a transform coefficient block. For example, the transform processing unit 206 may apply a horizontal / vertical transform combination, which may include a discrete cosine transform (DCT), an asymmetric discrete sine transform (ADST), a flipped ADST (e.g., an ADST in reverse order), and an identity transform (IDTX). When the identity transform is used, the transform is skipped in one of the vertical or horizontal directions. In some examples, the transform process may be skipped.

[0298] The quantization unit 208 may quantize the transform coefficients in the transform coefficient block to produce a quantized transform coefficient block. The quantization unit 208 may quantize the transform coefficients of the transform coefficient block according to a quantization parameter (QP) value associated with the current block. The video encoder 200 (e.g., via the mode selection unit 202) may adjust the degree of quantization applied to the transform coefficient block associated with the current block by adjusting the QP value associated with the CU. Quantization may introduce information loss, and therefore, the quantized transform coefficients may have a lower precision than the original transform coefficients generated by the transform processing unit 206.

[0299] The inverse quantization unit 210 and the inverse transform processing unit 212 may apply inverse quantization and inverse transform to the quantized transform coefficient block, respectively, to reconstruct a residual block from the transform coefficient block. The reconstruction unit 214 may generate a reconstructed block corresponding to the current block (although possibly with a certain degree of distortion) based on the reconstructed residual block and the prediction block generated by the mode selection unit 202. For example, the reconstruction unit 214 may add samples of the reconstructed residual block to corresponding samples from the prediction block generated by the mode selection unit 202 to generate a reconstructed block.

[0300] Filter unit 216 may perform one or more filter operations on the reconstructed blocks. For example, filter unit 216 may perform a deblocking operation to reduce blocking artifacts along the edges of the CU. In some examples, the operations of filter unit 216 may be skipped.

[0301] When operating according to AV1, filter unit 216 may perform one or more filter operations on the reconstructed blocks. For example, filter unit 216 may perform a deblocking operation to reduce blocking artifacts along the edges of the CU. In other examples, filter unit 216 may apply a constrained directional enhancement filter (CDEF), which may be applied after deblocking and may include applying a non-separable nonlinear low-pass directional filter based on an estimated edge direction. Filter unit 216 may also include a loop recovery filter applied after CDEF, and may include a separable symmetric normalized Wiener filter or a dual self-guided filter.

[0302] The video encoder 200 stores the reconstructed blocks in the DPB 218. For example, in an example where the operation of the filter unit 216 is not performed, the reconstruction unit 214 can store the reconstructed blocks to the DPB 218. In an example where the operation of the filter unit 216 is performed, the filter unit 216 can store the filtered reconstructed blocks to the DPB 218. The motion estimation unit 222 and the motion compensation unit 224 can retrieve a reference picture formed by the reconstructed (and potentially filtered) blocks from the DPB 218 to perform inter-frame prediction on blocks of subsequently encoded pictures. In addition, the intra-frame prediction unit 226 can use the reconstructed blocks in the DPB 218 of the current picture to perform intra-frame prediction on other blocks in the current picture.

[0303] In general, the entropy coding unit 220 may entropy encode syntax elements received from other functional components of the video encoder 200. For example, the entropy coding unit 220 may entropy encode the quantized transform coefficient blocks from the quantization unit 208. As another example, the entropy coding unit 220 may entropy encode the prediction syntax elements (e.g., motion information for inter-frame prediction or intra-frame mode information for intra-frame prediction) from the mode selection unit 202. The entropy coding unit 220 may perform one or more entropy coding operations on the syntax elements (as another example of video data) to generate entropy-encoded data. For example, the entropy coding unit 220 may perform a context-adaptive variable length coding (CAVLC) operation, a CABAC operation, a variable-to-variable (V2V) length coding operation, a syntax-based context-adaptive binary arithmetic coding (SBAC) operation, a probability interval partitioning entropy (PIPE) coding operation, an exponential Golomb coding operation, or another type of entropy coding operation on the data. In some examples, the entropy coding unit 220 may operate in a bypass mode, in which the syntax elements are not entropy encoded.

[0304] The video encoder 200 may output a bitstream including entropy-encoded syntax elements required to reconstruct a block of a slice or a picture. Specifically, the entropy encoding unit 220 may output a bitstream.

[0305] According to AV1, the entropy coding unit 220 may be configured as a symbol-to-symbol adaptive multi-symbol arithmetic codec. The syntax elements in AV1 include an alphabet of N elements, and the context (e.g., a probability model) includes a set of N probabilities. The entropy coding unit 220 may store the probabilities as an n-bit (e.g., 15-bit) cumulative distribution function (CDF). The entropy coding unit 220 may perform recursive scaling to update the context with an update factor based on the size of the alphabet.

[0306] The operations described above are described with respect to blocks. Such descriptions should be understood as operations for luma codec blocks and / or chroma codec blocks. As described above, in some examples, the luma codec block and the chroma codec block are the luma component and the chroma component of the CU. In some examples, the luma codec block and the chroma codec block are the luma component and the chroma component of the PU.

[0307] In some examples, operations performed with respect to luma codec blocks need not be repeated for chroma codec blocks. As an example, operations for identifying a motion vector (MV) and a reference picture for a luma codec block need not be repeated to identify the MV and reference picture for a chroma block. In fact, the MV of the luma codec block may be scaled to determine the MV of the chroma block, and the reference picture may be the same. As another example, the intra prediction process may be the same for luma codec blocks and chroma codec blocks.

[0308] The video encoder 200 represents an example of a device configured to encode video data, the device including a memory configured to store the video data, and one or more processing units implemented in circuits and configured to partition a codec block into sub-blocks, generate initial sub-block motion vectors for a plurality of sub-blocks, refine the initial sub-block motion vectors for the plurality of sub-blocks using decoder-side motion vector refinement to produce refined sub-block motion vectors for the plurality of sub-blocks, and perform linear regression on the refined sub-block motion vectors and coordinates of the plurality of sub-blocks to derive an affine motion model.

[0309] Figure 6 is a block diagram illustrating an example video decoder 300 that may perform the techniques of this disclosure. Figure 6It is provided for the purpose of explanation and does not limit the techniques as widely illustrated and described in this disclosure. For the purpose of explanation, this disclosure describes a video decoder 300 according to the techniques of VVC and HEVC. However, the techniques of this disclosure can be performed by video codec devices configured for other video codec standards.

[0310] exist Figure 6 In the example of , the video decoder 300 includes a codec picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transform processing unit 308, a reconstruction unit 310, a filter unit 312, and a DPB 314. Any or all of the CPB memory 320, the entropy decoding unit 302, the prediction processing unit 304, the inverse quantization unit 306, the inverse transform processing unit 308, the reconstruction unit 310, the filter unit 312, and the DPB 314 can be implemented in one or more processors or in processing circuits. For example, the units of the video decoder 300 can be implemented as one or more circuits or logic elements, as part of a hardware circuit, or as part of a processor, ASIC, or FPGA. In addition, the video decoder 300 may include additional or alternative processors or processing circuits to perform these and other functions.

[0311] The prediction processing unit 304 includes a motion compensation unit 316 and an intra prediction unit 318. The prediction processing unit 304 may include additional units that perform predictions according to other prediction modes. As an example, the prediction processing unit 304 may include a palette unit, an intra-block copy unit (which may form part of the motion compensation unit 316), an affine unit, a linear model (LM) unit, etc. In other examples, the video decoder 300 may include more, fewer, or different functional components.

[0312] When operating according to AV1, the motion compensation unit 316 may be configured to decode codec blocks of video data (e.g., both luma and chroma codec blocks) using translational motion compensation, affine motion compensation, OBMC, and / or composite inter-intra prediction, as described above. The intra prediction unit 318 may be configured to decode codec blocks of video data (e.g., both luma and chroma codec blocks) using directional intra prediction, non-directional intra prediction, recursive filter intra prediction, CFL, IBC, and / or color palette mode, as described above.

[0313] According to the technology of the present disclosure, as explained above, the motion compensation unit 316 can be configured to determine the affine motion model using a regression-based technique. In one example, the video decoder 300 can be configured to segment the codec block into sub-blocks. The motion compensation unit 316 can generate initial sub-block motion vectors for multiple sub-blocks and refine the initial sub-block motion vectors for the multiple sub-blocks using decoder-side motion vector refinement to produce refined sub-block motion vectors for the multiple sub-blocks. The motion compensation unit 316 can perform linear regression on the refined sub-block motion vectors and the coordinates of the multiple sub-blocks to derive the affine motion model. The motion compensation unit 316 can then decode the codec block using the affine motion model. For example, the motion compensation unit 316 can determine the control point motion vectors from the derived affine motion model and decode the codec block using the determined control point motion vectors.

[0314] CPB memory 320 may store video data, such as an encoded video bitstream, to be decoded by components of video decoder 300. The video data may be, for example, stored from computer readable medium 110 ( Figure 1 ) obtains video data stored in CPB memory 320. CPB memory 320 may include a CPB that stores encoded video data (e.g., syntax elements) from an encoded video bitstream. In addition, CPB memory 320 may store video data other than syntax elements of encoded and decoded pictures, such as temporary data representing outputs from various units of video decoder 300. DPB 314 typically stores decoded pictures, and video decoder 300 may output and / or use decoded pictures as reference video data when decoding subsequent data or pictures of the encoded video bitstream. CPB memory 320 and DPB 314 may be formed by any of a variety of memory devices, such as DRAM, including SDRAM, MRAM, RRAM or other types of memory devices. CPB memory 320 and DPB 314 may be provided by the same memory device or a separate memory device. In various examples, CPB memory 320 may be on-chip with other components of video decoder 300, or off-chip relative to those components.

[0315] Additionally or alternatively, in some examples, video decoder 300 may retrieve the video from memory 120 ( Figure 1 ) retrieves the encoded and decoded video data. That is, memory 120 may store data, as discussed above with respect to CPB memory 320. Likewise, when some or all of the functionality of video decoder 300 is implemented in software to be executed by processing circuitry of video decoder 300, memory 120 may store instructions to be executed by video decoder 300.

[0316] Figure 6The various units shown in FIG. 3 are shown to aid in understanding the operations performed by the video decoder 300. The units may be implemented as fixed function circuits, programmable circuits, or a combination thereof. Figure 5 , fixed-function circuits refer to circuits that provide specific functionality and are preset in the operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functionality in the operations that can be performed. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by the instructions of the software or firmware. Fixed-function circuits can execute software instructions (for example, to receive parameters or output parameters), but the type of operations performed by the fixed-function circuits is generally immutable. In some examples, one or more of the units may be different circuit blocks (fixed-function or programmable), and in some examples, one or more of the units may be integrated circuits.

[0317] The video decoder 300 may include an ALU, an EFU, a digital circuit, an analog circuit, and / or a programmable core formed by a programmable circuit. In an example where the operation of the video decoder 300 is performed by software executed on a programmable circuit, an on-chip or off-chip memory may store instructions (e.g., object code) of the software received and executed by the video decoder 300.

[0318] Entropy decoding unit 302 may receive encoded video data from the CPB and entropy decode the video data to reproduce syntax elements. Prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, and filter unit 312 may generate decoded video data based on syntax elements extracted from the bitstream.

[0319] Typically, the video decoder 300 reconstructs a picture on a block-by-block basis. The video decoder 300 may perform a reconstruction operation on each block individually (where the block currently being reconstructed (ie, decoded) may be referred to as a "current block").

[0320] The entropy decoding unit 302 may entropy decode syntax elements defining quantized transform coefficients of the quantized transform coefficient block, as well as transform information, such as a quantization parameter (QP) and / or a transform mode indication. The inverse quantization unit 306 may use the QP associated with the quantized transform coefficient block to determine a degree of quantization, and likewise, determine a degree of inverse quantization to be applied by the inverse quantization unit 306. The inverse quantization unit 306 may, for example, perform a bitwise left shift operation to inverse quantize the quantized transform coefficients. The inverse quantization unit 306 may thereby form a transform coefficient block including the transform coefficients.

[0321] After the inverse quantization unit 306 forms the transform coefficient block, the inverse transform processing unit 308 may apply one or more inverse transforms to the transform coefficient block to generate a residual block associated with the current block. For example, the inverse transform processing unit 308 may apply an inverse DCT, an inverse integer transform, an inverse Karhunen-Loeve transform (KLT), an inverse rotational transform, an inverse directional transform, or another inverse transform to the transform coefficient block.

[0322] In addition, prediction processing unit 304 generates a prediction block based on the prediction information syntax element entropy decoded by entropy decoding unit 302. For example, if the prediction information syntax element indicates that the current block is inter-predicted, then motion compensation unit 316 can generate the prediction block. In this case, the prediction information syntax element can indicate a reference picture in DPB 314 from which to retrieve the reference block, and a motion vector that identifies the position of the reference block in the reference picture relative to the position of the current block in the current picture. Motion compensation unit 316 can generally be substantially similar to the process described with respect to motion compensation unit 224 ( Figure 5 ) performs the inter-frame prediction process in the manner described.

[0323] As another example, if the prediction information syntax element indicates that the current block is intra-predicted, then the intra-prediction unit 318 can generate a prediction block according to the intra-prediction mode indicated by the prediction information syntax element. Again, the intra-prediction unit 318 can generally be substantially similar to the intra-prediction unit 226 ( Figure 5 The intra prediction process is performed in the manner described in the above description. The intra prediction unit 318 can retrieve data of neighboring samples of the current block from the DPB 314.

[0324] The reconstruction unit 310 may reconstruct the current block using the prediction block and the residual block. For example, the reconstruction unit 310 may add samples of the residual block to corresponding samples of the prediction block to reconstruct the current block.

[0325] The filter unit 312 may perform one or more filter operations on the reconstructed block. For example, the filter unit 312 may perform a deblocking operation to reduce blocking artifacts along the edges of the reconstructed block. The operations of the filter unit 312 may not necessarily be performed in all examples.

[0326] The video decoder 300 may store the reconstructed block in the DPB 314. For example, in an example where the operation of the filter unit 312 is not performed, the reconstruction unit 310 may store the reconstructed block to the DPB 314. In an example where the operation of the filter unit 312 is performed, the filter unit 312 may store the filtered reconstructed block to the DPB 314. As discussed above, the DPB 314 may provide reference information (such as samples of the current picture for intra-frame prediction and samples of previously decoded pictures for subsequent motion compensation) to the prediction processing unit 304. In addition, the video decoder 300 may output a decoded picture (e.g., a decoded video) from the DPB 314 for subsequent display on a display device (such as a video device). Figure 1 is presented on a display device 118).

[0327] In this manner, the video decoder 300 represents an example of a video decoding device that includes a memory configured to store video data, and one or more processing units implemented in circuits and configured to partition a codec block into sub-blocks, generate initial sub-block motion vectors for a plurality of sub-blocks, refine the initial sub-block motion vectors for the plurality of sub-blocks using decoder-side motion vector refinement to produce refined sub-block motion vectors for the plurality of sub-blocks, and perform linear regression on the refined sub-block motion vectors and coordinates of the plurality of sub-blocks to derive an affine motion model.

[0328] Figure 7 1 is a flowchart illustrating an example method for encoding a current block according to the technology of the present disclosure. The current block may be or include a current CU. Although with respect to the video encoder 200 ( Figure 1 and Figure 5 ), but it should be understood that other devices may be configured to perform the same Figure 7 A similar approach to the one used in this paper.

[0329] In this example, the video encoder 200 initially predicts the current block (350). For example, the video encoder 200 may form a prediction block for the current block. The video encoder 200 may then calculate a residual block for the current block (352). In order to calculate the residual block, the video encoder 200 may calculate the difference between the original uncoded block and the prediction block for the current block. The video encoder 200 may then transform the residual block and quantize the transform coefficients of the residual block (354). Next, the video encoder 200 may scan the quantized transform coefficients of the residual block (356). During or after the scan, the video encoder 200 may entropy encode the transform coefficients (358). For example, the video encoder 200 may encode the transform coefficients using CAVLC or CABAC. The video encoder 200 may then output the entropy encoded data of the block (360).

[0330] Figure 8 1 is a flowchart illustrating an example method for decoding a current block of video data according to the techniques of the present disclosure. The current block may be or include a current CU. Although with respect to the video decoder 300 ( Figure 1 and Figure 6 ), but it should be understood that other devices may be configured to perform the same Figure 8 A similar approach to the one used in this paper.

[0331] The video decoder 300 may receive entropy-encoded data of the current block, such as entropy-encoded prediction information and entropy-encoded data of transform coefficients of a residual block corresponding to the current block (370). The video decoder 300 may entropy decode the entropy-encoded data to determine the prediction information of the current block and reproduce the transform coefficients of the residual block (372). The video decoder 300 may predict the current block (374), for example, using an intra-frame or inter-frame prediction mode indicated by the prediction information of the current block to calculate a prediction block of the current block. The video decoder 300 may then inverse scan the reproduced transform coefficients (376) to create a quantized transform coefficient block. The video decoder 300 may then inverse quantize the transform coefficients and apply an inverse transform to the transform coefficients to generate a residual block (378). The video decoder 300 may finally decode the current block (380) by combining the prediction block and the residual block.

[0332] Fig. 9 is a flowchart illustrating another example method for encoding a current block according to the techniques of this disclosure. Fig. 9 The technology can be Figure 7 The video encoder 200 including the motion estimation unit 22 and the motion compensation unit 224 may be configured to perform Fig. 9 technology.

[0333] The video encoder 200 may be configured to partition a codec block into sub-blocks (900) and generate initial sub-block motion vectors for a plurality of sub-blocks (902). In one example, the video encoder 200 may generate initial sub-block motion vectors based on control point motion vectors representing an input affine motion model. The video encoder 200 may determine the control point motion vectors from an affine candidate merge list. In other examples, the video encoder 200 may generate initial sub-block motion vectors based on bi-predicted motion vectors.

[0334] The video encoder 200 may also be configured to refine the initial sub-block motion vectors for the plurality of sub-blocks using decoder-side motion vector refinement to generate refined sub-block motion vectors for the plurality of sub-blocks 904. For example, the video encoder 200 may independently refine each initial sub-block motion vector using decoder-side motion vector refinement to generate refined sub-block motion vectors for the plurality of sub-blocks.

[0335] The video encoder 200 may then perform linear regression on the refined sub-block motion vectors and the coordinates of the plurality of sub-blocks to derive an affine motion model (906). The video encoder 200 may then encode the codec block using the affine motion model (908). For example, the video encoder 200 may derive a refined control point motion vector for the codec block based on the affine motion model and encode the codec block using the refined control point motion vector.

[0336] In other examples, the video encoder 200 may also be configured to derive motion vector offsets using the initial sub-block motion vectors and decoder-side motion vector refinement, wherein the decoder-side motion vector refinement includes determining a first cost for the motion vector offsets. The video encoder 200 may also be configured to determine a second cost for an affine motion model. The video encoder 200 may determine to use motion vector offsets or an affine motion model based on the first cost and the second cost.

[0337] The video encoder 200 may then encode the codec block using the affine motion model based on the first cost being greater than the second cost. The video encoder 200 may encode the codec block using the motion vector offset based on the first cost being less than the second cost. Using the motion vector offset may include adding the motion vector offset to the control point motion vector associated with the initial affine motion model to generate a refined control point motion vector, and encoding the codec block using the refined control point motion vector.

[0338] Fig.10 is a flowchart illustrating another example method for decoding a current block according to the techniques of this disclosure. Fig.10 The technology can be Figure 8 The video decoder 300 including the motion compensation unit 316 may be configured to perform Fig.10 technology.

[0339] The video decoder 300 may be configured to partition the codec block into sub-blocks (950) and generate initial sub-block motion vectors for the plurality of sub-blocks (952). In one example, the video decoder 300 may generate the initial sub-block motion vectors based on control point motion vectors representing an input affine motion model. The video decoder 300 may determine the control point motion vectors from an affine candidate merge list. In other examples, the video decoder 300 may generate the initial sub-block motion vectors based on bi-predicted motion vectors.

[0340] The video decoder 300 may also be configured to refine the initial sub-block motion vectors for the plurality of sub-blocks using decoder-side motion vector refinement to generate refined sub-block motion vectors for the plurality of sub-blocks (954). For example, the video decoder 300 may independently refine each initial sub-block motion vector using decoder-side motion vector refinement to generate refined sub-block motion vectors for the plurality of sub-blocks.

[0341] The video decoder 300 may then perform linear regression on the refined sub-block motion vectors and the coordinates of the plurality of sub-blocks to derive an affine motion model (956). The video decoder 300 may then decode the codec block using the affine motion model (958). For example, the video decoder 300 may derive a refined control point motion vector for the codec block based on the affine motion model and decode the codec block using the refined control point motion vector.

[0342] In other examples, the video decoder 300 may also be configured to derive a motion vector offset using the initial sub-block motion vector and a decoder-side motion vector refinement, wherein the decoder-side motion vector refinement includes determining a first cost for the motion vector offset. The video decoder 300 may also be configured to determine a second cost for an affine motion model. The video decoder 300 may determine whether to use a motion vector offset or an affine motion model based on the first cost and the second cost.

[0343] The video decoder 300 may then decode the codec block using the affine motion model based on the first cost being greater than the second cost. The video decoder 300 may decode the codec block using the motion vector offset based on the first cost being less than the second cost. Using the motion vector offset may include adding the motion vector offset to the control point motion vector associated with the initial affine motion model to generate a refined control point motion vector, and decoding the codec block using the refined control point motion vector.

[0344] The following numbered clauses illustrate one or more aspects of the devices and techniques described in this disclosure.

[0345] Aspect 1A - A method of encoding and decoding video data, the method comprising: dividing a codec block into sub-blocks; generating initial sub-block motion vectors for a plurality of sub-blocks; refining the initial sub-block motion vectors for the plurality of sub-blocks using bilateral matching to produce refined sub-block motion vectors for the plurality of sub-blocks; performing linear regression on the refined sub-block motion vectors and coordinates of the plurality of sub-blocks to derive an affine motion model; and encoding and decoding the codec block using the affine motion model.

[0346] Aspect 2A-The method according to aspect 1A, wherein the multiple sub-blocks are all sub-blocks of the codec block.

[0347] Aspect 3A-The method according to aspect 1A, wherein the plurality of sub-blocks are a subset of the sub-blocks of the codec block.

[0348] Aspect 4A-The method according to aspect 1A further includes: deriving the control point motion vector of the codec block based on the affine motion model.

[0349] Aspect 5A - A method for encoding and decoding video data, the method comprising: dividing a codec block into sub-blocks; generating initial sub-block motion vectors for a plurality of sub-blocks; refining the initial sub-block motion vectors for the plurality of sub-blocks using bilateral matching to produce refined sub-block motion vectors for the plurality of sub-blocks; determining an affine motion model based on the refined sub-block motion vectors; and encoding and decoding the codec block using the affine motion model.

[0350] Aspect 6A-The method according to aspect 5A, wherein the plurality of sub-blocks are all sub-blocks of the codec block.

[0351] Aspect 7A-The method according to aspect 5A, wherein the plurality of sub-blocks are a subset of the sub-blocks of the codec block.

[0352] Aspect 8A - The method according to aspect 5A, wherein determining the affine motion model based on the refined sub-block motion vector comprises: determining a control point motion vector of the codec block based on the refined sub-block motion vector.

[0353] Aspect 9A - A method according to any one of aspects 1A-8A, wherein encoding and decoding includes decoding.

[0354] Aspect 10A - A method according to any one of aspects 1A-8A, wherein encoding and decoding includes encoding.

[0355] Aspect 11A - An apparatus for encoding and decoding video data, the apparatus comprising one or more components for performing the method according to any of aspects 1A to 10A.

[0356] Aspect 12A - The apparatus of aspect 11A, wherein the one or more components include one or more processors implemented in circuitry.

[0357] Aspect 13A-The apparatus according to any of Aspects 11A and 12A, further comprising a memory for storing video data.

[0358] Aspect 14A - The apparatus of any of Aspects 11A to 13A, further comprising a display configured to display the decoded video data.

[0359] Aspect 15A—The device of any of Aspects 11A-14A, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

[0360] Aspect 16A - The apparatus of any of Aspects 11A-15A, wherein the apparatus comprises a video decoder.

[0361] Aspect 17A - The apparatus of any of Aspects 11A-16A, wherein the apparatus comprises a video encoder.

[0362] Aspect 18A - A computer-readable storage medium having stored thereon instructions which, when executed, cause one or more processors to perform a method according to any of aspects 1A-10A.

[0363] Aspect 19A - An apparatus for encoding video data, the apparatus comprising: means for dividing a codec block into sub-blocks; means for generating initial sub-block motion vectors for a plurality of sub-blocks; means for refining the initial sub-block motion vectors for the plurality of sub-blocks using bilateral matching to produce refined sub-block motion vectors for the plurality of sub-blocks; means for performing linear regression on the refined sub-block motion vectors and coordinates of the plurality of sub-blocks to derive an affine motion model; and means for encoding and decoding the codec block using the affine motion model.

[0364] Aspect 20A - An apparatus for encoding video data, the apparatus comprising: means for dividing a codec block into sub-blocks; means for generating initial sub-block motion vectors for a plurality of sub-blocks; means for refining the initial sub-block motion vectors for the plurality of sub-blocks using bilateral matching to produce refined sub-block motion vectors for the plurality of sub-blocks; means for determining an affine motion model based on the refined sub-block motion vectors; and means for encoding and decoding the codec block using the affine motion model.

[0365] Aspect 1B—A method of decoding video data, the method comprising: partitioning a codec block into sub-blocks; generating initial sub-block motion vectors for a plurality of sub-blocks; refining the initial sub-block motion vectors for the plurality of sub-blocks using decoder-side motion vector refinement to produce refined sub-block motion vectors for the plurality of sub-blocks; and performing linear regression on the refined sub-block motion vectors and coordinates of the plurality of sub-blocks to derive an affine motion model.

[0366] Aspect 2B-The method according to aspect 1B, wherein generating the initial sub-block motion vector comprises: generating the initial sub-block motion vector based on a control point motion vector representing an input affine motion model.

[0367] Aspect 3B-The method according to aspect 2B also includes: determining the control point motion vector from the affine candidate merge list.

[0368] Aspect 4B-The method according to aspect 1B, wherein generating the initial sub-block motion vector comprises: generating the initial sub-block motion vector based on a bi-prediction motion vector.

[0369] Aspect 5B - The method according to any one of aspects 1B-4B further includes: decoding the codec block using an affine motion model.

[0370] Aspect 6B-A method according to aspect 5B, wherein decoding the codec block using the affine motion model comprises: deriving a refined control point motion vector of the codec block based on the affine motion model; and decoding the codec block using the refined control point motion vector.

[0371] Aspect 7B - A method according to any one of Aspects 1B-6B, wherein refining the initial sub-block motion vectors of the multiple sub-blocks using decoder-side motion vector refinement to generate refined sub-block motion vectors for the multiple sub-blocks includes: independently refining each initial sub-block motion vector using decoder-side motion vector refinement to generate refined sub-block motion vectors for the multiple sub-blocks.

[0372] Aspect 8B-The method according to any one of Aspects 1B-7B also includes: deriving a motion vector offset using an initial sub-block motion vector and a decoder-side motion vector refinement, wherein the decoder-side motion vector refinement includes determining a first cost of the motion vector offset; determining a second cost of the affine motion model; and determining to use the motion vector offset or the affine motion model based on the first cost and the second cost.

[0373] Aspect 9B-The method according to aspect 8B further includes: based on the first cost being greater than the second cost, decoding the codec block using an affine motion model.

[0374] Aspect 10B-The method according to aspect 8B further includes: based on the first cost being less than the second cost, decoding the codec block using the motion vector offset.

[0375] Aspect 11B-A method according to aspect 10B, wherein decoding the codec block using the motion vector offset includes: adding the motion vector offset to the control point motion vector associated with the initial affine motion model to generate a refined control point motion vector; and decoding the codec block using the refined control point motion vector.

[0376] Aspect 12B - A method according to any of aspects 1B-11B, wherein the plurality of sub-blocks are all sub-blocks of the codec block.

[0377] Aspect 13B - A method according to any of aspects 1B to 11B, wherein the plurality of sub-blocks are a subset of the sub-blocks of the codec block.

[0378] Aspect 14B—A device configured to decode video data, the device comprising: a memory; and one or more processors in communication with the memory, the one or more processors configured to: partition a codec block into sub-blocks; generate initial sub-block motion vectors for a plurality of sub-blocks; refine the initial sub-block motion vectors for the plurality of sub-blocks using decoder-side motion vector refinement to produce refined sub-block motion vectors for the plurality of sub-blocks; and perform linear regression on the refined sub-block motion vectors and coordinates of the plurality of sub-blocks to derive an affine motion model.

[0379] Aspect 15B - The apparatus according to aspect 14B, wherein to generate the initial sub-block motion vector, the one or more processors are further configured to: generate the initial sub-block motion vector based on the control point motion vector representing the input affine motion model.

[0380] Aspect 16B - The apparatus of aspect 15B, wherein the one or more processors are further configured to: determine a control point motion vector from the affine candidate merge list.

[0381] Aspect 17B - The apparatus according to aspect 14B, wherein, to generate the initial sub-block motion vector, the one or more processors are further configured to: generate the initial sub-block motion vector based on the bi-predicted motion vector.

[0382] Aspect 18B - An apparatus according to any one of Aspects 14B to 17B, wherein the one or more processors are further configured to: decode the codec block using an affine motion model.

[0383] Aspect 19B - An apparatus according to Aspect 18B, wherein, in order to decode the codec block using an affine motion model, one or more processors are further configured to: derive a refined control point motion vector of the codec block based on the affine motion model; and decode the codec block using the refined control point motion vector.

[0384] Aspect 20B - An apparatus according to any one of Aspects 14B-19B, wherein, in order to refine the initial sub-block motion vectors of multiple sub-blocks using decoder-side motion vector refinement to generate refined sub-block motion vectors for the multiple sub-blocks, the one or more processors are further configured to: independently refine each initial sub-block motion vector using decoder-side motion vector refinement to generate refined sub-block motion vectors for the multiple sub-blocks.

[0385] Aspect 21B - An apparatus according to any one of Aspects 14B-20B, wherein the one or more processors are further configured to: derive a motion vector offset using an initial sub-block motion vector and a decoder-side motion vector refinement, wherein the decoder-side motion vector refinement includes determining a first cost of the motion vector offset; determining a second cost of an affine motion model; and determining to use the motion vector offset or the affine motion model based on the first cost and the second cost.

[0386] Aspect 22B - The apparatus according to Aspect 21B, wherein the one or more processors are further configured to: decode the codec block using an affine motion model based on the first cost being greater than the second cost.

[0387] Aspect 23B. The apparatus of aspect 21B, wherein the one or more processors are further configured to: decode the codec block using the motion vector offset based on the first cost being less than the second cost.

[0388] Aspect 24B - An apparatus according to Aspect 23B, wherein, in order to use the motion vector offset to decode the codec block, one or more processors are further configured to: add the motion vector offset to the control point motion vector associated with the initial affine motion model to generate a refined control point motion vector; and decode the codec block using the refined control point motion vector.

[0389] Aspect 25B - An apparatus according to any of aspects 14B-24B, wherein the plurality of sub-blocks are all sub-blocks of the codec block.

[0390] Aspect 26B - An apparatus according to any of aspects 14B-24B, wherein the plurality of sub-blocks are a subset of the sub-blocks of the codec block.

[0391] Aspect 27B—An apparatus configured to decode video data, the apparatus comprising: means for partitioning a codec block into sub-blocks; means for generating initial sub-block motion vectors for a plurality of sub-blocks; means for refining the initial sub-block motion vectors for the plurality of sub-blocks using decoder-side motion vector refinement to produce refined sub-block motion vectors for the plurality of sub-blocks; and means for performing linear regression on the refined sub-block motion vectors and coordinates of the plurality of sub-blocks to derive an affine motion model.

[0392] Aspect 28B - The apparatus according to Aspect 27B further includes: a component for deriving a motion vector offset using the initial sub-block motion vector and a decoder-side motion vector refinement, wherein the decoder-side motion vector refinement includes determining a first cost for the motion vector offset; a component for determining a second cost for an affine motion model; and a component for determining whether to use the motion vector offset or the affine motion model based on the first cost and the second cost.

[0393] Aspect 29B - A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to: partition a codec block into sub-blocks; generate initial sub-block motion vectors for a plurality of sub-blocks; refine the initial sub-block motion vectors for the plurality of sub-blocks using decoder-side motion vector refinement to produce refined sub-block motion vectors for the plurality of sub-blocks; and perform linear regression on the refined sub-block motion vectors and coordinates of the plurality of sub-blocks to derive an affine motion model.

[0394] Aspect 30B-A non-transitory computer-readable storage medium according to Aspect 29B, wherein the instructions further cause one or more processors to: derive a motion vector offset using an initial sub-block motion vector and a decoder-side motion vector refinement, wherein the decoder-side motion vector refinement comprises determining a first cost of the motion vector offset; determining a second cost of an affine motion model; and determining to use the motion vector offset or the affine motion model based on the first cost and the second cost.

[0395] It should be appreciated that, depending on the example, certain actions or events of any of the techniques described herein may be performed in a different order, may be added, combined, or omitted entirely (e.g., not all described actions or events are required to practice the techniques). Furthermore, in some examples, actions or events may be performed simultaneously rather than sequentially, for example, through multithreading, interrupt processing, or multiple processors.

[0396] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or sent via a computer-readable medium as one or more instructions or codes and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium including any medium that facilitates, for example, the transfer of a computer program from one place to another according to a communication protocol. In this manner, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for the implementation of the techniques described in the present disclosure. A computer program product may include a computer-readable medium.

[0397] By way of example and not limitation, such computer-readable storage media may include one or more of RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and can be accessed by a computer. In addition, any connection is appropriately referred to as a computer-readable medium. For example, if instructions are sent from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (such as infrared, radio, and microwave), then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology (such as infrared, radio, and microwave) is included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other temporary media, but are actually directed to non-temporary tangible storage media. As used herein, disks and optical disks include compact disks (CDs), laser optical disks, optical optical disks, digital versatile disks (DVDs), floppy disks, and Blu-ray disks, where disks typically reproduce data magnetically, while optical disks reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0398] Instructions may be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. Therefore, the terms "processor" and "processing circuitry" as used herein may refer to any of the foregoing structures or any other structure suitable for the implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided in dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. In addition, these techniques may be fully implemented in one or more circuits or logic elements.

[0399] The techniques of the present disclosure may be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chipsets). Various components, modules, or units are described in the present disclosure to emphasize functional aspects of devices configured to perform the techniques of the present disclosure, but do not necessarily need to be implemented by different hardware units. Specifically, as described above, the various units may be combined in a codec hardware unit in conjunction with appropriate software and / or firmware, or provided by a collection of interoperable hardware units (including one or more processors as described above).

[0400] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method for decoding video data, the method comprising: Splitting the codec block into sub-blocks; generating initial sub-block motion vectors for a plurality of sub-blocks; refining initial sub-block motion vectors for the plurality of sub-blocks using decoder-side motion vector refinement to generate refined sub-block motion vectors for the plurality of sub-blocks; as well as A linear regression is performed on the refined sub-block motion vectors and the coordinates of the plurality of sub-blocks to derive an affine motion model.

2. The method according to claim 1, wherein: Generating the initial sub-block motion vector comprises: The initial sub-block motion vectors are generated based on control point motion vectors representing an input affine motion model.

3. The method according to claim 2, further comprising: The control point motion vector is determined from an affine candidate merge list.

4. The method according to claim 1, wherein: Generating the initial sub-block motion vector comprises: The initial sub-block motion vector is generated based on the bi-predicted motion vector.

5. The method according to claim 1, further comprising: The codec block is decoded using the affine motion model.

6. The method according to claim 5, wherein: Decoding the codec block using the affine motion model includes: deriving a refined control point motion vector of the codec block based on the affine motion model; and The codec block is decoded using the refined control point motion vector.

7. The method according to claim 1, wherein: Refining initial sub-block motion vectors for the plurality of sub-blocks using decoder-side motion vector refinement to generate refined sub-block motion vectors for the plurality of sub-blocks, comprising: Each of the initial sub-block motion vectors is independently refined using decoder-side motion vector refinement to produce refined sub-block motion vectors for the plurality of sub-blocks.

8. The method according to claim 1, further comprising: deriving a motion vector offset using the initial sub-block motion vector and a decoder-side motion vector refinement, wherein the decoder-side motion vector refinement comprises determining a first cost for the motion vector offset; determining a second cost for the affine motion model; and Using the motion vector offset or the affine motion model is determined based on the first cost and the second cost.

9. The method according to claim 8, further comprising: Based on the first cost being greater than the second cost, the codec block is decoded using the affine motion model.

10. The method according to claim 8, further comprising: Based on the first cost being less than the second cost, the codec block is decoded using the motion vector offset.

11. The method according to claim 10, wherein: Decoding the codec block using the motion vector offset includes: adding the motion vector offset to a control point motion vector associated with an initial affine motion model to generate a refined control point motion vector; and The codec block is decoded using the refined control point motion vector.

12. The method according to claim 1, wherein: The multiple sub-blocks are all sub-blocks of the codec block.

13. The method according to claim 1, wherein: The plurality of sub-blocks are a subset of the sub-blocks of the codec block.

14. An apparatus configured to decode video data, the apparatus comprising: Memory; as well as one or more processors in communication with the memory, the one or more processors configured to: Splitting the codec block into sub-blocks; generating initial sub-block motion vectors for a plurality of sub-blocks; refining initial sub-block motion vectors for the plurality of sub-blocks using decoder-side motion vector refinement to generate refined sub-block motion vectors for the plurality of sub-blocks; as well as A linear regression is performed on the refined sub-block motion vectors and the coordinates of the plurality of sub-blocks to derive an affine motion model.

15. The device according to claim 14, wherein: To generate the initial sub-block motion vector, the one or more processors are further configured to: The initial sub-block motion vectors are generated based on control point motion vectors representing an input affine motion model.

16. The device according to claim 15, wherein: The one or more processors are further configured to: The control point motion vector is determined from an affine candidate merge list.

17. The device according to claim 14, wherein: To generate the initial sub-block motion vector, the one or more processors are further configured to: The initial sub-block motion vector is generated based on the bi-predicted motion vector.

18. The device according to claim 14, wherein: The one or more processors are further configured to: The codec block is decoded using the affine motion model.

19. The device according to claim 18, wherein: To decode the codec block using the affine motion model, the one or more processors are further configured to: deriving a refined control point motion vector of the codec block based on the affine motion model; and The codec block is decoded using the refined control point motion vector.

20. The device according to claim 14, wherein: To refine the initial sub-block motion vectors for the plurality of sub-blocks using decoder-side motion vector refinement to generate refined sub-block motion vectors for the plurality of sub-blocks, the one or more processors are further configured to: Each of the initial sub-block motion vectors is independently refined using decoder-side motion vector refinement to produce refined sub-block motion vectors for the plurality of sub-blocks.

21. The device according to claim 14, wherein: The one or more processors are further configured to: deriving a motion vector offset using the initial sub-block motion vector and a decoder-side motion vector refinement, wherein the decoder-side motion vector refinement comprises determining a first cost for the motion vector offset; determining a second cost for the affine motion model; and Using the motion vector offset or the affine motion model is determined based on the first cost and the second cost.

22. The device according to claim 21, wherein The one or more processors are further configured to: Based on the first cost being greater than the second cost, the codec block is decoded using the affine motion model.

23. The device according to claim 21, wherein The one or more processors are further configured to: Based on the first cost being less than the second cost, the codec block is decoded using the motion vector offset.

24. The device according to claim 23, wherein: To decode the codec block using the motion vector offset, the one or more processors are further configured to: adding the motion vector offset to a control point motion vector associated with an initial affine motion model to generate a refined control point motion vector; and The codec block is decoded using the refined control point motion vector.

25. The device according to claim 14, wherein: The multiple sub-blocks are all sub-blocks of the codec block.

26. The device according to claim 14, wherein: The plurality of sub-blocks are a subset of the sub-blocks of the codec block.

27. A method of encoding video data, the method comprising: Splitting the codec block into sub-blocks; generating initial sub-block motion vectors for a plurality of sub-blocks; refining initial sub-block motion vectors for the plurality of sub-blocks using decoder-side motion vector refinement to generate refined sub-block motion vectors for the plurality of sub-blocks; performing linear regression on the refined sub-block motion vectors and the coordinates of the plurality of sub-blocks to derive an affine motion model; and The codec block is encoded using the affine motion model.

28. The method according to claim 27, wherein: Encoding the codec block using the affine motion model includes: deriving a refined control point motion vector of the codec block based on the affine motion model; and The codec block is encoded using the refined control point motion vector.

29. An apparatus configured to encode video data, the apparatus comprising: Memory; as well as one or more processors in communication with the memory, the one or more processors configured to: Splitting the codec block into sub-blocks; generating initial sub-block motion vectors for a plurality of sub-blocks; refining initial sub-block motion vectors for the plurality of sub-blocks using decoder-side motion vector refinement to generate refined sub-block motion vectors for the plurality of sub-blocks; performing linear regression on the refined sub-block motion vectors and the coordinates of the plurality of sub-blocks to derive an affine motion model; and The codec block is encoded using the affine motion model.

30. The device according to claim 29, wherein: To encode the codec block using the affine motion model, the one or more processors are further configured to: deriving a refined control point motion vector of the codec block based on the affine motion model; and The codec block is encoded using the refined control point motion vector.

Images