Gradient-based prediction refinement for video coding

Abstract

The present disclosure describes gradient-based prediction refinement. A video coder (e.g., a video encoder or a video decoder) determines one or more prediction blocks for inter-predicting a current block (e.g., based on one or more motion vectors for the current block). In gradient-based prediction refinement, the video coder modifies one or more samples of the prediction blocks based on various factors such as displacement in the horizontal direction, a horizontal gradient, displacement in the vertical direction, and a vertical gradient. The present disclosure provides a gradient-based prediction refinement (e.g., identical) in which the precision level of the displacement (e.g., at least one of horizontal or vertical displacement) is unified for different prediction modes (e.g., including affine mode and bidirectional optical flow (BDOF) mode).

Description

Technology Field

[0001] The present application claims priority to U.S. application No. 16 / 874,057 filed on May 14, 2020, which claims the benefit of U.S. provisional application No. 62 / 849,352 filed on May 17, 2019, and the entire contents of each of these applications are incorporated herein by reference.

[0002] Technology field

[0003] The present disclosure relates to video encoding and video decoding. Background Technology

[0004] Digital video capabilities can be integrated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite wireless phones, so-called "smartphones," video teleconferencing devices, video streaming devices, etc. Digital video devices implement video coding techniques such as those described in standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265 / High Efficiency Video Coding (HEVC), and extensions of such standards. By implementing such video coding techniques, video devices may more efficiently transmit, receive, encode, decode, and / or store digital video information.

[0005] Video coding techniques include spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to reduce or eliminate redundancy inherent in video sequences. For block-based video coding, a video slice (e.g., a video picture or a portion of a video picture) may be partitioned into video blocks, which may also be referred to as coding tree units (CTUs), coding units (CUs), and / or coding nodes. Video blocks in an intra-coded (I) slice of a picture are encoded using spatial prediction for reference samples in neighboring blocks of the same picture. Video blocks in an inter-coded (P or B) slice of a picture may use spatial prediction for reference samples in neighboring blocks of the same picture, or temporal prediction for reference samples in other reference pictures. Pictures may be referred to as frames, and reference pictures may be referred to as reference frames. means of solving the problem

[0006] Generally, the present disclosure describes techniques for gradient-based prediction refinement. A video coder (e.g., a video encoder or a video decoder) determines one or more prediction blocks for inter-predicting the current block (e.g., based on one or more motion vectors for the current block). In gradient-based prediction refinement, the video coder modifies one or more samples of the prediction blocks based on various factors such as displacement in the horizontal direction, horizontal gradient, displacement in the vertical direction, and vertical gradient.

[0007] For example, a motion vector identifies a prediction block. Displacement in the horizontal direction (also referred to as horizontal displacement) refers to a change in the x-coordinate of the motion vector (e.g., delta), and displacement in the vertical direction (also referred to as vertical displacement) refers to a change in the y-coordinate (e.g., delta). The horizontal gradient refers to the result of applying a filter to a first set of samples in the prediction block, and the vertical gradient refers to the result of applying a filter to a second set of samples in the prediction block.

[0008] The exemplary techniques described in this disclosure provide gradient-based prediction refinement that is unified (e.g., identical) for prediction modes in which the precision level of displacement (e.g., at least one of horizontal or vertical displacement) is different. For example, for a first prediction mode (e.g., affine mode), motion vectors may be at a first precision level, and for a second prediction mode (e.g., bidirectional optical flow (BDOF)), motion vectors may be at a second precision level. Accordingly, the vertical and horizontal displacements for motion vectors used in the affine mode and motion vectors used in the BDOF may be different. In this disclosure, a video coder may be configured to round (e.g., round up or round down) the vertical and horizontal displacements for motion vectors so that the precision level of the displacements is the same regardless of the prediction mode (e.g., so that the vertical and horizontal displacements for the affine mode and BDOF have the same precision level).

[0009] By rounding the precision levels of displacements, exemplary techniques can improve the overall operation of the video coder. For example, gradient-based predictive refinement involves multiplication and shift operations. If the precision levels of displacements differ for different modes, different logic circuits may be required to support different precision levels (e.g., a logic circuit configured for one precision level may not be suitable for other precision levels). Since the precision levels for displacements are the same for different modes, the same logic circuit can be reused for blocks, resulting in a smaller overall logic circuit and reduced power consumption because there is no need to power unused logic circuits.

[0010] In some examples, techniques for determining displacement may be based on information already available in the video decoder. For example, the way the video decoder determines horizontal or vertical displacement may be based on information available to the video decoder to inter-predict the current block according to the inter-predict mode. Furthermore, there may be specific inter-predict modes that are disabled for specific block types (e.g., based on size). In some examples, these inter-predict modes that were disabled for specific block types may be enabled for these block types, but the prediction blocks for such blocks may be modified using the exemplary techniques described in this disclosure.

[0011] In one example, the present disclosure describes a method for decoding video data, the method comprising: determining a prediction block for inter-predicting a current block; determining horizontal and vertical displacements for gradient-based prediction refinement of one or more samples of the prediction block; rounding the horizontal and vertical displacements to the same level of precision for different inter-prediction modes, including affine mode and BDOF mode; determining one or more refinement offsets based on the rounded horizontal and vertical displacements; modifying one or more samples of the prediction block based on the determined one or more refinement offsets to generate a modified prediction block; and restoring the current block based on the modified prediction block.

[0012] In one example, the present disclosure describes a method for encoding video data, the method comprising: determining a prediction block for inter-predicting a current block; determining horizontal and vertical displacements for gradient-based prediction refinement of one or more samples of the prediction block; rounding the horizontal and vertical displacements to the same level of precision for different inter-prediction modes, including affine mode and BDOF mode; determining one or more refinement offsets based on the rounded horizontal and vertical displacements; modifying one or more samples of the prediction block based on the determined one or more refinement offsets to generate a modified prediction block; determining residual values ​​representing the difference between the current block and the modified prediction block; and signaling information representing the residual values.

[0013] In one example, the present disclosure describes a device for coding video data, the device comprising a memory and a processing circuit configured to store one or more samples of a prediction block. The processing circuit is configured to determine a prediction block for inter-predicting a current block, determine horizontal and vertical displacements for gradient-based prediction refinement of one or more samples of the prediction block, round the horizontal and vertical displacements to the same level of precision for different inter-prediction modes including affine mode and BDOF mode, determine one or more refinement offsets based on the rounded horizontal and vertical displacements, modify one or more samples of the prediction block based on the determined one or more refinement offsets to generate a modified prediction block, and code the current block based on the modified prediction block.

[0014] In one example, the present disclosure describes a computer-readable storage medium storing instructions, the instructions which, when executed, cause one or more processors to determine a prediction block for inter-predicting a current block, to determine horizontal and vertical displacements for gradient-based prediction refinement of one or more samples of the prediction block, to round the horizontal and vertical displacements to the same level of precision for different inter-prediction modes including affine mode and BDOF mode, to determine one or more refinement offsets based on the rounded horizontal and vertical displacements, to modify one or more samples of the prediction block based on the determined one or more refinement offsets to generate a modified prediction block, and to code the current block based on the modified prediction block.

[0015] In one example, the present disclosure describes a device for coding video data, the device comprising: means for determining a prediction block for inter-predicting a current block; means for determining horizontal and vertical displacements for gradient-based prediction refinement of one or more samples of the prediction block; means for rounding the horizontal and vertical displacements to the same level of precision for different inter-prediction modes, including an affine mode and a BDOF mode; means for determining one or more refinement offsets based on the rounded horizontal and vertical displacements; means for modifying one or more samples of the prediction block based on the determined one or more refinement offsets to generate a modified prediction block; and means for coding a current block based on the modified prediction block.

[0016] Details of one or more examples are described in the accompanying drawings and the description below. Other features, purposes, and advantages will be apparent from the description, drawings, and claims. Brief explanation of the drawing

[0017] FIG. 1 is a block diagram illustrating an exemplary video encoding and decoding system capable of performing the techniques of the present disclosure. FIGS. 2a and 2b are conceptual diagrams illustrating an exemplary quadtree binary tree (QTBT) structure and a corresponding coding tree unit (CTU). FIG. 3 is a block diagram illustrating an exemplary video encoder capable of performing the techniques of the present disclosure. FIG. 4 is a block diagram illustrating an exemplary video decoder capable of performing the techniques of the present disclosure. Figure 5 is a block diagram illustrating an extended coding unit (CU) area used in a bidirectional optical flow (BDOF). Figure 6 is a conceptual diagram illustrating an example of selecting subblock motion vectors (MVs). Figure 7 is a flowchart illustrating an exemplary method for coding video data. Specific details for implementing the invention

[0018] The present invention relates to gradient-based prediction refinement. In gradient-based prediction refinement, a video coder (e.g., a video encoder or a video decoder) determines a prediction block for a current block based on motion vectors as part of inter-prediction, and modifies (e.g., refines) samples of the prediction block to generate modified prediction samples (e.g., refined prediction samples). The video encoder signals residual values ​​representing the difference between the modified prediction samples and the current block. The video decoder performs the same operations as those performed by the video encoder to modify the samples of the prediction block to generate modified prediction samples. The video decoder adds residual values ​​to the modified prediction samples to restore the current block.

[0019] One exemplary method for modifying samples in a prediction block is for the video coder to determine one or more refinement offsets and add the samples in the prediction block to the refinement offsets. One exemplary method for generating refinement offsets is based on gradients and motion vector displacements. The gradients may be determined from gradient filters applied to the samples in the prediction block.

[0020] Examples of motion vector displacements include horizontal displacement and vertical displacement relative to the motion vector. Horizontal displacement may be a value added to or subtracted from the x-coordinate of the motion vector, and vertical displacement may be a value added to or subtracted from the y-coordinate of the motion vector. For example, the horizontal displacement is Δv x It may also be referred to as, where, v x is the x-coordinate of the motion vector, and the vertical displacement is Δv yIt may also be referred to as, where, v y is the y-coordinate of the motion vector.

[0021] The precision level of the motion vector of the current block may differ for different inter-prediction modes. For example, the coordinates of the motion vector (e.g., x or y coordinates) may include an integer part and a fractional part. The fractional part is referred to as the sub-pel part of the motion vector because the integer part of the motion vector identifies the actual pixel in the reference picture containing the prediction block, and the sub-pel part of the motion vector adjusts the motion vector to identify the position between the pixels in the reference picture.

[0022] The precision level of a motion vector is based on the sub-pel portion of the motion vector and represents the granularity of the motion vector's movement from the actual pixels in the reference picture. For example, if the sub-pel portion of the x-coordinate is 0.5, the motion vector is half the distance between two horizontal pixels in the reference picture. If the sub-pel portion of the x-coordinate is 0.25, the motion vector is one-quarter the distance between two horizontal pixels, and so on. In these examples, the precision level of the motion vector may be equal to the sub-pel portion (e.g., the precision level is 0.5, 0.25, etc.).

[0023] In some examples, the precision level of the horizontal and vertical displacements may be based on the way the motion vector was generated or the precision level of the motion vector. For example, in some examples such as the merge mode, which is a form of inter-prediction mode, the sub-parts of the x and y coordinates of the motion vector may be the horizontal and vertical displacements, respectively. In another example, such as for the affine mode, which is a form of inter-prediction, the motion vector may be based on corner point motion vectors, and the horizontal and vertical displacements may be determined based on the corner point motion vectors.

[0024] The precision levels of horizontal and vertical displacements may differ for different inter-prediction modes. For example, for some inter-prediction modes, horizontal and vertical displacements may be more precise than for other inter-prediction modes (e.g., the precision level is 1 / 16 for the second prediction mode) (e.g., the precision level is 1 / 128 for the first prediction mode).

[0025] In implementation, the video coder may need to include different logic circuits to handle different precision levels. Performing gradient-based prediction refinement involves multiplication, shift operations, addition, and other arithmetic operations. A logic circuit configured for one precision level for horizontal or vertical displacements may not be able to process horizontal and vertical displacements of a higher precision level. Accordingly, some video coders include one set of logic circuits for performing gradient-based prediction refinement for one inter-prediction mode in which horizontal and vertical displacements have a first precision level, and a different set of logic circuits for performing gradient-based prediction refinement for another inter-prediction mode in which horizontal and vertical displacements have a second precision level.

[0026] However, having different logic circuits for performing gradient-based prediction refinement for different inter-prediction modes not only increases the size of the video coder but also results in additional logic circuits that utilize additional power. For example, when the current block is inter-predicted in the first mode, a first set of logic circuits for gradient-based prediction refinement is used. However, a second set of logic circuits for gradient-based prediction refinement for different inter-prediction modes still receives power.

[0027] The present disclosure describes examples of techniques for rounding precision levels for horizontal and vertical displacements to the same precision level for different inter-prediction modes. For example, a video coder may round a first displacement (e.g., a first horizontal or first vertical displacement) having a first precision level for a first block inter-predicted in a first inter-prediction mode to a set precision level, and may round a second displacement (e.g., a second horizontal or second vertical displacement) having a second precision level for a second block inter-predicted in a second inter-prediction mode to the same set precision level. Described in another way, the video coder may round at least one of the horizontal and vertical displacements to the same precision level for different inter-prediction modes. As an example, the first inter-prediction mode may be an affine mode, and the second inter-prediction mode may be a bidirectional optical flow (BDOF).

[0028] In this way, rather than having different logic circuits for different inter-prediction modes, the same logic circuit may be used for gradient-based prediction refinement for different inter-prediction modes. For example, the logic circuit of a video coder may be configured to perform gradient-based prediction refinement for horizontal and vertical displacements having a set precision level. The video coder may also round the horizontal and vertical displacements so that the precision level of the rounded horizontal and vertical displacements is the same as the set precision level, thereby allowing the same logic circuit to perform gradient-based prediction refinement for different inter-prediction modes.

[0029] FIG. 1 is a block diagram illustrating an exemplary video encoding and decoding system (100) capable of performing the techniques of the present disclosure. The techniques of the present disclosure generally relate to coding (encoding and / or decoding) video data. Generally, video data includes any data for processing video. Accordingly, video data may include raw, unencoded video, encoded video, decoded (e.g., restored) video, and video metadata such as signaling data.

[0030] As illustrated in FIG. 1, the system (100) includes, in this example, a source device (102) that provides encoded video data to be decoded and displayed by a destination device (116). In particular, the source device (102) provides the video data to the destination device (116) via a computer-readable medium (110). The source device (102) and the destination device (116) may include any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets, e.g., smartphones, televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, broadcast receiver devices, set-top boxes, etc. In some cases, the source device (102) and the destination device (116) may be equipped for wireless communication and thus may be referred to as wireless communication devices.

[0031] In the example of FIG. 1, the source device (102) includes a video source (104), memory (106), a video encoder (200), and an output interface (108). The destination device (116) includes an input interface (122), a video decoder (300), memory (120), and a display device (118). According to the present disclosure, the video encoder (200) of the source device (102) and the video decoder (300) of the destination device (116) may be configured to apply techniques for gradient-based predictive refinement. Thus, the source device (102) represents an example of a video encoding device, while the destination device (116) represents an example of a video decoding device. In other examples, the source device and the destination device may include other components or arrangements. For example, the source device (102) may receive video data from an external video source, such as an external camera. Likewise, the destination device (116) may interface with an external display device rather than including an integrated display device.

[0032] The system (100) as illustrated in FIG. 1 is merely one example. In general, any digital video encoding and / or decoding device may perform techniques for gradient-based predictive refinement. The source device (102) and the destination device (116) are merely examples of such coding devices in which the source device (102) generates coded video data for transmission to the destination device (116). The present disclosure refers to a "coding" device as a device that performs the coding (encoding and / or decoding) of data. Accordingly, the video encoder (200) and the video decoder (300) represent examples of coding devices, in particular a video encoder and a video decoder, respectively. In some examples, the devices (102, 116) may operate in a substantially symmetric manner such that each of the devices (102, 116) includes video encoding and decoding components. Accordingly, the system (100) may support unidirectional or bidirectional video transmission between video devices (102, 116) for, for example, video streaming, video playback, video broadcasting, or video calling.

[0033] Generally, the video source (104) represents a source of video data (i.e., raw, unencoded video data) and provides a sequential series of pictures of video data (also referred to as “frames”) to a video encoder (200) that encodes data for the pictures. The video source (104) of the source device (102) may include a video capture device such as a video camera, a video archive containing previously captured raw video, and / or a video feed interface for receiving video from a video content provider. As an additional alternative, the video source (104) may generate computer graphics-based data as the source video, or as a combination of live video, archived video, and computer-generated video. In each case, the video encoder (200) encodes the captured, pre-captured, or computer-generated video data. The video encoder (200) may rearrange the pictures from the received order (sometimes referred to as the “display order”) to the coding order for coding. The video encoder (200) may generate a bitstream containing encoded video data. Then, the source device (102) may output the encoded video data onto a computer-readable medium (110) through an output interface (108) for reception and / or extraction by, for example, the input interface (122) of the destination device (116).

[0034] The memory (106) of the source device (102) and the memory (120) of the destination device (116) represent general-purpose memories. In some examples, the memories (106, 120) may store raw video data, e.g., raw video from a video source (104) and raw, decoded video data from a video decoder (300). Additionally or alternatively, the memories (106, 120) may store software instructions executable by, for example, the video encoder (200) and the video decoder (300), respectively. Although the video encoder (200) and the video decoder (300) are shown separately in this example, it should be understood that the video encoder (200) and the video decoder (300) may also include internal memories for functionally similar or equivalent purposes. Furthermore, the memories (106, 120) may store encoded video data, for example, output from the video encoder (200) and input to the video decoder (300). In some examples, portions of the memories (106, 120) may be allocated as one or more video buffers to store, for example, raw, decoded, and / or encoded video data.

[0035] The computer-readable medium (110) may represent any type of medium or device capable of transmitting encoded video data from a source device (102) to a destination device (116). In one example, the computer-readable medium (110) represents a communication medium that enables the source device (102) to transmit the encoded video data directly to the destination device (116) in real time, for example, via a radio frequency network or a computer-based network. According to a communication standard such as a wireless communication protocol, the output interface (108) may modulate a transmission signal containing the encoded video data, and the input interface (122) may modulate a received transmission signal. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful for facilitating communication from a source device (102) to a destination device (116).

[0036] In some examples, the source device (102) may output encoded data from the output interface (108) to the storage device (112). Similarly, the destination device (116) may access the encoded data from the storage device (112) through the input interface (122). The storage device (112) may include any of the various distributed or locally accessed data storage media, such as a hard drive, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data.

[0037] In some examples, the source device (102) may output encoded video data to a file server (114) or other intermediate storage device that may store the encoded video generated by the source device (102). The destination device (116) may access the stored video data from the file server (114) via streaming or downloading. The file server (114) may be any type of server device capable of storing the encoded video data and transmitting the encoded video data to the destination device (116). The file server (114) may represent a web server (e.g., for a website), a file transfer protocol (FTP) server, a content delivery network device, or a NAS (network attached storage) device. The destination device (116) may access the encoded video data from the file server (114) via any standard data connection, including an internet connection. This may include a wireless channel (e.g., Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both of these, suitable for accessing encoded video data stored on the file server (114). The file server (114) and the input interface (122) may be configured to operate according to a streaming transmission protocol, a download transmission protocol, or a combination thereof.

[0038] The output interface (108) and input interface (122) may represent wireless transmitters / receivers, modems, wired networking components (e.g., Ethernet cards), wireless communication components operating according to any of the various IEEE 802.11 standards, or other physical components. In examples where the output interface (108) and input interface (122) include wireless components, the output interface (108) and input interface (122) may be configured to transmit data, such as encoded video data, according to cellular communication standards such as 4G, 4G-LTE (Long-Term Evolution), LTE Advanced, 5G, etc. In some examples where the output interface (108) includes a wireless transmitter, the output interface (108) and the input interface (122) may be configured to transmit data, such as encoded video data, according to other wireless standards such as IEEE 802.11 specifications, IEEE 802.15 specifications (e.g., ZigBee™), Bluetooth™ standards, etc. In some examples, the source device (102) and / or the destination device (116) may include individual system-on-a-chip (SoC) devices. For example, the source device (102) may include an SoC device for performing functions attributed to the video encoder (200) and / or the output interface (108), and the destination device (116) may include an SoC device for performing functions attributed to the video decoder (300) and / or the input interface (122).

[0039] The techniques of the present disclosure may be applied to video coding by supporting any of various multimedia applications, such as over-the-air television broadcasts, cable television transmissions, satellite television transmissions, internet streaming video transmissions, e.g., DASH (dynamic adaptive streaming over HTTP), digital video encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications.

[0040] The input interface (122) of the destination device (116) receives an encoded video bitstream from a computer-readable medium (110) (e.g., a storage device (112), a file server (114), etc.). The encoded video bitstream computer-readable medium (110) may include signaling information defined by a video encoder (200), which is also used by a video decoder (300), such as syntax elements having values ​​that describe the processing and / or characteristics of video blocks or other coded units (e.g., slices, pictures, groups of pictures, sequences, etc.). A display device (118) displays decoded pictures of the decoded video data to a user. The display device (118) may represent any of the various display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or other types of display devices.

[0041] Although not illustrated in FIG. 1, in some examples, the video encoder (200) and the video decoder (300) may each be integrated with an audio encoder and / or an audio decoder, and may include suitable MUX-DEMUX units or other hardware and / or software to handle multiplexed streams containing both audio and video in a common data stream. Where applicable, the MUX-DEMUX units may conform to other protocols such as the ITU H.223 multiplexer protocol or the User Datagram Protocol (UDP).

[0042] The video encoder (200) and the video decoder (300) may each be implemented as any of various suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. Where the techniques are partially implemented in software, the device may store instructions for the software on a suitable non-transient computer-readable medium and execute those instructions in hardware using one or more processors to perform the techniques of the present disclosure. Each of the video encoder (200) and the video decoder (300) may be included in one or more encoders or decoders, any one of which may be integrated as part of a combined encoder / decoder (CODEC) in an individual device. The device including the video encoder (200) and / or video decoder (300) may include an integrated circuit, a microprocessor, and / or a wireless communication device such as a cellular phone.

[0043] The video encoder (200) and video decoder (300) may operate according to a video coding standard such as ITU-T H.265, also referred to as High Efficiency Video Coding (HEVC), or extensions thereof, such as multi-view and / or scalable video coding extensions. Alternatively, the video encoder (200) and video decoder (300) may operate according to other proprietary or industry standards such as ITU-T H.266, also referred to as Versatile Video Coding (VVC). Recent drafts of VVC standards include "Versatile Video Coding (Draft 4)" by Bross et al., ITU-T SG 16 WP 3 and the Joint Video Experts Team (JVET) of ISO / IEC JTC 1 / SC 29 / WG 11, 13 th Meeting: Marrakech, MA, 9-18 January 2019, JVET-M1001-v5 (hereinafter referred to as "VVC Draft 4"). More recent drafts of the VVC standard include "Versatile Video Coding (Draft 8)" by Bross et al., and the JVET (Joint Video Experts Team) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 17 th Meeting: Brussels, BE, 7-17 January 2020, JVET-Q2001-vD (hereinafter referred to as "VVC Draft 8") is described. However, the techniques of the present disclosure are not limited to any specific coding standard.

[0044] Generally, the video encoder (200) and the video decoder (300) may perform block-based coding of pictures. The term “block” generally refers to a structure containing data to be processed (e.g., to be encoded, to be decoded, or otherwise to be used in the encoding and / or decoding process). For example, a block may contain a two-dimensional matrix of samples of luminance and / or chrominance data. Generally, the video encoder (200) and the video decoder (300) may code video data represented in a YUV (e.g., Y, Cb, Cr) format. That is, rather than coding red, green, and blue (RGB) data for samples of a picture, the video encoder (200) and the video decoder (300) may code luminance and chrominance components, wherein the chrominance components may include both red and blue chrominance components. In some examples, the video encoder (200) converts the received RGB-formatted data into a YUV representation prior to encoding, and the video decoder (300) converts the YUV representation into an RGB format. Alternatively, pre- and post-processing units (not shown) may perform these conversions.

[0045] The present disclosure may generally refer to the coding of pictures (e.g., encoding and decoding) to include a process of encoding or decoding data of the pictures. Similarly, the present disclosure may refer to the coding of blocks of pictures to include a process of encoding or decoding data for the blocks, e.g., prediction and / or residual coding. An encoded video bitstream generally includes a series of values ​​for syntax elements representing coding decisions (e.g., coding modes) and partitioning of the pictures into blocks. Accordingly, references to coding a picture or block should generally be understood as coding values ​​for the syntax elements forming the picture or block.

[0046] HEVC defines various blocks including coding units (CUs), prediction units (PUs), and transformation units (TUs). According to HEVC, a video coder (such as a video encoder (200)) partitions coding tree units (CTUs) into CUs according to a quadtree structure. That is, the video coder partitions CTUs and CUs into four identical non-nested squares, and each node of the quadtree has either zero or four child nodes. Nodes without child nodes may be referred to as "leaf nodes," and the CUs of such leaf nodes may contain one or more PUs and / or one or more TUs. The video coder may further partition the PUs and TUs. For example, in HEVC, a residual quadtree (RQT) represents the partitioning of TUs. In HEVC, PUs represent inter-prediction data, while TUs represent residual values. Intra-predicted CUs include intra-predicted information such as intra-mode indications.

[0047] As another example, the video encoder (200) and video decoder (300) may be configured to operate according to VVC. According to VVC, a video encoder (such as the video encoder (200)) partitions a picture into multiple coding tree units (CTUs). The video encoder (200) may partition the CTUs according to a tree structure such as a quadtree binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure eliminates the concepts of multiple partition types, such as the separation between CUs, PUs, and TUs in HEVC. The QTBT structure includes two levels: a first level partitioned according to quadtree partitioning, and a second level partitioned according to binary tree partitioning. The root node of the QTBT structure corresponds to a CTU. The leaf nodes of binary trees correspond to coding units (CUs).

[0048] In an MTT partitioning structure, blocks may be partitioned using quadtree (QT) partitions, binary tree (BT) partitions, and one or more types of tripletree (TT) partitions. A tripletree partition is a partition in which a block is divided into three subblocks. In some examples, a tripletree partition divides a block into three subblocks without dividing the original block through a centroid. The partitioning types in MTT (e.g., QT, BT, and TT) may be symmetric or asymmetric.

[0049] In some examples, the video encoder (200) and the video decoder (300) may use a single QTBT or MTT structure to represent each of the luminance and chrominance components, whereas in other examples, the video encoder (200) and the video decoder (300) may use two or more QTBT or MTT structures, such as one QTBT / MTT structure for the luminance component and another QTBT / MTT structure for both of the chrominance components (or two QTBT / MTT structures for individual chrominance components).

[0050] The video encoder (200) and video decoder (300) may be configured to use quadtree partitioning, QTBT partitioning, MTT partitioning, or other partitioning structures per HEVC. For the purposes of explanation, the description of the techniques of the present disclosure is presented with respect to QTBT partitioning. However, it should be understood that the techniques of the present disclosure may also be applied to video coders configured to use quadtree partitioning or other types of partitioning.

[0051] The present disclosure may interchangeably use "NxN" and "N by N" to refer to the sample dimensions of a block (such as a CU or other video block) in terms of vertical and horizontal dimensions, e.g., 16x16 samples or 16 by 16 samples. Generally, a 16×16 CU will have 16 samples in the vertical direction (y = 16) and 16 samples in the horizontal direction (x = 16). Similarly, an N×N CU generally has N samples in the vertical direction and N samples in the horizontal direction, where N represents a non-negative integer value. Samples in a CU may be arranged in rows and columns. Furthermore, CUs do not necessarily have the same number of samples in the horizontal direction as in the vertical direction. For example, CUs may contain N×M samples, where M is not necessarily equal to N.

[0052] A video encoder (200) encodes video data for CUs and other information representing prediction and / or residual information. The prediction information indicates how the CU will be predicted to form a prediction block for the CU. The residual information generally represents sample-by-sample differences between the samples of the CU prior to encoding and the prediction block.

[0053] To predict the CU, the video encoder (200) may generally form a prediction block for the CU through inter-prediction or intra-prediction. Inter-prediction generally refers to predicting the CU from data of a previously coded picture, whereas intra-prediction generally refers to predicting the CU from data of the same picture that was previously coded. To perform inter-prediction, the video encoder (200) may generate a prediction block using one or more motion vectors. The video encoder (200) may generally perform motion search to identify a reference block that closely matches the CU, for example, in terms of the differences between the CU and the reference block. The video encoder (200) may calculate a difference metric using the sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared differences (MSD), or other such difference calculations to determine whether the reference block closely matches the current CU. In some examples, the video encoder (200) may predict the current CU using unidirectional prediction or bidirectional prediction.

[0054] Some examples of VVC also provide an affine motion compensation mode that may be considered as an inter-prediction mode. In the affine motion compensation mode, the video encoder (200) may determine two or more motion vectors representing non-translational motion, such as zoom in or out, rotation, perspective motion, or other irregular motion types.

[0055] To perform intra-prediction, the video encoder (200) may select an intra-prediction mode to generate prediction blocks. Some examples of VVC provide 67 intra-prediction modes, including planar mode and DC mode, as well as various directional modes. Generally, the video encoder (200) selects an intra-prediction mode that describes neighbor samples for the current block (e.g., a block of CU) to predict samples of the current block. Such samples may generally be located above, above and to the left of the current block, or to the left of the current block, in the same picture as the current block, assuming that the video encoder (200) codes the CTUs and CUs in raster scan order (from left to right, from top to bottom).

[0056] The video encoder (200) encodes data indicating the prediction mode for the current block. For example, for inter-prediction modes, the video encoder (200) may encode motion information for the corresponding mode as well as data indicating which of the various available inter-prediction modes is used. For unidirectional or bidirectional inter-prediction, for example, the video encoder (200) may encode motion vectors using an Advanced Motion Vector Prediction (AMVP) or merge mode. The video encoder (200) may also encode motion vectors for an affine motion compensation mode using similar modes.

[0057] Following a prediction such as an intra-prediction or inter-prediction of a block, the video encoder (200) may calculate residual values ​​for the block. Residual values, such as residual blocks, represent sample-by-sample differences between the block and the prediction block for the block, formed using the corresponding prediction mode. The video encoder (200) may apply one or more transformations to the residual blocks to generate transformed data in a transformation domain instead of a sample domain. For example, the video encoder (200) may apply a Discrete Cosine Transform (DCT), an Integer Transform, a Wavelet Transform, or a conceptually similar transformation to the residual video data. Additionally, the video encoder (200) may apply a quadratic transformation following a first transformation, such as a Mode-Dependent Inseparable Quadratic Transform (MDNSST), a Signal-Dependent Transform, or a Karhunen-Loeve Transform (KLT). The video encoder (200) generates transformation coefficients following the application of one or more transformations.

[0058] As mentioned above, following any transformations to generate transformation coefficients, the video encoder (200) may perform quantization of the transformation coefficients. Quantization generally refers to a process in which transformation coefficients are quantized to provide additional compression, such that the amount of data used to represent those coefficients is reduced as much as possible. By performing the quantization process, the video encoder (200) may reduce the bit depth associated with some or all of the coefficients. For example, the video encoder (200) may round down an n-bit value to an m-bit value during quantization, where n is greater than m. In some examples, to perform quantization, the video encoder (200) may perform a bitwise right-shift of the value to be quantized.

[0059] Following quantization, the video encoder (200) may scan the transform coefficients to generate a one-dimensional vector from a two-dimensional matrix containing the quantized transform coefficients. The scan may be designed to place higher energy (and therefore lower frequency) coefficients at the front of the vector and lower energy (and therefore higher frequency) transform coefficients at the rear of the vector. In some examples, the video encoder (200) may utilize a predefined scan order to scan the quantized transform coefficients to generate a serialized vector, and then entropy-encode the quantized transform coefficients of the vector. In other examples, the video encoder (200) may perform an adaptive scan. After scanning the quantized transform coefficients to form a one-dimensional vector, the video encoder (200) may entropy-encode the one-dimensional vector, for example, according to context-adaptive binary arithmetic coding (CABAC). The video encoder (200) may also entropy-encode values ​​for syntax elements describing metadata associated with the encoded video data for use by the video decoder (300) in decoding the video data.

[0060] To perform CABAC, the video encoder (200) may assign a context within a context model to the symbol to be transmitted. The context may, for example, be related to whether the neighbor values ​​of the symbol are zero values. The probability determination may be based on the context assigned to the symbol.

[0061] The video encoder (200) may additionally generate syntax data, such as block-based syntax data, picture-based syntax data, and sequence-based syntax data, for the video decoder (300), such as picture headers, block headers, slice headers, or other syntax data, such as sequence parameter sets (SPS), picture parameter sets (PPS), or video parameter sets (VPS). Likewise, the video decoder (300) may decode such syntax data to determine how to decode the corresponding video data.

[0062] In this way, the video encoder (200) may generate a bitstream containing syntax elements describing the partitioning of the encoded video data, for example, into blocks of a picture (e.g., CUs), and prediction and / or residual information for the blocks. Ultimately, the video decoder (300) may receive the bitstream and decode the encoded video data.

[0063] Generally, the video decoder (300) performs a reversible process to decode the encoded video data of the bitstream as performed by the video encoder (200). For example, the video decoder (300) may decode values ​​for the syntax elements of the bitstream using CABAC in a reversible manner, but substantially similar to the CABAC encoding process of the video encoder (200). The syntax elements may define the CUs of the CTU by defining partitioning information for the picture's CTUs and partitioning of each CTU according to a corresponding partition structure, such as a QTBT structure. The syntax elements may additionally define prediction and residual information for blocks of video data (e.g., CUs).

[0064] Residual information may be represented, for example, by quantized transform coefficients. The video decoder (300) may inversely quantize and inversely transform the quantized transform coefficients of the block to reconstruct a residual block for the block. The video decoder (300) forms a prediction block for the block using a signaled prediction mode (intra- or inter-prediction) and associated prediction information (e.g., motion information for inter-prediction). Then, the video decoder (300) may reconstruct the original block by combining the prediction block and the residual block (on a sample-by-sample basis). The video decoder (300) may perform additional processing, such as performing a deblocking process to reduce visual artifacts along the boundaries of the block.

[0065] The present disclosure may generally refer to "signaling" specific information, such as syntax elements. The term "signaling" may generally refer to the communication of syntax elements and / or other data values ​​used to decode encoded video data. That is, the video encoder (200) may signal values ​​for syntax elements in a bitstream. Generally, signaling refers to generating values ​​in a bitstream. As mentioned above, the source device (102) may transmit the bitstream to the destination device (116) non-real-time or substantially real-time, as may occur when storing syntax elements in the storage device (112) for subsequent retrieval by the destination device (116).

[0066] According to the techniques of the present disclosure, the video encoder (200) and the video decoder (300) may be configured to perform gradient-based prediction refinement. As described above, as part of inter-predicting the current block, the video encoder (200) and the video decoder (300) may determine one or more prediction blocks for the current block (e.g., based on one or more motion vectors). In gradient-based prediction refinement, the video encoder (200) and the video decoder (300) modify one or more samples of the prediction block (e.g., including all samples).

[0067] For example, in gradient-based prediction refinement, location The inter-prediction samples at (e.g., samples from a prediction block) are at the location Offset derived from displacement in the horizontal direction, horizontal gradient, displacement in the vertical direction, and vertical gradient ( It is refined by ). In one example, predictive refinement is It is described as follows, where, is a horizontal gradient, and is the vertical gradient, and is the displacement in the horizontal direction, and is the displacement in the vertical direction.

[0068] The gradient of an image is a measure of the directional change in the image's intensity or color. For example, the gradient value is based on the rate of change in color or intensity in the direction that has the maximum change in color or intensity based on neighboring samples. As an example, the gradient value is larger when the rate of change is relatively high than when it is relatively low.

[0069] Furthermore, the prediction block for the current block may be a reference picture different from the current picture containing the current block. The video encoder (200) and the video decoder (300) determine an offset (e.g., based on sample values ​​in the reference picture) ) may also be determined (e.g., the gradient is determined based on sample values ​​in the reference picture). In some examples, the values ​​used to determine the gradient may be values ​​within the prediction block itself or values ​​generated based on the values ​​of the prediction block (e.g., interpolated, rounded, etc. values ​​generated from values ​​within the prediction block). Also, in some examples, the values ​​used to determine the gradient may be outside the prediction block and within the reference picture, or may be generated from samples outside the prediction block and within the reference picture (e.g., interpolation, rounding, etc.).

[0070] However, in some examples, the video encoder (200) and the video decoder (300) may determine the offset based on sample values ​​in the current picture. In some examples, such as intra-block copy, the current picture and the reference picture are the same picture.

[0071] Displacement (e.g., vertical and / or horizontal displacement) may be determined based on the inter-prediction mode. In some examples, the displacement is determined based on motion parameters. As described in more detail, for the decoder-side motion resolution mode, the displacement may be based on samples in the reference picture. For other inter-prediction modes, the displacement may not be based on samples in the reference picture, but exemplary techniques are not so limited, and samples in the reference picture may be used to determine the displacement. Various methods for determining vertical and / or horizontal displacements may exist, and techniques are not limited to a specific method for determining vertical and / or horizontal displacements.

[0072] The following describes exemplary methods for performing gradient calculations. For example, regarding gradient filters, in one example, a Sobel filter can be used for gradient calculation. The gradients are calculated as follows:

[0073]

[0074] and

[0075]

[0076] In some examples, a [1, 0, -1] filter is applied. The gradients can also be calculated as follows: and In some examples, other gradient filters, such as the Canny filter, may be applied.

[0077] For gradient normalization, the calculated gradients may be normalized before being used in deriving the refinement offset (e.g., before calculating ΔI), or normalization may be performed after deriving the refinement offset. A rounding process may also be applied during normalization. For example, if a [1, 0, -1] filter is applied, normalization is performed by adding 1 to the input value and then right-shifting by 1. If the input is scaled by powers from 2 to N, 1 < <N 을 가산한 다음 (N+1) 만큼의 우측 시프트에 의해 정규화가 수행된다.

[0078] Regarding gradients at boundaries, the gradients at the boundaries of the prediction block may be calculated by expanding the prediction block by S / 2 at each boundary, where S is the filtering step for gradient calculation. In one example, expanded prediction samples are generated by using the same motion vector as the prediction block for inter-prediction (motion compensation). In some examples, expanded prediction samples are generated by using the same motion vector but using a shorter filter for the interpolation process in motion compensation. In some examples, expanded prediction samples are generated by using a rounded motion vector for integer motion compensation. In some examples, expanded prediction samples are generated by padding, where padding is performed by copying boundary samples. In some examples, if the prediction block is generated by subblock-based motion compensation, the expanded prediction samples are generated by using the motion vector of the nearest subblock. In some examples, if the prediction block is generated by subblock-based motion compensation, the expanded prediction samples are generated by using a single representative motion vector. In one example, the representative motion vector may be a motion vector at the center of the prediction block. In one example, the representative motion vector may be derived by averaging the motion vectors of the boundary subblocks.

[0079] Subblock-based gradient derivation may also be applied to facilitate parallel processing or pipeline-friendly design in hardware. The width and height of subblocks, denoted as sbW and sbH, can be determined as follows: sbW = min(blkW, SB_WIDTH) and sbH = min(blkH, SB_HEIGHT). In this equation, blkW and blkH are the width and height of the prediction block, respectively. SB_WIDTH and SB_HEIGHT are two predetermined variables. In one example, both SB_WIDTH and SB_HEIGHT are equal to 16.

[0080] Regarding horizontal and vertical displacements, the horizontal and vertical displacements used in the derivation of rectification ( and ) can be determined in some examples by relying on the inter-prediction mode. However, exemplary techniques are not limited to determining horizontal and vertical displacements based on the inter-prediction mode.

[0081] For small block size inter-mode (e.g., inter-predicted small size blocks), the inter-predict mode for small blocks may be disabled or restricted to reduce worst-case memory bandwidth. For example, inter-predict for blocks 4x4 or smaller may be disabled, and bidirectional prediction for 4x8, 8x4, 4x16, and 16x4 may be disabled. Memory bandwidth may increase due to the interpolation process for those small blocks. Integer motion compensation without interpolation may still be applied to those small blocks without increasing worst-case memory bandwidth.

[0082] In one or more exemplary techniques, inter-prediction may be enabled for some or all of their small blocks, but integer motion compensation and gradient-based prediction refinement may also be enabled. The motion vector is first rounded into an integer motion vector for motion compensation. Then, the remainder of the rounding, that is, the sub-part of the motion vector, is for gradient-based prediction refinement. and It is used as. For example, if the motion vector for a small block is (2.25, 5.75), the integer motion vector used for motion compensation will be (2, 6), and the horizontal displacement (e.g., Δv x (i,j)) will be 0.25, and the vertical displacement (e.g., Δv y (i,j)) will be 0.75. In this example, the precision level of the horizontal and vertical displacements is 0.25 (or ¼). For example, the horizontal and vertical displacements can be incremented in steps of 0.25.

[0083] In some examples, for small block size inter-modes, gradient-based predictive refinement may be available, but only when the small size blocks are inter-predicted in merge mode. Examples of merge mode are described below. In some examples, for small size inter-modes, gradient-based predictive refinement may be disabled for blocks having integer motion modes. In integer motion modes, one or more motion vectors (e.g., signaled motion vectors) are integers. In some examples, for larger size blocks as well, if the blocks are inter-predicted in integer motion modes, gradient-based predictive refinement may be disabled for such blocks.

[0084] For a normal merging mode, which is an example of an inter-prediction mode in which motion information is derived from spatially or temporally neighbored coded blocks, and ... may be the remainder of a motion vector rounding process (e.g., similar to the above example of motion vector (2.25, 5.75)). In one example, a temporal motion vector predictor is derived by scaling the motion vector in the temporal motion buffer according to different picture order counts between the current picture and reference pictures. The rounding process may be performed to round the scaled motion vector to a specific precision. The remainder and It can be used as. The precision of the remainder (i.e., the precision level of horizontal and vertical displacements) can be predefined and may be higher than the precision of the motion vector prediction. For example, if the motion vector precision is 1 / 16, the remainder precision is 1 / (16*MaxBlkSize), where MaxBlkSize is the maximum block size. Described in another way, the precision level for horizontal and vertical displacements (e.g., Δv x and Δv y ) is 1 / (16*MaxBlkSize).

[0085] For the merge with motion vector difference (MMVD) mode, which is an example of an inter-prediction mode, the motion vector difference is signaled along with a merge index to represent motion information. In some techniques, the motion vector difference (e.g., the difference between the actual motion vector and the motion vector predictor) has the same precision as the motion vector. In one or more examples described in this disclosure, the motion vector difference may be allowed to have a higher precision. The signaled motion vector difference is first rounded to the motion vector precision, and the motion vector indicated by the merge index is added to generate the final motion vector for motion compensation. In one or more examples, the remainder after rounding (e.g., the difference between the rounded value of the motion vector difference and the original value of the motion vector difference) is used as horizontal and vertical displacements for gradient-based prediction refinement (e.g., and It may also be used as. In some examples, and It can be signaled as candidates for motion vector difference.

[0086] For the decoder-side motion vector refinement mode, motion compensation using the original motion vectors is performed to generate original bidirectional prediction blocks, the difference between the List 0 prediction and the List 1 prediction is calculated, and HistorigIt is denoted as. List 0 refers to a first reference picture list (RefPicList0) containing a list of reference pictures that can potentially be used for inter-prediction. List 1 refers to a second reference picture list (RefPicList1) containing a list of reference pictures that can potentially be used for inter-prediction. Then, the motion vectors in List 0 and List 1 are rounded to nearest integer positions. That is, motion vectors referencing pictures in List 0 are rounded to nearest integer positions, and motion vectors referencing pictures in List 1 are rounded to nearest integer positions. The search algorithm uses new integer motion vectors for motion compensation to find the minimum distortion between the blocks of pictures identified in the List 0 prediction and the blocks of pictures identified in List 1 ( DistNew It is used to search within a range of integer displacements to find pairs of displacements having ). If DistNew is smaller than DistOrig, the new integer motion vectors are used for prediction refinement in both List 0 and List 1 predictions. and It is fed to the Bidirectional Optical Flow (BDOF) to derive. Otherwise, BDOF is performed on the original List 0 and List 1 predictions for prediction refinement.

[0087] For affine modes, the motion field can be derived for each pixel (e.g., the motion vector can be determined on a per-pixel basis). However, a 4x4-based motion field is used for affine motion compensation to reduce complexity and memory bandwidth. For example, rather than determining the motion vector on a per-pixel basis, the motion vector is determined for subblocks, where one subblock is 4x4 as an example. Some other subblock sizes, e.g., 4x2, 2x4, or 2x2, may also be used. In one or more examples, gradient-based predictive refinement may be used to improve affine motion compensation. The gradient of the block can be calculated as described above. Affine Motion Model: This is given, where a, b, c, d, e, and f are values ​​determined by the video encoder (200) and video decoder (300) based on the length and width of the block and control point motion vectors, as some examples. The values ​​for a, b, c, d, e, and f may be signaled in some examples.

[0088] The following describes some exemplary ways of determining a, b, c, d, e, and f. In a video coder (e.g., video encoder (200) or video decoder (300)), a picture is partitioned into subblocks for block-based coding in affine mode. An affine motion model for a block also includes three motion vectors (MVs) at three different positions that are not on the same line ( , and It can be described by ). The three positions are typically referred to as control points, and the three motion vectors are referred to as control point motion vectors (CPMVs). If the three control points are at the three corners of the block, the affine motion can be described as follows:

[0089]

[0090] Here, blkW and blkH are the width and height of the block.

[0091] For affine mode, the video encoder (200) and the video decoder (300) may determine a motion vector for each subblock using the representative coordinates of the subblock (e.g., the center position of the subblock). In one example, the block is partitioned into non-overlapping subblocks. The block width is blkW, the block height is blkH, the subblock width is sbW, the subblock height is sbH, and then there are blkH / sbH rows of subblocks and blkW / sbW subblocks in each row. For a 6-parameter affine motion model, the i-th row (0 <= i <blkW / sbW) 및 j번째 열 (0<=j<blkH / sbH) 에서의 서브블록에 대한 모션 벡터 (서브블록 MV 로서 지칭됨) 는 다음과 같이 도출된다:

[0092]

[0093] From the above equation, the variables a, b, c, d, e, and f may also be defined as follows:

[0094]

[0095] For an affine mode, which is an example of an inter-prediction mode, the video encoder (200) and the video decoder (300) may determine displacements (e.g., horizontal or vertical displacements) by at least one of the following methods. The following are examples and should not be considered limiting. There may be other ways in which the video encoder (200) and the video decoder (300) may determine displacements (e.g., horizontal or vertical displacements) for the affine mode.

[0096] For 4x4 subblock-based affine motion compensation, for 2x2-based displacement derivation, the displacements in each 2x2 subblock are identical. For each 4x4 subblock, regarding the four 2x2 subblocks within the 4x4 is calculated as follows:

[0097]

[0098] To derive 1x1 displacements, displacements are derived for each sample. In a 4x4 grid, the coordinates of the top-left sample may be (0,0), in which case, is derived as follows:

[0099] .

[0100] In some examples, the division by 2 implemented as a right shift operation may be moved to a refinement offset calculation. For example, horizontal and vertical displacements (e.g., Δv x and Δv y Rather than performing a division operation by 2 when deriving ), the video encoder (200) and the video decoder (300) may perform a division operation by 2 as part of determining ΔI (e.g., a resolution offset).

[0101] For 4x2 subblock-based affine motion compensation, the motion field for storing motion vectors is still 4x4; however, the affine motion compensation is 4x2. The motion vector (MV) for a 4x4 subblock is It could be, and in this case, the MV for the left 4x2 motion compensation is And, the MV for the left 4x2 motion compensation is am.

[0102] For 2x2-based displacement derivation, the displacements in each 2x2 subblock are identical. For each 4x2 subblock, regarding the two 2x2 subblocks within the 4x4 is calculated as follows:

[0103]

[0104] To derive 1x1 displacements, displacements are derived for each sample. If the coordinates of the top-left sample in a 4x2 grid are (0,0), then is derived as follows:

[0105] .

[0106] The division by 2, which may be implemented as a right shift operation, can be moved to a refinement offset calculation. For example, horizontal and vertical displacements (e.g., Δv x and Δv y Rather than performing a division operation by 2 when deriving ), the video encoder (200) and the video decoder (300) may perform a division operation by 2 as part of determining ΔI (e.g., a resolution offset).

[0107] For 2x4 subblock-based affine motion compensation, the motion field for storing motion vectors is still 4x4, but the affine motion compensation is 2x4. The MV for a 4x4 subblock is It could be, and in this case, the MV for the left 4x2 motion compensation is And, the MV for the left 4x2 motion compensation is am.

[0108] For the derivation of 2x2-based displacements, the displacements in each 2x2 subblock are identical. For each 2x4 subblock, regarding the two 2x2 subblocks within the 2x4 is calculated as follows:

[0109]

[0110] For 1x1 displacement derivation, in 1x1-based displacement derivation, displacements are derived for each sample. In 2x4, the coordinates of the top-left sample may be (0,0), in this case, It can also be derived as follows:

[0111] .

[0112] The division by 2, which may be implemented as a right shift operation, can be moved to a refinement offset calculation. For example, horizontal and vertical displacements (e.g., Δv x and Δv y Rather than performing a division operation by 2 when deriving ), the video encoder (200) and the video decoder (300) may perform a division operation by 2 as part of determining ΔI (e.g., a resolution offset).

[0113] The following describes the prediction refinement for affine modes. After subblock-based affine motion compensation is performed, the prediction signal can be refined by adding an offset derived based on the gradient of the prediction signal and pixel-level motion. The offset at position (m,n) can be calculated as follows:

[0114]

[0115] Here are the horizontal gradients of the predicted signals, respectively. is the vertical gradient. and is the difference in the x and y components between the motion vector calculated at the position pixel location (m,n) and the subblock MV. Let the coordinates of the top-left sample of the subblock be (0,0), and the center of the subblock is ( Let ). Given affine motion parameters a, b, c, and d, and can be derived as follows:

[0116]

[0117] In a control-point-based affine motion model, the affine motion parameters a, b, c, and d are calculated from CPMVs as follows:

[0118]

[0119]

[0120] The following describes Bidirectional Optical Flow (BDOF). The Bidirectional Optical Flow (BDOF) tool is included in VTM4. BDOF was previously referred to as BIO. BDOF can also be used to refine the bidirectional prediction signals of Coding Units (CUs) at the 4×4 subblock level. The BDOF model is based on the optical flow concept, which assumes that the object's motion is smooth. For each 4×4 subblock, motion refinement ( ) is calculated by minimizing the difference between L0 and L1 predicted samples (e.g., predicted samples from the reference picture in the first reference picture list L0 and predicted samples from the reference picture in the second reference picture list L1). Motion resolution is then used to adjust the bidirectional predicted sample values ​​in a 4x4 subblock. The following steps are applied to the BDOF process.

[0121] First, the horizontal and vertical gradients of the two prediction signals ( and , ) is computed by directly calculating the difference between these two neighboring samples, that is,

[0122]

[0123] Here, is a list Coordinates of the predicted signal in This is a sample value from.

[0124] Subsequently, the auto- and cross-correlations of the gradients (S1, S2, S3, S5, and S6) are calculated as follows:

[0125]

[0126] In the equation,

[0127]

[0128] In the equation, Ω is a 6×6 window around a 4×4 subblock.

[0129] Motion Jeong Se-hwa ( ) is then derived using cross- and auto-correlation terms using the following:

[0130]

[0131] Here is the floor function. Based on motion refinement and gradients, the following adjustments are calculated for each sample in a 4×4 subblock:

[0132]

[0133] Finally, the BDOF samples of CU are calculated by adjusting the bidirectional samples as follows:

[0134]

[0135] In other words, rounding offset

[0136] These values ​​are selected so that the multipliers in the BDOF process do not exceed 15 bits, and the maximum bit unit of the intermediate parameters in the BDOF process is maintained within 32 bits.

[0137] To derive gradient values, the list outside the current CU boundaries ( Some prediction samples from ) ) needs to be generated. As exemplified in Fig. 5, BDOF in VTM4 uses a single extended row / column around the CU boundaries. To control the computational complexity of generating prediction samples outside the boundaries, prediction samples in the extended area (white positions) are generated by taking reference samples from nearby integer positions directly (using floor() operations on the coordinates) without interpolation, and a regular 8-tap motion compensation interpolation filter is used to generate prediction samples within the CU (gray positions). These extended sample values ​​are used only for gradient calculation. For the remaining steps in the BDOF process, if arbitrary sample and gradient values ​​outside the CU boundaries are required, these samples are padded (i.e., repeated) from their nearby neighbors.

[0138] The following describes the precision of displacements and gradients. In some examples, the same precision for horizontal and vertical displacements may be used in all modes. Precision may be predefined or signaled in high-level syntax. Thus, if horizontal and vertical displacements are derived from different modes with different precisions, the horizontal and vertical displacements are rounded to a predefined precision. Examples of predefined precisions include 1 / 4, 1 / 8, 1 / 16, 1 / 32, 1 / 64, 1 / 128, etc.

[0139] As explained above, precision, also referred to as the precision level, refers to horizontal and vertical displacements (e.g., Δv x and Δv y) may also indicate how precise it is, where horizontal and vertical displacements may be determined using one or more of the examples described above or using some other techniques. Generally, precision levels are defined as decimals (e.g., 0.25, 0.125, 0.0625, 0.03125, 0.015625, 0.0078125, etc.) or fractions (e.g., 1 / 4, 1 / 8, 1 / 16, 1 / 32, 1 / 64, 1 / 128, etc.). For example, for a 1 / 4 precision level, horizontal or vertical displacements may be expressed as increments of 0.25 (e.g., 0.25, 0.5, or 0.75). For a 1 / 8 precision level, horizontal and vertical displacements may be expressed in increments of 0.125 (e.g., 0.125, 0.25, 0.325, 0.5, 0.625, 0.75, or 0.825). As can be seen, the lower the numerical value of the precision level (e.g., 1 / 8 is smaller than 1 / 4), the greater the granularity of the increments and the more precisely the values ​​can be presented (e.g., for a 1 / 4 precision level, displacements are rounded to the nearest 1 / 4, but for a 1 / 8 precision level, displacements are rounded to the nearest 1 / 8).

[0140] Since horizontal and vertical displacements may have different precision levels for different inter-prediction modes, the video encoder (200) and the video decoder (300) may be configured to include different logic circuits to perform gradient-based prediction refinement for different inter-prediction modes. As described above, to perform gradient-based prediction refinement, the video encoder (200) and the video decoder (300) may perform the following operations: , here, g x and g yare respectively the first gradient based on the first set of samples of the prediction block and the second gradient based on the second set of samples of the prediction block, and Δv x and Δv y These are horizontal and vertical displacements, respectively. As can be seen, for gradient-based prediction refinement, the video encoder (200) and video decoder (300) may need to utilize memory not only to perform multiplication and addition operations, but also to store temporary results used in the calculation.

[0141] However, the capability of the logic circuit section for performing mathematical operations (e.g., multiplier circuit section, adder circuit section, memory register) may be limited to the precision level in which the logic circuit section is configured. For example, a logic circuit section configured for a first precision level may not be able to perform operations necessary for gradient prediction refinement in a second precision level where horizontal or vertical displacement is more precise.

[0142] Accordingly, some techniques utilize different sets of logic circuits configured for different precision levels to perform gradient-based prediction refinement for different inter-prediction modes. For example, a first set of logic circuits may be configured to perform gradient-based prediction refinement for inter-prediction modes with a horizontal and / or vertical displacement of 0.25, and a second set of logic circuits may be configured to perform gradient-based prediction refinement for inter-prediction modes with a horizontal and / or vertical displacement of 0.125. Having these different sets of circuits not only increases the overall size of the video encoder (200) and video decoder (300) but also potentially wastes power.

[0143] In some of the examples described in this disclosure, the same gradient calculation process may be used for all inter-prediction modes. Alternatively, the same logic circuit may be used to perform gradient-based prediction refinement for different inter-prediction modes. For example, the precision of the gradient may be maintained identically for prediction refinement in all inter-prediction modes. In some examples, for the precision of the displacements and gradients, exemplary techniques may ensure that the same (or unified) prediction refinement process can be applied to different modes, and that the same prediction refinement module can be applied to different modes.

[0144] As an example, the video encoder (200) and the video decoder (300) may be configured to round at least one of the horizontal and vertical displacements to the same precision level for different inter-prediction modes (e.g., same for affine mode and BDOF). For example, if the precision level to which the horizontal and vertical displacements are rounded is 0.015625 (1 / 64), then, if the precision level of the horizontal and / or vertical displacements is 1 / 4 for one inter-prediction mode, the precision level of the horizontal and / or vertical displacements is rounded to 1 / 64. If the precision level of the horizontal and / or vertical displacements is 1 / 128, the precision level of the horizontal and / or vertical displacements is rounded to 1 / 64.

[0145] In this way, the logic circuit for gradient-based prediction refinement can be reused for different inter-prediction modes. For example, in the above example, the video encoder (200) and the video decoder (300) may include a logic circuit for a precision level of 0.125, and this logic circuit can be reused for different inter-prediction modes because the precision level of horizontal and / or vertical displacements is rounded to 0.125.

[0146] In some examples, where rounding is not performed according to the techniques described in this disclosure, logic circuits for multiplication and accumulation type operations may be reused if the logic circuits are designed to have a relatively high-level precision (e.g., a logic circuit designed for a specific precision level for multiplication may process multiplication operations for values ​​of a lower precision level). However, for shift operations, a logic circuit designed for a specific precision may not be able to process shift operations for values ​​of a lower precision level. It may be possible to reuse logic circuits, including those for shift operations for different inter-prediction modes, using the exemplary techniques described in this disclosure where rounding techniques are described.

[0147] In one example, the predictive refinement offset is derived as follows:

[0148]

[0149] In the above equation, the offset is 1 << (shift-1), and the shift is determined by a predefined precision of the displacement and gradient and is fixed for different modes. In some examples, the offset is equal to 0.

[0150] In some examples, the mode may include one or more of the modes described above for horizontal and vertical displacements, such as small block size inter mode, normal merge mode, merge with motion vector difference, decoder-side motion vector refinement mode, and affine mode. The mode may also include the bidirectional optical flow (BDOF) described above.

[0151] Separate refinement may exist for each prediction direction. For example, in the case of bidirectional prediction, prediction refinement may be performed separately for each prediction direction. The result of the refinement may be clipped to a specific range to ensure the same bit width as the prediction without refinement. For example, the refinement result is clipped to a 16-bit range. As described above, exemplary techniques may also be applied to BDOF, where displacements in two different directions are assumed to be on the same motion trajectory.

[0152] The following describes an N-bit (e.g., 16-bit) multiplication constraint. To reduce the complexity of gradient-based prediction refinement, multiplications may be maintained within N bits (e.g., 16 bits). Gradients and displacements must be representable in 16 bits or less, in this example. Otherwise, the gradient or displacement is quantized to be within 16 bits, in this example. For example, a right shift may be applied to maintain a 16-bit representation.

[0153] The following is the refinement offset ( Explains ) and clipping of refinement results. Refinement offset ( ) is clipped to a specific range. In one example, the range is determined by the range of the original predicted signal. The range of may be the same as the range of the original prediction signal, or it may be a scaled range. The scale can be 1 / 2, 1 / 4, 1 / 8, etc. The refinement result is clipped to have the same range as the original prediction signal (e.g., the range of samples in the prediction block). The equation for performing clipping is as follows:

[0154]

[0155] predSamplesL0 and predSamplesL1 are the prediction samples in each unidirectional prediction direction. bdofOffset is the refinement offset derived by BDOF. Offset4 = 1 << (shift4-1) and Clip3(min, max, x) is a function that clips the value of x to be within the range min to max.

[0156] In this way, the video encoder (200) and the video decoder (300) may be configured to determine a prediction block for inter-predicting the current block. For example, the video encoder (200) and the video decoder (300) may determine a motion vector or block vector pointing to the prediction block (e.g., for an intra-block copy mode).

[0157] The video encoder (200) and the video decoder (300) may determine at least one of horizontal or vertical displacements for gradient-based prediction refinement of one or more samples of a prediction block. Examples of horizontal displacements are Δv x and examples of vertical displacement are Δv y is. In some examples, the video encoder (200) and the video decoder (300) may determine at least one of horizontal or vertical displacements for gradient-based prediction shaping of one or more samples of a prediction block based on an inter-prediction mode (e.g., as two examples, using the exemplary techniques above for affine modes, Δv x and Δv y Determine Δv using the exemplary techniques above for the merge mode x and Δv y (Determining).

[0158] According to one or more examples, the video encoder (200) and the video decoder (300) may round at least one of the horizontal and vertical displacements to the same precision level for different inter-prediction modes. Examples of different inter-prediction modes include affine modes and BDOF. For example, the precision level for a first horizontal or vertical displacement to perform gradient-based prediction refinement for a first block inter-prediction in a first inter-prediction mode may be at the first precision level, and the precision level for a second horizontal or vertical displacement to perform gradient-based prediction refinement for a second block inter-prediction in a second inter-prediction mode may be at the second precision level. The video encoder (200) and the video decoder (300) may be configured to round a first precision level for a first horizontal or vertical displacement to a precision level, and to round a second precision level for a first horizontal or vertical displacement to the same precision level.

[0159] In some examples, the precision level may be predefined (e.g., stored in advance on the video encoder (200) and video decoder (300)) or signaled (e.g., defined by the video encoder (200) and signaled to the video decoder (300)). In some examples, the precision level may be 1 / 64.

[0160] The video encoder (200) and the video decoder (300) may be configured to determine one or more refinement offsets based on at least one rounded of horizontal or vertical displacements. For example, the video encoder (200) and the video decoder (300) may determine ΔI(i,j) for each sample of the prediction block using at least one individual rounded of the horizontal or vertical displacements. That is, the video encoder (200) and the video decoder (300) may determine a refinement offset for each sample of the prediction block. In some examples, the video encoder (200) and the video decoder (300) may utilize the rounded horizontal and vertical displacements to determine the refinement offset (e.g., ΔI).

[0161] As described, to perform gradient-based prediction refinement, the video encoder (200) and the video decoder (300) may determine a first gradient based on a first set of samples of one or more samples of a prediction block (e.g., g x Determine (i,j), where the first set of samples is g x (The samples used to determine (i,j)), a second gradient may also be determined based on a second set of samples of one or more samples of the prediction block (e.g., g y Determine (i,j), where the second set of samples is g y (samples used to determine (i,j)). The video encoder (200) and video decoder (300) may also determine the clarification offsets based on the rounded horizontal and vertical displacements and the first and second gradients.

[0162] The video encoder (200) and the video decoder (300) may modify one or more samples of a prediction block based on one or more determined refinement offsets to generate a modified prediction block (e.g., one or more modified samples forming the modified prediction block). For example, the video encoder (200) and the video decoder (300) may add or subtract ΔI(i,j) from I(i,j), where I(i,j) refers to a sample in the prediction block located at position (i,j). In some examples, the video encoder (200) and the video decoder (300) may clip one or more refinement offsets (e.g., clipping ΔI(i,j). The video encoder (200) and video decoder (300) may modify one or more samples of the prediction block based on one or more clipped refinement offsets.

[0163] For encoding, the video encoder (200) may determine residual values ​​(e.g., of a residual block) representing the difference between the current block and the modified prediction block (e.g., based on modified samples of the modified prediction block) and signal information representing the residual values. For decoding, the video decoder (300) may receive information representing the residual values ​​and restore the current block based on the modified prediction block (e.g., modified samples of the modified prediction block) and the residual values ​​(e.g., by adding residual values ​​to the modified samples).

[0164] FIGS. 2a and 2b are conceptual diagrams illustrating an exemplary quadtree binary tree (QTBT) structure (130) and a corresponding coding tree unit (CTU) (132). Solid lines represent quadtree splits, and dotted lines represent binary tree splits. At each split (i.e., non-leaf) node of the binary tree, a flag is signaled to indicate which split type (i.e., horizontal or vertical) is used; in this example, 0 indicates a horizontal split and 1 indicates a vertical split. For quadtree splits, there is no need to indicate the split type because the quadtree nodes split the block horizontally and vertically into four sub-blocks of the same size. Accordingly, the video encoder (200) may encode the syntax elements (such as splitting information) for the region tree levels (i.e., solid lines) of the QTBT structure (130) and the syntax elements (such as splitting information) for the prediction tree levels (i.e., dotted lines) of the QTBT structure (130), and the video decoder (300) may decode them. The video encoder (200) may encode the video data, such as prediction and transformation data, for the CUs represented by the end leaf nodes of the QTBT structure (130), and the video decoder (300) may decode it.

[0165] Generally, the CTU (132) of FIG. 2b may be associated with parameters that define the sizes of blocks corresponding to the nodes of the QTBT structure (130) at the first and second levels. These parameters may include CTU size (indicating the size of the CTU (132) in the samples), minimum quadtree size (MinQTSize, indicating the minimum allowed quadtree leaf node size), maximum binary tree size (MaxBTSize, indicating the maximum allowed binary tree root node size), maximum binary tree depth (MaxBTDepth, indicating the maximum allowed binary tree depth), and minimum binary tree size (MinBTSize, indicating the minimum allowed binary tree leaf node size).

[0166] The root node of the QTBT structure corresponding to the CTU may have four child nodes at the first level of the QTBT structure, each of which may be partitioned according to quadtree partitioning. That is, the nodes at the first level are either leaf nodes (without child nodes) or have four child nodes. An example of the QTBT structure (130) illustrates such nodes by including child nodes and parent nodes with solid lines for branches. The nodes at the first level may be further partitioned by individual binary trees, provided they are not larger than the maximum allowed binary tree root node size (MaxBTSize). The binary tree splitting of a node may be repeated until the nodes resulting from the split reach the minimum allowed binary tree leaf node size (MinBTSize) or the maximum allowed binary tree depth (MaxBTDepth). An example of the QTBT structure (130) illustrates such nodes by having dotted lines for branches. Binary tree leaf nodes are referred to as coding units (CUs) used for prediction (e.g., intra-picture or inter-picture prediction) and transformation without any additional partitioning. As discussed above, CUs may also be referred to as "video blocks" or "blocks".

[0167] In one example of a QTBT partitioning structure, the CTU size is set to 128x128 (luminance samples and two corresponding 64x64 chroma samples), MinQTSize is set to 16x16, MaxBTSize is set to 64x64, MinBTSize (for both width and height) is set to 4, and MaxBTDepth is set to 4. Quadtree partitioning is first applied to the CTU to generate quadtree leaf nodes. Quadtree leaf nodes can have sizes ranging from 16x16 (i.e., MinQTSize) to 128x128 (i.e., CTU size). If a leaf quadtree node is 128x128, it will not be further partitioned by the binary tree because its size exceeds MaxBTSize (i.e., 64x64 in this example). Otherwise, the leaf quadtree nodes will be further partitioned by the binary tree. Therefore, the quadtree leaf node is also the root node of the binary tree and has a binary tree depth of 0. If the binary tree depth reaches MaxBTDepth (4 in this example), further partitioning is not allowed. If a binary tree node has a width equal to MinBTSize (4 in this example), it implies that further horizontal partitioning is not allowed. Similarly, a binary tree node with a height equal to MinBTSize implies that further vertical partitioning is not allowed for that binary tree node. As mentioned above, the leaf nodes of the binary tree are referred to as CUs and are further processed according to prediction and transformation without further partitioning.

[0168] FIG. 3 is a block diagram illustrating an exemplary video encoder (200) capable of performing the techniques of the present disclosure. FIG. 3 is provided for illustrative purposes and should not be construed as limiting the techniques as broadly illustrated and described in the present disclosure. For illustrative purposes, the present disclosure describes a video encoder (200) in the context of video coding standards such as the H.266 video coding standard and the HEVC video coding standard currently under development. However, the techniques of the present disclosure are not limited to these video coding standards and are generally applicable to video encoding and decoding.

[0169] In the example of FIG. 3, the video encoder (200) includes a video data memory (230), a mode selection unit (202), a residual generation unit (204), a conversion processing unit (206), a quantization unit (208), an inverse quantization unit (210), an inverse conversion processing unit (212), a restoration unit (214), a filter unit (216), a decoded picture buffer (DPB) (218), and an entropy encoding unit (220). Any or all of the video data memory (230), the mode selection unit (202), the residual generation unit (204), the conversion processing unit (206), the quantization unit (208), the inverse quantization unit (210), the inverse conversion processing unit (212), the restoration unit (214), the filter unit (216), the DPB (218), and the entropy encoding unit (220) may be implemented in one or more processors or in a processing circuit. Furthermore, the video encoder (200) may include additional or alternative processors or processing circuits to perform these and other functions.

[0170] The video data memory (230) may store video data to be encoded by components of the video encoder (200). The video encoder (200) may receive video data stored in the video data memory (230) from, for example, a video source (104) (Fig. 1). The DPB (218) may serve as a reference picture memory for storing reference video data for use in predicting subsequent video data by the video encoder (200). The video data memory (230) and the DPB (218) may be formed by any of the various memory devices, such as dynamic random access memory (DRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices, including synchronous DRAM (SDRAM). The video data memory (230) and the DPB (218) may be provided by the same memory device or by separate memory devices. In various examples, the video data memory (230) may be on-chip with other components of the video encoder (200) as exemplified, or off-chip with respect to those components.

[0171] In the present disclosure, a reference to the video data memory (230) should not be interpreted as being limited to memory inside the video encoder (200) unless specifically described as such, or to memory outside the video encoder (200) unless specifically described as such. Rather, a reference to the video data memory (230) should be understood as a reference memory that stores video data received by the video encoder (200) for encoding (e.g., video data for the current block to be encoded). The memory (106) of FIG. 1 may also provide temporary storage of outputs from various units of the video encoder (200).

[0172] Various units of FIG. 3 are illustrated to aid in understanding the operations performed by the video encoder (200). The units may be implemented as fixed-function circuits, programmable circuits, or a combination thereof. Fixed-function circuits refer to circuits that provide specific functions and are pre-configured for the operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functions in the operations that can be performed. For example, programmable circuits may execute software or firmware that causes the programmable circuits to operate in a manner defined by the commands of the software or firmware. Fixed-function circuits may execute software commands (e.g., to receive parameters or output parameters), but the types of operations performed by the fixed-function circuits are generally invariant. In some examples, one or more of the units may be separate circuit blocks (fixed-function or programmable), and in some examples, one or more units may be integrated circuits.

[0173] The video encoder (200) may include arithmetic logic units (ALUs), basic function units (EFUs), digital circuits, analog circuits, and / or programmable cores formed from programmable circuits. In examples where the operations of the video encoder (200) are performed using software executed by the programmable circuits, memory (106) (FI. 1) may store object code of software that the video encoder (200) receives and executes, or other memory (not shown) within the video encoder (200) may store such instructions.

[0174] The video data memory (230) is configured to store received video data. The video encoder (200) may extract a picture of video data from the video data memory (230) and provide the video data to the residual generation unit (204) and the mode selection unit (202). The video data in the video data memory (230) may be raw video data to be encoded.

[0175] The mode selection unit (202) includes a motion estimation unit (222), a motion compensation unit (224), an intra-prediction unit (226), and a gradient-based prediction refinement (GBPR) unit (227). The mode selection unit (202) may include additional function units to perform video prediction according to different prediction modes. As examples, the mode selection unit (202) may include a palette unit, an intra-block copy unit (which may be part of the motion estimation unit (222) and / or the motion compensation unit (224)), an affine unit, a linear model (LM) unit, etc.

[0176] Although the GBPR unit (227) is exemplified as being separate from the motion estimation unit (222) and the motion compensation unit (224), in some examples, the GBPR unit (227) may be part of the motion estimation unit (222) and / or the motion compensation unit (224). The GBPR unit (227) is depicted separately from the motion estimation unit (222) and the motion compensation unit (224) for ease of understanding and should not be considered as limiting.

[0177] The mode selection unit (202) generally adjusts multiple encoding passes to test combinations of encoding parameters and the resulting rate-distortion values ​​for such combinations. The encoding parameters may include partitioning of CTUs into CUs, prediction modes for CUs, transformation types for residual values ​​of CUs, quantization parameters for residual values ​​of CUs, etc. The mode selection unit (202) may ultimately select a combination of encoding parameters that has superior rate-distortion values ​​compared to other tested combinations.

[0178] The video encoder (200) partitions a picture extracted from the video data memory (230) into a series of CTUs and may encapsulate one or more CTUs within a slice. The mode selection unit (202) may partition the CTUs of the picture according to a tree structure, such as the quadtree structure or QTBT structure of HEVC described above. As described above, the video encoder (200) may form one or more CUs by partitioning the CTUs according to the tree structure. Such CUs may also generally be referred to as "video blocks" or "blocks."

[0179] Generally, the mode selection unit (202) also controls its components (e.g., motion estimation unit (222), motion compensation unit (224), intra-prediction unit (226), and GBPR unit (227)) to generate a prediction block for the current block (e.g., the current CU, or the overlapping part of PU and TU in HEVC). For inter-prediction of the current block, the motion estimation unit (222) may perform motion search to identify one or more closely matching reference blocks in one or more reference pictures (e.g., one or more previously coded pictures stored in DPB (218). In particular, the motion estimation unit (222) may calculate a value indicating how similar a potential reference block is to the current block, for example, according to the sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared differences (MSD), etc. The motion estimation unit (222) may generally perform these calculations using sample-by-sample differences between the reference block being considered and the current block. The motion estimation unit (222) may also identify a reference block having the lowest value resulting from these calculations, indicating the reference block that most closely matches the current block.

[0180] The motion estimation unit (222) may form one or more motion vectors (MVs) that define the positions of reference blocks in reference pictures for the position of the current block in the current picture. Then, the motion estimation unit (222) may provide the motion vectors to the motion compensation unit (224). For example, for unidirectional inter-prediction, the motion estimation unit (222) may provide a single motion vector, whereas for bidirectional inter-prediction, the motion estimation unit (222) may provide two motion vectors. Then, the motion compensation unit (224) may generate a prediction block using the motion vectors. For example, the motion compensation unit (224) may extract data of the reference block using the motion vectors. As another example, if the motion vector has fractional sample precision, the motion compensation unit (224) may interpolate the values ​​for the prediction block according to one or more interpolation filters. Furthermore, for bidirectional inter-prediction, the motion compensation unit (224) may extract data for two reference blocks identified by individual motion vectors and combine the extracted data, for example, through sample-by-sample averaging or weighted averaging.

[0181] As another example, for intra-prediction or intra-prediction coding, the intra-prediction unit (226) may generate a prediction block from samples adjacent to the current block. For example, for directional modes, the intra-prediction unit (226) may generally generate a prediction block by mathematically combining the values ​​of neighboring samples and populating these calculated values ​​in a direction defined across the current block. As another example, for DC mode, the intra-prediction unit (226) may calculate the average of neighboring samples for the current block and generate a prediction block containing this resulting average for each sample of the prediction block.

[0182] The GBPR unit (227) may be configured to perform the exemplary techniques described in this disclosure for gradient-based prediction refinement. For example, the GBPR unit (227), together with the motion compensation unit (224), may determine a prediction block for inter-predicting the current block (e.g., based on a motion vector determined by the motion estimation unit (222)). The GBPR unit (227) determines horizontal and vertical displacements (e.g., Δv) for gradient-based prediction refinement of one or more samples of the prediction block. x and Δv y ) may also be determined. As an example, the GBPR unit (227) may determine an inter-prediction mode based on decisions made by the mode selection unit (202) to inter-prediction the current block. In some examples, the GBPR unit (227) may determine horizontal and vertical displacements based on the determined inter-prediction mode.

[0183] The GBPR unit (227) may round horizontal and vertical displacements to the same precision level for different inter-prediction modes. For example, the current block may be the first current block, the prediction block may be the first prediction block, the horizontal and vertical displacements may be the first horizontal and vertical displacements, and the rounded horizontal and vertical displacements may be the first rounded horizontal and vertical displacements. In some examples, the GBPR unit (227) may determine a second prediction block for inter-predicting the second current block and determine second horizontal and vertical displacements for gradient-based prediction refinement of one or more samples of the second prediction block. To generate the second rounded horizontal and vertical displacements, the GBPR unit (227) may round the second horizontal and vertical displacements to the same precision level as the first horizontal and vertical displacements.

[0184] In some cases, the inter-prediction mode for inter-predicting the first current block and the inter-prediction mode for the second current block may be different. For example, among the different inter-prediction modes, the first mode is an affine mode, and among the different inter-prediction modes, the second mode is a bidirectional optical flow (BDOF) mode.

[0185] The precision level at which horizontal and vertical displacements are rounded may be predefined and stored for use by the GBPR unit (227), or the GBPR unit (227) may determine the precision level and the video encoder (200) may signal the precision level. As an example, the precision level is 1 / 64.

[0186] The GBPR unit (227) may determine one or more rectification offsets based on rounded horizontal and vertical displacements. For example, the GBPR unit (227) determines a first gradient based on a first set of samples of one or more samples of a prediction block (e.g., using the samples of the prediction block described above g x (determining (i,j)), a second gradient may also be determined based on a second set of samples of one or more samples of the prediction block (e.g., using the samples of the prediction block described above, g y (determining (i,j)). The GBPR unit (227) may determine one or more shaping offsets based on rounded horizontal and vertical displacements and first and second gradients. In some examples, the GBPR unit (227) may clip one or more shaping offsets if the value of one or more shaping offsets is too high (e.g., greater than a threshold).

[0187] The GBPR unit (227) may modify one or more samples of a prediction block based on one or more determined refinement offsets or one or more clipped refinement offsets to generate a modified prediction block (e.g., one or more modified samples forming a modified prediction block). For example, the GBPR unit (227) It can also be determined, where, g x (i,j) is the first gradient with respect to one or more samples located at (i,j), and Δv x (i,j) is the rounded horizontal displacement for one or more samples located at (i,j), and g y (i,j) is the second gradient with respect to the samples of one or more samples located at (i,j), and Δv y (i,j) is the rounded vertical displacement of one or more samples located at (i,j). In some examples, Δv x and Δv y may be the same for each of the samples (i,j) of the prediction block.

[0188] The resulting modified samples may form a prediction block (e.g., a modified prediction block) in gradient-based prediction refinement. That is, the modified prediction block is used as a prediction block in gradient-based prediction refinement. The mode selection unit (202) provides the prediction block to the residual generation unit (204). The residual generation unit (204) receives the raw, unencoded version of the current block from the video data memory (230) and the prediction block from the mode selection unit (202). The residual generation unit (204) calculates the sample-by-sample differences between the current block and the prediction block. The resulting sample-by-sample differences define the residual block for the current block. In some examples, the residual generation unit (204) may also determine the differences between sample values ​​in the residual block to generate the residual block using residual differential pulse code modulation (RDPCM). In some examples, the residual generation unit (204) may be formed using one or more subtractor circuits that perform binary subtraction.

[0189] In the examples where the mode selection unit (202) partitions the CUs into PUs, each PU may be associated with a luminance prediction unit and a corresponding chroma prediction unit. The video encoder (200) and the video decoder (300) may support PUs of various sizes. As indicated above, the size of the CU may refer to the size of the luminance coding block of the CU, and the size of the PU may refer to the size of the luminance prediction unit of the PU. Assuming that the size of a particular CU is 2Nx2N, the video encoder (200) may support PU sizes of 2Nx2N or NxN for intra-prediction, and may support symmetric PU sizes of 2Nx2N, 2NxN, Nx2N, NxN, etc. for inter-prediction. The video encoder (200) and video decoder (300) may also support asymmetric partitioning for PU sizes of 2NxnU, 2NxnD, nLx2N, and nRx2N for inter-prediction.

[0190] In examples where the mode selection unit (202) does not further partition the CU into PUs, each CU may be associated with a luminance coding block and a corresponding chroma coding block. As described above, the size of the CU may refer to the size of the luminance coding block of the CU. The video encoder (200) and the video decoder (300) may support CU sizes of 2Nx2N, 2NxN, or Nx2N.

[0191] For other video coding techniques such as intra-block copy mode coding, affine mode coding, and linear model (LM) mode coding, as some examples, the mode selection unit (202) generates a prediction block for the current block being encoded through individual units associated with the coding techniques. In some examples, such as palette mode coding, the mode selection unit (202) may not generate a prediction block, but instead generates syntax elements indicating a method for restoring the block based on a selected palette. In such modes, the mode selection unit (202) may provide these syntax elements to the entropy encoding unit (220) to be encoded.

[0192] As described above, the residual generation unit (204) receives video data for the current block and the corresponding prediction block. Then, the residual generation unit (204) generates a residual block for the current block. To generate the residual block, the residual generation unit (204) calculates the sample-by-sample differences between the current block and the prediction block.

[0193] The transformation processing unit (206) applies one or more transformations to the residual block to generate a block of transformation coefficients (referred to herein as a “transformation coefficient block”). The transformation processing unit (206) may form the transformation coefficient block by applying various transformations to the residual block. For example, the transformation processing unit (206) may apply the Discrete Cosine Transform (DCT), the Directional Transform, the Karhunen-Loeve Transform (KLT), or a conceptually similar transformation to the residual block. In some examples, the transformation processing unit (206) may perform multiple transformations on the residual block, such as a first transformation and a second transformation, such as a rotation transformation. In some examples, the transformation processing unit (206) does not apply transformations to the residual block.

[0194] The quantization unit (208) may quantize the transformation coefficients in the transformation coefficient block to generate a quantized transformation coefficient block. The quantization unit (208) may quantize the transformation coefficients of the transformation coefficient block according to the quantization parameter (QP) value associated with the current block. The video encoder (200) may adjust the degree of quantization applied to the coefficient blocks associated with the current block by adjusting the QP value associated with the CU (e.g., via the mode selection unit (202)). Quantization may introduce a loss of information, and thus, the quantized transformation coefficients may have lower precision than the original transformation coefficients generated by the transformation processing unit (206).

[0195] The inverse quantization unit (210) and the inverse transform processing unit (212) may each apply inverse quantization and inverse transforms to the quantized transform factor block to restore a residual block from the transform factor block. The restoration unit (214) may generate a restored block corresponding to the current block (potentially having some degree of distortion) based on the prediction block generated by the mode selection unit (202) and the restored residual block. For example, the restoration unit (214) may generate a restored block by adding samples of the restored residual block to corresponding samples from the prediction block generated by the mode selection unit (202).

[0196] The filter unit (216) may perform one or more filter operations on the restored blocks. For example, the filter unit (216) may perform deblocking operations to reduce blocking artifacts along the edges of the CUs. In some examples, the operations of the filter unit (216) may be skipped.

[0197] The video encoder (200) stores the restored blocks in the DPB (218). For example, in examples where the operations of the filter unit (216) are not required, the restoration unit (214) may store the restored blocks in the DPB (218). In examples where the operations of the filter unit (216) are required, the filter unit (216) may store the filtered restored blocks in the DPB (218). The motion estimation unit (222) and the motion compensation unit (224) may take a reference picture from the DPB (218) formed from the restored (and potentially filtered) blocks and inter-predict blocks of subsequently encoded pictures. Additionally, the intra-prediction unit (226) may use the restored blocks in the DPB (218) of the current picture to intra-predict other blocks in the current picture.

[0198] Generally, the entropy encoding unit (220) may entropy encode syntax elements received from other functional components of the video encoder (200). For example, the entropy encoding unit (220) may entropy encode blocks of quantized transform factors from the quantization unit (208). As another example, the entropy encoding unit (220) may entropy encode prediction syntax elements from the mode selection unit (202) (e.g., motion information for inter-prediction or intra-mode information for intra-prediction). The entropy encoding unit (220) may perform one or more entropy encoding operations on syntax elements, which are other examples of video data, to generate entropy-encoded data. For example, the entropy encoding unit (220) may perform a context-adaptive variable-length coding (CAVLC) operation, a CABAC operation, a V2V (variable-to-variable) length coding operation, a syntax-based context-adaptive binary arithmetic coding (SBAC) operation, a probability interval partitioning entropy (PIPE) coding operation, an exponential-Golomb encoding operation, or other types of entropy encoding operations on the data. In some examples, the entropy encoding unit (220) may operate in a bypass mode where syntax elements are not entropy encoded.

[0199] The video encoder (200) may output a bitstream containing entropy-encoded syntax elements necessary to restore blocks of a picture or slice. In particular, the entropy encoding unit (220) may output a bitstream.

[0200] The operations described above are described with respect to blocks. Such descriptions should be understood as operations for Luma coding blocks and / or Chroma coding blocks. As described above, in some examples, the Luma coding blocks and Chroma coding blocks are Luma and Chroma components of CU. In some examples, the Luma coding blocks and Chroma coding blocks are Luma and Chroma components of PU.

[0201] In some examples, operations performed on a luminal coding block do not need to be repeated for a chroma coding block. As an example, operations to identify the motion vector (MV) and reference picture for a luminal coding block do not need to be repeated to identify the MV and reference picture for chroma blocks. Rather, the MV for the luminal coding block may be scaled to determine the MV for the chroma blocks, and the reference picture may be the same. As another example, the intra-prediction process may be the same for the luminal coding blocks and the chroma coding blocks.

[0202] FIG. 4 is a block diagram illustrating an exemplary video decoder (300) that may perform the techniques of the present disclosure. FIG. 4 is provided for illustrative purposes and is not limited to techniques as broadly illustrated and described in the present disclosure. For illustrative purposes, the present disclosure describes a video decoder (300) and describes it according to the techniques of VVC and HEVC. However, the techniques of the present disclosure may be performed by video coding devices configured for other video coding standards.

[0203] In the example of FIG. 4, the video decoder (300) includes a coded picture buffer (CPB) memory (320), an entropy decoding unit (302), a prediction processing unit (304), an inverse quantization unit (306), an inverse transformation processing unit (308), a restoration unit (310), a filter unit (312), and a decoded picture buffer (DPB) (314). Any or all of the CPB memory (320), the entropy decoding unit (302), the prediction processing unit (304), the inverse quantization unit (306), the inverse transformation processing unit (308), the restoration unit (310), the filter unit (312), and the DPB (314) may be implemented in one or more processors or in a processing circuit. Furthermore, the video decoder (300) may include additional or alternative processors or processing circuits to perform these and other functions.

[0204] The prediction processing unit (304) includes a motion compensation unit (316), an intra-prediction unit (318), and a gradient-based prediction refinement (GBPR) unit (319). The prediction processing unit (304) may include additional units to perform predictions according to different prediction modes. As examples, the prediction processing unit (304) may include a palette unit, an intra-block copy unit (which may form part of the motion compensation unit (316)), an affine unit, a linear model (LM) unit, etc. In other examples, the video decoder (300) may include more, fewer, or different functional components.

[0205] Although the GBPR unit (319) is exemplified as being separate from the motion compensation unit (316), in some examples, the GBPR unit (319) may be part of the motion compensation unit (316). The GBPR unit (319) is depicted separately from the motion compensation unit (316) for ease of understanding and should not be considered as limiting.

[0206] The CPB memory (320) may store video data, such as an encoded video bitstream, to be decoded by components of the video decoder (300). The video data stored in the CPB memory (320) may be obtained, for example, from a computer-readable medium (110) (Fig. 1). The CPB memory (320) may include a CPB that stores encoded video data (e.g., syntax elements) from the encoded video bitstream. Additionally, the CPB memory (320) may store video data other than syntax elements of the encoded picture, such as transient data representing outputs from various units of the video decoder (300). The DPB (314) generally stores decoded pictures that the video decoder (300) may output and / or use as reference video data when decoding subsequent data or pictures of the encoded video bitstream. CPB memory (320) and DPB (314) may be formed by any of the various memory devices, such as dynamic random access memory (DRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices, including synchronous DRAM (SDRAM). CPB memory (320) and DPB (314) may be provided by the same memory device or by separate memory devices. In various examples, CPB memory (320) may be on-chip with respect to other components of the video decoder (300) or off-chip with respect to those components.

[0207] Additionally or alternatively, in some examples, the video decoder (300) may retrieve coded video data from memory (120) (Fig. 1). That is, the memory (120) may store data as discussed above in CPB memory (320). Likewise, the memory (120) may store instructions to be executed by the video decoder (300) when some or all of the functions of the video decoder (300) are implemented in software to be executed by the processing circuit of the video decoder (300).

[0208] The various units illustrated in FIG. 4 are exemplified to aid in understanding the operations performed by the video decoder (300). The units may be implemented as fixed-function circuits, programmable circuits, or a combination thereof. Similar to FIG. 3, fixed-function circuits refer to circuits that provide specific functions and are pre-configured for the operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functions in the operations that can be performed. For example, programmable circuits may execute software or firmware that causes the programmable circuits to operate in a manner defined by instructions of the software or firmware. Fixed-function circuits may execute software instructions (e.g., to receive parameters or output parameters), but the types of operations performed by the fixed-function circuits are generally invariant. In some examples, one or more of the units may be separate circuit blocks (fixed-function or programmable), and in some examples, one or more units may be integrated circuits.

[0209] The video decoder (300) may include ALUs, EFUs, digital circuits, analog circuits, and / or programmable cores formed from programmable circuits. In examples where the operations of the video decoder (300) are performed by software running on the programmable circuits, on-chip or off-chip memory may store instructions (e.g., object code) of the software that the video decoder (300) receives and executes.

[0210] The entropy decoding unit (302) receives encoded video data from the CPB and may regenerate syntax elements by entropy decoding the video data. The prediction processing unit (304), the inverse quantization unit (306), the inverse transform processing unit (308), the restoration unit (310), and the filter unit (312) may generate decoded video data based on syntax elements extracted from the bitstream.

[0211] Generally, the video decoder (300) restores the picture on a block-by-block basis. The video decoder (300) may also perform restoration operations for each block individually (wherein the block currently being restored, i.e., being decoded, may be referred to as the “current block”).

[0212] The entropy decoding unit (302) may entropy decode not only conversion information such as quantization parameters (QP) and / or conversion mode indication(s), but also syntax elements defining the quantized conversion coefficients of the quantized conversion coefficient block. The inverse quantization unit (306) may use the QP associated with the quantized conversion coefficient block to determine the degree of quantization and, similarly, the degree of inverse quantization to be applied by the inverse quantization unit (306). The inverse quantization unit (306) may, for example, perform a bit-by-bit left-shift operation to inversely quantize the quantized conversion coefficients. By doing so, the inverse quantization unit (306) may form a conversion coefficient block containing the conversion coefficients.

[0213] After the inverse quantization unit (306) forms the transform coefficient block, the inverse transform processing unit (308) may apply one or more inverse transforms to the transform coefficient block to generate a residual block associated with the current block. For example, the inverse transform processing unit (308) may apply an inverse DCT, an inverse integer transform, an inverse KLT (Karhunen-Loeve transform), an inverse rotation transform, an inverse directional transform, or other inverse transforms to the coefficient block.

[0214] Furthermore, the prediction processing unit (304) generates a prediction block according to the prediction information syntax elements entropy-decoded by the entropy decoding unit (302). For example, if the prediction information syntax elements indicate that the current block is inter-predicted, the motion compensation unit (316) may generate a prediction block. In this case, the prediction information syntax elements may indicate a motion vector identifying the position of the reference block in the reference picture as well as the position of the current block in the current picture, as well as the reference picture in the DPB (314) from which the reference block is to be extracted. The motion compensation unit (316) may generally perform the inter-predicting process in a manner substantially similar to that described for the motion compensation unit (224) (Fig. 3).

[0215] As another example, if the prediction information syntax elements indicate that the current block is intra-predicted, the intra-predict unit (318) may generate a prediction block according to the intra-predict mode indicated by the prediction information syntax elements. Again, the intra-predict unit (318) may generally perform the intra-predict process in a manner substantially similar to that described for the intra-predict unit (226) (Fig. 3). The intra-predict unit (318) may extract data of neighbor samples for the current block from the DPB (314).

[0216] As another example, if the prediction information syntax elements indicate that gradient-based prediction refinement is enabled, the GBPR unit (319) may modify samples of the prediction block to create a modified prediction block used to restore the current block (e.g., to create modified samples that form the modified prediction block).

[0217] The GBPR unit (319) may be configured to perform the exemplary techniques described in this disclosure for gradient-based prediction refinement. For example, the GBPR unit (319), together with the motion compensation unit (316), may determine a prediction block for inter-predicting the current block (e.g., based on a motion vector determined by the prediction processing unit (304)). The GBPR unit (319) determines horizontal and vertical displacements (e.g., Δv) for gradient-based prediction refinement of one or more samples of the prediction block. x and Δv y ) may also be determined. As an example, the GBPR unit (319) may determine an inter-prediction mode based on prediction information syntax elements to inter-predict the current block. In some examples, the GBPR unit (319) may determine horizontal and vertical displacements based on the determined inter-prediction mode.

[0218] The GBPR unit (319) may round horizontal and vertical displacements to the same precision level for different inter-prediction modes. For example, the current block may be the first current block, the prediction block may be the first prediction block, the horizontal and vertical displacements may be the first horizontal and vertical displacements, and the rounded horizontal and vertical displacements may be the first rounded horizontal and vertical displacements. In some examples, the GBPR unit (319) may determine a second prediction block for inter-predicting the second current block and determine second horizontal and vertical displacements for gradient-based prediction refinement of one or more samples of the second prediction block. To generate the second rounded horizontal and vertical displacements, the GBPR unit (319) may round the second horizontal and vertical displacements to the same precision level as the first horizontal and vertical displacements.

[0219] In some cases, the inter-prediction mode for inter-predicting the first current block and the inter-prediction mode for the second current block may be different. For example, among the different inter-prediction modes, the first mode is an affine mode, and among the different inter-prediction modes, the second mode is a bidirectional optical flow (BDOF) mode.

[0220] The precision level at which horizontal and vertical displacements are rounded may be predefined and stored for use by the GBPR unit (319), or the GBPR unit (319) may receive information indicating the precision level in the signaled information (e.g., precision level is signaled). As an example, the precision level is 1 / 64.

[0221] The GBPR unit (319) may determine one or more rectification offsets based on rounded horizontal and vertical displacements. For example, the GBPR unit (319) determines a first gradient based on a first set of samples of one or more samples of a prediction block (e.g., using the samples of the prediction block described above g x (determining (i,j)), a second gradient may also be determined based on a second set of samples of one or more samples of the prediction block (e.g., using the samples of the prediction block described above, g y (determining (i,j)). The GBPR unit (319) may determine one or more shaping offsets based on rounded horizontal and vertical displacements and first and second gradients. In some examples, the GBPR unit (319) may clip one or more shaping offsets if the value of one or more shaping offsets is too high (e.g., greater than a threshold).

[0222] The GBPR unit (319) may modify one or more samples of a prediction block based on one or more determined refinement offsets or one or more clipped refinement offsets to generate a modified prediction block (e.g., one or more modified samples forming a modified prediction block). For example, the GBPR unit (319) It can also be determined, where, g x (i,j) is the first gradient with respect to one or more samples located at (i,j), and Δv x (i,j) is the rounded horizontal displacement for one or more samples located at (i,j), and g y (i,j) is the second gradient with respect to the samples of one or more samples located at (i,j), and Δv y (i,j) is the rounded vertical displacement of one or more samples located at (i,j). In some examples, Δv x and Δv y may be the same for each of the samples (i,j) of the prediction block.

[0223] The resulting modified samples may form a modified prediction block in gradient-based prediction refinement. That is, the modified prediction block may be used as a prediction block in gradient-based prediction refinement. The restoration unit (310) may restore the current block using the prediction block and the residual block. For example, the restoration unit (310) may restore the current block by adding samples of the residual block to the corresponding samples of the prediction block.

[0224] The filter unit (312) may perform one or more filter operations on the restored blocks. For example, the filter unit (312) may perform deblocking operations to reduce blocking artifacts along the edges of the restored blocks. The operations of the filter unit (312) are not necessarily performed in all examples.

[0225] The video decoder (300) may store the restored blocks in the DPB (314). As discussed above, the DPB (314) may provide reference information to the prediction processing unit (304), such as samples of previously decoded pictures for subsequent motion compensation and samples of the current picture for intra-prediction. Furthermore, the video decoder (300) may output the decoded pictures from the DPB for subsequent presentation onto a display device such as the display device (118) of FIG. 1.

[0226] According to the first technique of this disclosure, a video coder (e.g., a video encoder (200) and / or a video decoder (300)) calculates a motion vector at position pixel position (m, n) and a subblock MV based on the subblock MVs (i.e., and It is also possible to derive the differences in the x and y components between ). For example, and When deriving, if affine motion parameters a, b, c, d, e, and f are calculated from CPMVS (control point motion vectors), it may be necessary to store the CPMVS of each block in a motion buffer. Since CPMVs have three MVs for each predicted direction instead of one MV as in normal inter-mode, this storage of the CPMVS of each block can significantly increase the buffer size. Accordingly, the present disclosure describes a method for a video coder based on sub-block MVs and Explains the process of performing the derivation of

[0227] For a 6-parameter affine model, three different subblock MVs may be selected that are not all in the same subblock row and column. In a 4-parameter affine model, two different subblock MVs are selected. In some examples, the selected subblock MVs are similar to the CPMVs described above, It can be used as, and here, and is in the same subblock row and and is in the same subblock column. After that, the parameter a Is Calculated as, and parameters b Is Calculated as, and parameters c Is Calculated as, and parameters d Is It is calculated as. In the case of a 4-parameter affine mode, the parameter a Is Calculated as, and parameters c Is It is calculated as, and parameter b is -c It is set to be the same as and , The parameter d is a It is set to be the same as . W is and It is the distance between, and H is and It is the distance between. However, in some examples, the three sub-block MVs are selected regardless of whether a 6-parameter affine model or a 4-parameter affine model is used.

[0228] The video coder selects subblock MVs such that W is equal to blkW / 2 and H is equal to blkH / 2. In one example, as illustrated in FIG. 6, is the subblock MV of the top-left subblock at position (0, 0), and is the subblock MV of the middle upper subblock at position (blkW / 2, 0), and is the subblock MV of the subblock MV of the middle left subblock at position (0, blkH / 2). In another example, is the subblock MV of the middle upper subblock at position (blkW / 2-sbW, 0), and is the subblock MV of the upper-right subblock at position (blkW-sbW, 0), and is the subblock MV of the intermediate central subblock at location (blkW / 2-sbW, blkH / 2).

[0229] According to the second technique of the present disclosure, a video coder and Clipping can also be performed. Gradient-based refinement offset calculation is and It can be estimated as being small. In this technique, the video coder uses a predefined threshold for the absolute value ( ) so as to be less than and You can also clip it.

[0230] As an example, a predefined threshold in offset calculation and gradient The multiplication between them may be configured so that it does not cause a buffer overflow. For example, if the budget for the multiplication result is 16 bits, the maximum absolute value is 1<<15 (1 bit for the sign), and or must not exceed 1<<15. Given that the gradient is expressed in k bits, is set to be the same as 1<<(15-k).

[0231] As another example, a predefined threshold (e.g., ) can also be set to the same value as in Bidirectional Optical Flow (BDOF), that is, am. It can also represent half pixels. and If the base unit for is 1 / q pixel, is q / 2.

[0232] As another example, a predefined threshold (e.g., ) Is It may also be set to be equal to the minimum value between 1 and (15-k).

[0233] According to the third technique of the present disclosure, a video coder and The precision of can also be set to the same precision as in BDOF. In one example, and The precision of is determined by shfit1 in Section 1.3. Therefore, and One unit of is 1 / (1< <shift1) 픽셀이다. 일 예에서, 시프트는 6 과 동일하게 설정된다. 다른 예에서, shift1 는 max( 2, 14 bitDepth) 와 동일하게 설정되고, bitDepth 는 인코딩 / 디코딩을 위한 비디오 신호의 내부 비트 심도이다.

[0234] According to the fourth technique of the present disclosure, the video coder may perform gradient calculation for predictive refinement of affine modes using the same process as in BDOF. As such, the same module of the video coder may be used for both gradient calculation for predictive refinement of affine modes and BDOF. However, the video coder may use different padding methods for prediction samples in the extended region.

[0235] As an example, a video coder may generate prediction samples in an extended area (white positions) by taking reference samples from nearby integer positions directly (using the floor() operation on the coordinates) without interpolation.

[0236] As another example, a video coder may generate prediction samples in an extended area (white positions) by taking reference samples from nearby integer positions directly (using a round() operation on the coordinates) without interpolation.

[0237] As another example, if arbitrary sample values ​​outside the subblock boundaries are needed, the video coder may pad (i.e., repeat) the necessary samples from their nearest neighbors. This can also be applied to gradient calculations in BDOF.

[0238] According to the fifth technique of the present disclosure, the video coder may perform clipping of the clarification results. In inter-prediction, the motion compensation prediction signal of a block is typically clipped to the same range as the original signal of the block. However, in bidirectional motion compensation, the motion compensation prediction signal for each direction is maintained at an intermediate precision and range to improve accuracy. After the weighted averaging process of bidirectional motion compensation, the results are rounded and clipped to the same range and precision as the original signal of the block. In this fifth technique, in the case of bidirectional prediction, the video coder may clip the results of the prediction clarification to have the same intermediate precision and range as in normal motion compensation. For example, the number of bits for the intermediate precision is 14, and the video coder may clip the prediction clarification results to the range of -(1<<14) to (1<<14).

[0239] FIG. 7 is a flowchart illustrating an exemplary method for coding video data. The current block may include a current CU. The example in FIG. 7 describes a processing circuit. Examples of processing circuits include fixed-function and / or programmable circuits for a video encoder (200), such as a GBPR unit (227), and a video decoder (300), such as a GBPR unit (319).

[0240] In one or more examples, memory may be configured to store samples of prediction blocks. For example, DPB (218) or DPB (314) may be configured to store samples of prediction blocks used for inter-prediction. An intra-block copy may be considered as an exemplary inter-prediction mode, in which case the block vector used for the intra-block copy is an example of a motion vector.

[0241] The processing circuit may determine a prediction block stored in memory to inter-predict the current block (350). The processing circuit may determine horizontal and vertical displacements (e.g., Δv) for gradient-based prediction refinement of one or more samples of the prediction block. x and Δv y ) may also be determined (352). As an example, the processing circuit may determine an inter-prediction mode for inter-predicting the current block. In some examples, the processing circuit may determine horizontal and vertical displacements based on the determined inter-prediction mode.

[0242] The processing circuit may round horizontal and vertical displacements to the same precision level for different inter-prediction modes (354). For example, the current block may be the first current block, the prediction block may be the first prediction block, the horizontal and vertical displacements may be the first horizontal and vertical displacements, and the rounded horizontal and vertical displacements may be the first rounded horizontal and vertical displacements. In some examples, the processing circuit may determine a second prediction block for inter-predicting the second current block and determine second horizontal and vertical displacements for gradient-based prediction refinement of one or more samples of the second prediction block. To generate the second rounded horizontal and vertical displacements, the processing circuit may round the second horizontal and vertical displacements to the same precision level as the first horizontal and vertical displacements.

[0243] In some cases, the inter-prediction mode for inter-predicting the first current block and the inter-prediction mode for the second current block may be different. For example, among the different inter-prediction modes, the first mode is an affine mode, and among the different inter-prediction modes, the second mode is a bidirectional optical flow (BDOF) mode.

[0244] The precision level at which horizontal and vertical displacements are rounded may be predefined or signaled. As an example, the precision level is 1 / 64.

[0245] The processing circuit may determine one or more refinement offsets based on rounded horizontal and vertical displacements (356). For example, the processing circuit determines a first gradient based on a first set of samples of one or more samples of a prediction block (e.g., using the samples of the prediction block described above g x(determining (i,j)), a second gradient may also be determined based on a second set of samples of one or more samples of the prediction block (e.g., using the samples of the prediction block described above, g y (determining (i,j)). The processing circuit may determine one or more refinement offsets based on rounded horizontal and vertical displacements and first and second gradients. In some examples, the processing circuit may clip one or more refinement offsets if the value of one or more refinement offsets is too high (e.g., greater than a threshold).

[0246] The processing circuit may modify one or more samples of a prediction block based on one or more determined refinement offsets or one or more clipped refinement offsets to generate a modified prediction block (e.g., one or more modified samples forming the modified prediction block) (358). For example, the processing circuit It can also be determined, where, g x (i,j) is the first gradient with respect to one or more samples located at (i,j), and Δv x (i,j) is the rounded horizontal displacement for one or more samples located at (i,j), and g y (i,j) is the second gradient with respect to the samples of one or more samples located at (i,j), and Δv y (i,j) is the rounded vertical displacement of one or more samples located at (i,j). In some examples, Δv x and Δv y may be the same for each of the samples (i,j) of the prediction block.

[0247] The processing circuit may code (e.g., encode or decode) the current block based on the modified prediction block (e.g., one or more modified samples of the modified prediction block) (360). For example, for video decoding, the processing circuit (e.g., video decoder (300)) may restore the current block based on the modified prediction block (e.g., by adding one or more modified samples to the received residual values). For video encoding, the processing circuit (e.g., video encoder (200)) may determine the residual values ​​(e.g., of the residual block) between the current block and the modified prediction block (e.g., one or more modified samples of the modified prediction block), and signal information indicating the residual values.

[0248] A non-limiting exemplary list of examples of the present disclosure is described below.

[0249] Example 1. A method for coding video data, the present method comprises: a step of performing subblock-based affine motion compensation to obtain a prediction signal for a current block of video data; and a step of refining the prediction signal by adding at least an offset to pixel locations within the prediction signal, wherein the values ​​of the offsets are derived based on the values ​​of a plurality of subblock motion vectors (MVs) for subblocks of the current block that are not all in the same subblock row or column of the current block.

[0250] Example 2. In the method of Example 1, and Based on ( m,n It further includes a step of determining the value of the offset in ).

[0251] Example 3. In the method of Example 2, the position of the predicted signal ( m,n The step of determining the value of the offset in ) includes the step of determining the value of the offset according to the following expression:

[0252]

[0253] In the equation is the horizontal gradient of the predicted signal and is the vertical gradient of the predicted signal.

[0254] Example 4. In the method of Example 2 or 3, based on the values ​​of a plurality of sub-block MVs and It further includes a step of determining the values ​​of.

[0255] Example 5. In the method of Example 4, and The step of determining the value of a Parameters, b Parameters, c parameters, and d Based on parameters and Includes a step of determining the value of.

[0256] Example 6. In the method of Example 5, a Parameters, b Parameters, c parameters, and d Based on parameters and The step of deriving the value of is according to the following equation and It includes a step of deriving,

[0257]

[0258] In the equation sbW represents the subblock width of the subblocks of the current block, and sbH represents the height of the subblocks of the current block, and ( m,n ) represents the pixel location within the current block.

[0259] Example 7. In the method of Example 5 or 6, according to the following formula a Parameters, b Parameters, cparameters, and d It includes a step for deriving parameters, and the affine motion model is expressed by six parameters:

[0260]

[0261] In the equation W represents the distance between the 1st MV of the affine motion model and the 2nd MV of the affine motion model. H represents the distance between the 1st MV of the affine motion model and the 3rd MV of the affine motion model.

[0262] Example 8. Any method of Examples 5-7 further comprises the step of deriving parameters a, b, c, and d according to the following equation, wherein the affine motion model is expressed by four parameters:

[0263]

[0264] W represents the distance between the 1st MV of the affine motion model and the 2nd MV of the affine motion model.

[0265] Example 9. In the method of Example 7 or Example 8, the first MV of the affine motion model is And, the 2nd MV of the affine motion model is And, the 3rd MV of the affine motion model is am.

[0266] Example 10. In any of the methods of Examples 2-9, to have an absolute value less than or equal to a predefined threshold. and It further includes a step of clipping.

[0267] Example 11. Any method of Examples 2-10 further includes the step of performing bidirectional optical flow (BDOF) correction on the prediction signals for the current block.

[0268] Example 12. With the same precision as used to perform BDOF refinement in the method of Example 11. and It further includes a step of saving.

[0269] Example 13. In the method of Example 11 or 12, the step of performing BDOF refinement includes the step of performing gradient calculation.

[0270] Example 14. In the method of Example 13, the step of performing gradient calculation to perform BDOF refinement uses the same process as calculating the horizontal gradient and / or vertical gradient of the prediction signal.

[0271] Example 15. Any method of Examples 1-14 further includes the step of clipping a refined prediction signal to have the same intermediate precision as in non-affine motion compensation.

[0272] Example 16. In the method of Example 15, the number of bits for intermediate precision n인 In this case, the step of clipping the refined prediction signal is to clip the refined prediction signal -(1<< n ) to (1<< n Includes a step of clipping to the range of ).

[0273] Example 17. In any of the methods of Examples 1-16, coding includes decoding.

[0274] Example 18. In any method of Examples 1-17, coding includes encoding.

[0275] Example 19. A device for coding video data, comprising one or more means for performing any of the methods of Examples 1-18.

[0276] Example 20. In the device of Example 19, one or more means include one or more processors implemented in the circuit portion.

[0277] Example 21. The device of Examples 19 and 20 includes a memory to store video data.

[0278] Example 22. Any of the devices of Examples 19-21 further includes a display configured to display decoded video data.

[0279] Example 23. In any of the devices of Examples 19-22, the device includes one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

[0280] Example 24. In any of the devices of Examples 19-23, the device includes a video decoder.

[0281] Example 25. In any of the devices of Examples 19-24, the device includes a video encoder.

[0282] Example 26. A computer-readable storage medium in which instructions are stored, wherein, when executed, the instructions cause one or more processors to perform any of the methods of Examples 1-18.

[0283] It should be recognized that, depending on the example, any specific operations or events of the techniques described herein may be performed in different sequences and may be added, merged, or removed in their entirety (e.g., not all described operations or events are essential for the implementation of the techniques). Furthermore, in certain examples, operations or events may be performed simultaneously rather than sequentially, for example, through multi-threaded processing, interrupt processing, or multiple processors.

[0284] In one or more examples, the described functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media corresponding to media of the same type as data storage media, or communication media including any medium that facilitates the transmission of a computer program from one place to another according to a communication protocol, for example. In this way, computer-readable media may generally correspond to (1) non-transient types of computer-readable storage media or (2) communication media such as signals or carrier waves. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to extract instructions, codes and / or data structures for the implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0285] As an example, not a limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and can be accessed by a computer. Additionally, any connection is appropriately named as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other ephemeral media, but instead are oriented toward non-ephemeral types of storage media. As used herein, disks and discs include compact discs (CDs), laser discs, optical discs, digital multifunction discs (DVDs), floppy discs, and Blu-ray discs, wherein disks typically reproduce data magnetically, but discs reproduce data optically using lasers. The above combinations should also be included within the scope of computer-readable media.

[0286] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or separate logic circuits. Accordingly, the terms “processor” and “processing circuit” as used herein may refer to any of the aforementioned structures or any other structures suitable for implementing the techniques described herein. Additionally, in some embodiments, the functions described herein may be provided within dedicated hardware and / or software modules integrated in codecs configured for or combined with encoding and decoding. Furthermore, the techniques may be fully implemented in one or more circuits or logic elements.

[0287] The techniques of the present disclosure may be implemented in a wide variety of devices or apparatus, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chip sets). While various components, modules, or units are described in the present disclosure to highlight functional aspects of devices configured to perform the disclosed techniques, they do not necessarily require implementation by different hardware units. Rather, as described above, various units may be combined into a codec hardware unit including one or more processors as described above, along with suitable software and / or firmware, or provided by a set of interoperable hardware units.

[0288] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

Claim 1 A method for decoding video data, comprising: determining a prediction block for inter-predicting a current block; determining horizontal and vertical displacements for gradient-based prediction refinement of one or more samples of the prediction block; rounding the horizontal and vertical displacements to the same precision level for different inter-prediction modes, including affine mode and bi-directional optical flow (BDOF) mode; determining a first gradient based on a first set of samples of the one or more samples of the prediction block; determining a second gradient based on a second set of samples of the one or more samples of the prediction block; and determining one or more refinement offsets based on the rounded horizontal and vertical displacements and the first and second gradients, wherein the step of determining the one or more refinement offsets Includes a step of determining, and g x (i,j) is the first gradient for the sample of the one or more samples located at (i,j), and Δv x (i,j) is the rounded horizontal displacement of one or more samples located at (i,j) with respect to the sample, and g y (i,j) is the second gradient with respect to the sample of the one or more samples located at (i,j), and Δv y A method for decoding video data, comprising: determining one or more refinement offsets, wherein (i,j) is a rounded vertical displacement of one or more samples located at (i,j); performing gradient-based prediction refinement by modifying one or more samples of the prediction block based on the determined one or more refinement offsets to generate a modified prediction block; and restoring the current block based on the modified prediction block. Claim 2 A method for decoding video data according to claim 1, further comprising the step of clipping the one or more refinement offsets, and modifying the one or more samples of the prediction block comprises modifying the one or more samples of the prediction block based on the clipped one or more refinement offsets. Claim 3 A method for decoding video data according to claim 1, further comprising the step of determining an inter-prediction mode for inter-predicting the current block, and the step of determining the horizontal and vertical displacements comprising the step of determining the horizontal and vertical displacements based on the determined inter-prediction mode. Claim 4 A method for decoding video data according to claim 1, wherein the precision level is 1 / 64. Claim 5 delete Claim 6 delete Claim 7 In claim 1, the prediction block is a first prediction block, the current block is a first current block, the horizontal and vertical displacements are first horizontal and vertical displacements, the one or more rectification offsets are first one or more rectification offsets, the rounded horizontal and vertical displacements are first rounded horizontal and vertical displacements, and the modified prediction block is a first modified prediction block; the method comprises the steps of: determining a second prediction block for inter-predicting a second current block; determining second horizontal and vertical displacements for gradient-based prediction rectification of one or more samples of the second prediction block; rounding the second horizontal and vertical displacements to the same precision level as the first horizontal and vertical displacements to generate second rounded horizontal and vertical displacements; and determining a second one or more rectification offsets based on the second rounded horizontal and vertical displacements.

2. A method for decoding video data, further comprising: a step of modifying one or more samples of the second prediction block based on one or more determined second refinement offsets to generate a modified prediction block; and a step of restoring the second current block based on the second modified prediction block. Claim 8 A method for encoding video data, comprising: determining a prediction block for inter-predicting a current block; determining horizontal and vertical displacements for gradient-based prediction refinement of one or more samples of the prediction block; rounding the horizontal and vertical displacements to the same precision level for different inter-prediction modes, including an affine mode and a bidirectional optical flow (BDOF) mode; determining a first gradient based on a first set of samples of the one or more samples of the prediction block; determining a second gradient based on a second set of samples of the one or more samples of the prediction block; and determining one or more refinement offsets based on the rounded horizontal and vertical displacements and the first and second gradients, wherein the step of determining the one or more refinement offsets Includes a step of determining, and g x (i,j) is the first gradient for the sample of the one or more samples located at (i,j), and Δv x (i,j) is the rounded horizontal displacement of one or more samples located at (i,j) with respect to the sample, and g y (i,j) is the second gradient with respect to the sample of the one or more samples located at (i,j), and Δv y A method for encoding video data, comprising: determining one or more refinement offsets, wherein (i,j) is a rounded vertical displacement of one or more samples located at (i,j); performing gradient-based prediction refinement by modifying one or more samples of the prediction block based on the determined one or more refinement offsets to generate a modified prediction block; determining residual values ​​representing the difference between the current block and the modified prediction block; and signaling information representing the residual values. Claim 9 A method for encoding video data according to claim 8, further comprising the step of clipping the one or more refinement offsets, and modifying the one or more samples of the prediction block, comprising modifying the one or more samples of the prediction block based on the clipped one or more refinement offsets. Claim 10 A method for encoding video data according to claim 8, further comprising the step of determining an inter-prediction mode for inter-predicting the current block, and the step of determining the horizontal and vertical displacements comprising the step of determining the horizontal and vertical displacements based on the determined inter-prediction mode. Claim 11 A method for encoding video data according to claim 8, wherein the precision level is 1 / 64. Claim 12 delete Claim 13 delete Claim 14 In claim 8, the prediction block is a first prediction block, the current block is a first current block, the horizontal and vertical displacements are first horizontal and vertical displacements, the one or more rectification offsets are first one or more rectification offsets, the rounded horizontal and vertical displacements are first rounded horizontal and vertical displacements, the modified prediction block is a first modified prediction block, and the residual values ​​include first residual values, and the method comprises: determining a second prediction block for inter-predicting a second current block; determining second horizontal and vertical displacements for gradient-based prediction rectification of one or more samples of the second prediction block; rounding the second horizontal and vertical displacements to the same precision level as the first horizontal and vertical displacements to generate second rounded horizontal and vertical displacements; and based on the second rounded horizontal and vertical displacements, the A method for encoding video data, further comprising: a step of determining one or more refinement offsets; a step of modifying one or more samples of the second prediction block based on the determined second refinement offsets to generate a second modified prediction block; a step of determining second residual values ​​representing the difference between the second current block and the second modified prediction block; and a step of signaling information representing the second residual values. Claim 15 A device for coding video data, comprising a memory configured to store one or more samples of a prediction block; The apparatus includes a processing circuit, wherein the processing circuit determines the prediction block for inter-predicting a current block; determines horizontal and vertical displacements for gradient-based prediction refinement of one or more samples of the prediction block; rounds the horizontal and vertical displacements to the same precision level for different inter-prediction modes, including affine mode and bidirectional optical flow (BDOF) mode; determines a first gradient based on a first set of samples of one or more samples of the prediction block; determines a second gradient based on a second set of samples of one or more samples of the prediction block; and performs gradient-based prediction refinement, wherein to perform the gradient-based prediction refinement, the processing circuit is configured to determine at least one or more refinement offsets based on the rounded horizontal and vertical displacements and the first and second gradients, and to determine the one or more refinement offsets, the processing circuit It is configured to determine, and g x (i,j) is the first gradient for the sample of the one or more samples located at (i,j), and Δv x (i,j) is the rounded horizontal displacement of one or more samples located at (i,j) with respect to the sample, and g y (i,j) is the second gradient with respect to the sample of the one or more samples located at (i,j), and Δv y A device for coding video data configured to perform gradient-based prediction refinement, wherein (i,j) is a rounded vertical displacement of one or more samples located at (i,j); modify one or more samples of the prediction block based on determined refinement offsets to generate a modified prediction block; and to code the current block based on the modified prediction block. Claim 16 A device for coding video data according to claim 15, wherein, for coding the current block, the processing circuit is configured to restore the current block based on the modified prediction block. Claim 17 A device for coding video data according to claim 15, wherein, for coding the current block, the processing circuit determines residual values ​​representing the difference between the current block and the modified prediction block; and is configured to signal information representing the residual values. Claim 18 A device for coding video data according to claim 15, wherein the processing circuit is configured to clip the one or more refinement offsets, and to modify the one or more samples of the prediction block, the processing circuit is configured to modify the one or more samples of the prediction block based on the clipped one or more refinement offsets. Claim 19 A device for coding video data according to claim 15, wherein the processing circuit is configured to determine an inter-prediction mode for inter-predicting the current block, and to determine the horizontal and vertical displacements, the processing circuit is configured to determine the horizontal and vertical displacements based on the determined inter-prediction mode. Claim 20 A device for coding video data, wherein the precision level is 1 / 64 in claim 15. Claim 21 delete Claim 22 delete Claim 23 In claim 15, the prediction block is a first prediction block, the current block is a first current block, the horizontal and vertical displacements are first horizontal and vertical displacements, the one or more refinement offsets are first one or more refinement offsets, the rounded horizontal and vertical displacements are first rounded horizontal and vertical displacements, the modified prediction block is a first modified prediction block, and the processing circuit determines a second prediction block for inter-predicting a second current block; determines second horizontal and vertical displacements for gradient-based prediction refinement of one or more samples of the second prediction block; to generate second rounded horizontal and vertical displacements, rounds the second horizontal and vertical displacements to the same precision level as the first horizontal and vertical displacements are rounded; and based on the second rounded horizontal and vertical displacements, determines a second one or more refinement offsets A device for coding video data configured to determine; modify one or more samples of the second prediction block based on one or more determined second refinement offsets to generate a second modified prediction block; and to code the second current block based on the second modified prediction block. Claim 24 A device for coding video data according to claim 15, further comprising a display configured to display decoded video data. Claim 25 A device for coding video data according to claim 15, further comprising a camera configured to capture the video data to be encoded. Claim 26 In claim 15, the device is a device for encoding video data, comprising one or more of a camera, a computer, a wireless communication device, a broadcast receiver device, or a set-top box. Claim 27 A non-transient computer-readable storage medium storing instructions, wherein the instructions, when executed, cause one or more processors to determine a prediction block for inter-predicting a current block; cause horizontal and vertical displacements for gradient-based prediction refinement of one or more samples of the prediction block; cause the horizontal and vertical displacements to be rounded to the same level of precision for different inter-prediction modes, including affine mode and bidirectional optical flow (BDOF) mode; cause the instructions to determine a first gradient based on a first set of samples of the one or more samples of the prediction block; cause the instructions to determine a second gradient based on a second set of samples of the one or more samples of the prediction block; and cause one or more refinement offsets based on the rounded horizontal and vertical displacements and the first and second gradients, wherein the instructions causing the one or more processors to determine the one or more refinement offsets cause the one or more processors to Includes commands that determine, and g x (i,j) is the first gradient for the sample of the one or more samples located at (i,j), and Δv x (i,j) is the rounded horizontal displacement of one or more samples located at (i,j) with respect to the sample, and g y (i,j) is the second gradient with respect to the sample of the one or more samples located at (i,j), and Δv y A non-transient computer-readable storage medium that determines one or more refinement offsets, wherein (i,j) is a rounded vertical displacement of one or more samples located at (i,j); and performs gradient-based predictive refinement, wherein the instructions for the one or more processors to perform the gradient-based predictive refinement include instructions for the one or more processors to modify one or more samples of the prediction block based on the determined one or more refinement offsets in order to generate a modified prediction block; and codes the current block based on the modified prediction block. Claim 28 A device for coding video data, comprising: means for determining a prediction block for inter-predicting a current block; means for determining horizontal and vertical displacements for gradient-based prediction refinement of one or more samples of the prediction block; means for rounding the horizontal and vertical displacements to the same precision level for different inter-prediction modes, including an affine mode and a bidirectional optical flow (BDOF) mode; means for determining a first gradient based on a first set of samples of the one or more samples of the prediction block; means for determining a second gradient based on a second set of samples of the one or more samples of the prediction block; and means for determining one or more refinement offsets based on the rounded horizontal and vertical displacements and the first and second gradients, wherein the means for determining the one or more refinement offsets Includes means for determining, and g x (i,j) is the first gradient for the sample of the one or more samples located at (i,j), and Δv x (i,j) is the rounded horizontal displacement of one or more samples located at (i,j) with respect to the sample, and g y (i,j) is the second gradient with respect to the sample of the one or more samples located at (i,j), and Δv y A device for coding video data, comprising: means for determining one or more refinement offsets, wherein (i,j) is a rounded vertical displacement of one or more samples located at (i,j); means for performing gradient-based prediction refinement, wherein the means for performing gradient-based prediction refinement includes means for modifying one or more samples of the prediction block based on the determined one or more refinement offsets to generate a modified prediction block; and means for coding the current block based on the modified prediction block.

Images