Early termination of motion vector refinement process in video coding

Abstract

A video decoder is configured to: determine a bidirectional motion vector for a current block of video data; and determine that a condition is satisfied with respect to the current block based on each component of the bidirectional motion vector of the current block being less than a threshold. The video decoder is further configured to: prematurely terminate the application of a motion vector refinement process to the bidirectional motion vector of the current block based on the condition being satisfied with respect to the current block. The video decoder is further configured to: determine a prediction block for the current block based on the bidirectional motion vector of the current block, and reconstruct the current block based on the prediction block for the current block.

Description

[0001] This application claims the benefit of U.S. Patent Application 16 / 952,736, filed November 19, 2020, and U.S. Provisional Patent Application 62 / 939,325, filed November 22, 2019, the entire contents of each of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to video encoding and video decoding. Background Technology

[0003] Digital video capabilities can be integrated into a wide variety of devices, including digital televisions, digital direct broadcasting systems, wireless broadcasting systems, personal digital assistants (PDAs), laptops or desktop computers, tablets, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radio phones, so-called "smartphones," video conferencing equipment, and video streaming devices. Digital video devices implement video decoding technologies, such as those described in standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 Part 10, Advanced Video Decoding (AVC), ITU-T H.265 / High-Efficiency Video Decoding (HEVC), and extensions to such standards. By implementing such video decoding technologies, video devices can more efficiently send, receive, encode, decode, and / or store digital video information.

[0004] Video decoding techniques include spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to reduce or remove inherent redundancy in video sequences. For block-based video decoding, video slices (i.e., video pictures or portions of video pictures) can be segmented into video blocks, which may also be referred to as decoding tree units (CTUs), decoding units (CUs), and / or decoding nodes. Video blocks in intra-frame decoded (I) slices of pictures are encoded using spatial predictions about reference samples in adjacent blocks within the same picture. Video blocks in inter-frame decoded (P or B) slices of pictures can use spatial predictions about reference samples in adjacent blocks within the same picture, or temporal predictions about reference samples in other reference pictures. Pictures may be referred to as frames, and reference pictures may be referred to as reference frames. Summary of the Invention

[0005] Typically, this disclosure describes techniques related to the early termination of the motion vector thinning process. The techniques disclosed herein can be applied to extensions of existing video codecs, such as High Efficiency Video Decoding (HEVC), Multi-Functional Video Decoding (VVC), and Basic Video Decoding (EVC), or can be applied to any future video decoding standard.

[0006] Motion vector thinning processes, such as bidirectional optical flow (BDOF) and decoder-side motion vector thinning (DMVR), are decoding tools that can be used during video decoding. For example, if an early termination condition for the motion vector thinning process is met, the video decoder (e.g., a video encoder or video decoder) can terminate the motion vector thinning process before it begins. Otherwise, if the early termination condition for the motion vector thinning process is not met, the video decoder performs the motion vector thinning process.

[0007] There may be one or more issues associated with applying motion vector refinement (MDR). When MDR is applied, the video decoder may add a relatively large amount of complexity and a relatively small amount of prediction accuracy. This could increase the overall complexity of the video decoder, increase costs, and potentially slow down the decoding process.

[0008] As described herein, a video decoder (e.g., a video encoder or video decoder) can determine whether a condition is met for the current block. For example, the video decoder can determine whether a condition is met for the current block based on each component of the bidirectional motion vector being less than a threshold. That is, the video decoder can terminate the motion vector refinement process early based on whether the absolute value of the bidirectional motion vector is small (i.e., less than a threshold) to help avoid applying the motion vector refinement process while adding a relatively large amount of complexity and a relatively small amount of prediction accuracy. In this way, the video decoder can terminate the application of the motion vector refinement process to the bidirectional motion vector of the current block early, which can potentially reduce the average case complexity on the decoder side.

[0009] In one example, a method for decoding video data includes: determining bidirectional motion vectors for a current block of video data; and determining conditions satisfied with respect to the current block based on each component of the bidirectional motion vectors of the current block being less than a threshold. The method further includes: prematurely terminating a motion vector thinning process applied to the bidirectional motion vectors of the current block based on the conditions satisfied with respect to the current block. The method also includes: determining a prediction block for the current block based on the bidirectional motion vectors of the current block; and reconstructing the current block based on the prediction block for the current block.

[0010] In another example, a device includes: a memory configured to store video data; and one or more processors implemented in the circuitry. The one or more processors are configured to: determine a bidirectional motion vector of a current block of video data; and determine a condition satisfied with respect to the current block based on each component of the bidirectional motion vector of the current block being less than a threshold. The one or more processors are configured to: prematurely terminate the motion vector refinement process applied to the bidirectional motion vector of the current block based on the condition satisfied with respect to the current block. The one or more processors are configured to: determine a prediction block for the current block based on the bidirectional motion vector of the current block; and reconstruct the current block based on the prediction block for the current block.

[0011] In one example, a computer-readable storage medium having instructions stored thereon, which, when executed, cause a processor to: determine the bidirectional motion vector of a current block of video data; and determine a condition satisfied with respect to the current block based on each component of the bidirectional motion vector of the current block being less than a threshold. The instructions further cause the processor to: prematurely terminate the motion vector thinning process applied to the bidirectional motion vector of the current block based on the condition satisfied with respect to the current block. The instructions further cause the processor to: determine a prediction block for the current block based on the bidirectional motion vector of the current block; and reconstruct the current block based on the prediction block for the current block.

[0012] In one example, an apparatus includes: components for determining bidirectional motion vectors of a current block of video data; and components for determining that a condition is met regarding the current block based on each component of the bidirectional motion vectors of the current block being less than a threshold. The apparatus further includes: components for prematurely terminating the application of a motion vector thinning process to the bidirectional motion vectors of the current block based on the condition being met. The apparatus also includes: components for determining a prediction block for the current block based on the bidirectional motion vectors of the current block, and components for reconstructing the current block based on the prediction block for the current block.

[0013] Details of one or more examples will be set forth in the accompanying drawings and the following description. Other features, objectives, and advantages will become apparent from the specification, drawings, and claims. Attached Figure Description

[0014] Figure 1 This is a block diagram illustrating an example video encoding and decoding system that can perform the techniques of this disclosure.

[0015] Figure 2A This is a conceptual diagram illustrating candidates for spatially adjacent motion vectors (MVs) used in merging patterns.

[0016] Figure 2B This is a conceptual diagram illustrating spatially adjacent MV candidates for the AMVP pattern.

[0017] Figure 3A This is a conceptual diagram illustrating an example Time Motion Vector Predictor (TMVP) candidate.

[0018] Figure 3B This is a conceptual diagram illustrating an example of MV scaling.

[0019] Figure 4 This is a conceptual diagram illustrating bilateral template matching.

[0020] Figure 5 This is a conceptual diagram illustrating an example extended decoding unit (CU) region used in bidirectional optical flow (BDOF).

[0021] Figure 6 This is a block diagram illustrating an example video encoder that can perform the techniques of this disclosure.

[0022] Figure 7 This is a block diagram illustrating an example video decoder that can perform the techniques disclosed herein.

[0023] Figure 8 This is a flowchart illustrating an example method for encoding the current block that can be performed by a video encoder according to one or more techniques of this disclosure.

[0024] Figure 9 This is a flowchart illustrating an example method for decoding a current block of video data, which can be performed by a video decoder according to one or more techniques disclosed herein.

[0025] Figure 10 This is a flowchart illustrating an example method that can be performed by a video decoder according to one or more techniques disclosed herein. Detailed Implementation

[0026] Motion vector refinement processes, such as bidirectional optical flow (BDOF) and decoder-side motion vector refinement (DMVR), are decoding tools that can be used in motion compensation processes during video decoding. For example, each of BDOF and DMVR can be associated with an early termination condition. If the early termination condition for BDOF is met, the video decoder (e.g., a video encoder or video decoder) can terminate the BDOF process before it begins. Otherwise, if the early termination condition for BDOF is not met, the video decoder performs the BDOF process. Similarly, if the early termination condition for DMVR is met, the video decoder can terminate the DMVR process before it begins. If the early termination condition for DMVR is not met, the video decoder performs the DMVR process.

[0027] From October 1 to 11, 2019, at the 16th meeting (JVET-P2001-v14) of the Joint Video Experts Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC29 / WG 11 in Geneva, Switzerland, Bross et al.'s "Versatile Video Coding (Draft 7)" (hereinafter referred to as "VVC Draft 7") is a draft of the VVC standard. In VVC Draft 7, early termination based on the sum of absolute differences (SAD) is applied to both BDOF and DMVR to reduce the average complexity on the decoder side. That is, there may be one or more problems related to the application of BDOF and / or DMVR, where the video decoder can apply BDOF and / or DMVR with a relatively large amount of added complexity and a relatively small amount of prediction accuracy. This could increase the overall complexity of the video decoder, increase costs, and potentially slow down the decoding process.

[0028] According to the techniques disclosed herein, early termination techniques for BDOF and / or DMVR can be improved by considering whether the bidirectional motion vectors are symmetrical (or nearly symmetrical) or whether the absolute values ​​of the bidirectional motion vectors are small (i.e., less than a threshold). This disclosure describes techniques that can provide solutions to one or more of the aforementioned problems and / or one or more other problems. For example, in one example, a video decoder (e.g., a video encoder or video decoder) can determine whether an early termination condition is met with respect to the current block. For example, the video decoder can determine that an early termination condition is met with respect to the current block based on each component of the bidirectional motion vector of the current block being less than a threshold. In this way, the video decoder can apply early termination of the motion vector refinement process applied to the bidirectional motion vectors of the current block based on whether the bidirectional motion vectors are symmetrical (or nearly symmetrical) and / or based on whether the absolute values ​​of the bidirectional motion vectors are small, which can potentially reduce the average case of decoder-side complexity with little or no loss in decoding accuracy.

[0029] Figure 1 This is a block diagram illustrating an example video encoding and decoding system 100 that can perform the techniques of this disclosure. The techniques of this disclosure generally relate to decoding (encoding and / or decoding) video data. Typically, video data includes any data used for processing video. Thus, video data can include raw, unencoded video, encoded video, decoded (e.g., reconstructed) video, and video metadata, such as signal notification data.

[0030] like Figure 1As shown, in this example, system 100 includes a source device 102 that provides encoded video data to be decoded and displayed by a destination device 116. Specifically, source device 102 provides video data to destination device 116 via computer-readable medium 110. Source device 102 and destination device 116 can include any of a wide range of devices, including: desktop computers, mobile devices (e.g., laptops, tablets, telephone receivers such as smartphones, cameras, etc.), set-top boxes, broadcast receiver devices, televisions, display devices, digital media players, video game consoles, video streaming devices, etc. In some cases, source device 102 and destination device 116 may be equipped for wireless communication and may therefore be referred to as wireless communication devices.

[0031] exist Figure 1 In the example, source device 102 includes a video source 104, memory 106, video encoder 200, and output interface 108. Destination device 116 includes an input interface 122, video decoder 300, memory 120, and display device 118. According to this disclosure, the video encoder 200 of source device 102 and the video decoder 300 of destination device 116 can be configured to apply technologies related to DMVR and BDOF. Therefore, source device 102 represents an example of a video encoding device, while destination device 116 represents an example of a video decoding device. In other examples, the source device and destination device may include other components or arrangements. For example, source device 102 may receive video data from an external video source such as an external camera. Similarly, destination device 116 may interface with an external display device rather than including an integrated display device.

[0032] like Figure 1 The system 100 shown is merely an example. Typically, any digital video encoding and / or decoding device can perform the techniques associated with DMVR and BDOF. Source device 102 and destination device 116 are merely examples of such decoding devices, where source device 102 generates decoded video data for transmission to destination device 116. This disclosure refers to a “decoding” device as a device that performs the decoding (encoding and / or decoding) of data. Thus, video encoder 200 and video decoder 300 represent examples of decoding devices, specifically a video encoder and a video decoder, respectively. In some examples, source device 102 and destination device 116 may operate in a substantially symmetrical manner, such that each device includes video encoding and decoding components. Therefore, system 100 can support one-way or two-way video transmission between source device 102 and destination device 116, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0033] Typically, video source 104 represents the source of video data (i.e., raw, unencoded video data) and provides a continuous series of pictures (also referred to as "frames") of video data to video encoder 200, which encodes the data for the pictures. Video source 104 of source device 102 may include video capture devices such as cameras, video archives containing previously captured raw video, and / or video feed interfaces receiving video from video content providers. As a further alternative, video source 104 may generate computer graphics-based data as source video, or a combination of live video, archived video, and computer-generated video. In each case, video encoder 200 encodes the captured, pre-captured, or computer-generated video data. Video encoder 200 may rearrange the pictures from the receiving order (sometimes referred to as "display order") into a decoding order for decoding. Video encoder 200 may generate a bitstream comprising encoded video data. The source device 102 can then output encoded video data to a computer-readable medium 110 via the output interface 108 for reception and / or acquisition by, for example, the input interface 122 of the destination device 116.

[0034] The memory 106 of source device 102 and the memory 120 of destination device 116 represent general-purpose memory. In some examples, memories 106 and 120 may store raw video data, such as raw video from video source 104 and raw, decoded video data from video decoder 300. Additionally or alternatively, memories 106 and 120 may store software instructions executable by, for example, video encoder 200 and video decoder 300. Although memories 106 and 120 are shown as separate from video encoder 200 and video decoder 300 in this example, it should be understood that video encoder 200 and video decoder 300 may also include internal memory for functionally similar or equivalent purposes. Furthermore, memories 106 and 120 may store encoded video data, such as data output from video encoder 200 and input to video decoder 300. In some examples, portions of memories 106 and 120 may be allocated as one or more video buffers, for example, to store raw, decoded, and / or encoded video data.

[0035] Computer-readable medium 110 can represent any type of medium or device capable of transmitting encoded video data from source device 102 to destination device 116. In one example, computer-readable medium 110 represents a communication medium that enables source device 102 to transmit encoded video data directly to destination device 116 in real time, for example, via a radio frequency network or a computer-based network. Depending on communication standards, such as wireless communication protocols, output interface 108 can modulate the transmitted signal including the encoded video data, and input interface 122 can demodulate the received transmitted signal. The communication medium can include any wireless or wired communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium can form part of a packet-based network, such as a local area network, wide area network, or global network (such as the Internet). The communication medium can include a router, switch, base station, or any other means that facilitates communication from source device 102 to destination device 116.

[0036] In some examples, computer-readable medium 110 may include storage device 112. Source device 102 may output encoded data to storage device 112 from output interface 108. Similarly, destination device 116 may access encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as hard disk drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0037] In some examples, computer-readable medium 110 may include file server 114 or another intermediate storage device that can store encoded video data generated by source device 102. Source device 102 may output encoded video data to file server 114 or another intermediate storage device that can store encoded video generated by source device 102. Destination device 116 may access the stored video data from file server 114 via streaming or download. File server 114 may be any type of server device capable of storing encoded video data and sending such encoded video data to destination device 116. File server 114 may represent a web server (e.g., for a website), a file transfer protocol (FTP) server, a content delivery network device, or a network attached storage (NAS) device. Destination device 116 may access the encoded video data from file server 114 via any standard data connection (including an Internet connection). This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of both suitable for accessing encoded video data stored on file server 114. File server 114 and input interface 122 can be configured to operate according to a streaming protocol, a downloading protocol, or a combination thereof.

[0038] Output interface 108 and input interface 122 can represent a wireless transmitter / receiver, a modem, a wired networking component (e.g., an Ethernet card), a wireless communication component operating according to any of a variety of IEEE 802.11 standards, or other physical components. In examples where output interface 108 and input interface 122 include wireless components, output interface 108 and input interface 122 can be configured to transmit data, such as encoded video data, according to cellular communication standards (such as 4G, 4G-LTE (Long Term Evolution), Advanced LTE, 5G, etc.). In some examples where output interface 108 includes a wireless transmitter, output interface 108 and input interface 122 can be configured to operate according to other wireless standards (such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee)). TM ),Bluetooth TM The source device 102 and / or destination device 116 may include corresponding system-on-chip (SoC) devices. For example, source device 102 may include an SoC device performing functions belonging to video encoder 200 and / or output interface 108, and destination device 116 may include an SoC device performing functions belonging to video decoder 300 and / or input interface 122.

[0039] The technology disclosed herein can be applied to video decoding to support any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, Internet streaming video transmission (such as Dynamic Adaptive Streaming (DASH) via HTTP), digital video encoded to a data storage medium, decoding digital video stored on a data storage medium, or other applications.

[0040] The input interface 122 of the destination device 116 can receive an encoded video bitstream from a computer-readable medium 110 (e.g., a communication medium, storage device 112, file server 114, etc.). The encoded video bitstream may include signaling information defined by the video encoder 200, such as syntax elements having values ​​describing the characteristics and / or processing of video blocks or other decoded units (e.g., slices, pictures, picture groups, sequences, etc.), which is also used by the video decoder 300. The display device 118 displays decoded images of the decoded video data to the user. The display device 118 may represent any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0041] Although Figure 1 Not shown, but in some examples, the video encoder 200 and video decoder 300 may each be integrated with the audio encoder and / or audio decoder, and may include appropriate MUX-DEMUX units or other hardware and / or software to process multiplexed streams of both audio and video in a common data stream. Where applicable, the MUX-DEMUX unit may conform to the ITU H.223 multiplexer protocol, or other protocols such as User Datagram Protocol (UDP).

[0042] Video encoder 200 and video decoder 300 can each be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When these technologies are implemented in part as software, the device may store instructions for the software in a suitable non-transitory computer-readable medium, and use one or more processors in hardware to execute these instructions to perform the technologies of this disclosure. Each of video encoder 200 and video decoder 300 may be included in one or more encoders or decoders, any one of which may be integrated as part of a combined encoder / decoder (codec (CODEC)) in the respective device. Devices including video encoder 200 and / or video decoder 300 may include integrated circuits, microprocessors, and / or wireless communication devices, such as cellular phones.

[0043] The video encoder 200 and video decoder 300 may operate according to video decoding standards such as ITU-TH.265, also known as High Efficiency Video Decoding (HEVC) or its extensions, such as Multi-View and / or Scalable Video Decoding Extensions. Alternatively, the video encoder 200 and video decoder 300 may operate according to other proprietary or industry standards such as ITU-TH.266, also known as Multi-Functional Video Decoding (VVC). VVC Draft 7 is the latest draft of the VVC standard. However, the technology disclosed herein is not limited to any particular decoding standard. From January 9 to 18, 2019, at the 15th meeting (JVET-M1002) of the Joint Video Experts Group (JVET) of ITU-TSG16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 in Marrakech, Morocco, J. Chen et al. presented “Algorithm description of Versatile Video Coding and Test Model 4 (VTM 4)”, which is an algorithm description and test model for VVC.

[0044] Typically, video encoder 200 and video decoder 300 can perform block-based decoding of images. The term "block" generally refers to a structure that includes data to be processed (e.g., encoded, decoded, or otherwise used in the encoding and / or decoding process). For example, a block may include a two-dimensional matrix of samples of luminance and / or chrominance data. Typically, video encoder 200 and video decoder 300 can decode video data represented in YUV (e.g., Y, Cb, Cr) format. That is, video encoder 200 and video decoder 300 can decode both luminance and chrominance components, where the chrominance components may include both red and blue chrominance components, rather than decoding red, green, and blue (RGB) data for samples of an image. In some examples, video encoder 200 converts received RGB formatted data to a YUV representation before encoding, and video decoder 300 converts the YUV representation to RGB format. Alternatively, preprocessing and post-processing units (not shown) can perform these conversions.

[0045] This disclosure can generally relate to the decoding (e.g., encoding and decoding) of images, including processes for encoding or decoding image data. Similarly, this disclosure can relate to the decoding of blocks of images, including processes for encoding or decoding data used for blocks, such as prediction and / or residual decoding. Encoded video bitstreams typically include a series of values ​​for syntax elements representing decoding decisions (e.g., decoding modes) and image-to-block segmentation. Therefore, references to decoded images or blocks should generally be understood as the decoded values ​​of the syntax elements used to form images or blocks.

[0046] HEVC defines various blocks, including decoding units (CUs), prediction units (PUs), and transform units (TUs). According to HEVC, a video decoder (such as a video encoder 200) partitions a decoding tree unit (CTU) into CUs based on a quadtree structure. That is, the video decoder partitions the CTU and CU into four equal, non-overlapping squares, and each node of the quadtree has zero or four child nodes. Nodes without child nodes can be called "leaf nodes," and the CU of a leaf node can include one or more PUs and / or one or more TUs. The video decoder can also partition PUs and TUs. For example, in HEVC, a residual quadtree (RQT) represents a partition of a TU. In HEVC, a PU represents inter-frame prediction data, while a TU represents residual data. Intra-predicted CUs include intra-frame prediction information, such as intra-frame mode indication.

[0047] As another example, video encoder 200 and video decoder 300 can be configured to operate according to VCC. According to VCC, the video decoder (such as video encoder 200) segments the image into multiple decoder tree units (CTUs). Video encoder 200 can segment CTUs according to a tree structure (such as a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure). The QTBT structure removes the concept of multiple segmentation types, such as the separation between CUs, PUs, and TUs in HEVC. The QTBT structure consists of two levels: a first level segmented according to quadtree segmentation and a second level segmented according to binary tree segmentation. The root node of the QTBT structure corresponds to a CTU. The leaf nodes of the binary tree correspond to decoder units (CUs).

[0048] In the MTT partitioning structure, blocks can be partitioned using quadtree (QT), binary tree (BT), and one or more types of triple tree (TT) partitioning (also known as ternary tree (TT)). Triple or ternary tree partitioning divides a block into three sub-blocks. In some examples, triple or ternary tree partitioning divides a block into three sub-blocks without using a center to separate the original block. The partitioning types in MTT (e.g., QT, BT, and TT) can be symmetric or asymmetric.

[0049] In some examples, the video encoder 200 and the video decoder 300 may use a single QTBT or MTT structure to represent each of the luma and chroma components, while in other examples, the video encoder 200 and the video decoder 300 may use two or more QTBT or MTT structures, such as one QTBT / MTT structure for the luma component and another QTBT / MTT structure for the two chroma components (or two QTBT / MTT structures for the respective chroma components).

[0050] The video encoder 200 and video decoder 300 can be configured to use per-HEVC quadtree segmentation, QTBT segmentation, or MTT segmentation or other segmentation structures. For illustrative purposes, the description of the techniques of this disclosure is presented in relation to QTBT segmentation. However, it should be understood that the techniques of this disclosure can also be applied to video decoders configured to use quadtree segmentation or other types of segmentation.

[0051] In an image, blocks (e.g., CTUs or CUs) can be grouped in various ways. As an example, a brick can refer to a rectangular area of ​​a row of CTUs within a specific tile in an image. A tile can be a rectangular area of ​​CTUs within a specific tile column and a specific tile row in an image. A tile column refers to a rectangular area of ​​CTUs with a height equal to the image height and a width specified by a syntax element (e.g., a syntax element in the image parameter set). A tile row refers to a rectangular area of ​​CTUs with a height specified by a syntax element (e.g., a syntax element in the image parameter set) and a width equal to the image width.

[0052] In some examples, a tile can be divided into multiple bricks, each brick containing one or more CTU rows within the tile. A tile that is not divided into multiple bricks can also be called a brick. However, a brick that is a proper subset of a tile cannot be called a tile.

[0053] The bricks in an image can also be arranged as slices. A slice can be an integer number of bricks in the image, and they can be exclusively contained in a single Network Abstraction Layer (NAL) unit. In some examples, a slice consists of several complete tiles or a continuous sequence of complete bricks that contain only one tile.

[0054] This disclosure uses "N×N" and "N multiplied by N" interchangeably to refer to the sample dimensions of a block (such as a CU or other video block) in the vertical and horizontal dimensions, for example, 16×16 samples or 16 by 16 samples. Typically, a 16×16 CU has 16 samples in the vertical direction (y = 16) and 16 samples in the horizontal direction (x = 16). Similarly, an N×N CU typically has N samples in the vertical direction and N samples in the horizontal direction, where N represents a non-negative integer value. Samples in a CU can be arranged in rows and columns. Furthermore, a CU does not necessarily need to have the same number of samples in the horizontal direction as in the vertical direction. For example, a CU can include N×M samples, where M is not necessarily equal to N.

[0055] The video encoder 200 encodes video data for the CU, representing prediction and / or residual information, as well as other information. The prediction information indicates how the CU should be predicted to form a prediction block for the CU. The residual information typically represents the sample-by-sample difference between the samples of the CU before encoding and the prediction block.

[0056] To predict the Cubic Frame (CU), the video encoder 200 typically forms prediction blocks for the CU using either inter-frame prediction or intra-frame prediction. Inter-frame prediction generally refers to predicting the CU from data in a previously decoded image, while intra-frame prediction generally refers to predicting the CU from data in a previously decoded image of the same frame. To perform inter-frame prediction, the video encoder 200 can use one or more motion vectors to generate prediction blocks. The video encoder 200 can typically perform a motion search to identify, for example, a reference block that closely matches the CU in terms of the difference between the CU and a reference block. The video encoder 200 can use sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared difference (MSD), or other such difference calculations to compute difference metrics to determine whether the reference block closely matches the current CU. In some examples, the video encoder 200 can use unidirectional or bidirectional prediction to predict the current CU.

[0057] According to the technology disclosed herein, video encoder 200 can determine whether a condition is met for the current block. For example, video encoder 200 can determine whether the condition is met for the current block based on the fact that each component of the bidirectional motion vector of the current block is less than a threshold. Based on the condition that the current block is met, video encoder 200 can prematurely terminate the motion vector thinning process (e.g., DMVR and / or BDOF) applied to the bidirectional motion vector of the current block. For example, video encoder 200 can avoid initiating and / or bypassing the motion vector thinning process (e.g., DMVR and / or BDOF) on the bidirectional motion vector of the current block.

[0058] Some examples of VVC also provide an affine motion compensation mode, which can be considered an inter-frame prediction mode. In the affine motion compensation mode, the video encoder 200 can determine two or more motion vectors representing non-translational motion (such as zooming in or out, rotation, perspective motion, or other irregular motion types).

[0059] To perform intra-frame prediction, the video encoder 200 can select an intra-frame prediction mode to generate prediction blocks. Some examples of VVC provide sixty-seven intra-frame prediction modes, including various orientation modes, as well as planar and DC modes. Typically, the video encoder 200 selects an intra-frame prediction mode that describes samples adjacent to the current block (e.g., a block of a CU), from which samples of the current block are predicted. Assuming the video encoder 200 decodes the CTU and CU in raster scan order (from left to right, from top to bottom), such samples are typically located above, to the upper left, or to the left of the current block in the same image as the current block.

[0060] The video encoder 200 encodes data representing the prediction mode used for the current block. For example, for inter-frame prediction modes, the video encoder 200 may encode data indicating which of the various available inter-frame prediction modes is used, as well as motion information for the corresponding mode. For example, for unidirectional or bidirectional inter-frame prediction, the video encoder 200 may use Advanced Motion Vector Prediction (AMVP) or merge modes to encode motion vectors. The video encoder 200 may use similar modes to encode motion vectors used for affine motion compensation modes.

[0061] Following prediction, such as intra-frame or inter-frame prediction of a block, the video encoder 200 can compute residual data for the block. The residual data (such as a residual block) represents the sample-by-sample difference between the block and the predicted block for that block, formed using the corresponding prediction mode. The video encoder 200 can apply one or more transforms to the residual block to produce transformed data in the transform domain rather than the sample domain. For example, the video encoder 200 can apply a Discrete Cosine Transform (DCT), integer transform, wavelet transform, or conceptually similar transforms to the residual video data. Furthermore, the video encoder 200 can apply secondary transforms after the first transform, such as Mode-dependent Inseparable Secondary Transform (MDNSST), Signal-dependent Transform, Karhunen-Loeve Transform (KLT), etc. The video encoder 200 produces transform coefficients after applying one or more transforms.

[0062] As described above, after any transformation that produces the transform coefficients, the video encoder 200 can perform quantization of the transform coefficients. Quantization generally refers to the process in which transform coefficients are quantized to minimize the amount of data used to represent them, thereby providing further compression. By performing the quantization process, the video encoder 200 can reduce the bit depth associated with some or all of the transform coefficients. For example, the video encoder 200 can round an n-bit value down to an m-bit value during quantization, where n is greater than m. In some examples, to perform quantization, the video encoder 200 can perform a bitwise right shift on the value to be quantized.

[0063] After quantization, the video encoder 200 can scan the transform coefficients to generate a one-dimensional vector from a two-dimensional matrix including the quantized transform coefficients. The scan can be designed to place higher-energy (and therefore lower-frequency) transform coefficients before the vector and lower-energy (and therefore higher-frequency) transform coefficients after the vector. In some examples, the video encoder 200 can scan the quantized transform coefficients using a predefined scan order to produce a serialized vector, and then entropy-encode the quantized transform coefficients of the vector. In other examples, the video encoder 200 can perform an adaptive scan. After scanning the quantized transform coefficients to form a one-dimensional vector, the video encoder 200 can entropy-encode the one-dimensional vector, for example, according to context-adaptive binary arithmetic decoding (CABAC). The video encoder 200 can also entropy-encode the values ​​of syntax elements used to describe metadata associated with the encoded video data for use by the video decoder 300 when decoding the video data.

[0064] To perform CABAC, the video encoder 200 can assign context within a context model to the symbols to be transmitted. Context can involve, for example, whether the neighboring values ​​of a symbol are zero. Probability determination can be based on the context assigned to the symbol.

[0065] The video encoder 200 can also generate syntax data for the video decoder 300, such as block-based syntax data, image-based syntax data, and sequence-based syntax data, for example, in image headers, block headers, slice headers, or other syntax data (such as sequence parameter sets (SPS), picture parameter sets (PPS), or video parameter sets (VPS)). The video decoder 300 can similarly decode this syntax data to determine how to decode the corresponding video data.

[0066] In this way, the video encoder 200 can generate a bitstream that includes encoded video data, such as syntax elements describing the segmentation of images into blocks (e.g., CUs) and prediction and / or residual information for the blocks. Finally, the video decoder 300 can receive the bitstream and decode the encoded video data.

[0067] Typically, the video decoder 300 performs the reverse process of the video encoder 200 to decode the encoded video data of the bitstream. For example, the video decoder 300 can use CABAC, in a manner substantially similar to but reversed by the CABAC encoding process of the video encoder 200, to decode the values ​​of syntax elements for the bitstream. Syntax elements can define segmentation information for segmenting images into CTUs and segmentation information for each CTU based on a corresponding segmentation structure (such as a QTBT structure), to define the CUs of the CTUs. Syntax elements can also define prediction and residual information for blocks (e.g., CUs) of the video data.

[0068] The residual information can be represented, for example, by quantized transform coefficients. The video decoder 300 can inversely quantize and inverse transform the quantized transform coefficients of the block to reconstruct the residual block for the block. The video decoder 300 uses a signaling prediction mode (intra-frame prediction or inter-frame prediction) and associated prediction information (e.g., motion information for inter-frame prediction) to form a prediction block for the block. The video decoder 300 can then combine the prediction block and the residual block (on a sample-by-sample basis) to reconstruct the original block. The video decoder 300 can perform additional processing, such as performing a deblocking process to reduce visual artifacts along the block boundaries.

[0069] According to the technology disclosed herein, video decoder 300 can determine whether a condition is met for the current block. For example, video decoder 300 can determine whether the condition is met for the current block based on the fact that each component of the bidirectional motion vector of the current block is less than a threshold. Based on the condition met for the current block, video decoder 300 can prematurely terminate the motion vector thinning process (e.g., DMVR and / or BDOF) applied to the bidirectional motion vector of the current block. For example, video decoder 300 can avoid initiating and / or bypassing the motion vector thinning process (e.g., DMVR and / or BDOF) on the bidirectional motion vector of the current block.

[0070] In HEVC, the largest decoding unit in a slice is called a Decoding Tree Block (CTB) or Decoding Tree Unit (CTU). A CTB contains a quadtree, and the nodes of the quadtree are decoding units. In the HEVC master profile, the size of the CTB can range from 16×16 to 64×64 (although an 8×8 CTB size is technically supported). The size of a Decoding Unit (CU) can be the same as the CTB, as small as 8×8. Each decoding unit decodes in a mode (e.g., inter-frame or intra-frame). When a CU is inter-frame decoded, it can be further divided into 2 or 4 Prediction Units (PUs), or become a single PU when further division is not applicable. When there are two PUs in a CU, these two PUs can be rectangles of half the size or two rectangles of 1 / 4 or 3 / 4 the size of the CU. When a CU is inter-frame decoded, each PU has a set of motion information derived from a unique inter-frame prediction mode.

[0071] In the HEVC standard, for a prediction unit (PU), there are two inter-frame prediction modes, named merging mode (skipping special cases considered as merging) and Advanced Motion Vector Prediction (AMVP) mode. In either AMVP or merging mode, a candidate list of motion vectors (MVs) is maintained for multiple motion vector predictors. The current PU's (multiple) motion vectors, as well as the reference index in merging mode, are generated by selecting a candidate from the MV candidate list.

[0072] The MV candidate list contains up to five candidates for the merge mode and only two candidates for the AMVP mode. Merge candidates can contain sets of motion information, such as motion vectors corresponding to both the reference picture list (list 0 and list 1) and the reference index. If a merge candidate is identified by the merge index, the reference picture and associated motion vectors for the prediction of the current block are determined. On the other hand, in AMVP mode, for each potential prediction direction from list 0 or list 1, the reference index needs to be explicitly signaled to the MV candidate list along with the MV predictor (MVP) index, since AMVP candidates only contain motion vectors. In AMVP mode, the predicted motion vectors can be further refined. The candidates for both modes are similarly derived from the same spatially and temporally adjacent blocks.

[0073] Spatial MV Candidates are from Figure 2A and Figure 2B The adjacent blocks shown are derived for a specific PU (PU0) (140), although the methods for generating candidates from blocks differ for merge and AMVP modes. Figure 2A This is a conceptual diagram illustrating spatially adjacent MV candidates for merging patterns. Figure 2B This is a conceptual diagram illustrating spatially adjacent MV candidates for the AMVP pattern.

[0074] In the merge mode, it is possible to utilize Figure 2A The numerical order shown derives up to four spatial MV candidates, and this order is as follows: left (0, A1), top (1, B1), top right (2, B0), bottom left (3, A0), and top left (4, B2), as... Figure 2A As shown.

[0075] In AMVP mode, adjacent blocks 142A-142E are divided into two groups: the left group consisting of blocks 0 and 1, and the upper group consisting of blocks 2, 3, and 4, as shown below. Figure 2B As shown. For each group, potential candidates in neighboring blocks of the same reference image as the reference image indicated by the signaled reference index have the highest priority to be selected to form the final candidates for that group. It is possible that no neighboring block contains motion vectors pointing to the same reference image. Therefore, if this candidate is not found, the first available candidate will be scaled to form the final candidate; thus, temporal distance differences can be compensated for.

[0076] If a Temporal Motion Vector Predictor (TMVP) candidate is enabled and available, it can be added to the MV candidate list, following the Spatial Motion Vector candidates. The motion vector derivation process for TMVP candidates is the same for both merge mode and AMVP mode; however, the target reference index for TMVP candidates in merge mode is always set to 0.

[0077] The primary block location used for TMVP candidate export is the lower right block outside the collocated PU (e.g., Figure 3A The block is shown as “T” to compensate for the deviation to the upper and left blocks used to generate spatially adjacent candidates. However, if the block is outside the current CTB line or motion information is unavailable, the block is replaced by the center block of the PU.

[0078] Figure 3A This is a conceptual diagram illustrating TMVP candidates. Figure 3A In the example, CU 160 includes a first PU 162 (denoted as PU0) and a second PU 164 (denoted as PU1). Figure 3AIn the example, the primary block position 166 used for TMVP candidate derivation is the lower-right block (displayed as block "T") outside the co-located PU, to compensate for the offset to the upper and left blocks used to generate spatially adjacent candidates. However, if the block at primary block position 166 is outside the current CTB line or motion information is unavailable, the block is replaced by the center block 168 of PU 602. The motion vector for the TMVP candidate is derived from the co-located PU of the co-located image, indicated at the slice level. The motion vector for the co-located PU is called the co-located MV. Similar to the temporal direct mode in H.264 / AVC, the co-located MV may need to be scaled to compensate for temporal distance differences in order to derive the TMVP candidate motion vector, for example, as... Figure 3B As shown.

[0079] Figure 3B This is a conceptual diagram illustrating MV scaling. Figure 3B In the example, the juxtaposed image 170 for the current image 172 includes a motion vector (i.e., the juxtaposed motion vector) that indicates the position in the juxtaposed reference image 174. The temporal distance between the juxtaposed image 170 and the juxtaposed reference image 174 is called the juxtaposed temporal distance. Furthermore, in the AMVP, the reference index for the current block of the current image 172 indicates the current reference image 176. The temporal distance between the current image 172 and the current reference image 176 is called the current temporal distance. The video decoder can use the juxtaposed motion vector as a temporal motion vector predictor (TMVP) for blocks in the current image 172, but can scale the juxtaposed motion vector based on the difference between the juxtaposed temporal distance and the current temporal distance.

[0080] Therefore, to determine the L0 (L0) AMVP candidate list, the video decoder can determine whether there are available neighboring blocks with L0 motion vectors in the left group. If a neighboring block exists, and the video decoder has access to motion information about the neighboring block, then the neighboring block can be considered available. Furthermore, if the left group includes available neighboring blocks with L0 motion vectors, the video decoder can include the L0 motion vectors in the L0 AMVP candidate list. Additionally, the video decoder can determine whether there are available neighboring blocks with L0 motion vectors in the upper group. If the upper group includes available neighboring blocks with L0 motion vectors, the video decoder can include the L0 motion vectors in the L0 AMVP candidate list. In some examples, if the left group and / or the upper group does not include available L0 motion vectors, the video decoder can include the L0 motion vectors of temporal neighbors. In some such examples, if temporal neighbors are unavailable or do not have L0 motion vectors, the video decoder can include zero-value motion vectors in the L0 AMVP candidate list. The video decoder can replace L0 with L1 to perform the same process to determine the list 1 (L1) AMVP candidate list.

[0081] The following presents several aspects of the merge and AMVP pattern.

[0082] Motion vector scaling: The value of a motion vector may be proportional to the distance between images during rendering time. As mentioned in this article, a motion vector associates a reference image and an image containing the motion vector (e.g., the containing image). When a video decoder (e.g., video encoder 200 or video decoder 300) uses motion vectors to predict other motion vectors, the video decoder can calculate the distance between the containing image and the reference image based on the Image Order Count (POC) value.

[0083] For a motion vector to be predicted, the included image and the reference image associated with that motion vector can be different. Therefore, a video decoder (e.g., video encoder 200 or video decoder 300) can calculate a new distance (based on POC). The video decoder can then scale the motion vector based on these two POC distances. For spatially adjacent candidates, the included image for both motion vectors is the same, while the reference image is different. When performing HEVC, the video decoder can apply motion vector scaling to both the TMVP and AMVP used for spatial and temporal adjacent candidates.

[0084] Artificial Motion Vector Candidate Generation: If the motion vector candidate list is incomplete, the video decoder (e.g., video encoder 200 or video decoder 300) can generate artificial motion vector candidates and insert them at the end of the list until the MV candidate list has all candidates. In merge mode, there are two types of artificial MV candidates: (1) combined candidates derived only from B-slices; (2) zero candidates only for AMVP if the first type does not provide enough artificial candidates.

[0085] For each pair of candidates that is already in the candidate list and has the necessary motion information, the video decoder (e.g., video encoder 200 or video decoder 300) can derive bidirectional combined motion vector candidates by combining the motion vector of the first candidate in the image of reference list 0 with the motion vector of the second candidate in the image of reference list 1.

[0086] Pruning for Candidate Insertion: Candidates from different blocks may happen to be identical, which reduces the efficiency of merging / AMVP candidate lists. Video decoders (e.g., video encoder 200 or video decoder 300) can apply pruning procedures to help prevent candidates from different blocks from being identical. When applying a pruning procedure, the video decoder can compare a candidate with other candidates in the current candidate list to avoid inserting the same candidate to some extent. To reduce complexity, the video decoder can apply only a limited number of pruning procedures instead of comparing every potential candidate with all other existing candidates.

[0087] Decoder-side motion vector refinement (DMVR) is a variation of decoder-side MV derivation techniques that avoids template-based refinement. When DMVR is applied, the video decoder (e.g., video encoder 200 or video decoder 300) can directly compute the initial dual-predicted MV (e.g., ... Figure 4 The bilateral matching cost between the single prediction reference blocks (denoted as I0(x+v0) and I1(x+v1), where x is the coordinate of the pixel within the current block) pointed to by v0 and v1 in the figure.

[0088] Figure 4 This is a conceptual diagram illustrating bilateral template matching. Figure 4 In the example, current image 180 includes current block 182. Current block 182 has an L0 motion vector that indicates the position corresponding to region 184 (denoted as Ref.0) in past reference image 186. Current block 182 also has an L1 motion vector that indicates the position corresponding to region 188 in future reference image 190. The video decoder (e.g., a video encoder or video decoder) performs a local search based on bilateral matching within a predefined 5×5 search range around the initial dual prediction MV. Following the raster scan order starting from the upper left of the 5×5 window, the DMVR process finds the optimal incremental MV (i.e., Δ). The optimal incremental MV is the incremental MV that results in the lowest bilateral matching cost, where the cost function is defined as the distortion between I0(x+v0+Δ) and I1(x+v1-Δ). The incremental MV is a tuple that includes x and y values ​​that are added to or subtracted from the x and y values ​​of the dual prediction MV. The optimal increment MV can be represented by Δ*. The distortion function used in VVC Draft 7 is the sum of absolute differences (SAD).

[0089] Then, the video decoder (e.g., video encoder 200 or video decoder 300) can further refine the output MV pairs (denoted as v0+Δ* and v1-Δ*, where Δ* is the optimized Δ within a predefined 5×5 window) with subpixel precision. The video decoder can then replace the original MV (v0) of the merged block with the resulting MV pairs. (0) and v1 (0) Then, the video decoder can perform motion compensation based on the refined video value (MV). Therefore, in Figure 4 In the example, the video decoder can determine regions 192 and 194 corresponding to the positions Ref0' and Ref1' indicated by the refined MV pairs.

[0090] From July 3 to 12, 2019, at the 15th meeting (JVET-O2001-vE) of the Joint Video Experts Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC29 / WG 11 in Gothenburg, Sweden, Bross et al. presented “Versatile Video Coding (Draft 6)” (hereinafter referred to as “VVC Draft 6”) (VTM-6.0). In the test model for VVC Draft 6, a video decoder (e.g., video encoder 200 or video decoder 300) can apply the DMVR process to the CU if each of the following conditions is met:

[0091] • CU-level merging mode with dual predictive MV.

[0092] • Relative to the current image, one reference image is in the past, and another reference image is in the future.

[0093] • The distances from the two reference images to the current image (e.g., the difference in point of view) are the same.

[0094] The CU is not decoded using affine mode or Advanced Time Motion Vector Prediction (ATMVP) merging mode.

[0095] The CU has more than 64 luminance samples.

[0096] • Both the CU height and CU width are greater than or equal to 8 luminance samples.

[0097] • The double prediction (BCW) weight index with CU weights indicates equal weights.

[0098] • For the current block, weighted prediction (WP) is not enabled, and

[0099] • For the current block, the combined inter-frame and intra-frame prediction (CIIP) mode is not used.

[0100] A video decoder (e.g., video encoder 200 or video decoder 300) can use bidirectional optical flow (BDOF) to refine the dual prediction signals of luminance samples in a 4×4 sub-block level CU. As its name suggests, the BDOF mode is based on the concept of optical flow, which assumes that the motion of an object is smooth. For each 4×4 sub-block, the video decoder can compute motion refinement (v) by minimizing the difference between the L0 and L1 prediction samples. x ,v y The video decoder can use motion refinement to adjust the sample values ​​of the double predictions in a 4×4 sub-block. The video decoder can apply the following steps during the BDOF process.

[0101] A video decoder (e.g., video encoder 200 or video decoder 300) can calculate the horizontal and vertical gradients of two predicted signals by directly computing the difference between two adjacent samples. and k = 0, 1, for example,

[0102]

[0103]

[0104] Among them I (k) (i,j) is the sample value at coordinate (i,j) of the predicted signal in list k (k=0,1), and shift1 is calculated based on the luminance bit depth bitDepth as shift1=max(2,14-bitDepth).

[0105] A video decoder (e.g., video encoder 200 or video decoder 300) can calculate the autocorrelation and cross-correlation of gradients S1, S2, S3, S5, and S6 as follows:

[0106] S1=∑ (i,j)∈Ω ψ x (i,j)·ψ x (i,j),S3=∑ (i,j)∈Ω θ(i,j)·ψ x (i,j)

[0107]

[0108]

[0109] in

[0110]

[0111]

[0112] θ(i,j)=(I (1) (i,j)>>n b )-(I (0) (i,j)>>n b )

[0113] Where Ω is a 6×6 window surrounding a 4×4 sub-block, and n a and n b The values ​​are set to min(5, bitDepth-7) and min(8, bitDepth-4), respectively.

[0114] Video decoders (e.g., video encoder 200 or video decoder 300) can derive motion refinement (v) using cross-correlation and autocorrelation terms, using the following formula. x ,v y ):

[0115]

[0116]

[0117] in th′ BIO =2 13-B . It is a floor function, and

[0118] Based on motion refinement and gradients, the video decoder (e.g., video encoder 200 or video decoder 300) can compute the following adjustments for each sample in a 4×4 sub-block:

[0119]

[0120] A video decoder (e.g., video encoder 200 or video decoder 300) can compute the BDOF samples of the CU by adjusting the double prediction samples as follows:

[0121] pred BDOF (x,y)=(I (0) (x,y)+I (1) (x,y)+b(x,y)+o offset )>>shift

[0122] Video decoders (e.g., video encoder 200 or video decoder 300) can select these values ​​such that the multiplier in the BDOF process does not exceed 15 bits and the maximum bit width of the intermediate parameters in the BDOF process is kept within 32 bits.

[0123] To derive the gradient values, the video decoder (e.g., video encoder 200 or video decoder 300) can generate some prediction samples I in list k (k = 0, 1) outside the current CU boundary. (k) (i,j). For example... Figure 5 As depicted, when performing BDOF, the video decoder can use an extended row / column around the boundary of CU 196. To control the computational complexity of generating prediction samples outside the boundary, the video decoder can generate prediction samples in the extended region (white area) by taking reference samples directly at nearby integer positions (using the floor() operation on the coordinates) without interpolation, and a normal 8-tap motion-compensated interpolation filter is used to generate prediction samples within the CU (gray area). The video decoder can use these extended sample values ​​only in gradient calculations. For the remaining steps in the BDOF process, if any samples and gradient values ​​outside the CU boundary are needed, the video decoder can fill in (i.e., repeat) samples and gradient values ​​from its nearest neighbor.

[0124] A video decoder (e.g., video encoder 200 or video decoder 300) can use BDOF to refine the dual prediction signal of a 4×4 sub-block level CU. The video decoder can apply BDOF to a CU if it meets all of the following conditions:

[0125] • Use a “true” double prediction mode to decode the CU, that is, one of the two reference images is displayed before the current image and the other is displayed after the current image.

[0126] ·CU does not use affine mode or ATMVP merge mode for decoding.

[0127] The CU has more than 64 luminance samples.

[0128] • Both the CU height and CU width are greater than or equal to 8 luminance samples.

[0129] • The BCW weight index indicates equal weights.

[0130] • WP is not enabled for the current CU.

[0131] • CIIP mode is not currently used for the CU.

[0132] In VVC Working Draft 7, when performing both BDOF and DMVR, the video decoder (e.g., video encoder 200 or video decoder 300) can apply SAD-based early termination to reduce the average complexity on the decoder side. However, when applying SAD-based early termination, the video decoder (e.g., video encoder 200 or video decoder 300) can apply BDOF and / or DMVR with a relatively large amount of added complexity and a relatively small amount of prediction accuracy. This can increase the overall complexity of the video decoder, increase costs, and potentially slow down the decoding process.

[0133] As described herein, a video decoder (e.g., video encoder 200 or video decoder 300) can determine whether conditions are met for the current block. For example, the video decoder can terminate early a DMVR with asymmetric bidirectional motion vectors. In some examples of performing early termination for a DMVR with asymmetric bidirectional motion vectors, the video decoder can terminate the DMVR early if the bidirectional motion vectors are not symmetrical to each other. That is, the video decoder can terminate early based on whether the bidirectional motion vectors are symmetrical (or nearly symmetrical) to help avoid applying BDOF and / or DMVR with a relatively large amount of added complexity and a relatively small amount of added prediction accuracy. In some examples, the video decoder can terminate early based on whether the absolute value of the bidirectional motion vectors is small (i.e., less than a threshold) to help avoid applying BDOF and / or DMVR with a relatively large amount of added complexity and a relatively small amount of added prediction accuracy. In this way, the video decoder can terminate early the motion vector refinement process applied to the bidirectional motion vectors of the current block, which can potentially reduce the average case of decoder-side complexity.

[0134] The following is a non-exclusive list of example conditions under which a video decoder may terminate a DMVR early in accordance with the techniques disclosed herein.

[0135] If MVL0x ≠ -1*MVL1x and MVL0y ≠ -1*MVL1y, then the video decoder (e.g., video encoder 200 or video decoder 300) can terminate the DMVR process prematurely. In this disclosure, MVL0x represents the x-component of the L0 motion vector, MVL0y represents the y-component of the L0 motion vector, MVL1x represents the x-component of the L1 motion vector, and MVL1y represents the y-component of the L1 motion vector. In other words, a video decoder (e.g., a video encoder 200 or a video decoder 300) can determine whether a condition (e.g., an early termination condition) is satisfied with the current block based on MVL0x not being equal to -1*MVL1x and MVL0y not being equal to -1*MVL1y, where MVL0x is the x-component of the list 0 motion vector of the current block's bidirectional motion vectors, MVL1x is the x-component of the list 1 motion vector of the current block's bidirectional motion vectors, MVL0y is the y-component of the list 0 motion vector of the current block's bidirectional motion vectors, and MVL1y is the y-component of the list 1 motion vector of the current block's bidirectional motion vectors.

[0136] If MVL0x ≠ -1*MVL1x or MVL0y ≠ -1*MVL1y, the video decoder (e.g., video encoder 200 or video decoder 300) can terminate the DMVR process early. As an example, the video decoder (e.g., video encoder 200 or video decoder 300) can determine whether a condition (e.g., an early termination condition) is satisfied with the current block based on MVL0x not being equal to -1*MVL1x or MVL0y not being equal to -1*MVL1y, where MVL0x is the x-component of the list 0 motion vector of the current block's bidirectional motion vectors, MVL1x is the x-component of the list 1 motion vector of the current block's bidirectional motion vectors, MVL0y is the y-component of the list 0 motion vector of the current block's bidirectional motion vectors, and MVL1y is the y-component of the list 1 motion vector of the current block's bidirectional motion vectors.

[0137] If for all Δx and Δy in the range from -N to N, MVL0x ≠ -1*(MVL1x+Δx) and MVL0y ≠ -1*(MVL1y+Δy), then a video decoder (e.g., video encoder 200 or video decoder 300) can terminate the DMVR process early, where N can be a sequence-level, picture-level, sub-picture-level, tile-level, or slice-level number or a predefined constant integer that signals the bitstream. For example, both Δx and Δy can be 32 (i.e., 2 pixels in spatial distance if the MV storage precision is 1 / 16 of a pixel), so if all Δx ∈ {-32,…,32} and Δy ∈ {-32,…,32} satisfy MVL0x ≠ -1*(MVL1x+Δx) and MVL0y ≠ -1*(MVL1y+Δy), then the DMVR can be terminated early. In other words, a video decoder (e.g., a video encoder 200 or a video decoder 300) can determine the condition (e.g., an early termination condition) that the current block satisfies based on the following: for all Δx and Δy in the range from -N to N, MVL0x is not equal to -1*(MVL1x+Δx) and MVL0y is not equal to -1*(MVL1y+Δy), where MVL0x is the x-component of the list 0 motion vector of the current block, MVL1x is the x-component of the list 1 motion vector of the current block, MVL0y is the y-component of the list 0 motion vector of the current block, MVL1y is the y-component of the list 1 motion vector of the current block, and N is a sequence-level, picture-level, sub-picture-level, tile-level, or slice-level number or a predefined constant integer that signals the sequence in the bit stream.

[0138] If for all Δx and Δy in the range from -N to N, MVL0x ≠ -1*(MVL1x+Δx) or MVL0y ≠ -1*(MVL1y+Δy), then a video decoder (e.g., video encoder 200 or video decoder 300) can terminate the DMVR process early, where N can be a sequence-level, picture-level, sub-picture-level, tile-level, or slice-level number or a predefined constant integer that signals the bitstream. For example, both Δx and Δy can be 32 (i.e., 2 pixels in spatial distance if the MV storage precision is 1 / 16 of a pixel), so if all Δx ∈ {-32,…,32} and Δy ∈ {-32,…,32} satisfy MVL0x ≠ -1*(MVL1x+Δx) or MVL0y ≠ -1*(MVL1y+Δy), then the DMVR can be terminated early. In other words, a video decoder (e.g., video encoder 200 or video decoder 300) can determine the condition (e.g., early termination condition) that the current block satisfies based on the fact that MVL0x is not equal to -1*(MVL1x+Δx) or MVL0y is not equal to -1*(MVL1y+Δy) for all Δx and Δy in the range from -N to N, where MVL0x is the x-component of the list 0 motion vector of the current block, MVL1x is the x-component of the list 1 motion vector of the current block, MVL0y is the y-component of the list 0 motion vector of the current block, MVL1y is the y-component of the list 1 motion vector of the current block, and N is a sequence-level, picture-level, sub-picture-level, tile-level, or slice-level number or a predefined constant integer that signals the sequence in the bit stream.

[0139] If the lengths of the motion vectors (MVs) are different from each other, the video decoder (e.g., video encoder 200 or video decoder 300) can terminate the DMVR process early. The length is defined as the P-norm of the motion vector, where p can be any non-negative integer. In other words, the video decoder (e.g., video encoder 200 or video decoder 300) can determine whether a condition (e.g., an early termination condition) is met with respect to the current block based on the fact that the lengths of the bidirectional motion vectors of the current block are different from each other. However, based on the fact that a condition is not met with respect to the second current block, the video decoder can apply a motion vector thinning process to the bidirectional motion vectors of the second block to determine the thinned bidirectional motion vectors of the second block, and determine the prediction block for the second block based on the thinned bidirectional motion vectors of the second block.

[0140] If the p-norms of the motion vectors (MVs) are completely different from each other, the video decoder (e.g., video encoder 200 or video decoder 300) can terminate the DMVR process early. Specifically, DMVR can be terminated early if the absolute difference between the corresponding p-norms of the bidirectional MVs falls outside a given threshold N (predefined or signaled at the sequence level, picture level, subpicture level, tile level, slice level, or CTU level). In other words, the video decoder (e.g., video encoder 200 or video decoder 300) can determine whether a condition (e.g., an early termination condition) is met for the current block based on the difference between the p-norms (e.g., lengths) of the bidirectional motion vectors of the current block being greater than the threshold.

[0141] If the p-norms of the motion vectors (MVs) are not completely different from each other, the video decoder (e.g., video encoder 200 or video decoder 300) can terminate the DMVR process early. Specifically, DMVR can be terminated early if the absolute difference between the corresponding p-norms of the bidirectional MVs falls within a given threshold N (predefined or signaled at the sequence level, picture level, subpicture level, tile level, slice level, or CTU level). In other words, the video decoder (e.g., video encoder 200 or video decoder 300) can determine whether a condition (e.g., an early termination condition) is met for the current block based on the difference between the p-norms (e.g., lengths) of the bidirectional motion vectors of the current block being less than the threshold.

[0142] Note that in some embodiments, the video decoder (e.g., video encoder 200 or video decoder 300) may apply the aforementioned early termination technique used in the DMVR process to terminate the BDOF process early.

[0143] In some examples of this disclosure, a video decoder (e.g., video encoder 200 or video decoder 300) can perform early termination for a DMVR with a small bidirectional MV. For example, the video decoder can terminate the DMVR early if the absolute value of each component (i.e., MVx and MVy) of a given bidirectional MV is less than a given threshold N, where N is an integer that may be predefined (e.g., 1, 2, 3, 4...) or signaled as a sequence-level, picture-level, subpicture-level, tile-level, slice-level, or CTU-level syntax element. Specifically, for N, if |MVL0x| < N, |MVL0y| < N, |MVL1x| < N, and |MVL1y| < N, the video decoder can terminate the DMVR process before it begins during the motion compensation phase. For example, N can be set to 4, so the DMVR is terminated early if |MVL0x| < 4, |MVL0y| < 4, |MVL1x| < 4, and |MVL1y| < 4. In another example, N can be set to 1, so that if MVL0x = 0, MVL0y = 0, MVL1x = 0 and MVL1y = 0, the DMVR can be prematurely terminated during motion compensation before its refinement process begins. In other words, the video decoder (e.g., video encoder 200 or video decoder 300) can determine whether a condition (e.g., an early termination condition) is met for the current block based on each component of the bidirectional motion vector of the current block being less than a threshold.

[0144] Note that in some examples, the same early termination method can be applied to BDOF to terminate BDOF itself early.

[0145] Therefore, according to examples of this disclosure, a video decoder (e.g., video encoder 200 or video decoder 300) can determine the bidirectional motion vectors of the current block of video data and determine whether a condition is satisfied with respect to the current block. The video decoder can prematurely terminate the motion vector refinement process applied to the bidirectional motion vectors of the current block based on the satisfaction of the condition, and determine a prediction block for the current block based on the bidirectional motion vectors of the current block. The video decoder can perform at least one of the following: reconstructing the current block based on the prediction block for the current block, or generating residual data for the current block based on the prediction block for the current block.

[0146] In some examples of this disclosure, the video decoder (e.g., video encoder 200 or video decoder 300) may use constrained motion vector (MV) refinement. In such examples, the video decoder may refine only one of the bidirectional MVs while leaving the other MV unchanged. For example, the video decoder may determine a first motion vector and a second motion vector for the current block of video data. The video decoder may determine a selected motion vector from the first and second motion vectors. The video decoder may generate a refined version of the selected motion vector by applying a motion vector refinement process to the selected motion vector. The video decoder may determine a prediction block for the current block based on the refined version of the selected motion vector and the one of the first and second motion vectors that is not the selected initial motion vector. The video decoder may perform at least one of the following: reconstructing the current block based on the prediction block for the current block or generating residual data for the current block based on the prediction block for the current block.

[0147] When performing a DMVR, video encoder 200 can specify whether to refine MVL0 or MVL1 and leave the other unchanged. Video encoder 200 can indicate which of the two MVs to refine by using sequence-level, picture-level, subpicture-level, tile-level, slice-level, CTU-level, or CU-level flags. In other words, the video encoder (e.g., video encoder 200) can include one or more syntax elements in the bitstream comprising an encoded representation of the video data that indicate either the first motion vector or the second motion vector of the current block is the selected motion vector. In some examples, video decoder (e.g., video decoder 300) can determine which of the first or second motion vectors of the current block is the selected motion vector based on one or more syntaxes signaled in the bitstream comprising the encoded representation of the video data.

[0148] When applying a DMVR, a video decoder (e.g., video encoder 200 or video decoder 300) can determine which of the MVL0 and MVL1 to be refined based on the length of the motion vector. In other words, the video decoder (e.g., video encoder 200 or video decoder 300) can determine the selected motion vector based on the lengths of the first motion vector and the second motion vector of the current block. The length is defined as the P-norm of the motion vector (e.g., the length itself), where p can be any non-negative integer. In some examples, the video decoder can refine the MV with the longer length while leaving the other unchanged. In another example, the video decoder can refine the MV with the shorter length while leaving the other unchanged.

[0149] When applying a DMVR, a video decoder (e.g., video encoder 200 or video decoder 300) can determine which of the MVL0 and MVL1 vectors needs to be refined based on the symmetry of MVL0 and MVL1. In other words, the video decoder can determine the selected motion vector, including by determining the selected motion vector based on the symmetry of the first motion vector and the second motion vector of the current block.

[0150] When two of the bidirectional motion vectors (MVs) are asymmetrical (e.g., MVL0 is not equal to -1*MVL1), a video decoder (e.g., video encoder 200 or video decoder 300) can refine the MV with the shorter (or longer in another embodiment) length while leaving the other unchanged; otherwise, if MVL0 equals -1*MVL1, both MVs are refined. In other words, the video decoder can select the shorter of the first and second motion vectors of the current block as the selected motion vector based on the asymmetry between them.

[0151] When applying a DMVR, a video decoder (e.g., video encoder 200 or video decoder 300) can determine which of the MVL0 and MVL1 should be refined based on their symmetry. In other words, the video decoder can determine the selected motion vector based on the symmetry between the first motion vector and the second motion vector of the current block. When the two MVs in a bidirectional stream are not perfectly symmetrical, the video decoder can refine the MV with the shorter (or longer in another embodiment) length while leaving the other unchanged; otherwise, both MVs are refined. When applying a DMVR, the video decoder can determine whether the two MVs in a bidirectional stream are not perfectly symmetrical based on the fact that for all Δx and Δy ranging from -N to N, MVL0 ≠ -1*(MVL1 + (Δx, Δy)), where N can be a sequence-level, picture-level, sub-picture-level, tile-level, or slice-level number or a predefined constant integer signaled in the bitstream. In other words, a video decoder can determine a selected motion vector based on MVL0x≠-1*(MVL1+(Δx, Δy)) for all Δx and Δy in the range from -N to N, where N is a sequence-level, picture-level, sub-picture-level, tile-level, slice-level number or a predefined constant integer that signals the sequence in the bit stream.

[0152] Note that in some examples of this disclosure, the video decoder (e.g., video encoder 200 or video decoder 300) may apply the same early termination and / or select one of the MVs to refine the BDOF process.

[0153] Therefore, according to one or more examples of this disclosure, a video decoder (e.g., video encoder 200 or video decoder 300) can determine the bidirectional motion vectors of the current block of video data. The video decoder can determine whether a condition is met regarding the current block. For example, the video decoder can determine that a condition is met based on each component of the bidirectional motion vectors of the current block being less than a threshold. The video decoder can prematurely terminate the motion vector thinning process applied to the bidirectional motion vectors of the current block based on the condition being met. The video decoder can determine a prediction block for the current block based on the bidirectional motion vectors of the current block. The video decoder can reconstruct the current block based on the prediction block for the current block and / or generate residual data for the current block based on the prediction block for the current block.

[0154] Figure 6 This is a block diagram illustrating an example video encoder 200 that can perform the techniques of this disclosure. Figure 6 This disclosure is provided for illustrative purposes and should not be construed as a limitation on the techniques extensively illustrated and described herein. For illustrative purposes, this disclosure describes a video encoder 200 in the context of video decoding standards such as the HEVC video decoding standard and the H.266 video decoding standard under development. However, the techniques disclosed herein are not limited to these video decoding standards and are generally applicable to video encoding and decoding.

[0155] exist Figure 6 In the example, the video encoder 200 includes a video data memory 230, a mode selection unit 202, a residual generation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a filter unit 216, a decoded picture buffer (DPB) 218, and an entropy coding unit 220. Any one or all of the video data memory 230, mode selection unit 202, residual generation unit 204, transform processing unit 206, quantization unit 208, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, filter unit 216, DPB 218, and entropy coding unit 220 can be implemented in one or more processors or processing circuits. Furthermore, the video encoder 200 may include additional or alternative processors or processing circuits to perform these and other functions.

[0156] The video data storage device 230 can store video data to be encoded by the components of the video encoder 200. The video encoder 200 can obtain data from, for example, a video source 104 (…). Figure 1The video encoder 200 receives video data stored in video data memory 230. DPB 218 can act as a reference picture memory storing reference video data for use by the video encoder 200 in predicting subsequent video data. Video data memory 230 and DPB 218 can be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Video data memory 230 and DPB 218 can be provided by the same memory device or separate memory devices. In various examples, video data memory 230 can be on-chip along with other components of the video encoder 200, as shown, or off-chip relative to those components.

[0157] In this disclosure, references to video data memory 230 should not be construed as being limited to memory within video encoder 200, unless specifically described therein, or to memory external to video encoder 200, unless specifically described therein. Rather, references to video data memory 230 should be understood as reference memory storing video data received by video encoder 200 for encoding (e.g., video data for the current block to be encoded). Figure 1 The memory 106 can also provide temporary storage for the outputs from various units of the video encoder 200.

[0158] It shows Figure 6 Various units are provided to aid in understanding the operations performed by the video encoder 200. These units can be implemented as fixed-function circuits, programmable circuits, or combinations thereof. Fixed-function circuits refer to circuits that provide specific functionality and are pre-programmable for performing operations. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functionality in the operations that can be performed. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by instructions from software or firmware. Fixed-function circuits can execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is generally constant. In some examples, one or more units can be different circuit blocks (fixed-function or programmable), and in some examples, one or more units can be integrated circuits.

[0159] The video encoder 200 may include an arithmetic logic unit (ALU), an elementary function unit (EFU), digital circuitry, analog circuitry, and / or a programmable core formed by programmable circuitry. In an example where the operation of the video encoder 200 is performed using software executed by programmable circuitry, memory 106 ( Figure 1The video encoder 200 may store instructions (e.g., object code) of the software received and executed by the video encoder 200, or another memory (not shown) within the video encoder 200 may store such instructions.

[0160] The video data storage unit 230 is configured to store received video data. The video encoder 200 can retrieve images of the video data from the video data storage unit 230 and provide the video data to the residual generation unit 204 and the mode selection unit 202. The video data in the video data storage unit 230 can be the raw video data to be encoded.

[0161] The mode selection unit 202 includes a motion estimation unit 222, a motion compensation unit 224, and an intra-frame prediction unit 226. The mode selection unit 202 may include additional functional units for performing video prediction based on other prediction modes. As an example, the mode selection unit 202 may include a palette unit, an intra-frame block copying unit (which may be part of the motion estimation unit 222 and / or the motion compensation unit 224), an affine unit, a linear model (LM) unit, etc. Figure 6 In the example, motion compensation unit 224 includes motion refinement unit 225.

[0162] The mode selection unit 202 typically coordinates multiple encoding processes to test combinations of encoding parameters and the resulting rate-distortion values ​​for those combinations. Encoding parameters may include the CTU-CU split, the prediction mode for the CU, the transformation type of the residual data for the CU, the quantization parameters of the residual data for the CU, etc. The mode selection unit 202 can ultimately select a combination of encoding parameters that yields a better rate-distortion value than other test combinations.

[0163] The video encoder 200 can segment images obtained from the video data storage 230 into a series of CTUs, and encapsulate one or more CTUs within a slice. The mode selection unit 202 can segment the CTUs of the image according to a tree structure (such as the quadtree structure or QTBT structure of HEVC described above). As mentioned above, the video encoder 200 can form one or more CUs by segmenting CTUs according to a tree structure. This CU can also generally be referred to as a "video block" or "block".

[0164] Typically, mode selection unit 202 also controls its components (e.g., motion estimation unit 222, motion compensation unit 224, and intra-prediction unit 226) to generate prediction blocks for the current block (e.g., the current CU, or, in HEVC, the overlapping portion of PU and TU). For inter-frame prediction of the current block, motion estimation unit 222 may perform a motion search to identify one or more closely matching reference blocks in one or more reference pictures (e.g., one or more previously decoded pictures stored in DPB 218). Specifically, motion estimation unit 222 may calculate values ​​indicating how similar a potential reference block is to the current block, for example, based on sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared error (MSD), etc. Motion estimation unit 222 may typically perform these calculations using the sample-by-sample difference between the current block and the considered reference blocks. Motion estimation unit 222 may identify the reference block with the lowest value obtained from these calculations, thereby indicating the reference block that most closely matches the current block.

[0165] Motion estimation unit 222 can generate one or more motion vectors (MVs) that define the position of a reference block in a reference image relative to the position of a current block in the current image. Motion estimation unit 222 can then provide the motion vectors to motion compensation unit 224. For example, for unidirectional inter-frame prediction, motion estimation unit 222 can provide a single motion vector, while for bidirectional inter-frame prediction, motion estimation unit 222 can provide two motion vectors. Motion compensation unit 224 can then use the motion vectors to generate prediction blocks. For example, motion compensation unit 224 can use the motion vectors to obtain data for the reference blocks. As another example, if the motion vectors have fractional-sample precision, motion compensation unit 224 can interpolate values ​​for the prediction blocks according to one or more interpolation filters. Furthermore, for bidirectional inter-frame prediction, motion compensation unit 224 can obtain data for the two reference blocks identified by the corresponding motion vectors and combine the obtained data, for example, by per-sample averaging or weighted averaging.

[0166] The motion refinement unit 225 of the motion compensation unit 224 can use motion refinement processes (such as, for example, BDOF and / or DMVR processes) to generate prediction blocks. Examples of BDOF and DMVR processes are described elsewhere in this disclosure. The motion refinement unit 225 can check one or more early termination conditions. If one or more early termination conditions are not met, the motion refinement unit 225 can continue to apply motion vector refinement processes (e.g., BDOF and / or DMVR processes) to generate prediction blocks. Otherwise, if the early termination conditions are met, the motion refinement unit 225 can generate prediction blocks without using the application of motion vector refinement processes (e.g., early termination of motion vector refinement process application). In some examples of this disclosure, one or more early termination conditions are modified relative to the early termination conditions set forth in VVC Draft 7. Specifically, as an example, early termination based on the sum of absolute differences (SAD) is applied to both BDOF and DMVR to reduce the average complexity on the decoder side, regardless of whether the bidirectional motion vectors are symmetrical (or nearly symmetrical) or whether the absolute values ​​of the bidirectional motion vectors are small (i.e., less than a threshold). Therefore, in some examples, the motion refinement unit 225 can perform the motion vector refinement process even when the bidirectional motion vectors are symmetrical (or nearly symmetrical) and / or when the absolute values ​​of the bidirectional motion vectors are small.

[0167] As another example, for intra-prediction or intra-prediction decoding, intra-prediction unit 226 can generate a prediction block from samples adjacent to the current block. For example, in directional mode, intra-prediction unit 226 can typically mathematically combine the values ​​of adjacent samples and fill these calculated values ​​across the current block in a defined direction to produce a prediction block. As another example, in DC mode, intra-prediction unit 226 can calculate the average of samples adjacent to the current block and generate a prediction block to include the resulting average for each sample in the prediction block.

[0168] Mode selection unit 202 provides a prediction block to residual generation unit 204. Residual generation unit 204 receives the raw, uncoded version of the current block from video data memory 230 and the prediction block from mode selection unit 202. Residual generation unit 204 calculates the sample-by-sample difference between the current block and the prediction block. The resulting sample-by-sample difference defines the residual block for the current block. In some examples, residual generation unit 204 may also determine the difference between sample values ​​in the residual block to generate the residual block using residual differential pulse decode-modulation (RDPCM). In some examples, residual generation unit 204 may be formed using one or more subtractor circuits that perform binary subtraction.

[0169] In the example where mode selection unit 202 divides a CU into PUs, each PU can be associated with a luma prediction unit and a corresponding chroma prediction unit. Video encoder 200 and video decoder 300 can support PUs of various sizes. As indicated above, the size of a CU can refer to the size of its luma decoding block, and the size of a PU can refer to the size of the luma prediction unit of the PU. Assuming a particular CU has a size of 2N×2N, then video encoder 200 can support PU sizes of 2N×2N or N×N for intra-frame prediction, and symmetric PU sizes of 2N×2N, 2N×N, N×2N, N×N, etc., for inter-frame prediction. Video encoder 200 and video decoder 300 can also support asymmetric segmentation of PU sizes of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-frame prediction.

[0170] In the example where mode selection unit 202 does not further divide the CU into PUs, each CU can be associated with a luma decoding block and a corresponding chroma decoding block. Similarly, the size of the CU can refer to the size of the luma decoding block of the CU. The video encoder 200 and video decoder 300 can support CU sizes of 2N×2N, 2N×N, or N×2N.

[0171] For other video decoding techniques, as a few examples such as intra-block copy mode decoding, affine mode decoding, and linear model (LM) mode decoding, mode selection unit 202 generates a prediction block for the current block being encoded via a corresponding unit associated with the decoding technique. In some examples, such as palette mode decoding, mode selection unit 202 may not generate a prediction block, but instead generate syntax elements indicating how the block should be reconstructed based on a selected palette. In such modes, mode selection unit 202 may provide these syntax elements to entropy coding unit 220 for encoding.

[0172] As described above, the residual generation unit 204 receives video data for the current block and the corresponding prediction block. The residual generation unit 204 then generates a residual block for the current block. To generate the residual block, the residual generation unit 204 calculates the sample-by-sample difference between the prediction block and the current block.

[0173] Transform processing unit 206 applies one or more transformations to the residual block to generate a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 can apply various transformations to the residual block to form the transform coefficient block. For example, transform processing unit 206 can apply discrete cosine transform (DCT), direction transformation, Caronan-Loy transform (KLT), or conceptually similar transformations to the residual block. In some examples, transform processing unit 206 can perform multiple transformations on the residual block, such as primary and secondary transformations, such as rotation transformations. In some examples, transform processing unit 206 does not apply any transformations to the residual block.

[0174] Quantization unit 208 can quantize the transform coefficients in a transform coefficient block to produce a quantized transform coefficient block. Quantization unit 208 can quantize the transform coefficients of the transform coefficient block based on the quantization parameter (QP) value associated with the current block. Video encoder 200 (e.g., via mode selection unit 202) can adjust the degree of quantization applied to the transform coefficient block associated with the current block by adjusting the QP value associated with the CU. Quantization may introduce information loss, and therefore, the quantized transform coefficients may have lower precision than the original transform coefficients generated by transform processing unit 206.

[0175] The inverse quantization unit 210 and the inverse transform processing unit 212 can apply inverse quantization and inverse transform to the quantized transform coefficient block, respectively, to reconstruct the residual block from the transform coefficient block. The reconstruction unit 214 can generate a reconstructed block corresponding to the current block based on the reconstructed residual block and the prediction block generated by the mode selection unit 202 (although it may have some degree of distortion). For example, the reconstruction unit 214 can add samples of the reconstructed residual block to the corresponding samples of the prediction block generated by the mode selection unit 202 to generate the reconstructed block.

[0176] Filter unit 216 can perform one or more filtering operations on the reconstructed block. For example, filter unit 216 can perform a deblocking operation to reduce block artifacts along the edges of the CU. The operation of filter unit 216 can be skipped in some examples.

[0177] The video encoder 200 stores the reconstructed blocks in the DPB 218. For example, in an example where the operation of the filter unit 216 is not required, the reconstruction unit 214 can store the reconstructed blocks in the DPB 218. In an example where the operation of the filter unit 216 is required, the filter unit 216 can store the filtered reconstructed blocks in the DPB 218. The motion estimation unit 222 and the motion compensation unit 224 can obtain a reference picture formed by the reconstructed (and possibly filtered) blocks from the DPB 218 to perform inter-frame prediction of blocks in subsequently encoded pictures. In addition, the intra-frame prediction unit 226 can use the reconstructed blocks in the DPB 218 of the current picture to perform intra-frame prediction of other blocks in the current picture.

[0178] Typically, entropy coding unit 220 can entropy code syntax elements received from other functional components of video encoder 200. For example, entropy coding unit 220 can entropy code quantized transform coefficient blocks from quantization unit 208. As another example, entropy coding unit 220 can entropy code predictive syntax elements (e.g., motion information for inter-frame prediction or intra-frame mode information for intra-frame prediction) from mode selection unit 202. Entropy coding unit 220 can perform one or more entropy coding operations on syntax elements, another example of video data, to generate entropy-coded data. For example, entropy coding unit 220 can perform context-adaptive variable-length decoding (CAVLC), CABAC, variable-to-variable (V2V) length decoding, syntax-based context-adaptive binary arithmetic decoding (SBAC), probabilistic interval partitioned entropy (PIPE) decoding, exponential Golomb coding, or another type of entropy coding operation on the data. In some examples, entropy coding unit 220 can operate in bypass mode.

[0179] The video encoder 200 can output a bitstream containing entropy-encoded syntax elements required to reconstruct slices or blocks of images. Specifically, the entropy coding unit 220 can output a bitstream.

[0180] The above operations are described relative to blocks. This description should be understood as operations applied to luma decoding blocks and / or chroma decoding blocks. As mentioned above, in some examples, the luma decoding block and chroma decoding block are the luma and chroma components of the CU. In some examples, the luma decoding block and chroma decoding block are the luma and chroma components of the PU.

[0181] In some examples, for the chroma decoding block, it is not necessary to repeat the operations performed relative to the luma decoding block. As an example, the operations for identifying the motion vector (MV) and reference image for the luma decoding block do not need to be repeated for identifying the MV and reference image for the chroma block. Instead, the MV for the luma decoding block can be scaled to determine the MV for the chroma block, and the reference image can be the same. As another example, the intra-frame prediction process can be the same for both the luma and chroma decoding blocks.

[0182] Video encoder 200 represents an example of a device configured to encode video data, including a memory configured to store the video data, and one or more processing units implemented in circuitry and configured to determine the bidirectional motion vectors of a current block of the video data. The one or more processing units may determine a condition for the current block based on each component of the bidirectional motion vectors being less than a threshold. The one or more processing units may prematurely terminate the motion vector thinning process applied to the bidirectional motion vectors of the current block based on the condition for the current block. The one or more processing units may determine a prediction block for the current block based on the bidirectional motion vectors of the current block. The one or more processing units may reconstruct the current block based on the prediction block for the current block, and / or generate residual data for the current block based on the prediction block for the current block.

[0183] Figure 7 This is a block diagram illustrating an example video decoder 300 that can perform the techniques disclosed herein. Figure 7 This disclosure is provided for illustrative purposes and does not limit the techniques as broadly illustrated and described herein. For illustrative purposes, this disclosure describes a video decoder 300 based on techniques of VVC and HEVC. However, the techniques of this disclosure can be implemented by video decoding devices configured for other video decoding standards.

[0184] exist Figure 7 In the example, the video decoder 300 includes a decoded picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transform processing unit 308, a reconstruction unit 310, a filter unit 312, and a decoded picture buffer (DPB) 314. Any one or all of the CPB memory 320, entropy decoding unit 302, prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, filter unit 312, and DPB 314 can be implemented in one or more processors or processing circuits. Furthermore, the video decoder 300 may include additional or alternative processors or processing circuits to perform these and other functions.

[0185] The prediction processing unit 304 includes a motion compensation unit 316 and an intra-prediction unit 318. The prediction processing unit 304 may include additional units to perform predictions based on other prediction modes. As an example, the prediction processing unit 304 may include a palette unit, an intra-block copying unit (which may form part of the motion compensation unit 316), an affine unit, a linear model (LM) unit, etc. In other examples, the video decoder 300 may include more, fewer, or different functional components. Figure 7 In the example, motion compensation unit 316 may include motion refinement unit 317.

[0186] CPB memory 320 can store video data to be decoded by components of video decoder 300, such as encoded video bitstreams. The video data stored in CPB memory 320 can be, for example, from computer-readable medium 110 (…). Figure 1 The CPB memory 320 may include a CPB that stores encoded video data (e.g., syntax elements) from the encoded video bitstream. Similarly, the CPB memory 320 may store video data other than the syntax elements of the decoded picture, such as temporary data representing the output from various units of the video decoder 300. The DPB 314 typically stores decoded pictures that the video decoder 300 may output as it decodes subsequent data or pictures from the encoded video bitstream and / or use as reference video data. The CPB memory 320 and DPB 314 may be formed of any of a variety of memory devices, such as DRAM, including SDRAM, MRAM, RRAM, or other types of memory devices. The CPB memory 320 and DPB 314 may be provided by the same memory device or separate memory devices. In various examples, the CPB memory 320 may be on-chip along with other components of the video decoder 300, or off-chip relative to those components.

[0187] Additionally or alternatively, in some examples, the video decoder 300 can be drawn from the memory 120 ( Figure 1 The decoded video data is obtained from the memory. That is, the memory 120 can store the data discussed above regarding the CPB memory 320. Similarly, the memory 120 can store instructions to be executed by the video decoder 300 when some or all of the functionality of the video decoder 300 is implemented in the software to be executed by the processing circuitry of the video decoder 300.

[0188] It shows Figure 7 The various units shown are to aid in understanding the operations performed by the video decoder 300. These units can be implemented as fixed-function circuits, programmable circuits, or combinations thereof. Similar to... Figure 6Fixed-function circuits refer to circuits that provide specific functionality and are pre-programmable for performing operations. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functionality in the operations they can perform. For example, a programmable circuit can execute software or firmware that causes it to operate in a manner defined by instructions from software or firmware. Fixed-function circuits can execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is usually constant. In some examples, one or more units may be different circuit blocks (fixed-function or programmable), and in some examples, one or more units may be integrated circuits.

[0189] The video decoder 300 may include an ALU, EFU, digital circuitry, analog circuitry, and / or a programmable core formed by programmable circuitry. In an example where the operation of the video decoder 300 is performed by software executed on the programmable circuitry, on-chip or off-chip memory may store instructions (e.g., object code) of the software received and executed by the video decoder 300.

[0190] Entropy decoding unit 302 can receive encoded video data from CPB and perform entropy decoding on the video data to reproduce syntax elements. Prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, and filter unit 312 can generate decoded video data based on syntax elements extracted from the bitstream.

[0191] Typically, the video decoder 300 reconstructs the image on a block-by-block basis. The video decoder 300 can perform the reconstruction operation on each block individually (where the block currently being reconstructed, i.e., the decoded block, can be referred to as the "current block").

[0192] Entropy decoding unit 302 can entropy decode the syntax elements of the quantized transform coefficients defining the quantized transform coefficient block, as well as transform information (such as quantization parameters (QP) and / or (multiple) transform mode indications). Inverse quantization unit 306 can use the QP associated with the quantized transform coefficient block to determine the degree of quantization, and similarly, determine the degree of inverse quantization to be applied by inverse quantization unit 306. Inverse quantization unit 306 can, for example, perform a bit-by-bit left shift operation to inverse quantize the quantized transform coefficients. Inverse quantization unit 306 can thus form a transform coefficient block including the transform coefficients.

[0193] After the inverse quantization unit 306 forms the transform coefficient block, the inverse transform processing unit 308 can apply one or more inverse transforms to the transform coefficient block to generate a residual block associated with the current block. For example, the inverse transform processing unit 308 can apply an inverse DCT, an inverse integer transform, an inverse Caronan-Loy transform (KLT), an inverse rotation transform, an inverse direction transform, or another inverse transform to the transform coefficient block.

[0194] Furthermore, prediction processing unit 304 generates prediction blocks based on prediction information syntax elements entropy decoded by entropy decoding unit 302. For example, if the prediction information syntax elements indicate that the current block is inter-frame predicted, motion compensation unit 316 can generate prediction blocks. In this case, the prediction information syntax elements may indicate a reference image in DPB 314 from which the reference block is obtained, and a motion vector that identifies the position of the reference block in the reference image relative to the position of the current block in the current image. Motion compensation unit 316 can generally be configured in a manner substantially similar to that relative to motion compensation unit 224 ( Figure 6 The method described is used to perform the inter-frame prediction process.

[0195] The motion refinement unit 317 of the motion compensation unit 316 can use motion refinement processes (such as, for example, BDOF and / or DMVR processes) to generate prediction blocks. Examples of BDOF and DMVR processes are described elsewhere in this disclosure. The motion refinement unit 317 can check one or more early termination conditions. If one or more early termination conditions are not met, the motion refinement unit 317 can continue to apply motion vector refinement processes (e.g., BDOF and / or DMVR processes) to generate prediction blocks. Otherwise, if the early termination conditions are met, the motion refinement unit 317 can generate prediction blocks without using the application of motion vector refinement processes (e.g., early termination of motion vector refinement process application). In some examples of this disclosure, one or more early termination conditions are modified relative to the early termination conditions set forth in VVC Draft 7. Specifically, as an example, early termination based on the sum of absolute differences (SAD) is applied to both BDOF and DMVR to reduce the average complexity on the decoder side, regardless of whether the bidirectional motion vectors are symmetrical (or nearly symmetrical) or whether the absolute values ​​of the bidirectional motion vectors are small (i.e., less than a threshold). Therefore, in some examples, the motion refinement unit 317 can perform the motion vector refinement process even when the bidirectional motion vectors are symmetrical (or nearly symmetrical) and / or when the absolute values ​​of the bidirectional motion vectors are small.

[0196] As another example, if the prediction information syntax element indicates that the current block is intra-predictable, then intra-prediction unit 318 can generate a prediction block according to the intra-prediction mode indicated by the prediction information syntax element. Again, intra-prediction unit 318 can generally be configured in a manner substantially similar to that of intra-prediction unit 226 ( Figure 6 The intra-prediction process is performed in the manner described. The intra-prediction unit 318 can obtain data of samples adjacent to the current block from the DPB 314.

[0197] Reconstruction unit 310 can use the prediction block and the residual block to reconstruct the current block. For example, reconstruction unit 310 can add samples from the residual block to the corresponding samples from the prediction block to reconstruct the current block.

[0198] Filter unit 312 can perform one or more filtering operations on the reconstructed block. For example, filter unit 312 can perform a deblocking operation to reduce block artifacts along the edges of the reconstructed block. The operation of filter unit 312 does not need to be performed in all examples.

[0199] The video decoder 300 can store the reconstructed blocks in the DPB 314. For example, in an example where the filter unit 312 is not operated, the reconstruction unit 310 can store the reconstructed blocks in the DPB 314. In an example where the filter unit 312 is operated, the filter unit 312 can store the filtered reconstructed blocks in the DPB 314. As discussed above, the DPB 314 can provide reference information to the prediction processing unit 304, such as samples of the current image for intra-frame prediction and previously decoded images for subsequent motion compensation. Furthermore, the video decoder 300 can output the decoded images (e.g., decoded video) from the DPB 314 for subsequent presentation on a display device (such as...). Figure 1 On the display device 118).

[0200] In this manner, video decoder 300 represents an example of a video decoding device, which includes a memory configured to store video data, and one or more processing units implemented in the circuit and configured to determine the bidirectional motion vector of the current block of video data. The one or more processing units may determine a condition for the current block based on each component of the bidirectional motion vector of the current block being less than a threshold. The one or more processing units may prematurely terminate the motion vector refinement process applied to the bidirectional motion vector of the current block based on the condition for the current block. The one or more processing units may determine a prediction block for the current block based on the bidirectional motion vector of the current block. The one or more processing units may reconstruct the current block based on the prediction block for the current block, and / or generate residual data for the current block based on the prediction block for the current block.

[0201] Figure 8 This is a flowchart illustrating an example method for encoding the current block. The current block may include the current CU. Although relative to video encoder 200 ( Figure 1 and Figure 4 The description is provided, but it should be understood that other devices can be configured to perform similar actions. Figure 8 The method.

[0202] In this example, video encoder 200 predicts the current block (350). For example, video encoder 200 may form a prediction block for the current block. As part of predicting the current block, video encoder 200 may use techniques disclosed herein (e.g., see...). Figure 10 The motion vectors are determined, and then the motion vectors can be used to determine the prediction block for the current block. For example, video encoder 200 (e.g., motion estimation unit 222) can determine the bidirectional motion vectors of the current block of video data. Video encoder 200 (e.g., motion refinement unit 225) can determine whether a condition is met for the current block. For example, video encoder 200 (e.g., motion refinement unit 225) can determine the condition is met for the current block based on each component of the bidirectional motion vector of the current block being less than a threshold. Based on the condition that the current block is met, video encoder 200 (e.g., motion refinement unit 225) can terminate the motion vector refinement process applied to the bidirectional motion vectors of the current block early. For example, video encoder 200 (e.g., motion compensation unit 224) can avoid applying or bypass DMVR and / or BDOF. Video encoder 200 (e.g., motion compensation unit 224) can determine the prediction block for the current block based on the bidirectional motion vectors of the current block.

[0203] In some examples, the video encoder 200 may determine a first motion vector and a second motion vector for the current block of video data; determine a selected motion vector from the first and second motion vectors; generate a refined version of the selected motion vector by applying a motion vector refinement process to the selected motion vector; and determine a prediction block for the current block based on the refined version of the selected motion vector and the one of the first and second motion vectors that is not the selected initial motion vector.

[0204] The video encoder 200 can then compute the residual block for the current block (352). To compute the residual block, the video encoder 200 can compute the difference between the original, uncoded block and the predicted block for the current block. The video encoder 200 can then transform and quantize the transform coefficients of the residual block (354). Next, the video encoder 200 can scan the quantized transform coefficients of the residual block (356). During or after the scan, the video encoder 200 can entropy-encode the transform coefficients (358). For example, the video encoder 200 can use CAVLC or CABAC to encode the transform coefficients. The video encoder 200 can then output the entropy-encoded data of the block (360).

[0205] Figure 9 This is a flowchart illustrating an example method for decoding the current block of video data. The current block may include the current CU. Although relative to video decoder 300 ( Figure 1 As described in Figure 3, however it should be understood that other devices can be configured to perform similar actions. Figure 9 The method.

[0206] The video decoder 300 can receive entropy-coded data for the current block, such as entropy-coded data for the transform coefficients of the residual block corresponding to the current block and entropy-coded prediction information (370). The video decoder 300 can entropy decode the entropy-coded data to determine the prediction information for the current block and the transform coefficients for reproducing the residual block (372).

[0207] The video decoder 300 may, for example, use an intra-frame or inter-frame prediction mode as indicated by the prediction information for the current block to predict the current block (374) to compute a predicted block for the current block. As part of predicting the current block, the video decoder 300 may use techniques disclosed herein (e.g., see...). Figure 10 The motion vectors are determined, and then the motion vectors can be used to determine the prediction block for the current block. For example, video decoder 300 (e.g., prediction processing unit 304) can determine the bidirectional motion vectors of the current block of video data. Video decoder 300 (e.g., motion refinement unit 317) can determine whether a condition is met for the current block. For example, video encoder 200 (e.g., motion refinement unit 317) can determine whether the condition is met for the current block based on each component of the bidirectional motion vector of the current block being less than a threshold. Based on the condition that the current block is met, video decoder 300 (e.g., motion refinement unit 317) can prematurely terminate the motion vector refinement process applied to the bidirectional motion vectors of the current block. For example, video decoder 300 (e.g., motion compensation unit 316) can avoid applying or bypass DMVR and / or BDOF. Video decoder 300 (e.g., motion compensation unit 316) can determine the prediction block for the current block based on the bidirectional motion vectors of the current block.

[0208] In some examples, the video decoder 300 may determine a first motion vector and a second motion vector for the current block of video data; determine a selected motion vector from the first and second motion vectors; generate a refined version of the selected motion vector by applying a motion vector refinement process to the selected motion vector; and determine a prediction block for the current block based on the refined version of the selected motion vector and the one of the first and second motion vectors that is not the selected initial motion vector.

[0209] The video decoder 300 can then inversely scan the reproduced transform coefficients (376) to create a block of quantized transform coefficients. The video decoder 300 can then inversely quantize and inverse transform the transform coefficients to produce a residual block (378). The video decoder 300 can finally decode the current block by combining the predicted block and the residual block (380).

[0210] Figure 10 This is a flowchart illustrating an example method that can be performed by a video decoder according to one or more techniques disclosed herein. Figure 10 The method can be performed by the video encoder 200 or the video decoder 300.

[0211] exist Figure 10 In the example, a video decoder (e.g., motion refinement unit 225 of video encoder 200 or motion refinement unit 317 of video decoder 300) can determine the bidirectional motion vector of the current block of video data (402). The video decoder can determine whether a condition is met with respect to the current block (404). For example, the video decoder can determine whether the condition is met with respect to the current block based on each component of the bidirectional motion vector of the current block being less than a threshold.

[0212] Based on the conditions that the current block meets, a video decoder (e.g., motion refinement unit 225 of video encoder 200 or motion refinement unit 317 of video decoder 300) can prematurely terminate the bidirectional motion vector application motion vector refinement process (406) on the current block. For example, the video decoder can avoid or bypass the bidirectional motion vector application motion vector refinement process on the current block in response to the conditions that the current block meets.

[0213] A video decoder (e.g., motion refinement unit 225 of video encoder 200 or motion refinement unit 317 of video decoder 300) can determine a prediction block (408) for the current block based on the bidirectional motion vectors of the current block. For example, the video decoder can determine a prediction block for the current block based on the bidirectional motion vectors of the current block that bypass the motion vector refinement process.

[0214] A video decoder (e.g., motion refinement unit 225 of video encoder 200 or motion refinement unit 317 of video decoder 300) can reconstruct the current block based on the predicted block for the current block, and / or a video encoder (e.g., motion refinement unit 225 of video encoder 200) can generate residual data (410) for the current block based on the predicted block for the current block.

[0215] The following is a non-limiting list of examples of one or more technologies based on this disclosure.

[0216] Clause 1. A method for decoding video data, the method comprising: determining bidirectional motion vectors of a current block of video data; determining whether a condition is satisfied with respect to the current block; prematurely terminating a motion vector refinement process applied to the bidirectional motion vectors of the current block based on the condition being satisfied with respect to the current block; determining a prediction block for the current block based on the bidirectional motion vectors of the current block; and performing at least one of the following: reconstructing the current block based on the prediction block for the current block, or generating residual data for the current block based on the prediction block for the current block.

[0217] Clause 2. The method of Clause 1, wherein the current block is a first block, and the method further comprises: determining bidirectional motion vectors of a second block of video data; determining whether a condition is satisfied with respect to the second block; applying a motion vector refinement process to the bidirectional motion vectors of the second block based on the condition not being satisfied with respect to the current block; determining a prediction block for the second block based on the refined bidirectional motion vectors of the current block; and performing at least one of the following: reconstructing the second block based on the prediction block for the second block, or generating residual data for the second block based on the prediction block for the second block.

[0218] Clause 3. The method according to any one of Clauses 1 to 2, wherein the motion vector refinement process is a decoder-side motion vector refinement (DMVR) process.

[0219] Clause 4. The method according to any one of Clauses 1 to 3, wherein the motion vector refinement process is a bidirectional optical flow (BDOF) process.

[0220] Clause 5. The method according to any one of Clauses 1 to 4, wherein determining whether the condition is satisfied with respect to the current block comprises: determining whether the condition is satisfied with respect to the current block based on MVL0x not being equal to -1*MVL1x and MVL0y not being equal to -1*MVL1y, wherein MVL0x is the x-component of the list 0 motion vector of the current block's bidirectional motion vectors, MVL1x is the x-component of the list 1 motion vector of the current block's bidirectional motion vectors, MVL0y is the y-component of the list 0 motion vector of the current block's bidirectional motion vectors, and MVL1y is the y-component of the list 1 motion vector of the current block's bidirectional motion vectors.

[0221] Clause 6. The method according to any one of Clauses 1 to 5, wherein determining whether the condition is satisfied with respect to the current block comprises: determining whether the condition is satisfied with respect to the current block based on MVL0x not being equal to -1*MVL1x or MVL0y not being equal to -1*MVL1y, wherein MVL0x is the x-component of the list 0 motion vector of the current block's bidirectional motion vectors, MVL1x is the x-component of the list 1 motion vector of the current block's bidirectional motion vectors, MVL0y is the y-component of the list 0 motion vector of the current block's bidirectional motion vectors, and MVL1y is the y-component of the list 1 motion vector of the current block's bidirectional motion vectors.

[0222] Clause 7. The method according to any one of Clauses 1 to 6, wherein determining whether the condition is satisfied with respect to the current block includes: based on all Δ values ​​in the range from -N to N. x and Δ y MVL0x is not equal to -1*(MVL1x+Δ x And MVL0y is not equal to -1*(MVL1y+Δ y ) to determine the conditions satisfied with the current block, where MVL0x is the x-component of the bidirectional motion vector list 0 of the current block, MVL1x is the x-component of the bidirectional motion vector list 1 of the current block, MVL0y is the y-component of the bidirectional motion vector list 0 of the current block, MVL1y is the y-component of the bidirectional motion vector list 1 of the current block, and N is a sequence-level, picture-level, sub-picture-level, tile-level, or slice-level number or a predefined constant integer that signals in the bit stream.

[0223] Clause 8. The method according to any one of Clauses 1 to 7, wherein determining whether the condition is satisfied with respect to the current block includes: based on all Δ values ​​in the range from -N to N. x and Δ y MVL0x is not equal to -1*(MVL1x+Δ x ) or MVL0y is not equal to -1*(MVL1y+Δ y ) to determine the conditions satisfied with the current block, where MVL0x is the x-component of the bidirectional motion vector list 0 of the current block, MVL1x is the x-component of the bidirectional motion vector list 1 of the current block, MVL0y is the y-component of the bidirectional motion vector list 0 of the current block, MVL1y is the y-component of the bidirectional motion vector list 1 of the current block, and N is a sequence-level, picture-level, sub-picture-level, tile-level, or slice-level number or a predefined constant integer that signals in the bit stream.

[0224] Clause 9. The method according to any one of Clauses 1 to 8, wherein determining whether the condition is satisfied with respect to the current block comprises: determining whether the condition is satisfied with respect to the current block based on the fact that the p-norms of the bidirectional motion vectors of the current block are different from each other.

[0225] Clause 10. The method according to any one of Clauses 1 to 9, wherein determining whether the condition is satisfied with respect to the current block comprises: determining whether the condition is satisfied with respect to the current block based on the difference between the p-norms of the bidirectional motion vectors of the current block being greater than a threshold.

[0226] Clause 11. The method according to any one of Clauses 1 to 10, wherein determining whether the condition is satisfied with respect to the current block comprises: determining whether the condition is satisfied with respect to the current block based on the difference between the p-norms of the bidirectional motion vectors of the current block being less than a threshold.

[0227] Clause 12. The method according to any one of Clauses 1 to 11, wherein determining whether the condition is satisfied with respect to the current block comprises: determining whether the condition is satisfied with respect to the current block based on each component of the bidirectional motion vector of the current block being less than a threshold.

[0228] Clause 13. A method for decoding video data, the method comprising: determining a first motion vector and a second motion vector of a current block of video data; determining a selected motion vector from the first and second motion vectors; generating a refined version of the selected motion vector by applying a motion vector refinement process to the selected motion vector; determining a prediction block for the current block based on the refined version of the selected motion vector and the first and second motion vectors that is not the selected initial motion vector; and performing at least one of: reconstructing the current block based on the prediction block for the current block, or generating residual data for the current block based on the prediction block for the current block.

[0229] Clause 14. The method according to Clause 13, wherein the motion vector refinement process is a decoder-side motion vector refinement (DMVR) process.

[0230] Clause 15. The method according to any one of Clauses 13 to 14, wherein the motion vector refinement process is a bidirectional optical flow (BDOF) process.

[0231] Clause 16. The method according to any one of Clauses 13 to 15 further includes: including one or more syntax elements indicating that a first motion vector or a second motion vector of the current block is a selected motion vector in the bitstream comprising an encoded representation of the video data.

[0232] Clause 17. The method according to any one of Clauses 13 to 15, wherein determining the selected motion vector comprises: determining which of the first motion vector or the second motion vector of the current block is the selected motion vector based on one or more syntaxes signaled in a bit stream comprising an encoded representation of video data.

[0233] Clause 18. The method according to any one of Clauses 13 to 17, wherein determining the selected motion vector comprises: determining the selected motion vector based on the lengths of the first motion vector of the current block and the second motion vector of the current block.

[0234] Clause 19. The method according to any one of Clauses 13 to 18, wherein determining the selected motion vector comprises: determining the selected motion vector based on the symmetry of the first motion vector and the second motion vector of the current block.

[0235] Clause 20. The method according to Clause 19, wherein determining the selected motion vector comprises: selecting the shorter of the first motion vector and the second motion vector of the current block as the selected motion vector based on the asymmetry between them.

[0236] Clause 21. The method according to any one of Clauses 13 to 20, wherein the method further comprises: making a determination based on the symmetry between a first motion vector of the current block and a second motion vector of the current block to determine a selected motion vector.

[0237] Clause 22. The method of determining the selected motion vector according to any one of Clauses 13 to 20 includes: based on all Δ values ​​in the range from -N to N. x and Δ y MVL0≠-1*(MVL1+(Δ x Δ y To make a determination, in order to determine the selected motion vector, where N is a sequence-level, picture-level, sub-picture-level, tile-level, slice-level number or a predefined constant integer that is signaled in the bit stream.

[0238] Clause 23. The method according to any one of Clauses 1 to 22, wherein decoding includes decoding.

[0239] Clause 24. The method according to any one of Clauses 1 to 22, wherein decoding includes encoding.

[0240] Clause 25. An apparatus for decoding video data, the apparatus comprising one or more components for performing a method according to any one of Clauses 1 to 22.

[0241] Clause 26. The device pursuant to Clause 25, wherein the one or more components include one or more processors implemented in a circuit.

[0242] Clause 27. The device pursuant to any one of Clauses 25 and 26 further includes a memory for storing video data.

[0243] Clause 28. The device pursuant to any one of Clauses 25 to 27 further includes a display configured to display decoded video data.

[0244] Clause 29. The device pursuant to any one of Clauses 25 to 28, wherein the device includes one or more of a camera, computer, mobile device, broadcast receiver device or set-top box.

[0245] Clause 30. An apparatus pursuant to any one of Clauses 25 to 29, wherein the apparatus includes a video decoder.

[0246] Clause 31. The device pursuant to any one of Clauses 25 to 30, wherein the device includes a video encoder.

[0247] Clause 32. A computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to perform a method according to any one of Clauses 1 to 22.

[0248] It will be recognized that, depending on the example, some actions or events of any of the techniques described herein may be performed in a different order, or may be added, combined, or omitted together (e.g., not all described actions or events are necessary for the practice of the technique). Furthermore, in some examples, actions or events may be performed concurrently by multithreading, interrupt handling, or multiple processors, rather than sequentially.

[0249] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, these functions may be stored or transmitted as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include: a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium that includes, for example, any medium facilitating the transfer of a computer program from one place to another according to a communication protocol. In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium accessible by one or more computers or one or more processors to obtain instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products may include computer-readable media.

[0250] For example, and not as a limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Similarly, any connection is properly referred to as a computer-readable medium. For example, coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave) are included in the definition of medium if the instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies (such as infrared, radio, and microwave). However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather refer to non-transient, tangible storage media. As used herein, disks and platters include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically magnetically reproduce data, while platters optically reproduce data using lasers. The above combinations should also be included within the scope of computer-readable media.

[0251] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the terms "processor" and "processing circuit" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Similarly, these techniques can be fully implemented within one or more circuit or logic elements.

[0252] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless handheld devices, integrated circuits (ICs), or IC sets (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but implementation by different hardware units is not necessarily required. Rather, as described above, various units can be combined in a codec hardware unit, or provided by a collection of interoperable hardware units including one or more processors as described above, combined with suitable software and / or firmware.

[0253] Various examples have been described. These and other examples are all within the scope of the following claims.

Claims

1. A method for decoding video data, the method comprising: Determine the bidirectional motion vector of the current block of the video data; The condition is determined based on whether each component of the bidirectional motion vector of the current block is less than a threshold, or whether the bidirectional motion vector of the current block is asymmetric. The motion vector refinement process for the bidirectional motion vector of the current block is terminated early based on the condition that the current block meets the above conditions. The predicted block for the current block is determined based on the bidirectional motion vector of the current block; as well as The current block is reconstructed based on the predicted block for the current block.

2. The method according to claim 1, wherein, Determine that the absolute value of each component of the bidirectional motion vector of the current block is less than the threshold, which satisfies the condition for the current block.

3. The method according to claim 1, wherein, The motion vector refinement process is the decoder-side motion vector refinement DMVR process.

4. The method according to claim 1, wherein, The motion vector refinement process is a bidirectional optical flow (BDOF) process.

5. The method according to claim 1, wherein, The current block is the first block, and the method further includes: Determine the bidirectional motion vector of the second block of the video data; Based on the fact that the second block does not meet the conditions, the motion vector refinement process is applied to the bidirectional motion vector of the second block to determine the refined bidirectional motion vector of the second block; The prediction block for the second block is determined based on the refined bidirectional motion vector of the second block; and The second block is reconstructed based on the predicted block for the second block.

6. The method according to claim 1, wherein, Determining that the current block satisfies the conditions includes: The condition is determined based on MVL0x not being equal to -1*MVL1x and MVL0y not being equal to -1*MVL1y, where MVL0x is the x-component of the list 0 motion vector of the bidirectional motion vector of the current block, MVL1x is the x-component of the list 1 motion vector of the bidirectional motion vector of the current block, MVL0y is the y-component of the list 0 motion vector of the bidirectional motion vector of the current block, and MVL1y is the y-component of the list 1 motion vector of the bidirectional motion vector of the current block.

7. The method according to claim 1, wherein, Determining that the current block satisfies the conditions includes: The condition is determined based on MVL0x not being equal to -1*MVL1x or MVL0y not being equal to -1*MVL1y, where MVL0x is the x-component of the list 0 motion vector of the bidirectional motion vectors of the current block, MVL1x is the x-component of the list 1 motion vector of the bidirectional motion vectors of the current block, MVL0y is the y-component of the list 0 motion vector of the bidirectional motion vectors of the current block, and MVL1y is the y-component of the list 1 motion vector of the bidirectional motion vectors of the current block.

8. The method according to claim 1, wherein, Determining that the current block satisfies the conditions includes: Based on all Δ in the range from -N to N x and Δ y MVL0x is not equal to -1*(MVL1x+Δ x And MVL0y is not equal to -1*(MVL1y+Δ y ) to determine whether the current block satisfies the condition, wherein MVL0x is the x-component of the list 0 motion vector of the bidirectional motion vector of the current block, MVL1x is the x-component of the list 1 motion vector of the bidirectional motion vector of the current block, MVL0y is the y-component of the list 0 motion vector of the bidirectional motion vector of the current block, MVL1y is the y-component of the list 1 motion vector of the bidirectional motion vector of the current block, and N is a sequence-level, picture-level, sub-picture-level, tile-level, slice-level number or a predefined constant integer that signals in the bit stream.

9. The method according to claim 1, wherein, Determining that the current block satisfies the conditions includes: Based on all Δ in the range from -N to N x and Δ y MVL0x is not equal to -1*(MVL1x+Δ x ) or MVL0y is not equal to -1*(MVL1y+Δ y ) to determine whether the current block satisfies the condition, wherein MVL0x is the x-component of the list 0 motion vector of the bidirectional motion vector of the current block, MVL1x is the x-component of the list 1 motion vector of the bidirectional motion vector of the current block, MVL0y is the y-component of the list 0 motion vector of the bidirectional motion vector of the current block, MVL1y is the y-component of the list 1 motion vector of the bidirectional motion vector of the current block, and N is a sequence-level, picture-level, sub-picture-level, tile-level, slice-level number or a predefined constant integer that signals in the bit stream.

10. The method according to claim 1, wherein, Determining that the current block satisfies the condition includes: determining that the current block satisfies the condition based on the fact that the lengths of the bidirectional motion vectors of the current block are different from each other.

11. The method according to claim 1, wherein, Determining that the current block satisfies the condition includes: determining that the current block satisfies the condition based on the difference between the lengths of the bidirectional motion vectors of the current block being greater than a threshold.

12. The method according to claim 1, wherein, Determining that the current block satisfies the condition includes: determining that the current block satisfies the condition based on the difference between the lengths of the bidirectional motion vectors of the current block being less than a threshold.

13. The method according to claim 1, further comprising: Residual data for the current block is generated based on the predicted block for the current block.

14. An apparatus for decoding video data, comprising: The memory is configured to store video data; as well as One or more processors are implemented in a circuit and configured to: Determine the bidirectional motion vector of the current block of the video data; The condition is determined based on whether each component of the bidirectional motion vector of the current block is less than a threshold, or whether the bidirectional motion vector of the current block is asymmetric. The motion vector refinement process for the bidirectional motion vector of the current block is terminated early based on the condition that the current block meets the above conditions. The predicted block for the current block is determined based on the bidirectional motion vector of the current block; as well as The current block is reconstructed based on the predicted block for the current block.

15. The device according to claim 14, wherein, Determine that the absolute value of each component of the bidirectional motion vector of the current block is less than the threshold, which satisfies the condition for the current block.

16. The device according to claim 14, wherein, The motion vector refinement process is the decoder-side motion vector refinement DMVR process.

17. The device according to claim 14, wherein, The motion vector refinement process is a bidirectional optical flow (BDOF) process.

18. The device according to claim 14, wherein, The current block is the first block, and the one or more processors are further configured to: Determine the bidirectional motion vector of the second block of the video data; Based on the fact that the second block does not meet the conditions, the motion vector refinement process is applied to the bidirectional motion vector of the second block to determine the refined bidirectional motion vector of the second block; The prediction block for the second block is determined based on the refined bidirectional motion vector of the second block; as well as The second block is reconstructed based on the predicted block for the second block.

19. The device according to claim 14, wherein, To determine whether the current block satisfies the condition, the one or more processors are configured to: The condition is determined based on MVL0x not being equal to -1*MVL1x and MVL0y not being equal to -1*MVL1y, where MVL0x is the x-component of the list 0 motion vector of the bidirectional motion vector of the current block, MVL1x is the x-component of the list 1 motion vector of the bidirectional motion vector of the current block, MVL0y is the y-component of the list 0 motion vector of the bidirectional motion vector of the current block, and MVL1y is the y-component of the list 1 motion vector of the bidirectional motion vector of the current block.

20. The device according to claim 14, wherein, To determine whether the current block satisfies the condition, the one or more processors are configured to: The condition is determined based on MVL0x not being equal to -1*MVL1x or MVL0y not being equal to -1*MVL1y, where MVL0x is the x-component of the list 0 motion vector of the bidirectional motion vectors of the current block, MVL1x is the x-component of the list 1 motion vector of the bidirectional motion vectors of the current block, MVL0y is the y-component of the list 0 motion vector of the bidirectional motion vectors of the current block, and MVL1y is the y-component of the list 1 motion vector of the bidirectional motion vectors of the current block.

21. The device according to claim 14, wherein, To determine whether the current block satisfies the condition, the one or more processors are configured to: The condition is determined to be satisfied with the current block based on the following: for all Δx and Δy in the range from -N to N, MVL0x is not equal to -1*(MVL1x+Δx) and MVL0y is not equal to -1*(MVL1y+Δy), where MVL0x is the x-component of the list 0 motion vector of the bidirectional motion vector of the current block, MVL1x is the x-component of the list 1 motion vector of the bidirectional motion vector of the current block, MVL0y is the y-component of the list 0 motion vector of the bidirectional motion vector of the current block, MVL1y is the y-component of the list 1 motion vector of the bidirectional motion vector of the current block, and N is a sequence-level, picture-level, sub-picture-level, tile-level, slice-level number or a predefined constant integer that signals notification in the bit stream.

22. The device according to claim 14, wherein, To determine whether the current block satisfies the condition, the one or more processors are configured to: The condition is determined to be satisfied with the current block based on the following: for all Δx and Δy in the range from -N to N, MVL0x is not equal to -1*(MVL1x+Δx) or MVL0y is not equal to -1*(MVL1y+Δy), where MVL0x is the x-component of the list 0 motion vector of the bidirectional motion vector of the current block, MVL1x is the x-component of the list 1 motion vector of the bidirectional motion vector of the current block, MVL0y is the y-component of the list 0 motion vector of the bidirectional motion vector of the current block, MVL1y is the y-component of the list 1 motion vector of the bidirectional motion vector of the current block, and N is a sequence-level, picture-level, sub-picture-level, tile-level, slice-level number or a predefined constant integer that signals notification in the bit stream.

23. The device according to claim 14, wherein, In order to determine that the condition is satisfied with respect to the current block, the one or more processors are configured to determine that the condition is satisfied with respect to the current block based on the fact that the lengths of the bidirectional motion vectors of the current block are different from each other.

24. The device according to claim 14, wherein, To determine whether the condition is met for the current block, the one or more processors are configured to determine whether the condition is met for the current block based on the difference between the lengths of the bidirectional motion vectors of the current block being greater than a threshold.

25. The device according to claim 14, wherein, To determine whether the condition is met for the current block, the one or more processors are configured to determine whether the condition is met for the current block based on the difference between the lengths of the bidirectional motion vectors of the current block being less than a threshold.

26. The device according to claim 14, wherein, The one or more processors are configured to generate residual data for the current block based on the predicted block for the current block.

27. The device of claim 14, further comprising a display configured to display the video data.

28. The device according to claim 14, wherein, The device includes one or more of a camera, computer, mobile device, broadcast receiver device, set-top box, integrated circuit, microprocessor, or wireless communication device.

29. The device of claim 14, further comprising a camera configured to capture the video data.

30. A computer-readable storage medium having instructions stored thereon, said instructions causing a processor, when executed: Determine the bidirectional motion vector of the current block of video data; The condition is determined based on whether each component of the bidirectional motion vector of the current block is less than a threshold, or whether the bidirectional motion vector of the current block is asymmetric. The motion vector refinement process for the bidirectional motion vector of the current block is terminated early based on the condition that the current block meets the above conditions. The predicted block for the current block is determined based on the bidirectional motion vector of the current block; as well as The current block is reconstructed based on the predicted block for the current block.

31. An apparatus for decoding video data, the apparatus comprising components for performing the method according to any one of claims 1 to 13.

32. A computer program product comprising computer-readable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 13.

Images