Hybrid inter bi-prediction in video coding

Abstract

A video decoder can be configured to: determine that the current block of video data is decoded in a dual predictive inter-frame mode; receive a first syntax element for identifying motion vector predictors from a first candidate list of motion vector predictors; receive a second syntax element for identifying motion vector differences; determine a first motion vector for the current block based on the motion vector predictors and the motion vector differences; determine a second motion vector for the current block from a second candidate list of motion vector predictors based on bilateral matching; and use the first and second motion vectors to determine a predicted block for the current block.

Description

Technical Field

[0001] This patent application claims the rights of the following applications: U.S. Provisional Patent Application No. 63 / 266,390, filed January 4, 2022; U.S. Provisional Patent Application No. 63 / 283,004, filed November 24, 2021; U.S. Provisional Patent Application No. 63 / 235,071, filed on August 19, 2021; U.S. Provisional Patent Application No. 63 / 215,257, filed June 25, 2021; the entire contents of each of the above applications are incorporated herein by reference.

[0002] This case concerns video encoding and video decoding. Prior Technology

[0003] Digital video capabilities can be incorporated into a wide variety of devices, including digital televisions, digital live broadcasting systems, wireless broadcasting systems, personal digital assistants (PDAs), laptops or desktop computers, tablets, e-book readers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite radio phones (so-called "smartphones"), video conferencing equipment, video streaming devices, etc. Digital video devices implement video decoding technologies (such as those defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 (Part 10, Advanced Video Decoding (AVC)), ITU-T H.265 / High Efficiency Video Decoding (HEVC), ITU-T H.266 / Various Video Decoding (VVC) and extensions to such standards, as well as proprietary video transcoders / formats (such as AOMedia Video 1 (AV1) developed by the Open Media Consortium). By implementing this video decoding technology, video devices can transmit, receive, encode, decode, and / or store digital video information more efficiently.

[0004] Video decoding techniques include spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences. For block-based video decoding, video slices (e.g., video pictures or portions of video pictures) can be segmented into video blocks, which may also be referred to as decoding tree units (CTUs), decoding units (CUs), and / or decoding nodes. Video blocks in a slice of a picture decoded intra-picture (I) are encoded using spatial prediction relative to reference samples in adjacent slices within the same picture. Video blocks in a slice of a picture decoded inter-picture (P or B) can use spatial prediction relative to reference samples in adjacent slices within the same picture, or temporal prediction relative to reference samples in other reference pictures. A picture may be referred to as a frame, and a reference picture may be referred to as a reference frame. Summary of the Invention

[0005] According to the technology described in this case, for blocks using dual prediction, one of the two motion vectors used for dual prediction can be determined using explicit signaling, while the other motion vector can be predicted without explicit signaling by using decoder-side motion derivation. The technique of using one explicitly signaled motion vector and one derived motion vector without signaling may be referred to herein as hybrid inter-frame dual prediction, or more simply as hybrid dual-mode.

[0006] Using a hybrid dual-mode approach, a first motion vector can be derived from a first candidate list, where the video decoder receives indices in the bitstream for candidate identification. Using the hybrid dual-mode approach, the video decoder can determine a second motion vector from a second candidate list, but instead of receiving explicit syntax for identifying the selected candidate in the bitstream, the video decoder can determine the candidate for the second motion vector based on the minimum bilateral matching error between a first reference block and a second reference block identified by the first motion vector. For each candidate in the second candidate list, the video decoder can locate the second reference block and determine the bilateral matching error between the second reference block and the first reference block. The video decoder can then select a motion vector from the second candidate list pointing to the second reference block as the second motion vector, which produces the minimum or lowest bilateral matching error relative to the first reference block. In some cases, the video decoder can use various decoder-side motion vector derivation techniques to further refine the selected motion vector.

[0007] By determining the first motion vector for the current block based on a motion vector predictor explicitly signaled and the motion vector difference, and determining the second motion vector for the current block based on bilateral matching from a list of second candidate motion vector predictors without explicit signaling, the technique of this invention can advantageously produce an improved rate-distortion tradeoff for dual-prediction blocks. For example, the technique of this invention can reduce the signaling management burden associated with dual prediction while still maintaining high prediction quality.

[0008] According to one example of the content of this case, a method for decoding video data includes the following steps: determining that the current block of the video data is decoded in a dual predictive frame mode; receiving a first syntax element for identifying motion vector predictors from a first candidate list of motion vector predictors; receiving a second syntax element for identifying motion vector differences; determining a first motion vector for the current block based on the motion vector predictors and the motion vector differences; determining a second motion vector for the current block from a second candidate list of motion vector predictors based on bilateral matching; and using the first motion vector and the second motion vector to determine a prediction block for the current block.

[0009] According to another example of the content of this case, an apparatus for decoding video data includes: a memory configured to store video data; and one or more processors implemented in a circuit system and configured to: determine that a current block of the video data is decoded in a dual predictive inter-frame mode; receive a first syntax element for identifying motion vector predictors from a first candidate list of motion vector predictors; receive a second syntax element for identifying motion vector differences; determine a first motion vector for the current block based on the motion vector predictors and the motion vector differences; determine a second motion vector for the current block from a second candidate list of motion vector predictors based on bilateral matching; and use the first motion vector and the second motion vector to determine a prediction block for the current block.

[0010] According to another example of the content of this case, an apparatus for encoding video data includes: a memory configured to store video data; and one or more processors implemented in a circuit system and configured to: determine that a current block of the video data is decoded in a dual predictive frame mode; determine a first motion vector for the current block; generate a first syntax element for identifying motion vector predictors from a first candidate list of motion vector predictors to be included in a bitstream of the encoded video data; generate a second syntax element for identifying a motion vector difference to be included in the bitstream of the encoded video data, wherein the motion vector predictor and the motion vector difference correspond to the first motion vector; determine a second motion vector for the current block from a second candidate list of motion vector predictors based on bilateral matching; use the first motion vector and the second motion vector to determine a prediction block for the current block; and encode the current block of the video data based on the prediction block.

[0011] According to another example of the content of this case, a computer-readable storage media storage instruction, when executed by one or more processors, causes the one or more processors to perform the following operations: determine that the current block of the video data is decoded in a dual predictive frame mode; receive a first syntax element for identifying motion vector predictors from a first candidate list of motion vector predictors; receive a second syntax element for identifying motion vector differences; determine a first motion vector for the current block based on the motion vector predictors and the motion vector differences; determine a second motion vector for the current block from a second candidate list of motion vector predictors based on bilateral matching; and use the first motion vector and the second motion vector to determine a prediction block for the current block.

[0012] According to another example of the content of this case, an apparatus for decoding video data includes: a component for determining that a current block of the video data is decoded in a dual predictive inter-frame mode; a component for receiving a first syntax element for identifying a motion vector predictor from a first candidate list of motion vector predictors; a component for receiving a second syntax element for identifying a motion vector difference; a component for determining a first motion vector for the current block based on the motion vector predictor and the motion vector difference; a component for determining a second motion vector for the current block from a second candidate list of motion vector predictors based on bilateral matching; and a component for determining a prediction block for the current block using the first motion vector and the second motion vector.

[0013] According to another example of the content of this case, an apparatus for encoding video data includes: means for determining a current block of the video data to be decoded in a double predictive inter-frame mode; means for determining a first motion vector for the current block; means for generating a first syntax element for identifying motion vector predictors from a first candidate list of motion vector predictors, to be included in a bitstream of the encoded video data; means for generating a second syntax element for identifying a motion vector difference, to be included in the bitstream of the encoded video data, wherein the motion vector predictor and the motion vector difference correspond to the first motion vector; means for determining a second motion vector for the current block from a second candidate list of motion vector predictors based on bilateral matching; means for using the first motion vector and the second motion vector to determine a prediction block for the current block; and means for encoding the current block of the video data based on the prediction block.

[0014] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, purposes, and advantages will be apparent from the description, drawings, and claims. Simple Explanation of the Diagram

[0015] Figure 1 is a block diagram illustrating an exemplary video encoding and decoding system capable of performing the techniques described in this case.

[0016] Figure 2A illustrates an example of spatially adjacent motion vector candidates for merging patterns.

[0017] Figure 2B illustrates an example of spatially adjacent motion vector candidates for the AMVP mode.

[0018] Figure 3A illustrates an example of a time motion vector predictor candidate.

[0019] Figure 3B illustrates an example of motion vector scaling.

[0020] Figure 4 illustrates an example of motion vectors proportional to time distance.

[0021] Figure 5 illustrates an example of a motion vector that is mirrored without considering time distance.

[0022] Figure 6 illustrates an example of a 3×3 square search pattern with a search range of [-8, 8].

[0023] Figure 7 illustrates an example of template matching performed over the search region around the initial motion vector.

[0024] Figure 8 illustrates an example of motion vector refinement on the decoding side.

[0025] Figure 9 illustrates an example of a valid candidate for the second predictor when performing hybrid inter-frame double prediction using bilateral matching.

[0026] Figure 10 illustrates an example of bilateral matching prediction using a fixed template.

[0027] Figure 11 illustrates an example of a full search with skipped search areas.

[0028] Figure 12 is a block diagram illustrating an exemplary video encoder capable of performing the techniques described in this case.

[0029] Figure 13 is a block diagram illustrating an exemplary video decoder that can perform the technology described in this case.

[0030] Figure 14 is a flowchart illustrating an exemplary procedure for encoding the current block according to the technology described in this case.

[0031] Figure 15 is a flowchart illustrating an exemplary procedure for decoding the current block according to the technology described in this case.

[0032] Figure 16 is a flowchart illustrating an exemplary procedure for encoding the current block according to the technology described in this case.

[0033] Figure 17 is a flowchart illustrating an exemplary procedure for decoding the current block according to the technology described in this case. Implementation

[0034] Video decoding (e.g., video encoding and / or video decoding) typically involves predicting blocks of video data from decoded blocks of video data within the same frame (e.g., intra-frame prediction) or from decoded blocks of video data in different frames (e.g., inter-frame prediction). In some cases, the video encoder also calculates residual data by comparing the predicted blocks with the original blocks. Therefore, the residual data represents the difference between the predicted block and the original block. To reduce the number of bits required to signal the residual data, the video encoder transforms and quantizes the residual data and signals the transformed and quantized residual data in the encoded bitstream. Compression achieved through the transformation and quantization process can be lossy, meaning that the transformation and quantization process may introduce distortion into the decoded video data.

[0035] The video decoder decodes the residual data and adds it to the prediction block to produce a reconstructed video block that matches the original video block more closely than the prediction block alone. Due to the loss introduced by transforming and quantizing the residual data, the first reconstructed block may have distortion or artifacts. A common type of artifact or distortion is called blockiness, where the boundaries of the blocks used to decode the video data are visible.

[0036] To further improve the quality of the decoded video, the video decoder can perform one or more filtering operations on the reconstructed video blocks. Examples of such filtering operations include deblocking filtering, sample self-adjusting offset (SAO) filtering, and self-adjusting loop filtering (ALF). The parameters used for these filtering operations can be determined by the video encoder and explicitly signaled in the encoded video bitstream, or they can be implicitly determined by the video decoder without needing to be explicitly signaled in the encoded video bitstream.

[0037] Video decoders can use unidirectional prediction or bidirectional prediction (also simply referred to as dual prediction) to predict blocks. In unidirectional prediction, the final predicted block is determined based on the predicted block identified by a single motion vector, while in dual prediction, the final predicted block is determined based on two predicted blocks identified by two motion vectors. This work describes techniques related to inter-frame dual prediction utilizing decoder-side motion vector derivation techniques (e.g., template matching, bilateral matching, decoder-side motion vector refinement).

[0038] As will be described in more detail below, when decoding blocks of video data using inter-frame prediction, the video encoder and video decoder can be configured to decode the blocks in various modes, such as merge mode and Advanced Motion Vector Predictor (AMVP) mode. In both merge mode and AMVP mode, the video encoder and video decoder are configured to generate a list of candidate motion vectors (e.g., a candidate list for a motion vector predictor). In merge mode, each candidate in the list includes motion information for predicting the block. The motion information may, for example, include one or more motion vectors and one or more reference image identifications. In AMVP mode, each AMVP candidate may include only the motion vectors that will be used as the motion vector predictor (MVP). A signal can also be used to indicate the motion vector difference (MVD) representing the difference between the MVP and the actual motion vectors.

[0039] AMVP has a high signal management overhead but generally produces good predictions. Combined modes have a relatively low signal management overhead, but on average do not produce prediction quality as good as AMVP. By enabling the video encoder and decoder to identify better motion vectors without an undesirable increase in signal management overhead, the technique described in this paper achieves a better rate-distortion tradeoff compared to existing techniques used to achieve dual prediction.

[0040] According to the technology described in this case, for blocks using dual prediction, one of the two motion vectors used for dual prediction can be determined using explicit signaling, while the other motion vector can be predicted without explicit signaling by using decoder-side motion derivation. The technique of using one explicitly signaled motion vector and one derived motion vector without signaling may be referred to herein as hybrid inter-frame dual prediction or hybrid dual-mode.

[0041] Using a hybrid dual-mode approach, a first motion vector can be derived using either AMVP or merge mode, where the video decoder receives indices in the bitstream for identifying candidates in either the merge or AMVP candidate list. Using the hybrid dual-mode approach, the video decoder can determine a second motion vector from a second candidate list, but instead of receiving explicit syntax for identifying the selected candidate in the bitstream, the video decoder can determine the candidate for the second motion vector based on the minimum bilateral matching error between a first reference block and a second reference block identified by the first motion vector. For each candidate in the second candidate list, the video decoder can locate the second reference block and determine the bilateral matching error between the second reference block and the first reference block. The video decoder can then select a motion vector from the second candidate list pointing to the second reference block as the second motion vector, which produces the minimum or lowest bilateral matching error relative to the first reference block. In some cases, the video decoder can use various decoder-side motion vector derivation techniques to further refine the selected motion vector.

[0042] In this context, bilateral matching error (also known as bilateral matching cost) represents the difference between two reference blocks on opposite sides of the current block in the reference image. Two relatively similar reference blocks typically have lower bilateral matching errors compared to two significantly different reference blocks. The similarity or difference between two reference blocks can be determined based on sampled values ​​from the two reference blocks using various techniques described herein.

[0043] By determining the first motion vector for the current block based on explicitly signaled MVP and MVD, and determining the second motion vector for the current block based on bilateral matching from a second candidate MVP list without explicit signaling, the technique of this invention can advantageously produce an improved rate-distortion tradeoff for dual-prediction blocks. For example, the technique of this invention can reduce the signaling management burden associated with dual prediction while still maintaining high prediction quality.

[0044] As will be explained in more detail below, the techniques described herein achieve a better rate-distortion tradeoff compared to existing techniques for achieving dual prediction, by enabling video encoders and decoders to identify better motion vectors without an undesirable increase in the burden of signal transmission management. The techniques described herein can be applied to any existing video transcoder, such as HEVC (High Efficiency Video Decoder), VVC (Multi-Functional Video Decoder), EVC (Essential Video Decoder), or efficient decoding tools in future video decoding standards.

[0045] Figure 1 is a block diagram illustrating an exemplary video encoding and decoding system 100 capable of performing the techniques described herein. In summary, the techniques described herein relate to decoding (encoding and / or decoding) video data. Typically, video data includes any data used for processing video. Therefore, video data can include raw, unencoded video, encoded video, decoded (e.g., reconstructed) video, and video relay data (e.g., signaling data).

[0046] As shown in Figure 1, in this example, system 100 includes a source device 102 that provides encoded video data to be decoded and displayed by a destination device 116. Specifically, the source device 102 provides the video data to the destination device 116 via computer-readable media 110. The source device 102 and destination device 116 can include any of a wide variety of devices, including desktop computers, notebook computers (i.e., laptops), mobile devices, tablets, set-top boxes, mobile phones such as smartphones, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, broadcast receiver devices, etc. In some cases, the source device 102 and destination device 116 can be configured for wireless communication and therefore can be referred to as wireless communication devices.

[0047] In the example of Figure 1, source device 102 includes a video source 104, memory 106, a video encoder 200, and an output interface 108. Destination device 116 includes an input interface 122, a video decoder 300, memory 120, and a display device 118. According to this invention, the video encoder 200 of source device 102 and the video decoder 300 of destination device 116 can be configured to apply techniques for mixed-frame double prediction. Therefore, source device 102 represents an example of a video encoding device, and destination device 116 represents an example of a video decoding device. In other examples, the source and destination devices may include other elements or arrangements. For example, source device 102 may receive video data from an external video source such as an external camera. Similarly, destination device 116 may interface with an external display device, rather than including an integrated display device.

[0048] The system 100 shown in Figure 1 is merely one example. Typically, any digital video encoding and / or decoding device can perform techniques for mixed-frame double prediction. Source device 102 and destination device 116 are merely examples of such decoding devices, where source device 102 generates decoded video data for transmission to destination device 116. In this context, "decoding" device refers to a device that performs the decoding (e.g., encoding and / or decoding) of data. Therefore, video encoder 200 and video decoder 300 represent examples of decoding devices (specifically, video encoder and video decoder). In some instances, source device 102 and destination device 116 can operate in a substantially symmetrical manner, such that each of source device 102 and destination device 116 includes elements for video encoding and decoding. Therefore, system 100 can support unidirectional or bidirectional video transmission between source device 102 and destination device 116, for example, for video streaming, video replay, video broadcasting, or video telephony.

[0049] Typically, video source 104 represents the source of video data (i.e., raw, unencoded video data) and provides a sequence of images (also referred to as "frames") of the video data to video encoder 200, which encodes the data used for the images. The video source 104 of source device 102 may include video capturing devices, such as a camera, a video archive unit containing previously captured raw video, and / or a video feed interface for receiving video from a video content provider. Alternatively, video source 104 may generate computer graphics-based data as source video, or a combination of live video, archived video, and computer-generated video. In each case, video encoder 200 may encode the captured, pre-captured, or computer-generated video data. Video encoder 200 may rearrange the images from the received order (sometimes referred to as "display order") to a decoding order for decoding. The video encoder 200 can generate a bitstream including encoded video data. Subsequently, the source device 102 can output the encoded video data to a computer-readable medium 110 via the output interface 108 so that it can be received and / or acquired by an input interface 122 of a destination device 116, for example.

[0050] The memory 106 of the source device 102 and the memory 120 of the destination device 116 represent general-purpose memory. In some instances, the memories 106 and 120 may store raw video data, such as raw video from the video source 104 and raw decoded video data from the video decoder 300. Alternatively, the memories 106 and 120 may store software instructions that can be executed by, for example, the video encoder 200 and the video decoder 300. Although in this example the memories 106 and 120 are shown separately from the video encoder 200 and the video decoder 300, it should be understood that the video encoder 200 and the video decoder 300 may also include internal memory for functionally similar or equivalent purposes. Furthermore, the memories 106 and 120 may store, for example, encoded video data output from the video encoder 200 and input to the video decoder 300. In some instances, portions of memory 106 and 120 may be allocated as one or more video buffers, for example, to store raw, decoded, and / or encoded video data.

[0051] Computer-readable media 110 can represent any type of media or device capable of transmitting encoded video data from source device 102 to destination device 116. In one example, computer-readable media 110 represents communication media that enables source device 102 to transmit encoded video data directly to destination device 116 in real time, for example, via a radio frequency network or a computer-based network. Output interface 108 can modulate the transmitted signal including encoded video data according to a communication standard such as a wireless communication protocol, and input interface 122 can demodulate the received transmitted signal according to a communication standard such as a wireless communication protocol. Communication media can include any wireless or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines. Communication media can form part of a packet-based network such as a local area network, a wide area network, or a global network such as the Internet. Communication media can include routers, switches, base stations, or any other devices that may be useful for facilitating communication from source device 102 to destination device 116.

[0052] In some instances, source device 102 can output encoded data from output interface 108 to storage device 112. Similarly, destination device 116 can access encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as hard disks, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data.

[0053] In some instances, source device 102 may output encoded video data to file server 114 or to another intermediate storage device that may store the encoded video data generated by source device 102. Destination device 116 may access the stored video data from file server 114 via streaming or downloading.

[0054] File server 114 can be any type of server device capable of storing encoded video data and transmitting such encoded video data to destination device 116. File server 114 can represent a web server (e.g., for a website), a server configured to provide file transfer protocol services (such as File Transfer Protocol (FTP) or Flute-based file delivery protocol), a Content Delivery Network (CDN) device, a Hypertext Transfer Protocol (HTTP) server, a Multimedia Broadcast Multicast Service (MBMS) or Enhanced MBMS (eMBMS) server, and / or a Network Attached Storage (NAS) device. File server 114 can additionally or alternatively implement one or more HTTP streaming protocols, such as HTTP-based Dynamic Self-Adjusting Streaming (DASH), HTTP Real-Time Streaming (HLS), Real-Time Streaming Protocol (RTSP), HTTP Dynamic Streaming, etc.

[0055] Destination device 116 can access encoded video data from file server 114 via any standard data connection (including an Internet connection). This connection may include a wireless channel (e.g., Wi-Fi connection), a wired connection (e.g., Digital Subscriber Line (DSL), cable modem, etc.), or a combination of both, suitable for accessing encoded video data stored on file server 114. Input interface 122 can be configured to operate according to any one or more of the various protocols discussed above for obtaining or receiving media data from file server 114, or other such protocols for obtaining media data.

[0056] Output interface 108 and input interface 122 can represent a wireless transmitter / receiver, a modem, a wired networking element (e.g., an Ethernet card), a wireless communication element operating according to any of the various IEEE 802.11 standards, or other physical elements. In instances where output interface 108 and input interface 122 include wireless elements, output interface 108 and input interface 122 can be configured to transmit data (such as encoded video data) according to cellular communication standards (such as 4G, 4G-LTE (Long Term Evolution), improved LTE, 5G, etc.). In some instances where output interface 108 includes a wireless transmitter, output interface 108 and input interface 122 can be configured to transmit data (such as encoded video data) according to other wireless standards (such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee™), the Bluetooth™ standard, etc.). In some instances, source device 102 and / or destination device 116 may include corresponding system-on-chip (SoC) elements. For example, source device 102 may include SoC elements for performing functions assigned to video encoder 200 and / or output interface 108, and destination device 116 may include SoC elements for performing functions assigned to video decoder 300 and / or input interface 122.

[0057] The technology described in this case can be applied to video decoding to support any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, internet streaming video transmission (such as HTTP-based Dynamic Self-Adjusting Streaming (DASH)), digital video encoded onto data storage media, decoding digital video stored on data storage media, or other applications.

[0058] The input interface 122 of the destination device 116 receives an encoded video bitstream from computer-readable media 110 (e.g., communication media, storage device 112, file server 114, etc.). The encoded video bitstream may include signal transmission information defined by the video encoder 200 and used by the video decoder 300, such as syntax elements having values ​​describing the characteristics and / or processing of video blocks or other decoding units (e.g., slices, pictures, picture groups, sequences, etc.). The display device 118 displays a decoded image of the decoded video data to the user. The display device 118 may represent any of various display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0059] Although not illustrated in Figure 1, in some instances, the video encoder 200 and the video decoder 300 may each be integrated with the audio encoder and / or audio decoder, and may include appropriate MUX-DEMUX units or other hardware and / or software to process multiplexed streams including both audio and video in a shared data stream.

[0060] The video encoder 200 and video decoder 300 can each be implemented as any of a variety of suitable encoder and / or decoder circuit systems, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), individual logic, software, hardware, firmware, or any combination thereof. When such techniques are partially implemented in software, the device can store instructions for the software in a suitable non-transitory computer-readable medium and use one or more processors, or hardware, to execute the instructions to perform the techniques described herein. Each of the video encoder 200 and video decoder 300 can be included in one or more encoders or decoders, and any encoder or decoder can be integrated as part of a combined encoder / decoder (CODEC) in the respective device. Devices including the video encoder 200 and / or video decoder 300 can include integrated circuits, microprocessors, and / or wireless communication devices (such as cellular phones).

[0061] The video encoder 200 and video decoder 300 may operate according to video decoding standards such as ITU-T H.265 (also known as the High Efficiency Video Decoding (HEVC) standard) or extensions thereof such as multiview and / or scalable video decoding extensions. Alternatively, the video encoder 200 and video decoder 300 may operate according to other proprietary or industry standards such as ITU-T H.266 (also known as Universal Video Decoding (VVC)). In other instances, the video encoder 200 and video decoder 300 may operate according to proprietary video transcoders / formats such as AOMedia Video 1 (AV1), extensions to AV1, and / or subsequent versions of AV1 (e.g., AV2)). In other instances, the video encoder 200 and video decoder 300 may operate according to other proprietary formats or industry standards. However, the technology described herein is not limited to any particular decoding standard or format. Typically, the video encoder 200 and the video decoder 300 can be configured to combine any video decoding technique that uses inter-frame double prediction to perform the content of this case.

[0062] Typically, the video encoder 200 and video decoder 300 can perform block-based decoding of an image. The term "block" generally refers to a structure comprising data to be processed (e.g., encoded, decoded, or otherwise used in encoding and / or decoding processes). For example, a block may include a two-dimensional matrix of samples of luminance and / or chrominance data. Typically, the video encoder 200 and video decoder 300 can decode video data represented in YUV (e.g., Y, Cb, Cr) format. That is, instead of decoding the sampled red, green, and blue (RGB) data used for the image, the video encoder 200 and video decoder 300 can decode both luminance and chrominance components, where the chrominance components may include both red hue and blue hue chrominance components. In some instances, the video encoder 200 converts the received RGB-formatted data to a YUV representation before encoding, and the video decoder 300 converts the YUV representation back to RGB format. Alternatively, the preprocessing unit and postprocessing unit (not shown) can perform these transformations.

[0063] In summary, the content of this case may relate to the decoding (e.g., encoding and decoding) of images to include procedures for encoding or decoding image data. Similarly, the content of this case may relate to the decoding of blocks of an image to include procedures for encoding or decoding data used for the blocks (e.g., prediction and / or residual decoding). Encoded video bitstreams typically include a series of values ​​for syntax elements representing decoding decisions (e.g., decoding modes) and segmenting the image into blocks. Therefore, references to decoding images or blocks should generally be understood as decoding the values ​​of syntax elements used to form images or blocks.

[0064] HEVC defines various blocks, including decoding units (CUs), prediction units (PUs), and transform units (TUs). According to HEVC, a video decoder (such as a video encoder 200) partitions the decoding tree unit (CTU) into CUs using a quadtree structure. That is, the video decoder partitions the CTU and CU into four equal, non-overlapping squares, and each node of the quadtree has zero or four child nodes. Nodes without child nodes can be called "leaf nodes," and the CU of such leaf nodes can include one or more PUs and / or one or more TUs. The video decoder can further partition the PUs and TUs. For example, in HEVC, a residual quadtree (RQT) represents a partition of the TU. In HEVC, the PU represents inter-frame prediction data, while the TU represents residual data. CUs predicted within a frame include intra-frame prediction information, such as intra-frame pattern indications.

[0065] As another example, the video encoder 200 and video decoder 300 can be configured to operate according to VVC. According to VVC, the video decoder (such as the video encoder 200) segments the image into a plurality of decoder tree units (CTUs). The video encoder 200 can segment the CTUs according to a tree structure (such as a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure). The QTBT structure eliminates the concept of multiple segmentation types, such as the separation between CUs, PUs, and TUs in HEVC. The QTBT structure includes two levels: a first level segmented according to quadtree segmentation, and a second level segmented according to binary tree segmentation. The root node of the QTBT structure corresponds to the CTU. The leaf nodes of the binary tree correspond to the decoder units (CUs).

[0066] In the MTT partitioning structure, blocks can be partitioned using quadtree (QT) partitioning, binary tree (BT) partitioning, and one or more types of ternary tree (TT) partitioning (also known as tripartite tree (TT)) partitioning. A ternary tree or tripartite tree partitioning is a partition in which a block is separated into three sub-blocks. In some instances, a ternary tree or tripartite tree partitioning divides a block into three sub-blocks without partitioning the original block through a center. The partitioning types in MTT (e.g., QT, BT, and TT) can be symmetric or asymmetric.

[0067] When operating according to the AV1 transcoder, the video encoder 200 and video decoder 300 can be configured to decode video data in blocks. In AV1, the largest decoded block that can be processed is called a superblock. In AV1, a superblock can be a 128x128 luminance sample or a 64x64 luminance sample. However, in subsequent video decoding formats (e.g., AV2), the superblock can be defined by different (e.g., larger) luminance sample sizes. In some instances, the superblock is the top level of the block quadtree. The video encoder 200 can further divide the superblock into smaller decoded blocks. The video encoder 200 can use square or non-square partitioning to divide the superblock and other decoded blocks into smaller blocks. Non-square blocks can include N / 2xN, NxN / 2, N / 4xN, and NxN / 4 blocks. The video encoder 200 and the video decoder 300 can perform separate prediction and transformation procedures on each of the decoded blocks.

[0068] AV1 also defines video data tiles. A tile is a rectangular array of superblocks that can be decoded independently of other tiles. That is, the video encoder 200 and video decoder 300 can encode and decode the decoding blocks within a tile separately, without using video data from other tiles. However, the video encoder 200 and video decoder 300 can perform filtering across tile boundaries. The tile size can be uniform or non-uniform. Tile-based decoding enables parallel processing and / or multithreading for both the encoder and decoder.

[0069] In some instances, the video encoder 200 and the video decoder 300 may use a single QTBT or MTT structure to represent each of the luminance and chrominance components, while in other instances, the video encoder 200 and the video decoder 300 may use two or more QTBT or MTT structures, such as one QTBT / MTT structure for the luminance component and another QTBT / MTT structure for the two chrominance components (or two QTBT / MTT structures for the respective chrominance components).

[0070] The video encoder 200 and the video decoder 300 can be configured to use quadtree partitioning, QTBT partitioning, MTT partitioning, superblock partitioning or other partitioning structures.

[0071] In some instances, a CTU includes a decoded tree block (CTB) of luminance samples from an image with three sampling arrays, two corresponding CTBs of chrominance samples, or a CTB of samples from a monochrome image or an image decoded using three separate color planes, and a syntax structure for decoding the samples. A CTB can be an NxN sampling block for some value of N, such that dividing the components into CTBs is a partition. Components can be from an array of three arrays (luminance and two chrominance) comprising an image in 4:2:0, 4:2:2, or 4:4:4 color formats, or a single sample from one of the three arrays, or from an array comprising an image in monochrome format, or a single sample from that array. In some instances, a decoded block is an MxN sampling block for some values ​​of M and N, such that dividing the CTB into decoded blocks is a partition.

[0072] Blocks (e.g., CTUs or CUs) can be categorized in various ways within an image. As an example, a brick can represent a rectangular region of a CTU row within a specific tile in an image. A tile can be a rectangular region of a CTU within a specific tile column and a specific tile row in an image. A tile column represents a rectangular region of a CTU, having a height equal to the height of the image and a width specified by a syntax element (e.g., as in an image parameter set). A tile row represents a rectangular region of a CTU, having a height specified by a syntax element (e.g., as in an image parameter set) and a width equal to the width of the image.

[0073] In some instances, a tile can be divided into multiple bricks, each of which may include one or more CTU rows within the tile. A tile that is not divided into multiple bricks can also be referred to as a brick. However, a brick that is a true subset of a tile may not be referred to as a tile. Bricks in an image can also be arranged as slices. A slice can be an integer number of bricks in the image, which can be uniquely contained within a single Network Abstraction Layer (NAL) unit. In some instances, a slice comprises several complete tiles or a continuous sequence of complete bricks comprising only one tile.

[0074] In this context, "NxN" and "N by N" can be used interchangeably to represent the sampling dimensions of a block (such as a CU or other video block) in the vertical and horizontal dimensions, for example, 16x16 samples or 16 by 16 samples. Typically, a 16x16 CU will have 16 samples in the vertical direction (y=16) and 16 samples in the horizontal direction (x=16). Similarly, an NxN CU typically has N samples in the vertical direction and N samples in the horizontal direction, where N represents a non-negative integer value. Samples in a CU can be arranged in rows and columns. Furthermore, a CU does not necessarily need to have the same number of samples in the horizontal direction as it does in the vertical direction. For example, a CU can include NxM samples, where M is not necessarily equal to N.

[0075] The video encoder 200 encodes video data for representation prediction and / or residual information, as well as other information, used for the CU. Prediction information indicates how the CU will be predicted to form a prediction block for the CU. Residual information typically represents the sample-by-sample difference between the sampled CU before encoding and the prediction block.

[0076] To predict the CU, the video encoder 200 can typically form prediction blocks for the CU via inter-frame prediction or intra-frame prediction. Inter-frame prediction typically represents predicting the CU based on data from a previously decoded image, while intra-frame prediction typically represents predicting the CU based on data from a previously decoded image of the same image. To perform inter-frame prediction, the video encoder 200 can use one or more motion vectors to generate prediction blocks. In this context, motion vectors typically refer to any information used to identify the position of a reference block in a reference image. A motion vector can be represented, for example, as an offset between the upper-left sample position of the current block in the current image and the upper-left sample position of the reference block in the reference image. As explained elsewhere in this description, in some decoding scenarios, the video encoder 200 can signal an initial motion vector used to identify the position of an initial reference block. Subsequently, the video decoder 300 can refine the motion vector by locating a refined reference block that is different from but generally close to the initial reference block.

[0077] The video encoder 200 typically performs motion search to identify, for example, a reference block that closely matches the CU in terms of the difference pattern between the CU and a reference block. The video encoder 200 can use the following to calculate difference metrics to determine whether the reference block closely matches the current CU: Sum of Absolute Differences (SAD), Sum of Squared Differences (SSD), Mean Absolute Difference (MAD), Mean Squared Difference (MSD), or other such difference calculations. In some instances, the video encoder 200 can use unidirectional or bidirectional prediction to predict the current CU.

[0078] Some instances of VVC also offer an affine motion compensation mode, which can be considered an inter-frame prediction mode. In affine motion compensation mode, the video encoder 200 can determine two or more motion vectors to represent non-translational motion (such as zooming in or out, rotation, perspective motion, or other irregular types of motion).

[0079] To perform intra-frame prediction, the video encoder 200 can select an intra-frame prediction mode to generate prediction blocks. Some instances of VVC provide sixty-seven intra-frame prediction modes, including various directional modes, as well as planar and DC modes. Typically, the video encoder 200 selects an intra-frame prediction mode that describes the adjacent samples of the current block (e.g., a block of the CU) to be predicted. Assuming the video encoder 200 decodes the CTU and CU in raster scan order (from left to right, from top to bottom), this sampling can typically be above, to the upper left, or to the left of the current block within the same image.

[0080] The video encoder 200 encodes data representing the prediction mode used for the current block. For example, for inter-frame prediction modes, the video encoder 200 can encode data indicating which of the various available inter-frame prediction modes is used, and motion information for the corresponding mode. For unidirectional or bidirectional inter-frame prediction, for example, the video encoder 200 can use AMVP or merging modes to encode motion vectors. The video encoder 200 can use similar modes to encode motion vectors used for affine motion compensation modes.

[0081] AV1 includes two common techniques for encoding and decoding blocks of video data. These two common techniques are intra-frame prediction (e.g., intra-frame prediction or spatial prediction) and inter-frame prediction (e.g., inter-frame prediction or temporal prediction). In the context of AV1, when using intra-frame prediction modes to predict blocks of the current frame of video data, the video encoder 200 and video decoder 300 do not use video data from other frames of the video data. For most intra-frame prediction modes, the video encoder 200 encodes the blocks of the current frame based on the difference between the sampled values ​​in the current block and the predicted values ​​generated from reference samples in the same frame. The video encoder 200 determines the predicted values ​​generated from the reference samples based on the intra-frame prediction mode.

[0082] After prediction, such as intra-frame prediction or inter-frame prediction for a block, the video encoder 200 can calculate residual data for that block. The residual data (such as a residual block) represents the sample-by-sample difference between the block and the prediction block used to form that block using the corresponding prediction pattern. The video encoder 200 can apply one or more transforms to the residual block to produce transformed data in the transform domain rather than the sampling domain. For example, the video encoder 200 can apply Discrete Cosine Transform (DCT), integer transform, wavelet transform, or conceptually similar transforms to the residual video data. Additionally, the video encoder 200 can apply a second transform after the first transform, such as Mode Dependent Inseparable Quadratic Transform (MDNSST), Signal Dependent Transform, Karhunen-Loeve Transform (KLT), etc. The video encoder 200 generates transform coefficients after applying one or more transforms.

[0083] As mentioned earlier, after any transformation to produce transform coefficients, the video encoder 200 can perform quantization on the transform coefficients. Quantization typically represents a procedure in which the transform coefficients are quantized to potentially reduce the amount of data used to represent the transform coefficients, thereby providing further compression. By performing the quantization procedure, the video encoder 200 can reduce the bit depth associated with some or all of the transform coefficients. For example, the video encoder 200 can round down an n-bit value to an m-bit value during quantization, where n is greater than m. In some instances, to perform quantization, the video encoder 200 can perform a bitwise right shift of the value to be quantized.

[0084] After quantization, the video encoder 200 can scan the transform coefficients to generate a one-dimensional vector from a two-dimensional matrix including the quantized transform coefficients. The scan can be designed to place higher-energy (and therefore lower-frequency) transform coefficients before the vector and lower-energy (and therefore higher-frequency) transform coefficients after the vector. In some instances, the video encoder 200 can use a predefined scan order to scan the quantized transform coefficients to produce a serialized vector, and then entropy encode the quantized transform coefficients of the vector. In other instances, the video encoder 200 can perform a self-adjusting scan. After scanning the quantized transform coefficients to form a one-dimensional vector, the video encoder 200 can entropy encode the one-dimensional vector, for example, according to context-adjusted binary arithmetic decoding (CABAC). The video encoder 200 can also entropy encode the values ​​of syntax elements used to describe relay data associated with the encoded video data for use by the video decoder 300 when decoding the video data.

[0085] To perform CABAC, the video encoder 200 can assign context within a context model to the symbols to be transmitted. The context can be related to, for example, whether the symbol's neighboring values ​​are zero. Probabilistic decisions can be based on the context assigned to the symbols.

[0086] The video encoder 200 can also generate syntax data (such as block-based syntax data, image-based syntax data, and sequence-based syntax data) or other syntax data (such as sequence parameter sets (SPS), image parameter sets (PPS), or video parameter sets (VPS)) for the video decoder 300, for example, from picture headers (PH), block headers, slice headers (SH). Similarly, the video decoder 300 can decode such syntax data to determine how to decode the corresponding video data.

[0087] In this manner, the video encoder 200 can generate a bitstream that includes encoded video data, such as syntax elements describing the segmentation of an image into blocks (e.g., CUs) and prediction and / or residual information for those blocks. Finally, the video decoder 300 can receive the bitstream and decode the encoded video data.

[0088] Typically, the video decoder 300 executes a procedure opposite to that executed by the video encoder 200 to decode the encoded video data of the bitstream. For example, the video decoder 300 may use CABAC to decode the values ​​of syntax elements for the bitstream in a manner substantially similar to, but opposite to, the CABAC encoding procedure of the video encoder 200. Syntax elements may define segmentation information for segmenting an image into CTUs and for segmenting each CTU according to a corresponding segmentation structure (such as a QTBT structure) to define the CUs of the CTU. Syntax elements may also define prediction and residual information for blocks (e.g., CUs) of the video data.

[0089] The residual information can be represented, for example, by quantized transform coefficients. The video decoder 300 can inversely quantize and inversely transform the quantized transform coefficients of a block to reproduce the residual block used for that block. The video decoder 300 uses a signal-informed prediction pattern (intra-frame prediction or inter-frame prediction) and associated prediction information (e.g., motion information for inter-frame prediction) to form a prediction block for that block. The video decoder 300 can then combine the prediction block and the residual block (on a sample-by-sample basis) to reproduce the original block. The video decoder 300 can perform additional processing, such as performing a deblocking procedure to reduce visual artifacts along the block boundaries.

[0090] In summary, this case may involve "signaling" certain information (such as syntax elements). The term "signaling" can generally refer to the transmission of values ​​for syntax elements and / or other data for decoding encoded video data. That is, the video encoder 200 can signal values ​​for syntax elements in the bitstream. Typically, signaling represents values ​​generated in the bitstream. As mentioned above, the source device 102 can transmit the bitstream to the destination device 116 substantially instantaneously or not instantaneously (such as when syntax elements are stored in the storage device 112 for later retrieval by the destination device 116).

[0091] In VVC, the largest decoding unit in a slice is called a Decoding Tree Block (CTB) or Decoding Tree Unit (CTU). A CTB contains a quadtree, binary tree, or ternary tree, whose nodes are decoding units. In the VVC main introduction, the size of a CTB can range from 4x4 to 128x128. A Decoding Unit (CU) can be the same size as the CTB or as small as 4x4. Each decoding unit is decoded using a mode (e.g., inter-frame or intra-frame). A CU decoded inter-frame (e.g., the current block decoded in inter-frame mode) can be further subdivided into 2, 3, or 4 Prediction Units (PUs), or into a single PU when further subdivision is not applicable. When a CU contains two PUs, the two PUs are evenly divided into rectangles of half the size. When a CU contains three PUs, the three rectangles have sizes of ¼, ½, and ¼ of the CU. When performing inter-frame decoding on the CU, each PU has a set of motion information derived using a unique inter-frame prediction pattern.

[0092] The video encoder 200 and video decoder 300 can be configured to perform motion vector prediction. In the VVC standard, for the prediction unit (PU), there are two inter-frame prediction modes, referred to as merging mode (skip mode is considered a special case of merging mode) and AMVP mode.

[0093] In AMVP mode, a motion vector candidate list (e.g., a candidate list for the motion vector predictor) is maintained for multiple MVPs. The motion vector for the current PU and a reference index in AMVP mode are generated by selecting a candidate from the motion vector candidate list. For inter-frame pattern prediction, the video encoder 200 and video decoder 300 can also generate two reference image lists (referred to as list 0 and list 1), from which one or more blocks are selected for inter-frame prediction. The motion vector candidate list contains at most two candidates for AMVP mode. In AMVP mode, since the AMVP candidates only contain motion vectors, for each potential prediction direction, the reference index is signaled along with the MVP index of the motion vector candidate list (e.g., the candidate list for the motion vector predictor). In AMVP mode, the predicted motion vectors can be further refined. Candidates for AMVP mode are derived from spatially and temporally adjacent blocks or from historical motion information tables.

[0094] Figure 2A is a conceptual diagram illustrating an example of spatially adjacent motion vector candidates for merging mode. The video decoder 300 generates a candidate list by adding motion information of spatially adjacent candidates to the candidate list. For a specific PU (PU0), spatial motion vector candidates are derived from the adjacent blocks shown in Figures 2A and 2B, but the procedure used to generate candidates from the blocks may differ for merging mode and AMVP mode. In merging mode, up to five spatial motion vector candidates can be derived for block 140 (PU0) in the order shown in Figure 2A. The video encoder 200 and video decoder 300 can be configured to use the following order: left (0), top (1), top right (2), bottom left (3), and top left (4), as shown in Figure 2A.

[0095] Figure 2B is a conceptual diagram illustrating an example of spatially adjacent motion vector candidates for AMVP. In AMVP mode, the adjacent blocks of block 142 (PU0) are divided into two groups: the left group including blocks 0 and 1, and the upper group including blocks 2, 3, and 4, as shown in Figure 2B. For each group, potential candidates in the adjacent blocks that reference the same reference image indicated by the signaled reference are prioritized to form the final candidates for that group. It is possible that none of the adjacent blocks contain motion vectors pointing to the same reference image. Therefore, if such a candidate cannot be found, the first available candidate can be scaled to form the final candidate, thereby allowing compensation for temporal distance differences.

[0096] The following describes temporal motion vector prediction. The video encoder 200 and video decoder 300 can be configured to add temporal motion vector predictor (TMVP) candidates (if enabled and available) to the motion vector candidate list after adding spatial motion vector candidates. The procedure for motion vector derivation for TMVP candidates may be the same for both merged mode and AMVP mode. However, in some instances, the target reference index for TMVP candidates in merged mode can be set to 0.

[0097] Figure 3A is a conceptual diagram illustrating an example of a TMVP candidate for block 144 (PU0). The main block position used for TMVP candidate derivation is the lower right block outside the co-located PU (shown as block "T" in Figure 3A) to compensate for the bias of the upper and left blocks used to generate spatially adjacent candidates. However, if this block is outside the current CTB line or motion information is unavailable, the center block of the PU is used to replace the block.

[0098] The video decoder 300 can derive motion vectors for TMVP candidates from the co-location PUs of the co-location images indicated at the slice level. The motion vectors used for the co-location PUs are called co-location MVs. If a block in the reference image and the current block each include at least one primitive corresponding to the same relative position in the reference image and the current image, then the block in the reference image can be considered, for example, as co-located with a block in the current image.

[0099] Figure 3B is a conceptual timing diagram illustrating an example of the motion vector scaling procedure 146. Similar to the direct temporal mode in AVC, to derive TMVP candidate motion vectors, the video decoder 300 can scale co-located motion vectors to compensate for temporal distance differences, as shown in Figure 3B. Motion vector scaling typically assumes that the value of the motion vector is proportional to the distance between the images in the rendering time. The motion vector associates two images (a reference image and an image containing the motion vector, i.e., the containing image). When using a motion vector to predict another motion vector, the distance between the containing image and the reference image is calculated based on the Picture Order Count (POC) value.

[0100] For a motion vector being predicted, the associated containing image and reference image may differ. Therefore, a new distance based on the Proof-of-Concept (POC) can be calculated, and the motion vector can be scaled based on the distances between the two POCs. For spatially adjacent candidates, the containing images used for two motion vectors are the same, while the reference images are different. In HEVC, motion vector scaling is applied to both TMVP and AMVP for spatially and temporally adjacent candidates.

[0101] The video encoder 200 and video decoder 300 can be configured to perform artificial motion vector candidate generation. If the motion vector candidate list is incomplete (e.g., less than a predetermined number of candidates), artificial motion vector candidates are generated and inserted at the end of the list until the list has a specified number of candidates.

[0102] In the merged mode, there are two types of artificial motion vector candidates: combined candidates derived only for B-slices, and zero motion vector candidates for AMVP when the combined candidates derived only for B-slices do not provide enough artificial candidates.

[0103] For each pair of candidates that are already in the candidate list and have the necessary motion information, a bidirectional combined motion vector candidate is derived by combining the motion vector of the first candidate in the image reference list 0 and the motion vector of the second candidate in the image reference list 1.

[0104] The video encoder 200 and video decoder 300 can be configured to perform a pruning procedure for candidate insertion. Motion vector candidates from different blocks may happen to be identical, which reduces the efficiency of merging / AMVP candidate lists. That is, two different motion vector candidates may point to two different blocks, but the values ​​of the two different motion vector candidates may be the same, resulting in redundant candidates. A pruning procedure is applied to remove redundant candidates. The pruning procedure compares one candidate in the current candidate list with other candidates to avoid inserting the same candidate in some instances. To reduce complexity, only a limited number of pruning procedures are applied instead of comparing every potential candidate with all other existing candidates.

[0105] The video encoder 200 and video decoder 300 can be configured to perform bilateral matching prediction. Bilateral matching (also known as bilateral merging) prediction is a merging pattern based on frame rate upconversion (FRUC) technology. When the video encoder 200 and video decoder 300 apply the BM pattern to a block, two initial motion vectors MV0 and MV1 are derived by selecting merging candidates from a constructed merging list using a signaled merging candidate index. Subsequently, the video encoder 200 and video decoder 300 perform a bilateral matching search around the initial block located by MV0 and MV1. The final MV0' and MV1' are derived based on the minimum bilateral matching error.

[0106] The MVD0 (denoted by MV0'–MV0) and MVD1 (denoted by MV1'–MV1) pointing to two reference blocks are proportional to the temporal distance (TD) between the current image and the two reference images (e.g., TD0 and TD1). Figure 4 illustrates an example of MVD0 and MVD1, where TD1 is four times TD0.

[0107] However, there is an alternative design where MVD0 and MVD1 are mirror images regardless of the time distance between TD0 and TD1. Figure 5 illustrates an example of mirrored MVD0 and MVD1, where TD1 is four times the length of TD0.

[0108] Bilateral matching performs a local search around the initial MV0 and MV1 to derive the final MV0' and MV1'. The local search applies a 3×3 square search pattern to iterate through the search range [-8, 8]. In each search iteration, the bilateral matching error of the eight surrounding motion vectors in the search pattern is calculated and compared with the bilateral matching error of the center MV. The motion vector with the smallest bilateral matching error becomes the new center motion vector in the next search iteration. The local search terminates when the current center motion vector has the minimum cost within the 3×3 square search pattern, or when the local search reaches a predefined maximum number of search iterations. Figure 6 illustrates an example of a 3×3 square search pattern in the search range [-8, 8].

[0109] The video encoder 200 and video decoder 300 can be configured to perform template matching (TM) prediction. TM prediction is a special merging mode based on FRUC technology. In TM prediction mode, motion information for blocks is not signaled but derived on the decoder side. TM prediction can be applied to both AMVP mode and general merging mode. In AMVP mode, MVP candidate selection is determined based on template matching technology used to identify candidates that result in the minimum difference between the current block template and the reference block template. In general merging mode, a TM mode flag can be signaled to indicate the use of TM, and TM can then be applied to the merging candidates indicated by the merging index for motion vector refinement.

[0110] Figure 7 illustrates an exemplary template matching procedure performed over a search area surrounding the initial MV. As shown in Figure 7, the video encoder 200 and video decoder 300 can be configured to use template matching to derive motion information of the current CU by finding the closest match between the current template 180 (the top and / or left adjacent block of the current CU) in the current image 182 and a template within a reference template 184 used for a reference block (of the same size as the template) in the reference image 186. Using AMVP candidates selected based on the initial matching error, the video encoder 200 and video decoder 300 can refine the MVP using template matching. Using merge candidates indicated by a merge index signaled by a merging indicator, the merged motion vectors corresponding to L0 and L1 can be independently refined via template matching. Subsequently, the less precise merged motion vectors can be further refined based on the more precise merged MV.

[0111] The video encoder 200 and video decoder 300 can be configured to implement the cost function. Motion-compensated interpolation is required when the motion vector points to a fractional sampling location. To reduce complexity, bilinear interpolation can be used instead of general 8-point junction DCT-IF interpolation for template matching to generate a template in the reference image. An exemplary matching cost C for template matching can be calculated as follows: in It is a weighting factor, which is empirically set to 4. and These indicate the motion vector for the current test and the initial motion vector (e.g., the MVP candidate in AMVP mode or the merged motion vector in merge mode). The sum of absolute differences (SAD) can be used as the matching cost for template matching.

[0112] When using TM, the video encoder 200 and video decoder 300 can be configured to refine the motion vector (e.g., the initial MV) using only luma samples. However, the motion vector based on the refinement decision can be used for both luma and chroma predictions in MC (motion compensation) inter-frame prediction. After determining the motion vector, the final MC can be performed using an 8-pin interpolation filter for luma and a 4-pin interpolation filter for chroma.

[0113] The video encoder 200 and video decoder 300 can be configured to implement a search procedure. Motion vector refinement can include a pattern-based motion vector search procedure with a template matching cost criterion and utilizing a hierarchy. Two search modes are supported for motion vector refinement: diamond search and cross search. The hierarchy specifies the iterative procedure for refining the MV, starting with a coarse MVD precision (e.g., quarter-pole (pel)) and ending with a finer precision (e.g., 1 / 8-pole). For example, quarter-pole motion vector precision implies performing a template matching procedure over a search area around the initial motion vector to identify the refined motion vector, where the search step size uses a quarter of the luminance sampling distance (or resolution) as the MVD precision (between the initial motion vector and the refined MV). The motion vector is searched directly with a quarter-pole luminance sampling MVD precision using the diamond pattern, followed by a search with a quarter-pole luminance sampling MVD precision using the cross pattern, and then further refined with an eighth-pole luminance sampling MVD precision using the cross pattern. The search range for motion vector refinement is set to be equal to (-8, +8) luminance samples around the initial MV. When the current block has two predictions, the two motion vectors are refined independently, and the best motion vector (in terms of matching cost) is then set as the prior to further refine other motion vectors using the BCW weight value.

[0114] The video encoder 200 and video decoder 300 can be configured to perform decoder-side motion vector refinement (DMVR). In VVC, DMVR can be applied to improve the accuracy of motion vectors in merged modes. In the dual prediction operation, refined motion vectors are searched around the initial motion vectors in reference image lists L0 and L1. The DMVR procedure calculates the distortion between two candidate blocks in reference image lists L0 and L1.

[0115] Figure 8 illustrates an example of motion vector refinement on the decoder side. As shown in Figure 8, the video encoder 200 and the video decoder 300 can be configured to compute the SAD between blocks 188 and 190 based on each motion vector candidate around the initial MV. Block 188 can be from a reference image in reference image list L0, and block 190 can be from a reference image in reference image list L1. The motion vector candidate with the lowest SAD becomes the refined motion vector and is used to generate the dual prediction signal. In the example of Figure 8, MV0 and MV1 represent the initial motion vectors, and MV0' and MV1' represent the refined motion vectors that have been refined by MV diff and -MV diff, respectively.

[0116] The refined motion vectors derived by the DMVR program are used to generate inter-frame prediction samples and also for temporal motion vector prediction for future image decoding. The original motion vectors are used in the deblocking process and also for spatial motion vector prediction for future CU decoding. DMVR is based on a sub-block merging mode with a predefined maximum processing unit of 16x16 luminance samples. When the width and / or height of a CU is greater than 16 luminance samples, the CU can be further divided into sub-blocks with a width and / or height equal to 16 luminance samples.

[0117] The video encoder 200 and video decoder 300 can be configured to implement a search scheme. In the DMVR, the search point revolves around an initial motion vector, and the motion vector offset is constrained by the MVD mirroring rule. In other words, any point checked by the DMVR represented by the candidate motion vector pair (MV0, MV1) follows the following two equations: in This represents the thinning offset between the initial motion vector and the thinned motion vector in one of the reference images. In DMVR, the thinning search range is two integer luminance samples from the initial MV. The search includes an integer sample offset search phase and a fractional sample thinning phase.

[0118] The video encoder 200 and video decoder 300 can be configured to perform a 25-point full search for integer sample offset search. First, the SAD of the initial motion vector pair is calculated. If the SAD of the initial motion vector pair is less than a threshold, the integer sample offset search phase of the DMVR is terminated. Otherwise, the SAD of the remaining 24 points is calculated and checked in raster scan order. The point with the minimum SAD is selected as the output of the integer sample offset search phase. To reduce the penalty for uncertainty in DMVR refinement, the original motion vectors can be selected for use during the DMVR procedure. The SAD between reference blocks referenced via the initial motion vector candidates is reduced by 1 / 4 of the SAD value.

[0119] Following the integer sample search, the video encoder 200 and video decoder 300 can be configured to perform fractional sample refinement. To save computational complexity, fractional sample refinement is derived by using the parametric error surface equation instead of performing an additional search using SAD comparison. Fractional sample refinement is conditionally triggered based on the output of the integer sample search stage. Fractional sample refinement is further applied when the integer sample search stage terminates with the minimum SAD at the center in either the first or second iterative search.

[0120] In subprime offset estimation based on parametric error surfaces, the cost at the center location and the cost at four adjacent locations from the center are used to fit a two-dimensional parabolic error surface equation of the following form: in( This corresponds to the fractional position with the minimum cost, and C corresponds to the minimum cost value. The above equation is calculated by solving it using the cost values ​​of the five search points. for: and The value is automatically constrained between -8 and 8 because all cost values ​​are positive and the minimum is E(0,0). This case corresponds to a half-priority offset of a 1 / 16-precision primitive motion vector in VVC. The calculated fraction ( Add to the integer distance to refine the motion vector to obtain the refinement increment MV of the sub-primitive precision.

[0121] The video encoder 200 and video decoder 300 can be configured to perform bilinear interpolation and sample padding. In VVC, the maximum resolution of the motion vector is 1 / 16 luminance sample. An 8-point interpolation filter is used to interpolate the samples at fractional positions. In DMVR, the search point surrounds an initial fractional primitive motion vector with an integer sample offset. Therefore, the samples at these fractional positions need to be interpolated for the DMVR search procedure. To reduce computational complexity, a bilinear interpolation filter is used to generate fractional samples for the search procedure in DMVR. In some instances, by using a bilinear filter with a 2-sample search range, DMVR does not access more reference samples compared to a normal motion compensation procedure. After obtaining the refined motion vector using the DMVR search procedure, a normal 8-point interpolation filter is applied to produce the final prediction. In order to avoid accessing more reference samples than normal MC procedures, the video encoder 200 and video decoder 300 can be configured to fill in samples that are not needed for interpolation procedures based on raw motion vectors but are needed for interpolation procedures based on refined MV.

[0122] The video encoder 200 and video decoder 300 can be configured to implement one or more enable conditions for DMVR. In one instance, the video encoder 200 and video decoder 300 can be configured to enable DMVR when all of the following conditions are met (e.g., present or true). • CU-level merging mode with dual predictive MV • Relative to the current image, one reference image is in the past, and another reference image is in the future. • The distances (e.g., POC difference) from the two reference images to the current image are the same. • The CU has more than 64 luminance samples. Both the CU height and CU width must be greater than or equal to 8 luminance samples. ·BCW weight index indicates equal weights ·WP not enabled for the current block • Combined intra-frame and inter-frame prediction (CIIP) modes are not used in the current block.

[0123] Existing techniques may present some potential problems. When using inter-frame dual prediction to predict AMVP mode decoded blocks, the final motion vector for each prediction direction (e.g., reference lists 0 and 1) is derived by applying MVD to the corresponding MVP, where the MVP is derived using a signal-notified index to select candidates from the AMVP candidate list, and the MVD is derived via motion estimation and signal-notified in the bitstream. Signal-notified MVD improves the accuracy of the final motion vector and the quality of the predictor at the expense of compression efficiency.

[0124] When the merge mode decoding block uses inter-frame dual prediction, the final motion vector for each prediction direction is derived from the MVP or refined MVP. An index is used to signal the MVP to select a candidate from the merge candidate list, and bilateral matching or template matching techniques are used to derive the refined MVP. Compared to the AMVP mode, the merge mode does not require signaling the MVD value because the motion field inferred from spatially or temporally adjacent merge candidates typically provides sufficient accuracy to achieve acceptable compression efficiency. However, for blocks with complex motion, the MVP candidates of the merge mode may not always be accurate. This paper introduces a hybrid dual-prediction inter-frame mode that achieves a better trade-off between motion accuracy and signaling management burden compared to existing dual-prediction inter-frame modes.

[0125] In a first instance, the video encoder 200 and video decoder 300 can be configured to perform hybrid inter-frame dual prediction (e.g., hybrid dual-mode) using bilateral matching. Using hybrid dual-mode, the video encoder 200 and video decoder 300 can be configured to derive a first motion vector by applying the MVD to the MVP. That is, the first motion vector equals the MVD plus the MVP. The video encoder 200 can derive the MVD via motion estimation and transmit the MVP and MVD to the video decoder 300 via signaling in the bitstream. In some instances, the video encoder 200 can signal the MVP as an index for selecting candidates from a first candidate list (e.g., a first candidate list of the motion vector predictor). As mentioned above, the first candidate list can be constructed as an AMVP candidate list or a merged candidate list, or it can be constructed in another manner.

[0126] Using a hybrid dual-mode approach, the video encoder 200 and video decoder 300 can be configured to derive a second motion vector by selecting an MVP from a second candidate list (e.g., a second candidate list of a motion vector predictor). The second candidate list may differ from the first candidate list. The video encoder 200 and video decoder 300 can be configured to determine the MVP selection based on the minimum bilateral matching error between a first reference block and a second reference block, where the first reference block is derived using the derived first motion vector, and the second reference block (for each MVP candidate in the second candidate list) is derived using MVP candidates from the second candidate list. The second candidate list can be constructed as an AMVP candidate list, a merged candidate list, or using some other construction. The MVP selection of the second predictor is further refined to obtain the minimum bilateral matching error between the first and second reference blocks.

[0127] In other words, the video encoder 200 and the video decoder 300 can be configured to derive a first motion vector by decoding the bitstream, and to derive a second motion vector using bilateral matching selection and refinement based on the candidate list in the absence of explicit signal transmission in the bitstream.

[0128] As mentioned above, as part of performing hybrid inter-frame prediction using bilateral matching, the video encoder 200 and video decoder 300 can be configured to derive a first predictor from an AMVP candidate list. A first motion vector for dual prediction can be determined from the MVD and MVP, where the MVP is the first predictor. The video encoder 200 and video decoder 300 can be configured to derive the MVP from a candidate list constructed as an AMVP candidate list. For example, the video decoder 300 can be configured to derive the first predictor as follows: 1) Receive the first flag (HybridBiFlag) in the bit stream to indicate whether the current block is decoded into a hybrid inter-frame double prediction block. 2) When HybridBiFlag equals 1 (indicating that the current block is decoded as a hybrid inter-frame double prediction block), the second flag (FirstPredictonReferenceList0Flag) is received in the bit stream, which indicates whether to signal MVD for reference list 0 or reference list 1. 3) Receive the index (FirstPredictorRefIdx) in the bit stream, which indicates which reference picture in the determined reference list (FirstPredictorRefList) is used to generate the first predictor. 4) Use the determined FirstPredictorRefList and FirstPredictorRefIdx to construct the AMVP candidate list. 5) Receive the index (FirstPredictorMvpIdx) in the bit stream to indicate which MVP (FirstPredictorMvp) in the AMVP candidate list will be used to generate the first predictor. 6) Receive MVD (FirstPredictorMvd) in the bit stream. 7) The final motion vector (FirstPredictorMv) of the first predictor is derived by applying (e.g., adding) FirstPredictorMvd to FirstPredictorMvp.

[0129] As described above, as part of performing hybrid inter-frame prediction using bilateral matching, the video encoder 200 and video decoder 300 can be configured to derive a second predictor from a merge candidate list. The video encoder 200 and video decoder 300 can use bilateral matching to derive a second motion vector to select the MVP from the merge candidate list based on the minimum bilateral matching error between the first and second reference blocks. The video encoder 200 and video decoder 300 can further refine the MVP to determine the minimum bilateral matching error between the first and second reference blocks. The video encoder 200 and video decoder 300 can be configured to use the first motion vector derived in the manner described above to determine the first reference block.

[0130] The video decoder 300 can be configured to derive the second final motion vector as follows: 1) Use FirstPredictorMvp, FirstPredictorRefList, and FirstPredictorRefIdx as described above to determine the first reference block (FirstRefBlock). 2) Construct candidate lists, such as merging candidate lists. 3) The reference list (SecondPredictorRefList) used for the second predictor is determined as SecondPredictorRefList = 1 - FirstPredictorRefList. That is, if L0 is used to determine the first predictor, then L1 is used to derive the second predictor, and vice versa. 4) The bilateral matching cost (costBM[i]) for merging each MVP candidate in the candidate list is derived as follows: a) For MVP candidate i (SecondPredictorMvpCand[i]) in the merge candidate list, determine whether SecondPredictorMvpCand[i] can be used as a second predictor for the current block based on a condition. The condition could be, for example: SecondPredictorMvpCand[i] has a predictor at SecondPredictorRefList such that the reference image index of SecondPredictorMvpCand[i] at SecondPredictorRefList is not equal to an invalid value (-1). In the example described below, this condition can be further restricted to the reference images of the first and second predictors being in different orientations relative to the current image. b) When it is determined that SecondPredictorMvpCand[i] can be used as the second predictor, the bilateral matching cost (costBM[i]) is derived by calculating the bilateral matching error between FirstRefBlock and the second reference block (SecondRefBlock[i]). SecondPredictorRefList, SecondPredictorRefIdx[i], and SecondPredictorMvp[i] are used to determine SecondRefBlock[i], where SecondPredictorRefIdx[i] and SecondPredictorMvp[i] are information derived from SecondPredictorMvpCand[i]. c) Otherwise, if it is determined that SecondPredictorMvpCand[i] cannot be used as a second predictor, then set costBM[i] to the maximum bilateral matching cost. 5) The second MVP index is determined to be equal to k by comparing costBM[k] with the minimum bilateral matching error of all merge candidates. 6) The final motion vector of the second predictor (SecondPredictorMv) is derived as follows via bilateral matching motion vector refinement: a) The initial motion vector is derived as SecondPredictorMvp[k]. b) Refine the bilateral matching motion vector to derive deltaMV, where the bilateral matching error between FirstRefBlock (in step 1) and SecondRefBlock[k]' is the minimum value within the bilateral matching search range or the minimum value within the local search region. SecondPredictor MVP[k] + deltaMV can be used as the motion vector to generate SecondRefBlock[k]'. c) The final motion vector of the second predictor is derived by setting SecondPredictorMv to be equal to SecondPredictorMvp[k]+deltaMV.

[0131] The video encoder 200 and video decoder 300 can be configured to generate and process syntax signaling interactions with other inter-frame prediction tools. As described above in the example of deriving the first predictor from an AMVP candidate list, the video encoder 200 can transmit and the video decoder 300 can receive a flag, HybridBiFlag, indicating whether the current block has been decoded as a hybrid inter-frame dual prediction block.

[0132] In some instances, the techniques described in this case include not enabling hybrid dual-mode prediction along with affine AMVP and symmetric motion vector difference (SMVD) prediction. In some instances, the video encoder 200 can be configured to signal the syntax element HybridBiFlag before the syntax element inter_pred_idc flag, where the inter_pred_idc flag indicates the prediction direction (interDir) of the current block. Therefore, when the decoded value of HybridBiFlag is true, the video decoder 300 can be configured to implicitly determine (e.g., in the absence of decoding a specific syntax element in the bitstream) the following parameters: The value of the syntax element interDir can be implicitly set to 3, which means that the prediction direction is bidirectional. The value of the syntax element `affineFlag` can be implicitly set to `false`, which means that affine prediction is disabled. • The value of the syntax element smvdMode can be implicitly set to false, which means that SMVD prediction is disabled.

[0133] In another instance, the video encoder 200 can be configured to signal the syntax element HybridBiFlag after affine AMVP and SMVD modes. In this case, the video decoder 300 can be configured to implicitly determine HybridBiFlag as false without decoding the bitstream when any of the following conditions are met: • If the syntax element `inter_pred_idc` is not equal to 3, this means that the current block is predicted using a single prediction. • When the syntax element `affineFlag` is equal to `true`, it means that affine prediction is enabled, or • When the syntax element smvdMode is equal to true, it means that SMVD prediction is enabled.

[0134] This case also describes the following technique: when the current block is decoded using a hybrid inter-frame double prediction block, the video decoder 300 can be configured to use a preset self-adjusting motion vector resolution (AMVR) and double prediction with CU-level weighted prediction (BCW) for the current block. That is, for a hybrid inter-frame double prediction block, the video decoder 300 can be configured to implicitly set the AMVR parameter to IMV_OFF and the BCW parameter to BCW_DEFAULT without decoding the bitstream.

[0135] As part of performing hybrid inter-frame double prediction using bilateral matching, the video encoder 200 and video decoder 300 can be configured to operate based on various enable conditions and other restrictions. In some instances, the techniques described herein include configuring the video encoder 200 and video decoder 300 to use several enable conditions to determine whether the hybrid inter-frame double prediction procedure applies to a block, a block in a slice, or a block in a picture. In response to a determination that the hybrid inter-frame double prediction procedure does not apply to a block (or is disabled), the video encoder 200 does not need to encode the value of HybridBiFlag, and the video decoder 300 does not need to decode the value of HybridBiFlag.

[0136] An example of an enabling condition is that the current block is a block of a bipredictable slice, a bipredictable sub-image, or a bipredictable picture. That is, if the current block is included in a slice, sub-image, or picture that does not support biprediction, the video encoder 200 and the video decoder 300 can skip processing the syntax element HybridBiFlag.

[0137] In some instances, the technology described in this case includes enabling hybrid inter-frame double prediction for a block when at least one reference image of the block is from the future relative to the current image in display order, and at least one reference image is from the past relative to the current image in display order. That is, hybrid inter-frame double prediction can be disabled if all reference images point to the future of the current image or all reference images are from the past of the current image. In this context, "past" refers to images that precede the current image in output order, and "future" refers to images that follow the current image in output order.

[0138] For `mvdL1ZeroFlag`, when the value of `mvdL1ZeroFlag` is equal to `true`, the video decoder 300 can be configured to implicitly deduce, without decoding the bitstream, that the MVD signaling to reference list 1 is set to zero MVD. In the current VVC specification, `mvdL1ZeroFlag` is a slice parameter.

[0139] In some instances, the technique described in this case includes: when a block is determined to be a hybrid inter-frame double prediction decoded block and the slice parameter mvdL1ZeroFlag is equal to true, the value of FirstPredictorOnReferenceList0Flag is implicitly determined to be true. In other words, for hybrid inter-frame double prediction, the video decoder 300 can be configured to determine the MVD signaled at reference list 0 without decoding the bitstream. The video decoder 300 can determine that a block is decoded as a hybrid inter-frame double prediction decoded block by decoding the value of HybridBiFlag, which is equal to true, from the bitstream.

[0140] In some instances, the techniques described in this case include disabling hybrid frame double prediction when the slice parameter mvdL1ZeroFlag for the current block is true. In this case, the video encoder 200 and the video decoder 300 can be configured to implicitly determine blocks for which the value of HybridBiFlag is false without decoding the bitstream.

[0141] The video encoder 200 and video decoder 300 can also be configured to operate according to other constraints. For example, in an instance with the final motion vector of the second predictor derivation procedure described above, there may be conditions for determining whether an MVP candidate in the merge candidate list can be used as a second predictor for the current block. In some instances, the technology described herein includes the following conditions: If the first predictor's reference comes from the future of the current block, then the second predictor's reference should come from the past of the current block. If the first predictor's reference comes from the past of the current block, then the second predictor's reference should come from the future of the current block.

[0142] In other words, the reference images for the two predictors should be located in different directions relative to the current image. This means that if the first reference image is displayed before the current image, then the second reference image should be displayed after the current image, and vice versa.

[0143] Figure 9 illustrates an instance of the first predictor (reference block 192) used for the current block 194 located in a reference picture (reference picture 1) that is past or before the current picture in the display order. There are five candidates (cand0 to cand4) for the second predictor; however, only three of these candidates (second reference block candidate 196) are applicable because these candidates are located in reference pictures (reference pictures 2 and 3) that are future or after the current picture in the display order.

[0144] An example of bilateral matching prediction with a fixed template will now be described. As described above regarding the derivation of the final motion vector for the second predictor, the bilateral matching motion vector refinement procedure for the second predictor may differ from the bilateral matching procedure in the current VVC standard. In the current VVC standard, bilateral matching motion vector refinement derives a mirrored deltaMV, which is applied to the two initial motion vectors of the reference block, as shown in Figure 5.

[0145] Figure 10 illustrates an example of bilateral matching prediction with a fixed paradigm. In some instances, the video encoder 200 and video decoder 300 can be configured to derive the final second motion vector using bilateral matching motion vector refinement with a fixed paradigm. The fixed paradigm (FirstRefBlock 193) can be derived via FirstPredictorRefList, FirstPredictorRefIdx, and FirstPredictorMvp, as described above in the example of deriving the first predictor from the AMVP candidate list.

[0146] The video encoder 200 and video decoder 300 can be configured to set the initial motion vector of the initial second reference block (195) to the determined SecondPredictorMvp[k] of SecondPredictorMvpCand[k]. The final second reference block 197 (SecondRefBlock[k]) is derived via SecondPredictorRefList, SecondPredictorRefIdx[k], and SecondPredictorMvp[k] + deltaMV. The video decoder 300 can be configured to derive deltaMV by searching within the bilateral matching search range.

[0147] In some instances, the search range can be predefined horizontally as [-8, 8] and vertically as [-8, 8]. In some instances, the search range can be self-adjusted based on the dimensions of the current block. Further details regarding the self-adjusting search range for bilateral match prediction are described in U.S. Patent Application No. 17 / 101,687, filed November 23, 2020, and published as U.S. Patent Publication No. 2022 / 0167001.

[0148] In some instances, the video encoder 200 and the video decoder 300 can be configured to derive the value of deltaMV as follows: 1) Derive an integer primitive precision motion vector with the minimum bilateral matching error between FirstRefBlock and SecondRefBlock[k] within the search range. 2) Based on the derived integer motion vector, derive a semi-primitive motion vector with the minimum bilateral matching error between FirstRefBlock and SecondRefBlock[k]. 3) The fractional motion vector with 1 / 16 primitive accuracy is derived by sub-primitive offset estimation based on the parameter error surface, as described above regarding the search scheme.

[0149] In some instances, for integer primitive-precision motion vector searches, the video encoder 200 and video decoder 300 can be configured to apply a 3×3 square search pattern to achieve a local minimum bilateral matching error. In some instances, for integer primitive-precision motion vector searches, the video encoder 200 and video decoder 300 can be configured to apply a full search within the bilateral matching search range to obtain the minimum bilateral matching error within the search range. In some instances, the search range can be categorized into several diamond-shaped search regions, and a search priority order can be assigned to each search region. In some instances, a full search can examine all search regions. In another instance, a full search skips a subset of search regions to reduce encoder / decoder complexity. Figure 11 illustrates an example of a full search with 5 diamond-shaped search regions, examining search regions 0 to 3 to derive the integer deltaMV, and skipping search region 4.

[0150] As part of performing hybrid inter-frame prediction using bilateral matching, the video encoder 200 and video decoder 300 can be configured to utilize an alternative first reference block. As in the example above, the first reference block (FirstRefBlock) is derived using FirstPredictorMvp, FirstPredictorRefList, and FirstPredictorRefIdx, as discussed above regarding the derivation of the first predictor from the AMVP candidate list. The motion vector used to output the first reference block is the MVP derived from the first predictor.

[0151] In some instances, the technique described in this case includes using an alternative first reference block derived via the final motion vector (FirstPredictorMv) of the first predictor. The video encoder 200 and the video decoder 300 can be configured to derive FirstPredictorMv by applying FirstPredictorMvp to FirstPredictorMvp.

[0152] As part of performing hybrid inter-frame dual prediction using bilateral matching, the video encoder 200 and video decoder 300 can be configured to perform an alternative decision on FirstPredictorMvp. Compared to the techniques described above for deriving the first predictor from the AMVP candidate list, FirstPredictorRefIdx and FirstPredictorMvpIdx can be signaled to derive FirstPredictorMvp by selecting an MVP from the MVP candidate list. In some instances, the techniques of this application include using alternative methods to implicitly determine FirstPredictorMvp without signaling. One exemplary technique is to select an MVP from the MVP candidate list based on the minimum template matching error.

[0153] As part of performing hybrid inter-frame dual prediction using bilateral matching, the video encoder 200 and video decoder 300 can be configured to perform alternative decisions to SecondPredictorMvp.

[0154] Compared to the techniques described above for deriving the second predictor from the merged candidate list, when deriving SecondPredictorMvp[k], the index of the MVP candidate list k is implicitly determined based on the minimum bilateral matching error. In some instances, the techniques described herein include using alternative methods to explicitly determine SecondPredictorMvp by signaling one or more indices used to select MVP candidates from the MVP candidate list. The decoding mode of the current block can be used to determine whether SecondPredictorMvp is implicitly determined based on the minimum bilateral matching error or explicitly determined using explicit signaling of one or more indices.

[0155] In another instance, when using a signaled index for selecting the MVP from the MVP candidate list to determine FirstPredictorMvp and / or SecondPredictorMvp, the video encoder 200 and video decoder 300 can be configured to sort the MVP candidate list to prioritize the most trusted MVP candidate. Doing so reduces the signaling cost of the MVP index. The sorting of the MVP candidate list can be based on bilateral matching error or template matching error. The example described above for deriving the second predictor from the merged candidate list can be considered a special case where the merged candidate list is sorted based on bilateral matching error, and the first MVP candidate in the sorted MVP candidate list is implicitly selected without signaling the index in the bitstream.

[0156] As part of performing hybrid inter-frame dual prediction using bilateral matching, the video encoder 200 and video decoder 300 can be configured to perform bilateral matching motion vector refinement for both the first predictor MVP and the second predictor MVP. The example illustration above uses a fixed template (generated from motion vector information of the first predictor) to refine the bilateral matching procedure of the second predictor MVP. In some instances, the technique described herein includes using bilateral matching to refine FirstPredictorMvp and the determined SecondPredictorMvp[k]. Bilateral matching motion vector refinement uses FirstPredictorMvp and SecondPredictorMvp[k] as initial motion vectors. The bilateral matching procedure searches around the initial motion vectors within a predetermined search range. The final motion vectors FirstPredictorMvp' and SecondPredictorMvp[k]' are derived based on the minimum bilateral matching error. The MVD between the final motion vector and the initial MV (denoted by (FirstPredictorMvp'-FirstPredictorMvp) and (SecondPredictorMvp[k]'-SecondPredictorMvp[k])) is proportional to the temporal distance between the current image and the two reference images, as described above regarding bilateral matching prediction. In some instances, mirrored MVD can be applied without considering the temporal distance, as shown in Figure 5.

[0157] As described above regarding the derivation of the first predictor from the AMVP candidate list, the final motion vector of the first predictor FirstPredictorMv is then derived by applying the derived FirstPredictorMvd to the refined first predictor MVP FirstPredictorMvp'.

[0158] The video encoder 200 and the video decoder 300 can derive the second final motion vector in a manner similar to that described above regarding the derivation of the second predictor from the merged candidate list.

[0159] As part of using bilateral matching to perform hybrid inter-frame dual prediction, the video encoder 200 and the video decoder 300 can be configured to perform a combination of known BM and single-pattern BM.

[0160] In some instances, this case describes the combination of bilateral matching refinement used for both the first and second predictor MVPs (as previously mentioned, referred to as conventional BM and bilateral matching refinement) used only for the second predictor MVP. This technique may be referred to as single-template BM.

[0161] In some instances, the video encoder 200 and the video decoder 300 can be configured to implicitly determine the use of a known BM or a single-template BM via a parameter derived from the first predictor.

[0162] For example, when the number of MVP candidates in the first MVP candidate list is greater than 1, the video decoder 300 can be configured to use the MVP index signaled by a signal to determine whether to use a known BM or a single-pattern BM. When FirstPredictorMvpIdx equals 0, the BM program uses the known BM; otherwise, the BM program uses the single-pattern BM.

[0163] In another instance, when the number of MVP candidates in the first MVP candidate list is equal to 1, the video decoder 300 can be configured to use a signal reference index to determine whether to use a conventional BM or a single-pattern BM. When FirstPredictorRefIdx is equal to 0, the BM procedure uses the conventional BM; otherwise, the BM procedure uses the single-pattern BM.

[0164] In another instance, the use of a conventional BM or a single-pattern BM can be determined by decoding the flags from the bitstream.

[0165] The video encoder 200 and video decoder 300 can be configured to perform advanced syntax signaling. As described above regarding enable conditions and other limitations, mixed inter-frame double prediction may not be applied to blocks in the current slice or picture when the enable conditions or limitations are not met. Several advanced syntax elements can be used to indicate whether mixed inter-frame double prediction will be applied.

[0166] In one instance, mixed inter-frame double prediction can be disabled for a decoded image or slice based on an available reference image. Certain conditions can be applied to the reference image to enable mixed inter-frame double prediction. Examples of such conditions may include: • There must be at least one pair of reference images, one of which is in the past relative to the current image, and the other is in the future relative to the current image. Neither of these two reference images is a long-term reference image. Both reference images have the same resolution as the current image.

[0167] The conditions listed above can be used individually or in combination. When these conditions are not met, mixed frame double prediction is disabled for the image / slice, therefore no signal is sent to the corresponding block-level syntax element for mixed frame double prediction, and it is inferred that mixed frame double prediction is not applied.

[0168] In another instance, if at least one of the conditions is not met, a slice / picture-level flag can be introduced for inter-frame double prediction as a bitstream consistency constraint to set the flag to 0.

[0169] In another instance, a bitstream consistency constraint is introduced into the existing signaling process, which means that mixed frame double prediction is not applied and the corresponding administrative burden is set to 0. This situation indicates that the procedure is not used if at least one of the above conditions is not met.

[0170] The video encoder 200 and the video decoder 300 can be configured to perform merge list filling. As described above regarding deriving the second predictor from the merge candidate list, when deriving the second predictor from the merge candidate list and the merge list has fewer than a predefined number of candidates, the filling procedure can be applied to the merge list construction.

[0171] In some instances, the reference image assigned to the padded zero-motion vector candidate can satisfy certain conditions for mixed inter-frame biprediction. Examples of such conditions may include one or more of the following (also discussed above): • There must be at least one pair of reference images, one of which is in the past relative to the current image, and the other is in the future relative to the current image. Neither of these two reference images is a long-term reference image. Both reference images have the same resolution as the current image.

[0172] By iterating through all possible combinations of reference images, all possible zero-motion vector candidates can be found, and those reference images that satisfy predefined conditions are identified in a certain order. For example, the first loop is performed on list 0, and the second loop on list 1, and vice versa. Other sorting is also possible. The procedure can be executed at the slice / image level, and the list of zero-motion vector candidates found is stored as a preset candidate list. At the block level, when the number of candidates is less than the predefined maximum number of candidates, the procedure can loop through the preset candidates to add them to the candidate list until the number of candidates reaches the predefined maximum value.

[0173] The video encoder 200 and video decoder 300 can be configured to adjust the number of reference images. As in the examples above regarding enable conditions and other limitations, when hybrid inter-frame double prediction is applied to the current block, the number of reference images can be less than the number of reference images signaled at the slice or image level.

[0174] In this example, the video encoder 200 and the video decoder 300 can be configured to examine each reference picture in the reference picture list to determine the number of reference pictures for the block to be decoded via hybrid inter-frame double prediction. A reference picture is determined as a candidate reference picture when it meets certain conditions for hybrid inter-frame double prediction. Examples of such conditions may include one or more of the following (also discussed above): • There must be at least one pair of reference images, one of which is in the past relative to the current image, and the other is in the future relative to the current image. Neither of these two reference images is a long-term reference image. Both reference images have the same resolution as the current image.

[0175] In one instance, the number of reference images signaled at the slice / image level is 2. If, after checking the above conditions, the first reference image meets the conditions while the second reference image does not, the video encoder 200 and the video decoder 300 can be configured to set the number of reference images to 1. The index of the reference image can be implicitly determined to be 0 instead of signaling it in the bitstream.

[0176] The video encoder 200 and video decoder 300 can be configured to operate according to the signaling conditions of FirstPredictorMvd. As described above regarding the derivation of the first predictor from the AMVP candidate list, in some instances, FirstPredictorMvd can always be signaled in the bitstream. The first final motion vector (FirstPredictorMv) is derived by applying FirstPredictorMvd to FirstPredictorMvp for the first predictor.

[0177] In some instances, FirstPredictorMvd is signaled in the bitstream when certain conditions are met. In some instances, the condition is checking if the value of FirstPredictorMvpIdx is equal to a first value K. For example, when FirstPredictorMvpIdx equals K, FirstPredictorMvd is signaled in the bitstream. The first value K can be an integer value, such as 0 or 1, or it can be a set of integer values, such as {0, 1, 2}. In some instances, the condition is checking if the value of FirstPredictorRefIdx is equal to a first value M. For example, when FirstPredictorRefIdx equals M, FirstPredictorMvd is signaled in the bitstream. The first value M can be an integer value (such as 0 or 1) or it can be a set of integer values ​​(such as {0, 1, 2}). In some instances, the condition is checking if the value of FirstPredictorMvpIdx equals the first value K and the value of FirstPredictorRefIdx equals the first value M. In other words, the condition is to check the combination of the values ​​of FirstPredictorMvpIdx and FirstPredictorRefIdx.

[0178] The video decoder 300 can be configured to determine the second predictive MVP via an index and a bilateral matching cost signaled by the signal. As described above regarding the derivation of the second predictor from the merge candidate list, the video decoder 300 can determine that the second MVP index is equal to k by comparing costBM[k] with the minimum bilateral matching error of all merge candidates.

[0179] In some instances, the second MVP index is determined via a signaled index, SecondPredictorMvpIdx, and the bilateral matching cost costBM[i] of each merge candidate. The value of SecondPredictorMvpIdx indicates the index of the values ​​of costBM[i] sorted in ascending order. When the value of SecondPredictorMvpIdx is equal to 0, the second MVP is determined to be equal to k, where costBM[k] has the minimum bilateral matching error among all merge candidates. When the value of SecondPredictorMvpIdx is equal to 1, the second MVP is determined to be equal to p, where costBM[p] has the second minimum bilateral matching error among all merge candidates.

[0180] The video encoder 200 and video decoder 300 can be configured to operate based on the signaling conditions of SecondPredictorMvpIdx. In this example, SecondPredictorMvpIdx is signaled in the bitstream when certain conditions are met. In some instances, the condition is to check whether the value of FirstPredictorMvpIdx is equal to a first value K. For example, when FirstPredictorMvpIdx equals K, SecondPredictorMvpIdx is signaled in the bitstream. The first value K can be an integer value, such as 0 or 1, or the first value K can be a set of integer values, such as {0, 1, 2}. In some instances, the condition is to check whether the value of FirstPredictorRefIdx is equal to a first value M. For example, when FirstPredictorRefIdx equals M, SecondPredictorMvpIdx is signaled in the bitstream. The first value M can be an integer value, such as 0 or 1, or it can be a set of integer values, such as {0, 1, 2}. In some instances, the condition checks whether the value of FirstPredictorMvpIdx is equal to the first value K and whether the value of FirstPredictorRefIdx is equal to the first value M. In other words, the condition checks the combination of the values ​​of FirstPredictorMvpIdx and FirstPredictorRefIdx. In some instances, the signaling condition for SecondPredictorMvpIdx is the same as the signaling condition for FirstPredictorMvd.

[0181] In a second example, the video encoder 200 and the video decoder 300 can be configured to perform hybrid inter-frame dual prediction using template matching. In this example, the technique described herein includes deriving a hybrid dual-mode second predictor via template matching.

[0182] As part of performing hybrid inter-frame double prediction using template matching, the video encoder 200 and video decoder 300 can be configured to derive a first predictor from the AMVP candidate list. The first predictor derivation procedure can be the same as described above regarding performing hybrid inter-frame double prediction using bilateral matching.

[0183] As part of performing hybrid inter-frame dual prediction using template matching, the video encoder 200 and video decoder 300 can be configured to derive a second predictor from a merge candidate list. In some instances, the final motion vector of the second predictor can be derived by selecting an MVP from the merge candidate list based on the minimum template matching error between the current block template and the reference block template using template matching. The selected MVP is further refined to obtain the minimum template matching error between the current block template and the reference block template. As described above regarding template matching prediction, the current block template can be derived from the reconstructed blocks above and / or to the left.

[0184] In some instances, a reference block template can be derived via a single prediction using information from candidates in a merge candidate list. In another instance, a reference block template is derived via a dual prediction that uses information from candidates in a merge candidate list and information from a derived first predictor, such as FirstPredictorMvp, FirstPredictorRefList, and FirstPredictorRefIdx, as discussed above regarding the derivation of a first predictor from an AMVP candidate list.

[0185] The final motion vector of the second predictor is derived as follows: 1) Derive FirstPredictorMvp, FirstPredictorRefList and FirstPredictorRefIdx as described above. 2) Construct a list of candidate mergers. 3) Derive the current block template (CurBlockTemplate) from the adjacent reconstructed blocks above and / or to the left. 4) Determine the reference list (SecondPredictorRefList) of the second predictor as SecondPredictorRefList = 1 - FirstPredictorRefList. 5) The template matching cost (costTM[i]) for each MVP candidate in the merged candidate list is derived as follows: a) At MVP candidate i (SecondPredictorMvpCand[i]) in the merge candidate list, determine whether SecondPredictorMvpCand[i] can be used as a second predictor for the current block by means of conditions (e.g., SecondPredictorMvpCand[i] has no predictor at SecondPredictorRefList (the reference image index of SecondPredictorMvpCand[i] at SecondPredictorRefList is equal to an invalid value / -1)). As described above regarding using bilateral matching to perform mixed frame double prediction, the conditions can be further restricted, namely, the reference images of the first predictor and the second predictor are in different directions relative to the current image, as determined by the corresponding reference image list and reference image index. b) When it is determined that SecondPredictorMvpCand[i] can be used as the second predictor, the template matching cost(costTM[i]) is derived by calculating the template matching error between CurBlockTemplate and the reference block template (RefBlockTemplate[i]). (i) The single prediction reference template RefBlockTemplate[i] is derived from the reconstructed blocks above and / or to the left of the reference block in the reference image. The reference image can be determined by SecondPredictorRefList and SecondPredictorRefIdx[i]. The position of the reference block can be determined by SecondPredictorMvp[i] and the position of the current block. (ii) The double prediction reference template RefBlockTemplate[i] is derived from the equal-weight double prediction of the first reference block template block and the second reference block template. The first reference block template can be the reconstructed block above and / or to the left of the first reference block in the first reference image. The first reference image can be determined by FirstPredictorRefList and FirstPredictorRefIdx. The position of the first reference block can be determined by FirstPredictorMvp and the position of the current block. The second reference block template can be the reconstructed block above and / or to the left of the second reference block in the second reference image. The second reference image can be determined by SecondPredictorRefList and SecondPredictorRefIdx[i]. The position of the second reference block can be determined by SecondPredictorMvp[i] and the position of the current block. c) Otherwise, (when it is determined that SecondPredictorMvpCand[i] cannot be used as the second predictor), set costTM[i] to the maximum template matching cost. 6) By comparing costTM[k], it is determined that the second MVP index equals k, which is the minimum template matching error among all merge candidates. 7) The final motion vector of the second predictor (SecondPredictorMv) is derived as follows through template matching motion vector refinement: a) The initial motion vector is derived as SecondPredictorMvp[k]. b) Template matching motion vector refinement to derive deltaMV, where the template matching error between CurBlockTemplate (in step 1) and RefBlockTemplate[k]' is the minimum value within the template matching search range or the minimum value within the local search region. RefBlockTemplate[k]' can be generated by using SecondPredictorMvp[k] + deltaMV as the motion vector. c) The final motion vector of the second predictor is derived by setting SecondPredictorMv to be equal to SecondPredictorMvp[k]+deltaMV.

[0186] The video encoder 200 and video decoder 300 can be configured to perform template matching. Template matching is a decoder-side motion vector derivation procedure that refines the motion information of the current CU by finding the nearest match between a template in the current image (e.g., the top and / or left adjacent blocks of the current CU) and a block in the reference image (i.e., of the same size as the template). As shown in Figure 7, a better motion vector is searched around the initial motion of the current CU within the primitive search range of [-8, +8]. The template matching procedure used in the following document: Y. Chen et al., "Description of SDR, HDR and 360° video coding technology proposal by Qualcomm and Technicolor – low and high complexity versions", Joint Video Exploration Group (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 10th meeting: San Diego, USA, April 10-20, 2018 (JVET-J0021), was modified as follows: the search step size is determined based on AMVR mode, and template matching can be cascaded with bilateral matching procedures in merge mode.

[0187] In AMVP mode, MVP candidates are determined based on template matching error, selecting those that achieve the minimum difference between the current block template and the reference block template. Template matching is then performed only for that specific MVP candidate for motion vector refinement. Template matching refines the MVP candidate using an iterative diamond search, starting with full primitive MVD precision (or 4 primitives for 4 primitive AMVR mode) within the [–8, +8] primitive search range. AMVP candidates can be further refined using a cross-search with full primitive MVD precision (or 4 primitives for 4 primitive AMVR mode), followed by sequential use of half-primitive and quarter-primitive precisions (as specified in Table 1) according to the AMVR mode. This search procedure ensures that the MVP candidate maintains the same motion vector precision indicated by the AMVR mode after the template matching procedure.

[0188] Table 1. Search patterns for AMVR and its merged mode. Search mode AMVR mode Merge mode 4 primitives Full image primitive Half-graph Quarter primitive AltIF=0 AltIF=1 4-element rhombus v 4-key intersection v Full-image rhombus v v v v v Full primitive intersection v v v v v Semi-primitive intersection v v v v Quarter primitive intersection v v 1 / 8 primitive intersection v

[0189] In merge mode, a similar search procedure is applied to merge candidates indicated by the merge index. As shown in Table 1, template matching may proceed up to 1 / 8 primitive MVD accuracy or skip to an MVD accuracy exceeding half a primitive MVD accuracy, depending on whether an alternative interpolation filter is used based on the merged motion information (which is used in AMVR with half-primitive mode). Furthermore, when template matching mode is enabled, template matching can operate as a separate procedure or an additional motion vector refinement procedure between block-based and sub-block-based bilateral matching procedures, depending on whether bilateral matching can be enabled based on its enable condition check.

[0190] The video encoder 200 and video decoder 300 can be configured to perform multi-pass DMVR as described in ECM. In some instances of ECM, multi-pass decoder-side motion vector refinement is applied. In the first pass, bilateral matching is applied to the decoded block. In the second pass, bilateral matching is applied to each 16x16 sub-block within the decoded block. In the third pass, motion vectors in each 8x8 sub-block are refined by applying bidirectional optical flow (BDOF). The refined motion vectors are stored for both spatial and temporal motion vector prediction.

[0191] The video encoder 200 and video decoder 300 can be configured to perform the first step using block-based bilateral matching motion vector refinement. In the first step, refined motion vectors are derived by applying bilateral matching to the decoded blocks. Similar to decoder-side motion vector refinement (DMVR), in the dual prediction operation, refined motion vectors are searched around two initial motion vectors (MV0 and MV1) in the reference image lists L0 and L1. Refined motion vectors (MV0_pass1 and MV1_pass1) are derived around the initial motion vectors based on the minimum bilateral matching cost between the two reference blocks in L0 and L1.

[0192] Bilateral matching performs a local search to derive the integer sampling precision intDeltaMV. The local search applies a 3×3 square search pattern to cycle through the search range [–sHor, sHor] in the horizontal direction and [–sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block dimension, and the maximum value of sHor and sVer is 8.

[0193] The bilateral matching cost is calculated as: bilCost = mvDistanceCost + sadCost. When the block size cbW*cbH is greater than 64, the MRSAD cost function is applied to remove the DC effect of distortion between reference blocks. When the bilCost at the center point of the 3×3 search pattern has the minimum cost, the intDeltaMV local search is terminated. Otherwise, the current minimum cost search point becomes the new center point of the 3×3 search pattern, and the search for the minimum cost continues until the video decoder 300 reaches the end of the search range.

[0194] The existing fractional sampling refinement is further applied to derive the final deltaMV. Subsequently, the video encoder 200 and video decoder 300 can derive the refined motion vector after the first step, as shown below: MV0_pass1 = MV0 + deltaMV MV1_pass1=MV1–deltaMV

[0195] The video encoder 200 and video decoder 300 can be configured to perform the second step using sub-block-based bilateral matching motion vector refinement. In the second step, refined motion vectors are derived by applying bilateral matching to 16×16 grid sub-blocks. For each sub-block, a refined motion vector is searched around the two motion vectors (MV0_pass1 and MV1_pass1) obtained in the first step in the reference image lists L0 and L1. The refined motion vectors (MV0_pass2(sbIdx2) and MV1_pass2(sbIdx2)) are derived based on the minimum bilateral matching cost between the two reference sub-blocks in L0 and L1.

[0196] For each subblock, a full search is performed on both sides to derive the integer sampling precision intDeltaMV. The full search has a search range of [–sHor, sHor] in the horizontal direction and a search range of [–sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block dimension, and the maximum value of sHor and sVer is 8.

[0197] The bilateral matching cost is calculated by applying a cost factor to the sum of absolute transformation differences (SATD) cost between two reference subblocks, e.g., bilCost = satdCost * costFactor. The search region (2 * sHor + 1) * (2 * sVer + 1) is divided into up to five diamond-shaped search regions, as shown in Figure 11. Each search region is assigned a costFactor determined by the distance (intDeltaMV) between each search point and the starting motion vector, and each diamond region is processed in order starting from the center of the search region. Within each region, search points are processed in raster scan order from the top left corner to the bottom right corner. The integer primitive full search terminates when the minimum bilCost within the current search region is less than a threshold (equal to sbW * sbH); otherwise, the integer primitive full search continues to the next search region until all search points have been checked. Figure 11 illustrates an exemplary diamond-shaped region within the search region.

[0198] The existing VVC DMVR fractional sampling refinement is further applied to derive the final deltaMV(sbIdx2). Subsequently, the video encoder 200 and video decoder 300 can derive the refined motion vector after the second step, as shown below: MV0_pass2(sbIdx2) = MV0_pass1 + deltaMV(sbIdx2) MV1_pass2(sbIdx2) = MV1_pass1 – deltaMV(sbIdx2)

[0199] The video encoder 200 and video decoder 300 can be configured to perform a third step using sub-block-based bidirectional optical flow (BDOF) motion vector refinement. In the third step, refined motion vectors are derived by applying BDOF to 8×8 grid sub-blocks. For each 8×8 sub-block, BDOF refinement is applied to derive scaled Vx and Vy without clipping from the refined motion vectors of the parent-child blocks in the second step. The derived bioMv(Vx, Vy) is rounded to 1 / 16 sample precision and clipped between -32 and 32.

[0200] Subsequently, the video encoder 200 and the video decoder 300 can derive the refined motion vectors (MV0_pass3(sbIdx3) and MV1_pass3(sbIdx3)) after the third step, as shown below: MV0_pass3(sbIdx3) = MV0_pass2(sbIdx2) + bioMv MV1_pass3(sbIdx3) = MV0_pass2(sbIdx2) – bioMv

[0201] In M. Coban, F. Le Léannec and J. StröM, "Algorithm description of Enhanced Compression Model 2 (ECM 2)", JVET-W2025, July 2021, the inter-frame decoding techniques of multi-step decoder-side motion vector refinement (MPDVMR) and template matching (template matching merging and template matching AMVP) have great potential for improvement in decoding efficiency.

[0202] The above description of a mixed-frame inter-prediction procedure for improving decoding efficiency has already been presented. This document also describes the interaction of the mixed-frame inter-prediction procedure with MPDMVR, template match merging, and template match AMVP procedures. Techniques for improving the decoding efficiency of the mixed-frame inter-prediction procedure are also described. The examples described below can be used individually or in any combination.

[0203] The first example of deriving a self-adjusting second predictor will now be described. As previously mentioned, the mixed-frame dual predictor procedure has a first predictor derived via a decoded bitstream and a second predictor derived without decoding the bitstream. In the second predictor derivation procedure, a matching error is calculated for each candidate. The candidate with the minimum matching error is selected. The matching error is presumably calculated via bilateral matching or template matching.

[0204] The first exemplary technique uses several parameters. These parameters are used in the second predictor derivation procedure. Some of these parameters are predefined at both the video encoder 200 and the video decoder 300. Some of these parameters are derived from the decoded bitstream.

[0205] The video encoder 200 and video decoder 300 can be configured to utilize matching function parameters. This example describes high-level parameters called MatchingFunctionHighLevel and low-level parameters called MatchingFunctionLowLevel. High-level parameters are parameters that can be used for video sequences, sets of images within video sequences, images, slices, sub-images, tiles, CTUs, or predetermined regions in a video transcoder. Low-level parameters are parameters used for blocks (e.g., decoding units or prediction units), where these parameters can be signaled in the bitstream or derived based on the matching function achieving a lower matching cost, regardless of which is possible.

[0206] The parameter indicates which matching function to use in the second predictor derivation procedure. • When the value of MatchingFunctionHighLevel is equal to the first value (e.g., 0), it indicates that the block can only use the template matching function. The value of MatchingFunctionLowLevel is determined to be equal to the fourth value (e.g., 0). The matching error for each candidate is calculated by using template matching for the current block. • When the value of MatchingFunctionHighLevel is equal to the second value (e.g., 1), it indicates that the block can only use the two-sided matching function. The value of MatchingFunctionLowLevel is determined to be equal to the fifth value (e.g., 1). The matching error for each candidate is calculated by using two-sided matching for the current block. • When the value of MatchingFunctionHighLevel equals the third value (e.g., 2), it indicates that the block can use either a bilateral matching function or a template matching function. The value of MatchingFunctionLowLevel is derived by decoding flags from the bitstream. ○ When the value of the decoded flag is equal to the sixth value (e.g., 0), the value of MatchingFunctionLowLevel is determined to be equal to the fourth value (e.g., 0). The matching error for each candidate is calculated by using template matching for the current block. ○ When the value of the decoded flag is equal to the seventh value (e.g., 1), the value of MatchingFunctionLowLevel is determined to be equal to the fifth value (e.g., 1). The matching error for each candidate is calculated by using bilateral matching over the current block.

[0207] In one instance, high-level parameters (such as MatchingFunctionHighLevel) can be signaled in a bitstream as parameters for SPS, PPS, PH, sub-image headers, SH, tiles, and / or CTU.

[0208] In one instance, the proposed high-level parameters are predefined and synchronized in both the encoder and decoder.

[0209] In one instance, when the matching function is determined to be a template match for the current block, but the current block has no template (e.g., the top and left adjacent blocks of the current block are not available), the matching error for each candidate for the current block is calculated by using bilateral matching.

[0210] The video encoder 200 and video decoder 300 can be configured to determine candidate indices for the second predictor parameters. This example proposes using a high-level parameter called MaxNumSecondPredCandHighLevel, a low-level parameter called NumSecondPredCandLowLevel, and a low-level parameter called SecondPredCandxLowLevel. The high-level parameter MaxNumSecondPredCandHighLevel is a parameter that can be used for video sequences, sets of images within video sequences, images, sub-images, tiles, slices, or CTUs. The low-level parameters are used for blocks.

[0211] The parameter MaxNumSecondPredCandHighLevel indicates the maximum number of candidates to select from the candidate list. The value of MaxNumSecondPredCandHighLevel ranges from 1 to K, where K is a predefined value and K can be greater than or equal to 1.

[0212] If the number of candidates in the candidate list of the current block is equal to N, and N is equal to or less than MaxNumSecondPredCandHighLevel, then set the value of NumSecondPredCandLowLevel to be equal to N.

[0213] Otherwise, (when N is greater than MaxNumSecondPredCandHighLevel), set the value of NumSecondPredCandLowLevel to be equal to MaxNumSecondPredCandHighLevel.

[0214] The parameter SecondPredCandIdxLowLevel indicates which candidate in the candidate list of the current block will be used as the candidate for the second predictor. The value of SecondPredCandIdxLowLevel ranges from 0 to (NumSecondPredCandLowLevel – 1). When the value of SecondPredCandIdxLowLevel is equal to 0, the candidate with the smallest matching error is used as the candidate for the second predictor. When the value of SecondPredCandIdxLowLevel is equal to 1, the candidate with the second smallest matching error is used as the candidate for the second predictor, and so on.

[0215] When the determined value of NumSecondPredCandLowLevel is equal to 1, the value of SecondPredCandxLowLevel is determined to be equal to 0.

[0216] When the determined value of NumSecondPredCandLowLevel is greater than 1, the value of SecondPredCandxLowLevel is derived by decoding the bit stream.

[0217] Therefore, in one instance of the content of this case, the video encoder 200 and the video decoder 300 can be configured to derive a first predictor for a mixed inter-frame prediction procedure, derive a second predictor for a mixed inter-frame prediction procedure based on high-level and low-level parameters, and decode blocks of video data using the mixed inter-frame prediction procedure according to the first and second predictors.

[0218] In one instance, advanced parameters are used for one or more of the following: video sequence, a collection of images within the video sequence, images, slices, sub-images, tiles, CTUs, or a predefined region.

[0219] In one instance, low-level parameters are used for blocks, including one or more of the decoding or prediction units.

[0220] In one instance, one or more of the high-level or low-level parameters indicate the matching function used to derive the second predictor.

[0221] In one instance, the advanced parameter indicates the maximum number of candidates from the candidate list.

[0222] In one instance, the second low-level parameter indicates candidates in the candidate list to be used to derive the second predictor.

[0223] A second instance of motion vector refinement will now be described. As previously mentioned, when the second predictor derivation procedure applies bilateral matching, the selected second candidate motion vector is refined by minimizing the bilateral matching error. The motion vector refinement procedure is skipped when the DMVR enable condition (e.g., the DMVR enable condition described above) is not met for the current block.

[0224] As mentioned above, when the second predictor derivation procedure applies template matching, the selected second candidate motion vector is refined by minimizing the template matching error. The motion vector refinement procedure is skipped when there is no template for the current block (e.g., the top and left adjacent blocks of the current block are unavailable).

[0225] The video encoder 200 and video decoder 300 can be configured to implement block-based motion vector refinement with a MatchingFunctionLowLevel indicator. The value of MatchingFunctionLowLevel is determined as described above in the first example of the derivation of the self-adjusting second predictor.

[0226] The value of MatchingFunctionLowLevel indicates that when the two-sided matching function is used in the second predictor derivation procedure, the selected candidate motion vectors are refined by minimizing the two-sided matching error.

[0227] The value of MatchingFunctionLowLevel indicates that when the template matching function is used in the second predictor derivation procedure, the selected candidate motion vectors are refined by minimizing the template matching error.

[0228] The video encoder 200 and video decoder 300 can be configured to prioritize motion vector refinement based on bilateral matching blocks. This example presents a predefined block-based motion vector refinement procedure independent of the value of MatchingFunctionLowLevel.

[0229] To implement the proposed block-based motion vector refinement procedure, the video encoder 200 and the video decoder 300 can be configured to perform the following steps in sequence: • Check if the DMVR enable conditions are met. If it is determined that the DMVR enable conditions are met, then apply the block-based DMVR motion vector refinement to the current block (e.g., as the first step of the BDMVR mentioned above). Otherwise, (when it is determined that the DMVR enable conditions are not met), check whether the current block template is available (e.g., the top and / or left adjacent blocks of the current block are available). If it is determined that the current block template is available, apply block-based template matching motion vector refinement to the current block. Otherwise, skip block-based motion vector refinement for the current block.

[0230] The video encoder 200 and video decoder 300 can be configured to prioritize pattern matching based on block-based motion vector refinement. This example describes a predefined block-based motion vector refinement procedure independent of the value of MatchingFunctionLowLevel.

[0231] To implement the proposed block-based motion vector refinement procedure, the video encoder 200 and the video decoder 300 can be configured to perform the following steps in sequence: • Check if the current block template is available (e.g., the top and / or left adjacent blocks of the current block are available). If it is determined that the current block template is available, then refine the block-based template matching motion vector and apply it to the current block. Otherwise, (when it is determined that the current block template is not available), check whether the DMVR enable conditions are met. If it is determined that the DMVR enable conditions are met, then refine the block-based DMVR motion vectors and apply them to the current block (e.g., as the first step of BDMVR as described above). Otherwise, skip block-based motion vector refinement for the current block.

[0232] In another instance, as described above regarding prioritizing the block-based motion vector refinement of bilateral matching and prioritizing the block-based motion vector refinement of template matching, the choice of prioritization procedure is indicated by flags in the bitstream at the sequence, picture, sub-picture, slice, tile, CTU, and / or block level.

[0233] In the second example, the video encoder 200 and the video decoder 300 can be configured to perform sub-block-based motion vector refinement. This example describes MPDMVR sub-block-based motion vector refinement for mixed inter-frame dual prediction decoding blocks.

[0234] In one instance, when the DMVR enablement conditions are met, a second step (e.g., as described above) and a third step (e.g., as described above) are applied to the current block to refine the motion vectors based on the sub-blocks.

[0235] In one instance, when the DMVR enablement conditions are met, the third step (e.g., as described above) is applied only to the current block to refine the motion vectors based on the sub-blocks.

[0236] The video encoder 200 and video decoder 300 can be configured to implement DMVR enable conditions. As in the second example concerning block-based motion vector refinement and sub-block-based motion vector refinement for the MatchingFunctionLowLevel indication, the DMVR enable conditions may differ from those described in the background above. For example: - The enabling condition may not have the condition that the distance (i.e., the difference in POC) from the two reference images to the current image is the same. - The enable conditions may not include conditions that constrain block size or block dimension. - The enable conditions may not include the conditions for constraining blocks in merge mode.

[0237] The video encoder 200 and video decoder 300 can be configured to implement enabled constraints. The constraints described above require that the reference images for the two predictors be located in different orientations relative to the current image. In this example, the constraint is applied based on the MatchingFunctionLowLevel determined as previously stated.

[0238] When the value of MatchingFunctionLowLevel indicates that bilateral matching is applied to the current block, this restriction is not applied to the current block.

[0239] When the value of MatchingFunctionLowLevel indicates that a pattern match is applicable to the current block, this restriction applies to the current block. In another instance, the restriction applies to the current block independently of the value of MatchingFunctionLowLevel. In yet another instance, the restriction does not apply to the current block.

[0240] An example of constructing the candidate list for the second predictor will now be discussed. This example introduces several constraints on the candidates in the second predictor candidate list.

[0241] General constraints will now be described. Before constructing the second predictor candidate list, the reference list FirstPredictorRefList for the first predictor and the reference list SecondPredictorRefList for the second predictor may have already been determined. The video encoder 200 and the video decoder 300 can be configured to implement the constraint that each candidate in the second predictor candidate list should have a valid motion vector at SecondPredictorRefList.

[0242] When the above enable restrictions are applied, the following restrictions may be applied: each candidate in the second predictor candidate list should have a valid motion vector that satisfies the enable restrictions.

[0243] In one instance, it is proposed that artificially generated candidates should not be added to the second predictor candidate list. For example, zero motion vector candidates should not be added to the second predictor candidate list. For example, average motion vector candidates (e.g., paired candidates of VVC or HEVC) should not be added to the second predictor candidate list.

[0244] The video encoder 200 and video decoder 300 can be configured to implement matching function-related restrictions. When the value of MatchingFunctionHighLevel indicates that the current block can use a bilateral matching or template matching function, the video decoder 300 can check candidates in a second candidate list to avoid selecting the same candidate by different matching functions.

[0245] In one instance, the bilateral matching function can be prioritized. To implement a second candidate list for the template matching function, the video encoder 200 and the video decoder 300 can be configured to perform the following steps in sequence: - Construct a second prediction candidate list (bilateral matching candidate list) that satisfies the constraints of the bilateral matching function as described above. - Calculate the bilateral matching error for each candidate in the bilateral matching candidate list. - Compare the bilateral matching errors and store the top N candidates with matching errors smaller than the other candidates. Here, N equals the value of NumSecondPredCandLowLevel, as discussed above regarding the candidate index for the second predictor parameter of the first instance. - Construct a second predictive candidate list (template matching candidate list) that satisfies the constraints of the template matching function as described above. A candidate should not be added to the template matching candidate list if it is the same as one stored in the previous steps.

[0246] In one instance, the template matching function can be prioritized. To implement a second candidate list for the bilateral matching function, the video encoder 200 and the video decoder 300 can be configured to perform the following steps in sequence: - Construct a second prediction candidate list (template matching candidate list) that satisfies the constraints of the above template matching function. - Calculate the template matching error for each candidate in the template matching candidate list. - Compare the template matching errors and store the top N candidates with matching errors smaller than the other candidates. Here, N equals the value of NumSecondPredCandLowLevel, as described above. - Construct a second prediction candidate list (bilateral matching candidate list) that satisfies the constraints of the bilateral matching function described above. A candidate should not be added to the bilateral matching candidate list if it is the same as one stored in the previous steps.

[0247] Figure 12 is a block diagram illustrating an exemplary video encoder 200 capable of performing the techniques described herein. Figure 12 is provided for illustrative purposes and should not be construed as limiting the techniques generally illustrated and described in this content. For illustrative purposes, this content describes a video encoder 200 based on VVC (ITU-T H.266 under development) and HEVC (ITU-T H.265) technologies. However, the techniques described herein can be performed by video encoding devices configured for other video decoding standards and video decoding formats, such as AV1 and subsequent versions of the AV1 video decoding format.

[0248] In the example of Figure 12, the video encoder 200 includes a video data memory 230, a mode selection unit 202, a residual generation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a filter unit 216, a decoded image buffer (DPB) 218, and an entropy encoding unit 220. Any or all of the video data memory 230, mode selection unit 202, residual generation unit 204, transform processing unit 206, quantization unit 208, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, filter unit 216, DPB 218, and entropy encoding unit 220 can be implemented in one or more processors or in a processing circuit system. For example, the units of the video encoder 200 can be implemented as one or more circuit or logic elements as part of a hardware circuit system, or as part of a processor, ASIC, or FPGA. In addition, the video encoder 200 may include additional or alternative processors or processing circuitry to perform these and other functions.

[0249] Video data memory 230 can store video data to be encoded by elements of video encoder 200. Video encoder 200 can receive video data stored in video data memory 230 from, for example, video source 104 (FIG. 1). DPB 218 can act as reference picture memory, storing reference video data for use when the video encoder 200 predicts subsequent video data. Video data memory 230 and DPB 218 can be formed of any of a variety of memory devices, such as dynamic random access memory (DRAM) (including synchronous DRAM (SDRAM)), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Video data memory 230 and DPB 218 can be provided by the same memory device or separate memory devices. In various instances, video data memory 230 can be on-chip (as shown) with other elements of video encoder 200, or off-chip relative to those elements.

[0250] In this context, references to video data memory 230 should not be construed as limited to memory within the video encoder 200 (unless specifically described therein), or to memory outside the video encoder 200 (unless specifically described therein). Rather, references to video data memory 230 should be understood as a reference memory storing video data received by the video encoder 200 for encoding (e.g., video data for the current block to be encoded). Memory 106 of FIG. 1 may also provide temporary storage for outputs from various units of the video encoder 200.

[0251] The various units illustrated in Figure 12 help to understand the operations performed by the video encoder 200. These units can be implemented as fixed-function circuits, programmable circuits, or a combination thereof. Fixed-function circuits represent circuits that provide specific functions and are pre-configured regarding the operations that can be performed. Programmable circuits represent circuits that can be programmed to perform various tasks and provide flexible functionality in terms of the operations that can be performed. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by the instructions of the software or firmware. Fixed-function circuits can execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is generally immutable. In some instances, one or more of these units can be different circuit blocks (fixed-function or programmable), and in some instances, one or more of the units can be integrated circuits.

[0252] The video encoder 200 may include an arithmetic logic unit (ALU), an essential function unit (EFU), digital circuitry, analog circuitry, and / or a programmable core, all formed by programmable circuitry. In an example where software executed by programmable circuitry is used to perform the operation of the video encoder 200, memory 106 (FIG. 1) may store instructions (e.g., object code) of the software received and executed by the video encoder 200, or another memory (not shown) within the video encoder 200 may store such instructions.

[0253] The video data memory 230 is configured to store received video data. The video encoder 200 can retrieve images of the video data from the video data memory 230 and provide the video data to the residual generation unit 204 and the mode selection unit 202. The video data in the video data memory 230 can be the raw video data to be encoded.

[0254] The mode selection unit 202 includes a motion estimation unit 222, a motion compensation unit 224, and an in-frame prediction unit 226. The mode selection unit 202 may include additional functional units to perform video prediction based on other prediction modes. As an example, the mode selection unit 202 may include a palette unit, an in-frame copy unit (which may be part of the motion estimation unit 222 and / or the motion compensation unit 224), an affine unit, a linear model (LM) unit, etc.

[0255] The mode selection unit 202 typically coordinates multiple encoding passes to test combinations of encoding parameters and the rate-distortion values ​​obtained for these combinations. Encoding parameters may include segmenting the CTU into CUs, the prediction mode for the CUs, the transformation type of the residual data for the CUs, and the quantization parameters of the residual data for the CUs. The mode selection unit 202 can ultimately select a combination of encoding parameters that yields a better rate-distortion value than other tested combinations.

[0256] The video encoder 200 can segment an image obtained from the video data memory 230 into a series of CTUs, and encapsulate one or more CTUs within a slice. The mode selection unit 202 can segment the image's CTUs according to a tree structure (such as the MTT structure, QTBT structure, superblock structure, or quadtree structure described above). As mentioned earlier, the video encoder 200 can form one or more CUs by segmenting CTUs according to a tree structure. Such CUs are also commonly referred to as "video blocks" or "blocks".

[0257] Typically, mode selection unit 202 also controls its components (e.g., motion estimation unit 222, motion compensation unit 224, and intra-frame prediction unit 226) to generate predicted blocks for the current block (e.g., the current CU, or the overlapping portion of PU and TU in HEVC). To perform inter-frame prediction on the current block, motion estimation unit 222 can perform a motion search to identify one or more closely matching reference blocks in one or more reference images (e.g., one or more previously decoded images stored in DPB 218). Specifically, motion estimation unit 222 can calculate values ​​representing the degree of similarity between a potential reference block and the current block, for example, based on SAD, SSD, MAD, MSD, etc. Motion estimation unit 222 can typically perform these calculations using the sample-by-sample difference between the current block and the considered reference blocks. Motion estimation unit 222 can identify the reference block with the lowest value obtained from these calculations, indicating the reference block that most closely matches the current block.

[0258] Motion estimation unit 222 can generate one or more motion vectors (MVs) that define the position of a reference block in a reference image relative to the position of the current block in the current image. Motion estimation unit 222 can then provide the motion vectors to motion compensation unit 224. For example, for unidirectional inter-frame prediction, motion estimation unit 222 can provide a single motion vector, while for bidirectional inter-frame prediction, motion estimation unit 222 can provide two motion vectors. Motion compensation unit 224 can then use the motion vectors to generate prediction blocks. For example, motion compensation unit 224 can use the motion vectors to obtain data for the reference blocks. As another example, if the motion vectors have fractional sampling precision, motion compensation unit 224 can interpolate the values ​​used for predicting blocks according to one or more interpolation filters. Furthermore, for bidirectional inter-frame prediction, motion compensation unit 224 can obtain data for two reference blocks identified by the corresponding motion vectors and combine the obtained data, for example, via sample-wise averaging or weighted averaging.

[0259] When operating according to the AV1 video decoding format, the motion estimation unit 222 and the motion compensation unit 224 can be configured to encode the decoded blocks of the video data (e.g., both the luminance decoded blocks and the chrominance decoded blocks) using translational motion compensation, affine motion compensation, overlapping block motion compensation (OBMC), and / or composite inter-frame and intra-frame prediction.

[0260] As another example, for in-frame prediction or in-frame prediction decoding, the in-frame prediction unit 226 can generate a prediction block based on samples adjacent to the current block. For example, for directional mode, the in-frame prediction unit 226 can typically mathematically combine the values ​​of adjacent samples and fill in these calculated values ​​across the current block in a defined direction to generate a prediction block. As another example, for DC mode, the in-frame prediction unit 226 can calculate the average of the adjacent samples of the current block and generate a prediction block to include the resulting average of each sample for the prediction block.

[0261] When operating according to the AV1 video decoding format, the in-frame prediction unit 226 can be configured to encode the decoded blocks of the video data (e.g., both luma-decoded blocks and chroma-decoded blocks) using directional in-frame prediction, non-directional in-frame prediction, recursive filter in-frame prediction, chroma prediction based on luminance (CFL) prediction, intra-block copy (IBC), and / or palette modes. The mode selection unit 202 may include additional functional units for performing video prediction according to other prediction modes.

[0262] Mode selection unit 202 provides the prediction block to residual generation unit 204. Residual generation unit 204 receives the original, uncoded version of the current block from video data memory 230 and the prediction block from mode selection unit 202. Residual generation unit 204 calculates the sample-by-sample difference between the current block and the prediction block. The resulting sample-by-sample difference defines the residual block for the current block. In some instances, residual generation unit 204 may also determine the difference between sample values ​​in the residual block to generate the residual block using Residual Differential Pulse Decoding Modulation (RDPCM). In some instances, residual generation unit 204 may be formed using one or more subtractor circuits performing binary subtraction.

[0263] In an instance where the mode selection unit 202 divides the CU into PUs, each PU can be associated with a luminance prediction unit and a corresponding chrominance prediction unit. The video encoder 200 and video decoder 300 can support PUs of various sizes. As noted above, the size of the CU can represent the size of the luminance decoding block of the CU, and the size of the PU can represent the size of the luminance prediction unit of the PU. Assuming a particular CU has a size of 2Nx2N, the video encoder 200 can support PU sizes of 2Nx2N or NxN for intra-frame prediction, and symmetrical PU sizes of 2Nx2N, 2NxN, Nx2N, NxN, or similar sizes for inter-frame prediction. The video encoder 200 and video decoder 300 can also support asymmetric segmentation for PU sizes of 2NxnU, 2NxnD, nLx2N, and nRx2N for inter-frame prediction.

[0264] In instances where the mode selection unit 202 does not further divide the CU into PUs, each CU can be associated with a luminance decoding block and a corresponding chrominance decoding block. As mentioned above, the size of the CU can represent the size of the luminance decoding block of the CU. The video encoder 200 and the video decoder 300 can support CU sizes of 2Nx2N, 2NxN, or Nx2N.

[0265] For other video decoding techniques (such as block-based copy mode decoding, affine mode decoding, and linear model (LM) mode decoding), mode selection unit 202 generates a prediction block for the current block being encoded via a corresponding unit associated with the decoding technique. In some instances (such as palette mode decoding), mode selection unit 202 may not generate a prediction block, but instead generate syntax elements indicating how the block should be reconstructed based on the selected palette. In this mode, mode selection unit 202 can provide these syntax elements to entropy coding unit 220 for encoding.

[0266] As mentioned above, the residual generation unit 204 receives video data for the current block and the corresponding prediction block. Subsequently, the residual generation unit 204 generates a residual block for the current block. To generate the residual block, the residual generation unit 204 calculates the sample-by-sample difference between the prediction block and the current block.

[0267] Transform processing unit 206 applies one or more transformations to the residual block to produce a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 can apply various transformations to the residual block to form the transform coefficient block. For example, transform processing unit 206 can apply discrete cosine transform (DCT), direction transform, Karhunen-Loeve transform (KLT), or conceptually similar transformations to the residual block. In some instances, transform processing unit 206 can perform multiple transformations on the residual block, such as primary and secondary transformations (e.g., rotation transformations). In some instances, transform processing unit 206 does not apply any transformations to the residual block.

[0268] When operating according to AV1, transform processing unit 206 can apply one or more transforms to the residual block to produce a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 can apply various transforms to the residual block to form the transform coefficient block. For example, transform processing unit 206 can apply a combination of horizontal / vertical transforms, which may include the Discrete Cosine Transform (DCT), the Asymmetric Discrete Sine Transform (ADST), the Inverted ADST (e.g., the Reverse ADST), and the Identity Transform (IDTX). When using the Identity Transform, the transform is skipped in either the vertical or horizontal direction. In some instances, transform processing can be skipped.

[0269] Quantization unit 208 can quantize the transform coefficients in the transform coefficient block to produce a quantized transform coefficient block. Quantization unit 208 can quantize the transform coefficients of the transform coefficient block based on the quantization parameter (QP) value associated with the current block. Video encoder 200 (e.g., via mode selection unit 202) can adjust the degree of quantization applied to the transform coefficient block associated with the current block by adjusting the QP value associated with the CU. Quantization may cause information loss, and therefore, the quantized transform coefficients may have lower precision than the original transform coefficients produced by transform processing unit 206.

[0270] The inverse quantization unit 210 and the inverse transform processing unit 212 can apply inverse quantization and inverse transform to the quantized transform coefficient block to reconstruct the residual block from the transform coefficient block. The reconstruction unit 214 can generate a reconstructed block corresponding to the current block (although potentially with some degree of distortion) based on the reconstructed residual block and the prediction block generated by the mode selection unit 202. For example, the reconstruction unit 214 can add a sample of the reconstructed residual block to a corresponding sample of the prediction block generated by the mode selection unit 202 to generate the reconstructed block.

[0271] Filter unit 216 can perform one or more filter operations on the reconstructed block. For example, filter unit 216 can perform a deblocking operation to reduce blocking artifacts along the edges of the CU. In some instances, the operation of filter unit 216 can be skipped.

[0272] When operating according to AV1, filter unit 216 can perform one or more filter operations on the reconstructed block. For example, filter unit 216 can perform a deblocking operation to reduce blocky artifacts along the edges of the CU. In other instances, filter unit 216 can apply a constrained directional enhancement filter (CDEF), which can be applied after deblocking, and can include an inseparable, nonlinear, low-pass directional filter applied based on the estimated edge direction. Filter unit 216 can also include a loop recovery filter applied after CDEF, and can include a separable symmetric normalized Wiener filter or a dual-self-guided filter.

[0273] The video encoder 200 stores the reconstructed blocks in the DPB 218. For example, in an instance where the operation of the filter unit 216 is not performed, the reconstruction unit 214 may store the reconstructed blocks in the DPB 218. In an instance where the operation of the filter unit 216 is performed, the filter unit 216 may store the filtered reconstructed blocks in the DPB 218. The motion estimation unit 222 and the motion compensation unit 224 may obtain a reference image formed from the reconstructed (and potentially filtered) blocks from the DPB 218 to perform inter-frame prediction of blocks in subsequently encoded images. In addition, the intra-frame prediction unit 226 may use the reconstructed blocks of the current image in the DPB 218 to perform intra-frame prediction of other blocks in the current image.

[0274] Typically, entropy coding unit 220 can entropy code syntax elements received from other functional elements of video encoder 200. For example, entropy coding unit 220 can entropy code quantized transform coefficient blocks from quantization unit 208. As another example, entropy coding unit 220 can entropy code predictive syntax elements (e.g., motion information for inter-frame prediction or intra-frame pattern information for intra-frame prediction) from mode selection unit 202. Entropy coding unit 220 can perform one or more entropy coding operations on syntax elements that are another instance of video data to produce entropy-coded data. For example, entropy coding unit 220 can perform context-adjustable variable-length decoding (CAVLC), CABAC, variable-to-variable (V2V) length decoding, syntax-based context-adjustable binary arithmetic decoding (SBAC), probabilistic interval partitioning entropy (PIPE) decoding, exponential Golomb coding, or another type of entropy coding operation on the data. In some instances, the entropy coding unit 220 can operate in a bypass mode where syntax elements are not entropy encoded.

[0275] The video encoder 200 can output a bitstream containing entropy-encoded syntax elements needed to reconstruct slices or blocks of an image. Specifically, the entropy encoding unit 220 can output a bitstream.

[0276] According to AV1, entropy coding unit 220 can be configured as a symbol-to-symbol self-adjusting multi-symbol arithmetic decoder. The syntax elements in AV1 include an N-element alphabet, and the context (e.g., a probability model) includes a set of N probabilities. Entropy coding unit 220 can store the probabilities as an n-bit (e.g., 15-bit) cumulative distribution function (CDF). Entropy coding unit 220 can perform recursive scaling to update the context using an update factor based on letter size.

[0277] The above operations are described for the blocks. This description should be understood as referring to the operations used for the luma decoding block and / or the chroma decoding block. As mentioned earlier, in some instances, the luma decoding block and the chroma decoding block are the luma and chroma components of the CU. In some instances, the luma decoding block and the chroma decoding block are the luma and chroma components of the PU.

[0278] In some instances, it is not necessary to repeat the operations performed for the chroma decoding block as with the luma decoding block. For example, it is not necessary to repeat the operations used to identify motion vectors and reference images for the luma decoding block to identify motion vectors and reference images for the chroma block. Instead, the motion vectors for the luma decoding block can be scaled to determine the motion vectors for the chroma block, and the reference image can be the same. As another example, the in-frame prediction procedure can be the same for both the luma and chroma decoding blocks.

[0279] Video encoder 200 represents an example of a device configured to encode video data, the device including: a memory configured to store video data; and one or more processing units implemented in a circuit system and configured to perform hybrid inter-frame double prediction according to the techniques described herein.

[0280] Figure 13 is a block diagram illustrating an exemplary video decoder 300 capable of performing the techniques described herein. Figure 13 is provided for illustrative purposes and does not limit the techniques generally illustrated and described in this document. For illustrative purposes, this document describes the video decoder 300 according to VVC (ITU-T H.266 under development) and HEVC (ITU-T H.265) technologies. However, the techniques described herein can be performed by video decoding devices configured for other video decoding standards.

[0281] In the example of Figure 13, the video decoder 300 includes a decoded picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transform processing unit 308, a reconstruction unit 310, a filter unit 312, and a decoded picture buffer (DPB) 314. Any or all of the CPB memory 320, entropy decoding unit 302, prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, filter unit 312, and DPB 314 can be implemented in one or more processors or in a processing circuitry system. For example, the units of the video decoder 300 can be implemented as one or more circuit or logic elements as part of a hardware circuitry system, or as part of a processor, ASIC, or FPGA. Furthermore, the video decoder 300 may include additional or alternative processors or processing circuitry systems to perform these and other functions.

[0282] The prediction processing unit 304 includes a motion compensation unit 316 and an in-frame prediction unit 318. The prediction processing unit 304 may include additional units that perform predictions based on other prediction modes. As examples, the prediction processing unit 304 may include a palette unit, an in-block copy unit (which may form part of the motion compensation unit 316), an affine unit, a linear model (LM) unit, etc. In other instances, the video decoder 300 may include more, fewer, or different functional elements.

[0283] When operating according to AV1, motion compensation unit 316 can be configured to decode the decoded blocks of video data (e.g., both luma and chroma decoded blocks) using translational motion compensation, affine motion compensation, OBMC, and / or composite inter-frame intra-frame prediction, as described above. Intra-frame prediction unit 318 can be configured to decode the decoded blocks of video data (e.g., both luma and chroma decoded blocks) using directional intra-frame prediction, non-directional intra-frame prediction, recursive filter intra-frame prediction, CFL prediction, intra-block copy (IBC), and / or palette mode, as described above.

[0284] According to the technology described in this case, the motion compensation unit 316 can be configured to predict the current block using hybrid inter-frame double prediction by determining a first motion vector from a bitstream of video data using signaled syntax elements, and by determining a second motion vector using bilateral matching and without explicit signal transmission.

[0285] CPB memory 320 can store video data, such as encoded video bitstreams, to be decoded by elements of video decoder 300. For example, video data stored in CPB memory 320 can be obtained from computer-readable media 110 (FIG. 1). CPB memory 320 may include a CPB storing encoded video data (e.g., syntax elements) from the encoded video bitstream. Furthermore, CPB memory 320 can store video data other than the syntax elements of the decoded image, such as temporary data representing the output from various units of video decoder 300. DPB 314 typically stores decoded images that video decoder 300 can output and / or use as reference video data when decoding subsequent data or images from the encoded video bitstream. CPB memory 320 and DPB 314 can be formed from any of a variety of memory devices, such as DRAM, including SDRAM, MRAM, RRAM, or other types of memory devices. CPB memory 320 and DPB 314 can be provided by the same memory device or separate memory devices. In various instances, CPB memory 320 can be on-chip with other components of the video decoder 300, or off-chip relative to those components.

[0286] Alternatively, in some instances, the video decoder 300 may retrieve the decoded video data from memory 120 (FIG. 1). That is, memory 120 may utilize CPB memory 320 to store data as discussed above. Similarly, when some or all of the functions of the video decoder 300 are implemented using software to be executed by the processing circuitry of the video decoder 300, memory 120 may store instructions to be executed by the video decoder 300.

[0287] The various units illustrated in Figure 13 help to understand the operations performed by the video decoder 300. These units can be implemented as fixed-function circuits, programmable circuits, or a combination thereof. Similar to Figure 12, a fixed-function circuit represents a circuit that provides a specific function and is pre-configured regarding the operations that can be performed. A programmable circuit represents a circuit that can be programmed to perform various tasks and provides flexible functionality in terms of the operations that can be performed. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by the instructions of the software or firmware. A fixed-function circuit can execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is generally immutable. In some instances, one or more of these units can be different circuit blocks (fixed-function or programmable), and in some instances, one or more of the units can be integrated circuits.

[0288] The video decoder 300 may include an ALU, EFU, digital circuitry, analog circuitry, and / or a programmable core formed by programmable circuitry. In an instance where the operation of the video decoder 300 is performed by software executing on the programmable circuitry, on-chip or off-chip memory may store instructions (e.g., object code) of the software received and executed by the video decoder 300.

[0289] Entropy decoding unit 302 can receive encoded video data from CPB and perform entropy decoding on the video data to reconstruct syntax elements. Prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, and filter unit 312 can generate decoded video data based on syntax elements extracted from the bitstream.

[0290] Typically, the video decoder 300 reconstructs the image on a block-by-block basis. The video decoder 300 can perform the reconstruction operation on each block individually (where the block currently being reconstructed (i.e., decoded) can be referred to as the "current block").

[0291] Entropy decoding unit 302 can entropy decode the syntax elements of the quantized transform coefficients used to define the quantized transform coefficient block, as well as transform information such as quantization parameters (QP) and / or transform mode indications. Inverse quantization unit 306 can use the QP associated with the quantized transform coefficient block to determine the quantization level, and similarly, determine the inverse quantization level applied by inverse quantization unit 306. Inverse quantization unit 306 can, for example, perform a bit-by-bit left shift operation to inverse quantize the quantized transform coefficients. Inverse quantization unit 306 can thus form a transform coefficient block including the transform coefficients.

[0292] After the inverse quantization unit 306 forms the transform coefficient block, the inverse transform processing unit 308 can apply one or more inverse transforms to the transform coefficient block to generate a residual block associated with the current block. For example, the inverse transform processing unit 308 can apply the inverse DCT, inverse integer transform, inverse Karhunen-Loeve transform (KLT), inverse rotation transform, inverse direction transform, or another inverse transform to the transform coefficient block.

[0293] Furthermore, the prediction processing unit 304 generates a prediction block based on the prediction information syntax elements entropy-decoded by the entropy decoding unit 302. For example, if the prediction information syntax elements indicate that the current block is predicted inter-frame, the motion compensation unit 316 can generate the prediction block. In this case, the prediction information syntax elements may indicate a reference image in the DPB 314 from which the reference block is to be obtained, and a motion vector for identifying the position of the reference block in the reference image relative to the position of the current block in the current image. The motion compensation unit 316 can generally perform the inter-frame prediction procedure in a manner substantially similar to that described with respect to the motion compensation unit 224 (FIG. 12).

[0294] As another example, if the prediction information syntax element indicates that the current block is predicted in-frame, then the in-frame prediction unit 318 can generate a predicted block according to the in-frame prediction pattern indicated by the prediction information syntax element. Again, the in-frame prediction unit 318 can generally perform the in-frame prediction procedure in a manner substantially similar to that described with respect to the in-frame prediction unit 226 (FIG. 12). The in-frame prediction unit 318 can obtain data from the adjacent samples of the current block from the DPB 314.

[0295] The reconstruction unit 310 can reconstruct the current block using the predicted block and the residual block. For example, the reconstruction unit 310 can reconstruct the current block by adding the sample of the residual block to the corresponding sample of the predicted block.

[0296] Filter unit 312 can perform one or more filter operations on the reconstructed block. For example, filter unit 312 can perform a deblocking operation to reduce blocking artifacts along the edges of the reconstructed block. The operation of filter unit 312 is not necessarily performed in all instances.

[0297] The video decoder 300 can store the reconstructed blocks in the DPB 314. For example, in an instance where the operation of the filter unit 312 is not performed, the reconstruction unit 310 can store the reconstructed blocks in the DPB 314. In an instance where the operation of the filter unit 312 is performed, the filter unit 312 can store the filtered reconstructed blocks in the DPB 314. As discussed above, the DPB 314 can provide reference information (such as samples of the current image for in-frame prediction and samples of previously decoded images for subsequent motion compensation) to the prediction processing unit 304. Furthermore, the video decoder 300 can output decoded images (e.g., decoded video) from the DPB 314 for subsequent presentation on a display device such as the display device 118 of FIG. 1.

[0298] In this manner, video decoder 300 represents an instance of a video decoding device, which includes: a memory configured to store video data; and one or more processing units implemented in a circuit system and configured to perform mixed-frame dual prediction according to the techniques described herein.

[0299] Figure 14 is a flowchart illustrating an exemplary procedure for encoding a current block according to the technology described in this case. The current block may include the current CU. Although video encoder 200 (Figures 1 and 12) has been described, it should be understood that other devices may be configured to execute a procedure similar to that of Figure 14.

[0300] In this example, the video encoder 200 initially predicts the current block (350). For example, the video encoder 200 may form a prediction block for the current block. In some cases, the video encoder 200 may use the hybrid inter-frame double prediction described herein to predict the current block. Subsequently, the video encoder 200 may compute a residual block for the current block (352). To compute the residual block, the video encoder 200 may compute the difference between the original uncoded block and the prediction block for the current block. Subsequently, the video encoder 200 may transform the residual block and quantize the transform coefficients of the residual block (354). Next, the video encoder 200 may scan the quantized transform coefficients of the residual block (356). During or after scanning, the video encoder 200 may entropy encode the transform coefficients (358). For example, the video encoder 200 may use CAVLC or CABAC to encode the transform coefficients. Subsequently, the video encoder 200 can output the entropy-encoded data (360) of the block.

[0301] Figure 15 is a flowchart illustrating an exemplary procedure for decoding a current block of video data according to the technology described in this case. The current block may include the current CU. Although video decoder 300 (Figures 1 and 13) has been described, it should be understood that other devices may be configured to execute a procedure similar to that of Figure 15.

[0302] The video decoder 300 can receive entropy-encoded data for the current block (such as entropy-encoded prediction information and entropy-encoded data for transform coefficients of the residual block corresponding to the current block) (370). The video decoder 300 can entropy decode the entropy-encoded data to determine the prediction information for the current block and reproduce the transform coefficients of the residual block (372). The video decoder 300 can predict the current block (374), for example, by using an intra-frame or inter-frame prediction mode indicated by the prediction information of the current block to calculate a prediction block for the current block. In some examples, the video decoder 300 can use hybrid inter-frame double prediction to predict the current block, as described herein. Subsequently, the video decoder 300 can inverse scan the reproduced transform coefficients (376) to construct a block of quantized transform coefficients. Subsequently, the video decoder 300 can inverse quantize the transform coefficients and apply the inverse transform to the transform coefficients to produce a residual block (378). Ultimately, the video decoder 300 can decode the current block by combining the predicted block and the residual block (380).

[0303] Figure 16 is a flowchart illustrating an exemplary program for encoding the current block according to the technology described in this case. The current block may be, for example, the current CU. Although video encoder 200 (Figures 1 and 12) has been described, it should be understood that other devices may be configured to execute programs similar to those in Figure 16.

[0304] The video encoder 200 determines that the current block of the video data is decoded in a dual predictive frame mode (400) and determines a first motion vector for the current block (402). To signal the first motion vector, the video encoder 200 generates a first syntax element for identifying an MVP from a first candidate list of MVPs, to be included in the bitstream of the encoded video data (404). The candidate list may be, for example, an AMVP candidate list as described above, or some other type of candidate list. The video encoder 200 generates a second syntax element for identifying an MVD, to be included in the bitstream of the encoded video data (406). The MVP and MVD correspond to the first motion vector. For example, the MVD added to the MVP may be equal to the first motion vector.

[0305] The video encoder 200 determines a second motion vector for the current block from a second candidate MVP list based on bilateral matching (408). To determine the second motion vector for the current block from the second candidate MVP list based on bilateral matching, the video encoder 200 can be configured to: locate a first reference block in a first reference image using a first motion vector; locate a second reference block in a second reference image using the corresponding motion vector of each of a plurality of candidates in the second candidate motion vector predictor list; and determine the bilateral matching error between the first and second reference blocks.

[0306] To determine the second motion vector for the current block from a list of second candidate MVPs based on bilateral matching, the video encoder 200 can be configured to: use a first motion vector to locate a first reference block in a first reference image; determine the bilateral matching cost error between the first reference block and a second reference block, which is located using the corresponding motion vector of each corresponding candidate. Subsequently, the video encoder 200 can identify the candidate with the minimum bilateral matching error from a plurality of candidates and derive the second motion vector from the motion vector of the identified candidate.

[0307] The second motion vector is the same as the identified candidate motion vector, or the video encoder 200 can further refine the identified candidate motion vectors to determine the second motion vector. For example, to derive the second motion vector from the identified candidate motion vectors, the video encoder 200 can be configured to perform bilateral matching motion vector refinement as described above or perform other types of decoder-side motion vector refinement. To perform bilateral matching motion vector refinement, the video encoder 200 can be configured to: identify an initial second reference block using the identified candidate motion vectors; search for a final second reference block within a search range around the initial second reference block, the final second reference block having a lower bilateral matching error relative to the first reference block compared to the bilateral matching error generated by the initial second reference block relative to the first reference block; and determine the second motion vector based on the final second reference block.

[0308] The video encoder 200 uses a first motion vector and a second motion vector to determine the prediction block (410) for the current block. For example, the video encoder 200 may form the prediction block by averaging (possibly with weights) the first reference block and the second reference block.

[0309] The video encoder 200 encodes the current block of video data based on prediction blocks (412). In order to encode the current block of video data based on prediction blocks, the video encoder 200 can generate a bitstream of encoded video data without including explicit signaling in the bitstream of encoded video data for identifying candidates in the second candidate motion vector predictor list.

[0310] The video encoder 200 can be configured to: determine, based on the decoding mode of the current block, whether the second motion vector is determined for the current block based on bilateral matching from a second candidate MVP list, or whether the second motion vector is determined for the current block using explicit signaling. The video encoder 200 can be configured to: generate a third syntax element to be included in the bitstream of the encoded video data; and set the third syntax element to a first value. The first value for the third syntax element indicates that the second motion vector is determined for the current block based on bilateral matching from a second candidate MVP list, and the second value for the third syntax element indicates that the second motion vector is determined for the current block using explicit signaling. The video encoder 200 can determine whether to enable double prediction for a slice containing the current block, and in response to the decision to enable double prediction for a slice containing the current block, generate a third syntax element to be included in the bitstream of the encoded video data.

[0311] Figure 17 is a flowchart illustrating an exemplary procedure for decoding a current block of video data according to the technology described in this case. The current block may be, for example, the current CU. Although video decoder 300 (Figures 1 and 13) has been described, it should be understood that other devices may be configured to execute a procedure similar to that of Figure 17.

[0312] The video decoder 300 determines that the current block of the video data is decoded using a dual predictive inter-frame mode (420). The video decoder 300 receives a first syntax element (422) for identifying the MVP from a first candidate list of MVPs, and a second syntax element (424) for identifying the MVD. For example, the first candidate list may be an AMVP candidate list as described above, or it may be some other type of candidate list. The video decoder 300 determines a first motion vector (426) for the current block based on the MVP and MVD.

[0313] The video decoder 300 determines a second motion vector (428) for the current block from a second candidate MVP list based on bilateral matching. The video decoder 300 can be configured to determine the second motion vector without explicit signaling for identifying candidates in the second candidate motion vector predictor list. To determine the second motion vector for the current block from the second candidate MVP list based on bilateral matching, the video decoder 300 can be configured to: locate a first reference block in a first reference image using a first motion vector; locate a second reference block in the second reference image using the corresponding motion vector of each of the plurality of candidates in the second candidate motion vector predictor list; and determine the bilateral matching error between the first and second reference blocks. The first reference image may, for example, be from a list of first reference images.

[0314] The video decoder 300 uses a first motion vector and a second motion vector to determine the prediction block (430) for the current block. As described in more detail with respect to Figures 13 and 15, the video decoder 300 can, for example, add the prediction block to the residual block to form a reconstructed block. The video decoder 300 can apply one or more filters to the reconstructed block and output the filtered, reconstructed block as part of a decoded picture of the video data for storage and / or display.

[0315] To determine a second motion vector for the current block from a second candidate MVP list based on bilateral matching, the video decoder 300 can be configured to: locate a first reference block in a first reference image using a first motion vector; and determine the bilateral matching cost error between the first and second reference blocks for each of a plurality of candidates in a second candidate motion vector predictor list, the second reference block being located using the corresponding motion vector of each corresponding candidate. The first reference image can be from a first reference image list, and the second reference image can be from a second reference image list. The video decoder 300 can identify the candidate with the minimum bilateral matching error from the plurality of candidates and derive the second motion vector from the motion vector of the identified candidate.

[0316] The second motion vector may be the same as the identified candidate motion vector, or the video encoder 200 may further refine the identified candidate motion vectors to determine the second motion vector. For example, to derive the second motion vector from the identified candidate motion vectors, the video decoder 300 may be configured to perform bilateral matching motion vector refinement as described above, or to perform other types of decoder-side motion vector refinement. To perform bilateral matching motion vector refinement, the video decoder 300 may be configured to: identify an initial second reference block using the identified candidate motion vectors; search for a final second reference block within a search range around the initial second reference block, the final second reference block having a lower bilateral matching error relative to the first reference block compared to the bilateral matching error generated by the initial second reference block relative to the first reference block; and determine the second motion vector based on the final second reference block.

[0317] The video decoder 300 can be configured to determine, based on the decoding mode of the current block, whether the second motion vector is determined for the current block based on bilateral matching from the second candidate MVP list, or whether the second motion vector is determined for the current block using explicit signaling. The video decoder 300 can determine whether to enable double prediction for slices containing the current block, and in response to the decision to enable double prediction for slices containing the current block, receive a third syntax element.

[0318] The video decoder 300 can be configured to receive a third syntax element set to a first value. The first value for the third syntax element indicates that the second motion vector is determined for the current block based on bilateral matching from the second candidate MVP list, and the second value for the third syntax element indicates that the second motion vector is determined for the current block using explicit signaling.

[0319] The following numbered clauses describe one or more forms of the equipment and technology described in this case.

[0320] Clause 1. A method for decoding video data, the method comprising the steps of: determining that a current block of the video data is decoded in a dual predictive frame mode; determining a first motion vector for the current block based on a motion vector predictor and motion vector difference; determining a second motion vector for the current block from a list of candidate motion vectors; and using the first motion vector and the second motion vector to determine a prediction block for the current block.

[0321] Clause 2A, the method according to Clause 1A, wherein determining the second motion vector for the current block from the list of candidate motion vectors comprises: determining the second motion vector in the absence of explicit signaling for identifying a candidate in the list of candidate motion vectors.

[0322] Clause 3A, the method pursuant to Clause 1A or 2A, also includes the steps of: receiving a first syntax element for identifying the motion vector predictor; and receiving a second syntax element for defining the motion vector difference.

[0323] Clause 4A, the method according to Clause 3A, wherein receiving the second syntax element for defining the motion vector difference comprises: receiving the second syntax element for defining the motion vector difference in response to determining that one or more conditions are true, the one or more conditions including at least one of the following: the value of the first syntax element is equal to a first value, or the reference image index is equal to a second value.

[0324] Clause 5A, the method of any one of Clauses 1A-4A, wherein determining the second motion vector for the current block from the list of candidate motion vectors comprises: selecting a candidate from the list of candidate motion vectors based on bilateral matching.

[0325] Clause 6A, the method pursuant to Clauses 1A, 3A, 4A and 5A, Clauses 1A, 3A and 5A, or Clauses 1A and 5A, also includes the steps of: receiving a third syntax element; selecting the candidate from the list of candidate motion vectors based on the value of the third syntax element and bilateral matching.

[0326] Clause 7A, the method of any one of Clauses 1A-6A, wherein determining the second motion vector for the current block from the list of candidate motion vectors comprises: selecting a candidate from the list of candidate motion vectors based on template matching.

[0327] Clause 8A, the method according to any one of Clauses 1A-7A, also includes the following steps: performing merge list filling to generate a list of candidate motion vectors.

[0328] Clause 9A, the method according to any one of Clauses 1A-8A, wherein decoding includes decoding.

[0329] Clause 10A, the method pursuant to any of Clauses 1A-8A, wherein decoding includes encoding.

[0330] Clause 11A, an apparatus for decoding video data, the apparatus comprising one or more components for performing the method pursuant to any one of Clauses 1A-10A.

[0331] Clause 12A, the device pursuant to Clause 11A, wherein the one or more components include one or more processors implemented in a circuit system.

[0332] The device pursuant to any one of Clauses 11A and 12A, under Clause 13A, also includes: memory for storing the video data.

[0333] Clause 14A, and any of the devices pursuant to Clauses 11A-13A, also include: a display configured to display decoded video data.

[0334] Clause 15A, Equipment under any one of Clauses 11A-14A, wherein the equipment includes one or more of a camera, computer, mobile device, broadcast receiver equipment or set-top box.

[0335] Clause 16A, or any of Clauses 11A-15A, wherein the device includes a video decoder.

[0336] Clause 17A, the device pursuant to any of Clauses 11A-16A, wherein the device includes a video encoder.

[0337] Clause 18A, a computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to perform a method pursuant to any one of Clauses 1A-10A.

[0338] Clause 1B, a method for decoding video data, the method comprising the steps of: deriving a first predictor for a mixed inter-frame prediction procedure; deriving a second predictor for the mixed inter-frame prediction procedure based on high-level and low-level parameters; and using the mixed inter-frame prediction procedure to decode blocks of video data according to the first predictor and the second predictor.

[0339] Clause 2B, the method according to Clause 1B, wherein the advanced parameter is used for one or more of the video sequence, the set of pictures of the video sequence, pictures, slices, sub-pictures, tiles, CTUs, or predetermined areas.

[0340] Clause 3B, the method pursuant to Clause 1B, wherein the low-level parameter is used in the block, including one or more of the decoding unit or prediction unit.

[0341] Clause 4B, the method according to Clause 1B, wherein one or more of the high-level parameters or the low-level parameters indicate the matching function for deriving the second predictor.

[0342] Clause 5B, the method according to Clause 1B, wherein the advanced parameter indicates the maximum number of candidates from the candidate list.

[0343] Clause 6B, the method according to Clause 5B, wherein the second low-level parameter indicates the candidates in the candidate list to be used to derive the second predictor.

[0344] Clause 7B, the method according to Clause 1B, wherein the value of the low-level parameter indicates that a bilateral matching function is used to derive the second predictor, the method also including the steps of refining the motion vector of the second predictor, including minimizing the bilateral matching error.

[0345] Clause 8B, the method according to Clause 1B, wherein the value of the low-level parameter indicates: the template matching function is used to derive the second predictor, the method also includes the steps of: refining the motion vector of the second predictor, including minimizing the template matching error.

[0346] Clause 9B, the method pursuant to Clause 1B, also includes the following steps: refining the motion vector of the second predictor using a block-based motion vector refinement procedure.

[0347] Clause 10B, the method pursuant to Clause 1B, also includes the following steps: refining the motion vector of the second predictor using a sub-block-based motion vector refinement procedure.

[0348] Clause 11B, the method pursuant to Clause 1B, also includes the following steps: determining the activation conditions for the mixed frame dual prediction procedure based on the value of the low-level parameter.

[0349] Clause 12B, the method pursuant to any of Clauses 1B-11B, wherein decoding includes encoding.

[0350] Clause 13B, an apparatus for decoding video data, the apparatus comprising one or more components for performing the method pursuant to any one of Clauses 1B-12B.

[0351] Clause 14B, the device pursuant to Clause 13B, wherein the one or more components include one or more processors implemented in a circuit system.

[0352] The device pursuant to any one of Clauses 13B and 14B also includes: memory for storing the video data.

[0353] Clause 16B, and any of the devices pursuant to Clauses 13B-15B, also include: a display configured to display decoded video data.

[0354] Clause 17B, Equipment under any one of Clauses 13B-16B, wherein the equipment includes one or more of a camera, computer, mobile device, broadcast receiver equipment or set-top box.

[0355] Clause 18B, or any of Clauses 13B-17B, wherein the device includes a video decoder.

[0356] Clause 19B, or any of Clauses 13B-18B, wherein the device includes a video encoder.

[0357] Clause 20B, a computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to perform the method pursuant to any one of Clauses 1B-12B.

[0358] Clause 21B, a method for decoding video data, the method comprising the steps of: deriving a first predictor for a mixed inter-frame prediction procedure; deriving a second predictor for the mixed inter-frame prediction procedure based on high-level and low-level parameters; and using the mixed inter-frame prediction procedure to decode blocks of video data according to the first predictor and the second predictor.

[0359] Clause 22B, the method pursuant to Clause 21B, wherein the advanced parameter is used for one or more of the following: video sequence, a collection of images of the video sequence, images, slices, sub-images, tiles, CTUs, or a predetermined area.

[0360] Clause 23B, the method of any one of Clauses 21B-22B, wherein the low-level parameter is used in the block, including one or more of the decoding unit or prediction unit.

[0361] Clause 24B, the method of any one of Clauses 21B-23B, wherein one or more of the high-level parameter or the low-level parameter indicates the matching function for deriving the second predictor.

[0362] Clause 25B, the method of any one of Clauses 21B-23B, wherein the advanced parameter indicates the maximum number of candidates from the candidate list.

[0363] Clause 26B, the method according to Clause 25B, wherein the second low-level parameter indicates candidates in the candidate list to be used to derive the second predictor.

[0364] Clause 27B, the method according to any one of Clauses 21B-26B, wherein the value of the low-level parameter indicates that a bilateral matching function is used to derive the second predictor, the method also comprising the steps of refining the motion vector of the second predictor, including minimizing the bilateral matching error.

[0365] Clause 28B, the method according to any one of Clauses 21B-27B, wherein the value of the low-level parameter indicates that: the template matching function is used to derive the second predictor, the method also includes the steps of: refining the motion vector of the second predictor, including minimizing the template matching error.

[0366] The method of Clause 29B, or any of Clauses 21B-28B, also includes the step of refining the motion vector of the second predictor using a block-based motion vector refinement procedure.

[0367] The method of Clause 30B, or any of Clauses 21B-29B, also includes the step of refining the motion vector of the second predictor using a sub-block-based motion vector refinement procedure.

[0368] Clause 31B, or any of the methods pursuant to Clauses 21B-30B, also includes the step of: determining the enabling conditions for the mixed frame dual prediction procedure based on the value of the low-level parameter.

[0369] Clause 1C, a method for decoding video data, the method comprising the steps of: determining that the current block of the video data is decoded in a dual predictive inter-frame mode;

[0370] The method receives a first syntax element for identifying motion vector predictors from a first candidate list of motion vector predictors; receives a second syntax element for identifying motion vector differences; determines a first motion vector for the current block based on the motion vector predictor and the motion vector difference; determines a second motion vector for the current block from a second candidate list of motion vector predictors based on bilateral matching; and uses the first motion vector and the second motion vector to determine a predicted block for the current block.

[0371] Clause 2C, the method according to Clause 1C, wherein determining the second motion vector for the current block from the second candidate motion vector predictor list comprises: determining the second motion vector without receiving explicit signaling for identifying candidates in the second candidate motion vector predictor list.

[0372] Clause 3C, the method according to Clause 1C, wherein determining the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching comprises: using the first motion vector to locate a first reference block in a first reference image, wherein the first reference image is from a first reference image list; for each of a plurality of candidates in the second candidate motion vector predictor list, using the corresponding motion vector of the corresponding candidate to locate a second reference block in a second reference image, wherein the second reference image is from a second reference image list; and determining the bilateral matching error between the first reference block and the second reference block.

[0373] Clause 4C, the method according to Clause 1C, wherein determining the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching comprises: using the first motion vector to locate a first reference block in a first reference image; determining, for each of a plurality of candidates in the second candidate motion vector predictor list, a bilateral matching cost error between the first reference block and a second reference block, the second reference block being located using the corresponding motion vector of each corresponding candidate; identifying the candidate with the minimum bilateral matching error from the plurality of candidates; and deriving the second motion vector from the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates.

[0374] Clause 5C, the method according to Clause 4C, wherein deriving the second motion vector from the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates includes: performing bilateral matching motion vector refinement.

[0375] Clause 6C, the method according to Clause 5C, wherein performing bilateral matching motion vector refinement includes: identifying an initial second reference block using the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates; searching for a final second reference block within a search range around the initial second reference block, the final second reference block producing a lower bilateral matching error relative to the first reference block compared to the bilateral matching error produced by the initial second reference block relative to the first reference block; and determining the second motion vector based on the final second reference block.

[0376] Clause 7C, the method pursuant to Clause 1C, also includes the following steps: determining, based on the decoding mode of the current block, whether the second motion vector is determined for the current block based on the bilateral matching from the list of second candidate motion vector predictors, or whether the second motion vector is determined for the current block using explicit signaling.

[0377] Clause 8C, the method according to Clause 1C, also includes the steps of: receiving a third syntax element, wherein a first value for the third syntax element indicates that the second motion vector is determined for the current block based on the bilateral matching from the list of second candidate motion vector predictors, and a second value for the third syntax element indicates that the second motion vector is determined for the current block using explicit signaling; and determining that the third syntax element is set to the first value.

[0378] Clause 9C, the method pursuant to Clause 8C, also includes the following steps: determining whether to enable double prediction for a slice containing the current block; and in response to the determination to enable double prediction for the slice containing the current block, receiving the third syntax element.

[0379] Clause 10C, an apparatus for decoding video data, the apparatus comprising: a memory configured to store the video data; one or more processors implemented in a circuit system and configured to: determine that a current block of the video data is decoded in a dual predictive inter-frame mode; receive a first syntax element for identifying motion vector predictors from a first candidate list of motion vector predictors; receive a second syntax element for identifying motion vector differences; determine a first motion vector for the current block based on the motion vector predictors and the motion vector differences; determine a second motion vector for the current block from a second candidate list of motion vector predictors based on bilateral matching; and determine a prediction block for the current block using the first motion vector and the second motion vector.

[0380] According to Clause 11C, the device pursuant to Clause 10C, wherein in order to determine the second motion vector for the current block from the list of second candidate motion vector predictors, the one or more processors are also configured to determine the second motion vector without receiving explicit signaling for identifying candidates in the list of second candidate motion vector predictors.

[0381] According to Clause 12C and the apparatus pursuant to Clause 10C, in order to determine the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching, the one or more processors are also configured to: use the first motion vector to locate a first reference block in a first reference image, wherein the first reference image is from a first reference image list; for each of the plurality of candidates in the second candidate motion vector predictor list, use the corresponding motion vector of the corresponding candidate to locate a second reference block in a second reference image, wherein the second reference image is from a second reference image list; and determine the bilateral matching error between the first reference block and the second reference block.

[0382] According to Clause 13C and the apparatus pursuant to Clause 10C, in order to determine the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching, the one or more processors are also configured to: use the first motion vector to locate a first reference block in a first reference image; determine, for each of a plurality of candidates in the second candidate motion vector predictor list, the bilateral matching cost error between the first reference block and the second reference block, the second reference block being located using the corresponding motion vector of each corresponding candidate; identify the candidate with the minimum bilateral matching error from the plurality of candidates; and derive the second motion vector from the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates.

[0383] According to Clause 14C, the apparatus pursuant to Clause 13C, wherein in order to derive the second motion vector from the motion vector of the candidate having the minimum bilateral matching error from the plurality of candidates, the one or more processors are also configured to perform bilateral matching motion vector refinement.

[0384] According to Clause 15C and the apparatus pursuant to Clause 14C, in order to perform bilateral matching motion vector refinement, the one or more processors are also configured to: identify an initial second reference block using the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates; search for a final second reference block within a search range around the initial second reference block, the final second reference block having a lower bilateral matching error relative to the first reference block compared to the bilateral matching error generated by the initial second reference block relative to the first reference block; and determine the second motion vector based on the final second reference block.

[0385] The apparatus of Clause 16C and Clause 10C, wherein the one or more processors are also configured to: determine, based on the decoding mode of the current block, whether the second motion vector is determined for the current block based on the bilateral matching from the list of second candidate motion vector predictors, or whether the second motion vector is determined for the current block using explicit signaling.

[0386] Clause 17C, the apparatus pursuant to Clause 10C, wherein the one or more processors are also configured to: receive a third syntax element, wherein a first value for the third syntax element indicates that the second motion vector is determined for the current block based on the bilateral matching from the list of second candidate motion vector predictors, and a second value for the third syntax element indicates that the second motion vector is determined for the current block using explicit signaling; and determine that the third syntax element is set to the first value.

[0387] The apparatus of Clause 18C and Clause 17C, wherein the one or more processors are also configured to: determine whether to enable double prediction for a slice containing the current block; and in response to the decision to enable double prediction for the slice containing the current block, receive the third syntax element.

[0388] Clause 19C, equipment pursuant to Clause 10C, wherein the equipment includes wireless communication equipment and also includes a receiver configured to receive encoded video data.

[0389] The device pursuant to Clause 20C and Clause 19C, wherein the wireless communication device includes a telephone mobile phone, and wherein the receiver is configured to demodulate a signal including the encoded video data in accordance with a wireless communication standard.

[0390] The equipment pursuant to Clause 21C and Clause 10C also includes: a display configured to display decoded video data.

[0391] Clause 22C, Equipment under Clause 10C, wherein the equipment includes one or more of a camera, computer, mobile device, broadcast receiver equipment or set-top box.

[0392] Clause 23C. An apparatus for encoding video data, the apparatus comprising: a memory configured to store the video data; one or more processors implemented in a circuit system and configured to: determine that a current block of the video data is decoded in a double predictive frame mode; determine a first motion vector for the current block; generate a first syntax element for identifying motion vector predictors from a first candidate list of motion vector predictors to be included in a bitstream of encoded video data; generate a second syntax element for identifying a motion vector difference to be included in the bitstream of encoded video data, wherein the motion vector predictor and the motion vector difference correspond to the first motion vector; determine a second motion vector for the current block from a second candidate list of motion vector predictors based on bilateral matching; determine a prediction block for the current block using the first motion vector and the second motion vector; and encode the current block of the video data based on the prediction block.

[0393] According to Clause 24C and Clause 23C, in order to encode the current block of video data based on the prediction block, the one or more processors are also configured to: generate a bitstream of the encoded video data without including explicit signaling in the bitstream of the encoded video data for identifying candidates in the second candidate motion vector predictor list.

[0394] According to Clause 25C and the apparatus pursuant to Clause 23C, in order to determine the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching, the one or more processors are also configured to: use the first motion vector to locate a first reference block in a first reference image, wherein the first reference image is from a first reference image list; for each of the plurality of candidates in the second candidate motion vector predictor list, use the corresponding motion vector of the corresponding candidate to locate a second reference block in a second reference image, wherein the second reference image is from a second reference image list; and determine the bilateral matching error between the first reference block and the second reference block.

[0395] According to Clause 26C and the apparatus of Clause 23C, in order to determine the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching, the one or more processors are also configured to: use the first motion vector to locate a first reference block in a first reference image; determine the bilateral matching cost error between the first reference block and the second reference block for each of a plurality of candidates in the second candidate motion vector predictor list, the second reference block being located using the corresponding motion vector of each corresponding candidate; identify the candidate with the minimum bilateral matching error from the plurality of candidates; and derive the second motion vector from the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates.

[0396] In accordance with Clause 27C and Clause 26C, the device wherein, in order to derive the second motion vector from the motion vector of the candidate having the minimum bilateral matching error from the plurality of candidates, the one or more processors are also configured to perform bilateral matching motion vector refinement.

[0397] According to Clause 28C and the apparatus pursuant to Clause 27C, in order to perform bilateral matching motion vector refinement, the one or more processors are also configured to: identify an initial second reference block using the motion vectors of the candidate with the minimum bilateral matching error from the plurality of candidates; search for a final second reference block within a search range around the initial second reference block, the final second reference block having a lower bilateral matching error relative to the first reference block compared to the bilateral matching error generated by the initial second reference block relative to the first reference block; and determine the second motion vector based on the final second reference block.

[0398] The apparatus of Clause 29C and Clause 23C, wherein the one or more processors are also configured to: determine, based on the decoding mode of the current block, whether the second motion vector is determined for the current block based on the bilateral matching from the list of second candidate motion vector predictors, or whether the second motion vector is determined for the current block using explicit signaling.

[0399] The apparatus of Clause 30C and Clause 23C, wherein the one or more processors are also configured to: generate a third syntax element to be included in the bitstream of the encoded video data, wherein a first value for the third syntax element indicates that the second motion vector is determined for the current block based on the bilateral matching from the list of second candidate motion vector predictors, and a second value for the third syntax element indicates that the second motion vector is determined for the current block using explicit signaling; and set the third syntax element to the first value.

[0400] The apparatus of Clause 31C and Clause 30C, wherein the one or more processors are also configured to: determine whether to enable double prediction for a slice containing the current block; and in response to the decision to enable the double prediction for the slice containing the current block, generate the third syntax element to be included in the bitstream of the encoded video data.

[0401] The equipment under Clause 32C and Clause 23C includes wireless communication equipment and transmitters configured to transmit encoded video data.

[0402] The device pursuant to Clause 33C and Clause 32C, wherein the wireless communication device includes a telephone mobile phone, and wherein the transmitter is configured to modulate a signal including the encoded video data in accordance with a wireless communication standard.

[0403] The equipment covered by Clause 34C and Clause 23C also includes: a camera configured to capture video data.

[0404] The equipment under Clause 35C and Clause 23C includes one or more of a camera, computer, mobile device, broadcast receiver equipment or set-top box.

[0405] Clause 36C. A computer-readable storage medium storing instructions, which, when executed by one or more processors, cause the one or more processors to: determine that a current block of video data is decoded in a dual predictive frame mode; receive a first syntax element for identifying motion vector predictors from a first candidate list of motion vector predictors; receive a second syntax element for identifying motion vector differences; determine a first motion vector for the current block based on the motion vector predictors and the motion vector differences; determine a second motion vector for the current block from a second candidate list of motion vector predictors based on bilateral matching; and determine a prediction block for the current block using the first motion vector and the second motion vector.

[0406] Clause 37C, computer-readable storage media pursuant to Clause 36C, wherein, in order to determine the second motion vector for the current block from the second candidate motion vector predictor list, the instructions cause one or more processors to determine the second motion vector without receiving explicit signaling for identifying candidates in the second candidate motion vector predictor list.

[0407] According to Clause 38C and the computer-readable storage medium pursuant to Clause 36C, in order to determine the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching, the instructions cause one or more processors to perform the following operations: using the first motion vector to locate a first reference block in a first reference image, wherein the first reference image is from a first reference image list; for each of a plurality of candidates in the second candidate motion vector predictor list, using the corresponding motion vector of the corresponding candidate to locate a second reference block in a second reference image, wherein the second reference image is from a second reference image list; and determining the bilateral matching error between the first reference block and the second reference block.

[0408] According to Clause 39C and the computer-readable storage medium pursuant to Clause 36C, in order to determine the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching, the instructions cause one or more processors to perform the following operations: using the first motion vector to locate a first reference block in a first reference image; determining, for each of a plurality of candidates in the second candidate motion vector predictor list, the bilateral matching cost error between the first reference block and the second reference block, the second reference block being located using the corresponding motion vector of each corresponding candidate; identifying the candidate with the minimum bilateral matching error from the plurality of candidates; and deriving the second motion vector from the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates.

[0409] According to Clause 40C and Clause 39C, a computer-readable storage medium wherein, in order to derive the second motion vector from the motion vector of the candidate having the minimum bilateral matching error from the plurality of candidates, the instructions cause one or more processors to perform bilateral matching motion vector refinement.

[0410] Clause 41C, a computer-readable storage medium pursuant to Clause 40C, wherein, in order to perform bilateral matching motion vector refinement, the instructions cause one or more processors to: identify an initial second reference block using the motion vectors of the candidate having the minimum bilateral matching error from the plurality of candidates; search for a final second reference block within a search range around the initial second reference block, the final second reference block having a lower bilateral matching error relative to the first reference block compared to the bilateral matching error generated by the initial second reference block relative to the first reference block; and determine the second motion vector based on the final second reference block.

[0411] Clause 42C, a computer-readable storage medium pursuant to Clause 36C, stores additional instructions that, when executed, cause one or more processors to: determine, based on the decoding mode of the current block, whether the second motion vector is determined for the current block based on the bilateral matching from the list of second candidate motion vector predictors, or whether the second motion vector is determined for the current block using explicit signaling.

[0412] Clause 43C, a computer-readable storage medium pursuant to Clause 36C, stores additional instructions that, when executed, cause one or more processors to: receive a third syntax element, wherein a first value for the third syntax element indicates that the second motion vector is determined for the current block based on the bilateral matching from the list of second candidate motion vector predictors, and a second value for the third syntax element indicates that the second motion vector is determined for the current block using explicit signaling; and determine that the third syntax element is set to the first value.

[0413] Clause 44C, a computer-readable storage medium pursuant to Clause 43C, stores additional instructions that, when executed, cause one or more processors to: determine whether to enable double prediction for a slice containing the current block; and, in response to the decision to enable double prediction for the slice containing the current block, receive the third syntax element.

[0414] Clause 1D, a method for decoding video data, the method comprising the steps of: determining that a current block of the video data is decoded in a dual predictive frame mode; receiving a first syntax element for identifying motion vector predictors from a first candidate list of motion vector predictors; receiving a second syntax element for identifying motion vector differences; determining a first motion vector for the current block based on the motion vector predictors and the motion vector differences; determining a second motion vector for the current block from a second candidate list of motion vector predictors based on bilateral matching; and using the first motion vector and the second motion vector to determine a prediction block for the current block.

[0415] Clause 2D, the method according to Clause 1D, wherein determining the second motion vector for the current block from the second candidate motion vector predictor list comprises: determining the second motion vector without receiving explicit signaling for identifying candidates in the second candidate motion vector predictor list.

[0416] According to Clause 3D, or the method of Clause 1D or 2D, determining the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching includes: using the first motion vector to locate a first reference block in a first reference image, wherein the first reference image is from a first reference image list; for each of a plurality of candidates in the second candidate motion vector predictor list, using the corresponding motion vector of the corresponding candidate to locate a second reference block in a second reference image, wherein the second reference image is from a second reference image list; and determining the bilateral matching error between the first reference block and the second reference block.

[0417] Clause 4D, the method according to Clause 1D or 2D, wherein determining the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching comprises: using the first motion vector to locate a first reference block in a first reference image; determining, for each of a plurality of candidates in the second candidate motion vector predictor list, a bilateral matching cost error between the first reference block and the second reference block, the second reference block being located using the corresponding motion vector of each corresponding candidate; identifying the candidate with the minimum bilateral matching error from the plurality of candidates; and deriving the second motion vector from the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates.

[0418] Clause 5D, the method according to Clause 4D, wherein deriving the second motion vector from the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates includes: performing bilateral matching motion vector refinement.

[0419] Clause 6D, the method according to Clause 5D, wherein performing bilateral matching motion vector refinement includes: identifying an initial second reference block using the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates; searching for a final second reference block within a search range around the initial second reference block, the final second reference block having a lower bilateral matching error relative to the first reference block compared to the bilateral matching error generated by the initial second reference block relative to the first reference block; and determining the second motion vector based on the final second reference block.

[0420] The method of Clause 7D, or any of Clauses 1D-6D, also includes the following steps: determining, based on the decoding mode of the current block, whether the second motion vector is determined for the current block based on the bilateral matching from the list of second candidate motion vector predictors, or whether the second motion vector is determined for the current block using explicit signaling.

[0421] The method under Clause 8D, or any of Clauses 1D-7D, also includes the steps of: receiving a third syntax element, wherein a first value for the third syntax element indicates that the second motion vector is determined for the current block based on the bilateral matching from the list of second candidate motion vector predictors, and a second value for the third syntax element indicates that the second motion vector is determined for the current block using explicit signaling; and determining that the third syntax element is set to the first value.

[0422] Clause 9D, the method pursuant to Clause 8D, also includes the following steps: determining whether to enable double prediction for the slice containing the current block; and in response to the determination to enable double prediction for the slice containing the current block, receiving the third syntax element.

[0423] Clause 10D, an apparatus for decoding video data, the apparatus comprising: a memory configured to store the video data; and one or more processors implemented in a circuit system and configured to: determine that a current block of the video data is decoded in a dual predictive frame mode; receive a first syntax element for identifying motion vector predictors from a first candidate list of motion vector predictors; receive a second syntax element for identifying motion vector differences; determine a first motion vector for the current block based on the motion vector predictors and the motion vector differences; determine a second motion vector for the current block from a second candidate list of motion vector predictors based on bilateral matching; and determine a prediction block for the current block using the first motion vector and the second motion vector.

[0424] According to Clause 11D, the device pursuant to Clause 10D, wherein in order to determine the second motion vector for the current block from the list of second candidate motion vector predictors, the one or more processors are also configured to determine the second motion vector without receiving explicit signaling for identifying candidates in the list of second candidate motion vector predictors.

[0425] According to Clause 12D, or the apparatus pursuant to Clause 10D or 11D, in order to determine the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching, the one or more processors are also configured to: use the first motion vector to locate a first reference block in a first reference image, wherein the first reference image is from a first reference image list; for each of a plurality of candidates in the second candidate motion vector predictor list, use the corresponding motion vector of the corresponding candidate to locate a second reference block in a second reference image, wherein the second reference image is from a second reference image list; and determine the bilateral matching error between the first reference block and the second reference block.

[0426] According to Clause 13D, or the apparatus pursuant to Clause 10D or 11D, in order to determine the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching, the one or more processors are also configured to: use the first motion vector to locate a first reference block in a first reference image; determine the bilateral matching cost error between the first reference block and the second reference block for each of a plurality of candidates in the second candidate motion vector predictor list, the second reference block being located using the corresponding motion vector of each corresponding candidate; identify the candidate with the minimum bilateral matching error from the plurality of candidates; and derive the second motion vector from the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates.

[0427] According to Clause 14D, the apparatus pursuant to Clause 13D, wherein in order to derive the second motion vector from the motion vector of the candidate having the minimum bilateral matching error from the plurality of candidates, the one or more processors are also configured to perform bilateral matching motion vector refinement.

[0428] According to Clause 15D and the apparatus pursuant to Clause 14D, in order to perform bilateral matching motion vector refinement, the one or more processors are also configured to: identify an initial second reference block using the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates; search for a final second reference block within a search range around the initial second reference block, the final second reference block having a lower bilateral matching error relative to the first reference block compared to the bilateral matching error generated by the initial second reference block relative to the first reference block; and determine the second motion vector based on the final second reference block.

[0429] The device under Clause 16D, or any of Clauses 10D-15D, wherein the one or more processors are also configured to: determine, based on the decoding mode of the current block, whether the second motion vector is determined for the current block based on the bilateral matching from the list of second candidate motion vector predictors, or whether the second motion vector is determined for the current block using explicit signaling.

[0430] The device pursuant to any one of Clauses 10D-16D, under Clause 17D, wherein the one or more processors are also configured to: receive a third syntax element, wherein a first value for the third syntax element indicates that the second motion vector is determined for the current block based on the bilateral matching from the list of second candidate motion vector predictors, and a second value for the third syntax element indicates that the second motion vector is determined for the current block using explicit signaling; and determine that the third syntax element is set to the first value.

[0431] The apparatus of Clause 18D, pursuant to Clause 17D, wherein the one or more processors are also configured to: determine whether to enable double prediction for a slice containing the current block; and in response to the decision to enable double prediction for the slice containing the current block, to receive the third syntax element.

[0432] The device under Clause 19D, or any of Clauses 10D-18D, wherein the device includes wireless communication equipment and also includes a receiver configured to receive encoded video data.

[0433] The device pursuant to Clause 20D and Clause 19D, wherein the wireless communication device includes a telephone mobile phone, and wherein the receiver is configured to demodulate a signal including the encoded video data in accordance with a wireless communication standard.

[0434] The equipment pursuant to any one of Clauses 10D-20D, under Clause 21D, also includes: a display configured to display decoded video data.

[0435] Clause 22D, or any of Clauses 10D-21D, wherein the device includes one or more of a camera, computer, mobile device, broadcast receiver device or set-top box.

[0436] Clause 23D, an apparatus for encoding video data, the apparatus comprising: a memory configured to store the video data; one or more processors implemented in a circuit system and configured to: determine that a current block of the video data is decoded in a dual predictive frame mode; determine a first motion vector for the current block; generate a first syntax element for identifying motion vector predictors from a first candidate list of motion vector predictors to be included in a bitstream of encoded video data; generate a second syntax element for identifying a motion vector difference to be included in the bitstream of encoded video data, wherein the motion vector predictor and the motion vector difference correspond to the first motion vector; determine a second motion vector for the current block from a second candidate list of motion vector predictors based on bilateral matching; determine a prediction block for the current block using the first motion vector and the second motion vector; and encode the current block of the video data based on the prediction block.

[0437] According to Clause 24D and Clause 23D, in order to encode the current block of video data based on the prediction block, the one or more processors are also configured to: generate a bitstream of the encoded video data without including explicit signaling in the bitstream of the encoded video data for identifying candidates in the second candidate motion vector predictor list.

[0438] According to Clause 25D, or the apparatus pursuant to Clause 23D or 24D, in order to determine the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching, the one or more processors are also configured to: use the first motion vector to locate a first reference block in a first reference image, wherein the first reference image is from a first reference image list; for each of a plurality of candidates in the second candidate motion vector predictor list, use the corresponding motion vector of the corresponding candidate to locate a second reference block in a second reference image, wherein the second reference image is from a second reference image list; and determine the bilateral matching error between the first reference block and the second reference block.

[0439] According to Clause 26D, or the apparatus pursuant to Clause 23D or 24D, wherein, in order to determine the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching, the one or more processors are also configured to: use the first motion vector to locate a first reference block in a first reference image; determine, for each of a plurality of candidates in the second candidate motion vector predictor list, the bilateral matching cost error between the first reference block and the second reference block, the second reference block being located using the corresponding motion vector of each corresponding candidate; identify the candidate with the minimum bilateral matching error from the plurality of candidates; and derive the second motion vector from the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates.

[0440] In accordance with Clause 27D and Clause 26D, the device wherein, in order to derive the second motion vector from the motion vector of the candidate having the minimum bilateral matching error from the plurality of candidates, the one or more processors are also configured to perform bilateral matching motion vector refinement.

[0441] According to Clause 28D and the apparatus pursuant to Clause 27D, in order to perform bilateral matching motion vector refinement, the one or more processors are also configured to: identify an initial second reference block using the motion vectors of the candidate with the minimum bilateral matching error from the plurality of candidates; search for a final second reference block within a search range around the initial second reference block, the final second reference block having a lower bilateral matching error relative to the first reference block compared to the bilateral matching error generated by the initial second reference block relative to the first reference block; and determine the second motion vector based on the final second reference block.

[0442] The device under Clause 29D, or any of Clauses 23D-28D, wherein the one or more processors are also configured to: determine, based on the decoding mode of the current block, whether the second motion vector is determined for the current block based on the bilateral matching from the list of second candidate motion vector predictors, or whether the second motion vector is determined for the current block using explicit signaling.

[0443] The apparatus of any one of Clauses 30D or 23D-29D, wherein the one or more processors are also configured to: generate a third syntax element to be included in the bitstream of the encoded video data, wherein a first value for the third syntax element indicates that the second motion vector is determined for the current block based on the bilateral matching from the list of second candidate motion vector predictors, and a second value for the third syntax element indicates that the second motion vector is determined for the current block using explicit signaling; and set the third syntax element to the first value.

[0444] The device under Clause 31D and Clause 30D, wherein the one or more processors are also configured to: determine whether to enable double prediction for a slice containing the current block; and in response to the decision to enable the double prediction for the slice containing the current block, generate the third syntax element to be included in the bitstream of the encoded video data.

[0445] The device under Clause 32D, or any of Clauses 23D-31D, wherein the device includes wireless communication equipment and also includes a transmitter configured to transmit encoded video data.

[0446] The device under Clause 33D and Clause 32D, wherein the wireless communication device includes a telephone mobile phone, and wherein the transmitter is configured to modulate a signal including the encoded video data in accordance with a wireless communication standard.

[0447] The equipment covered by Clause 34D and any of Clauses 23D-33D also includes: a camera configured to capture video data.

[0448] The device under Clause 35D, or any of Clauses 23D-34D, wherein the device includes one or more of a camera, computer, mobile device, broadcast receiver device or set-top box.

[0449] Clause 36D, a computer-readable storage medium storing instructions, which, when executed by one or more processors, cause the one or more processors to: determine that a current block of video data is decoded in a dual predictive frame mode; receive a first syntax element for identifying motion vector predictors from a first candidate list of motion vector predictors; receive a second syntax element for identifying motion vector differences; determine a first motion vector for the current block based on the motion vector predictors and the motion vector differences; determine a second motion vector for the current block from a second candidate list of motion vector predictors based on bilateral matching; and determine a prediction block for the current block using the first motion vector and the second motion vector.

[0450] Clause 37D, computer-readable storage media pursuant to Clause 36D, wherein, in order to determine the second motion vector for the current block from the second candidate motion vector predictor list, the instructions cause one or more processors to determine the second motion vector without receiving explicit signaling for identifying candidates in the second candidate motion vector predictor list.

[0451] According to Clause 38D, or a computer-readable storage medium pursuant to Clauses 36D or 37D, in order to determine the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching, the instructions cause one or more processors to: locate a first reference block in a first reference image using the first motion vector, wherein the first reference image is from a first reference image list; locate a second reference block in a second reference image using the corresponding motion vector of each of a plurality of candidates in the second candidate motion vector predictor list, wherein the second reference image is from a second reference image list; and determine the bilateral matching error between the first reference block and the second reference block.

[0452] According to Clause 39D, or a computer-readable storage medium pursuant to Clauses 36D or 37D, in order to determine the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching, the instructions cause one or more processors to perform the following operations: using the first motion vector to locate a first reference block in a first reference image; determining, for each of a plurality of candidates in the second candidate motion vector predictor list, the bilateral matching cost error between the first reference block and the second reference block, the second reference block being located using the corresponding motion vector of each corresponding candidate; identifying the candidate with the minimum bilateral matching error from the plurality of candidates; and deriving the second motion vector from the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates.

[0453] According to Clause 40D and Clause 39D, a computer-readable storage medium wherein, in order to derive the second motion vector from the motion vector of the candidate having the minimum bilateral matching error from the plurality of candidates, the instructions cause one or more processors to perform bilateral matching motion vector refinement.

[0454] Clause 41D, a computer-readable storage medium pursuant to Clause 40D, wherein, in order to perform bilateral matching motion vector refinement, the instructions cause one or more processors to: identify an initial second reference block using the motion vectors of the candidate with the minimum bilateral matching error from the plurality of candidates; search for a final second reference block within a search range around the initial second reference block, the final second reference block having a lower bilateral matching error relative to the first reference block compared to the bilateral matching error generated by the initial second reference block relative to the first reference block; and determine the second motion vector based on the final second reference block.

[0455] Clause 42D, or any of Clauses 36D-41D, a computer-readable storage medium storing additional instructions that, when executed, cause one or more processors to: determine, based on the decoding mode of the current block, whether the second motion vector is determined for the current block based on the bilateral matching from the list of second candidate motion vector predictors, or whether the second motion vector is determined for the current block using explicit signaling.

[0456] A computer-readable storage medium, pursuant to any one of Clauses 43D or 36D-42D, stores additional instructions that, when executed, cause one or more processors to: receive a third syntax element, wherein a first value for the third syntax element indicates that the second motion vector was determined for the current block based on the bilateral matching from the list of second candidate motion vector predictors, and a second value for the third syntax element indicates that the second motion vector was determined for the current block using explicit signaling; and determine that the third syntax element is set to the first value.

[0457] Clause 44D, a computer-readable storage medium pursuant to Clause 43D, stores additional instructions that, when executed, cause one or more processors to: determine whether to enable double prediction for a slice containing the current block; and, in response to the decision to enable double prediction for the slice containing the current block, receive the third syntax element.

[0458] It should be recognized that, depending on the instance, certain actions or events of any technique described herein may be performed in a different order, or may be added, combined, or omitted entirely (e.g., not all described actions or events are necessary for implementing such techniques). Furthermore, in some instances, actions or events may be performed concurrently rather than sequentially, for example via multithreading, interrupt handling, or multiple processors.

[0459] In one or more instances, the described functionality may be implemented using hardware, software, firmware, or any combination thereof. If implemented in software, such functionality may be stored as one or more instructions or code on or transmitted via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium, which corresponds to tangible media such as data storage media or communication media, including, for example, any media that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this manner, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to obtain instructions, code, and / or data structures for implementing the techniques described herein. A computer program product may include a computer-readable medium.

[0460] For example, and not as a limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other media capable of storing desired program code in the form of instructions or data structures, and any other media accessible by a computer. Furthermore, any connection is appropriately referred to as computer-readable media. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (e.g., infrared, radio, and microwave), then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology (e.g., infrared, radio, and microwave) is included in the definition of media. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather refer instead to non-transient tangible storage media. As used herein, magnetic disks and optical disks include CDs, laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs, where magnetic disks typically copy data magnetically, while optical discs use lasers to copy data optically. Combinations of the above should also be included within the scope of computer-readable media.

[0461] Instructions can be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or individual logic circuit systems. Therefore, the terms "processor" and "processing circuit system" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some cases, the functions described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined transcoder. Furthermore, these techniques can be implemented entirely within one or more circuit or logic elements.

[0462] The technology described herein can be implemented in a wide variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various elements, modules, or units are described in this description to emphasize the functional characteristics of devices configured to perform the disclosed technology, but they do not necessarily need to be implemented via different hardware units. Specifically, as mentioned above, various units can be combined in a transcoder hardware unit, or provided by a collection of interoperable hardware units (including one or more processors as described above) combined with appropriate software and / or firmware.

[0463] The individual instances have been described. These and other instances fall within the scope of the attached request item.

[0464] 100: Video Encoding and Decoding System 102: Source device 104: Video Source 106: Memory 108: Output Interface 110: Computer-readable media 112: Storage device 114: Archive Server 116: Destination Equipment 118: Display device 120: Memory 122: Input Interface 144: Block 146: Motion Vector Scaling Procedure 180: Current Template 182: Current image 184: Reference Template 186: Reference Image 188: Block 190: Block 192: Reference Block 193: Fixed Template 194: Current block 195: Initial Second Reference Block 196: Second Reference Block Candidate 197: Final Second Reference Block 200: Video Encoder 202: Mode Selection Unit 204: Residual Generation Unit 206: Transformation Processing Unit 208: Quantization unit 210: Inverse quantization unit 212: Inverse Transform Processing Unit 214: Reconfiguration Unit 216: Filter Unit 218:DPB 220: Entropy Coding Unit 222: Motion Estimation Unit 224: Motion Compensation Unit 226: In-frame prediction unit 230: Video Data Memory 300: Video Decoder 302: Entropy Decoding Unit 304: Predictive Processing Unit 306: Inverse quantization unit 308: Inverse Transformation Processing Unit 310: Reconfiguration Unit 312: Filter Unit 314:DPB 316: Motion Compensation Unit 318: In-frame prediction unit 320: CPB Memory 350: Steps 352: Steps 354: Steps 356: Steps 358: Steps 360: Steps 370: Steps 372: Steps 374: Steps 376: Steps 378: Steps 380: Steps 400: Steps 402: Steps 404: Steps 406: Steps 408: Steps 410: Steps 412: Steps 420: Steps 422: Steps 424: Steps 426: Steps 428: Steps 430: Steps MV0: Initial motion vector MV0': Final motion vector MV1: Initial motion vector MV1': Final motion vector PU0: Specific PU T: Block TD0: Time Distance TD1: Time Distance TMVP: Time Motion Vector Predictor

[0465] Domestic storage information (please note in order of storage institution, date, and number) none Overseas storage information (please note in the order of storage country, institution, date, and number) none

Claims

1. A method for decoding video data, the method comprising the steps of: determining that a current block of the video data is decoded in a bidirectional predictive frame mode; receiving a first syntax element, wherein the first syntax element includes a flag having one of only two values, wherein a first value of the two values ​​indicates that a first motion vector system for a bidirectional predictive block is to be received in a bitstream of the video data, and a second motion vector system for the bidirectional predictive block is to be received in the bitstream of the video data, and wherein a second value of the two values ​​indicates that the first motion vector system for the bidirectional predictive block is to be received in the bitstream of the video data, and the second motion vector system for the bidirectional predictive block is determined not to receive explicit signaling in the bitstream of the video data; receiving a second syntax element for identifying a motion vector predictor from a first candidate list of motion vector predictors; receiving a third syntax element for identifying a motion vector difference; A first motion vector for the current block is determined based on the motion vector predictor and the motion vector difference; in response to the first syntax having the second value, a second motion vector for the current block is determined from a second list of candidate motion vector predictors based on bilateral matching; and a predicted block for the current block is determined using the first motion vector for the current block and the second motion vector for the current block.

2. The method according to claim 1, wherein the step of determining the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching includes the following steps: using the first motion vector for the current block to locate a first reference block in a first reference image, wherein the first reference image is from a first reference image list; for each of a plurality of candidates in the second candidate motion vector predictor list, using a corresponding motion vector of the corresponding candidate to locate a second reference block in a second reference image, wherein the second reference image is from a second reference image list; and determining a bilateral matching error between the first reference block and the second reference block.

3. The method according to claim 1, wherein the step of determining the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching includes the following steps: using the first motion vector for the current block to locate a first reference block in a first reference image; determining a bilateral matching cost error between the first reference block and a second reference block for each of a plurality of candidates in the second candidate motion vector predictor list, the second reference block being located using a corresponding motion vector of each corresponding candidate; identifying a candidate with a minimum bilateral matching error from the plurality of candidates; and deriving the second motion vector for the current block from the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates.

4. The method according to request 3, wherein the step of deriving the second motion vector for the current block from the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates includes the following steps: performing bilateral matching motion vector refinement.

5. The method according to claim 4, wherein the step of performing the bilateral matching motion vector refinement includes the following steps: identifying an initial second reference block using the motion vectors of the candidate with the minimum bilateral matching error from the plurality of candidates; searching for a final second reference block within a search range around the initial second reference block, the final second reference block producing a lower bilateral matching error relative to the first reference block compared to the bilateral matching error produced by the initial second reference block relative to the first reference block; and determining the second motion vector for the current block based on the final second reference block.

6. The method according to request item 1 also includes the following steps: determining whether to enable double prediction for all slices containing the current block; and in response to the determination to enable double prediction for the slice containing the current block, receiving the first syntax element.

7. An apparatus for decoding video data, the apparatus comprising: A memory configured to store the video data; and one or more processors implemented in a circuit system and configured to: determine that a current block of the video data is decoded in a bidirectional predictive frame mode; receive a first syntax element, wherein the first syntax element includes a flag having one of only two values, wherein a first value of the two values ​​indicates that a first motion vector system for a bidirectional predictive block is to be received in a bitstream of the video data, and a second motion vector system for the bidirectional predictive block is to be received in the bitstream of the video data, and wherein a second value of the two values ​​indicates that the first motion vector system for the bidirectional predictive block is to be received in the bitstream of the video data, and the second motion vector system for the bidirectional predictive block is determined not to receive explicit signaling in the bitstream of the video data; receive a second syntax element for identifying a motion vector predictor from a first candidate list of motion vector predictors; and receive a third syntax element for identifying a motion vector difference. A first motion vector for the current block is determined based on the motion vector predictor and the motion vector difference; in response to the first syntax having the second value, a second motion vector for the current block is determined from a second list of candidate motion vector predictors based on bilateral matching; and a predicted block for the current block is determined using the first motion vector for the current block and the second motion vector for the current block.

8. The device according to request 7, wherein, in order to determine the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching, the one or more processors are also configured to: locate a first reference block in a first reference image using the first motion vector for the current block, wherein the first reference image is from a first reference image list; locate a second reference block in a second reference image using a corresponding motion vector of the corresponding candidate for each of the plurality of candidates in the second candidate motion vector predictor list, wherein the second reference image is from a second reference image list; and determine a bilateral matching error between the first reference block and the second reference block.

9. The device according to claim 7, wherein, in order to determine the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching, the one or more processors are also configured to: locate a first reference block in a first reference image using the first motion vector for the current block; determine a bilateral matching cost error between the first reference block and a second reference block for each of a plurality of candidates in the second candidate motion vector predictor list, the second reference block being located using a corresponding motion vector of each corresponding candidate; identify a candidate with a minimum bilateral matching error from the plurality of candidates; and derive the second motion vector for the current block from the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates.

10. The device according to request 9, wherein, in order to derive the second motion vector for the current block from the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates, the one or more processors are also configured to perform bilateral matching motion vector refinement.

11. The device according to claim 10, wherein, in order to perform the bilateral matching motion vector refinement, the one or more processors are also configured to: identify an initial second reference block using the motion vectors of the candidate with the minimum bilateral matching error from the plurality of candidates; search for a final second reference block within a search range around the initial second reference block, the final second reference block producing a lower bilateral matching error relative to the first reference block compared to the bilateral matching error produced by the initial second reference block relative to the first reference block; and determine the second motion vector for the current block based on the final second reference block.

12. The device according to request item 7, wherein the one or more processors are also configured to: determine whether to enable double prediction for all slices containing the current block; and in response to the decision to enable the double prediction for the slice containing the current block, receive the first syntax element.

13. The device according to claim 7, wherein the device includes a wireless communication device and a receiver configured to receive encoded video data.

14. The device according to claim 13, wherein the wireless communication device includes a telephone, and wherein the receiver is configured to demodulate a signal including the encoded video data according to a wireless communication standard.

15. The equipment according to claim 7 also includes: A display configured to display decoded video data.

16. The device according to claim 7, wherein the device includes one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

17. An apparatus for encoding video data, the apparatus comprising: A memory configured to store the video data; one or more processors implemented in a circuit system and configured to: determine that a current block of the video data is decoded in a bi-predictive inter-frame mode; determine a first motion vector for the current block; generate a first syntax element for identifying a motion vector predictor from a first candidate list of motion vector predictors, to be included in a bitstream of the encoded video data; generate a second syntax element for identifying a motion vector difference, to be included in the bitstream of the encoded video data, wherein the motion vector predictor and the motion vector difference correspond to the first motion vector; determine a second motion vector for the current block from a second candidate list of motion vector predictors based on bilateral matching; determine a prediction block for the current block using the first motion vector and the second motion vector for the current block; and encode the current block of the video data based on the prediction block; Generate a third syntax element for inclusion in the bitstream of encoded video data, wherein the third syntax element includes a flag having one of only two values, wherein a first value of the two values ​​indicates that the second motion vector system for the current block is determined for the current block from the list of second candidate motion vector predictors based on the bilateral matching, and a second value of the two values ​​indicates that the second motion vector system for the current block is determined for the current block using explicit signaling; and set the third syntax element to the first value.

18. The apparatus according to claim 17, wherein, in order to encode the current block of video data based on the prediction block, the one or more processors are also configured to: generate a bitstream of the encoded video data without including explicit signaling in the bitstream of the encoded video data for identifying a candidate in the second candidate motion vector predictor list.

19. The device according to claim 17, wherein, in order to determine the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching, the one or more processors are also configured to: use the first motion vector to locate a first reference block in a first reference image, wherein the first reference image is from a first reference image list; for each of the plurality of candidates in the second candidate motion vector predictor list, use a corresponding motion vector of the corresponding candidate to locate a second reference block in a second reference image, wherein the second reference image is from a second reference image list; and determine a bilateral matching error between the first reference block and the second reference block.

20. The apparatus according to claim 17, wherein, in order to determine the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching, the one or more processors are also configured to: use the first motion vector to locate a first reference block in a first reference image; determine a bilateral matching cost error between the first reference block and a second reference block for each of a plurality of candidates in the second candidate motion vector predictor list, the second reference block being located using a corresponding motion vector of each corresponding candidate; identify a candidate with a minimum bilateral matching error from the plurality of candidates; and derive the second motion vector for the current block from the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates.

21. The device according to request 20, wherein, in order to derive the second motion vector for the current block from the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates, the one or more processors are also configured to perform bilateral matching motion vector refinement.

22. The apparatus according to claim 21, wherein, in order to perform the bilateral matching motion vector refinement, the one or more processors are also configured to: identify an initial second reference block using the motion vectors of the candidate with the minimum bilateral matching error from the plurality of candidates; search for a final second reference block within a search range around the initial second reference block, the final second reference block producing a lower bilateral matching error relative to the first reference block compared to the bilateral matching error produced by the initial second reference block relative to the first reference block; and determine the second motion vector for the current block based on the final second reference block.

23. The device according to request 17, wherein the one or more processors are also configured to: determine, based on a decoding mode of the current block, whether the second motion vector for the current block is determined for the current block based on the bilateral matching from the list of second candidate motion vector predictors, or whether the second motion vector for the current block is determined for the current block using explicit signaling.

24. The device according to request item 17, wherein the one or more processors are also configured to: determine whether to enable double prediction for all slices containing the current block; and in response to the decision to enable the double prediction for the slice containing the current block, generate the third syntax element to be included in the bitstream of the encoded video data.

25. The device according to claim 17, wherein the device includes a wireless communication device and a transmitter configured to transmit encoded video data.

26. The device according to claim 25, wherein the wireless communication device includes a telephone, and wherein the transmitter is configured to modulate a signal including the encoded video data according to a wireless communication standard.

27. The equipment according to claim 17 also includes: A camera configured to capture video data.

28. The device according to claim 17, wherein the device includes one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

29. A computer-readable storage medium storing instructions, which, when executed by one or more processors, cause the one or more processors to: determine that a current block of video data is decoded in a bidirectional predictive frame mode; receive a first syntax element, wherein the first syntax element includes a flag having one of only two values, wherein a first value of the two values ​​indicates that a first motion vector system for a bidirectional predictive block is to be received in a bitstream of the video data, and a second motion vector system for the bidirectional predictive block is to be received in the bitstream of the video data, and wherein a second value of the two values ​​indicates that the first motion vector system for the bidirectional predictive block is to be received in the bitstream of the video data, and the second motion vector system for the bidirectional predictive block is determined not to receive explicit signaling in the bitstream of the video data; receive a second syntax element for identifying a motion vector predictor from a first candidate list of motion vector predictors; Receive a third syntax element for identifying a motion vector difference; in response to the first syntax having the second value, determine a first motion vector for the current block based on the motion vector predictor and the motion vector difference; determine a second motion vector for the current block from a second list of candidate motion vector predictors based on bilateral matching; and determine a predicted block for the current block using the first motion vector and the second motion vector for the current block.

30. The computer-readable storage medium according to request 29, wherein, in order to determine the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching, the instructions cause one or more processors to perform the following operations: using the first motion vector for the current block to locate a first reference block in a first reference image, wherein the first reference image is from a first reference image list; for each of a plurality of candidates in the second candidate motion vector predictor list, using a corresponding motion vector of the corresponding candidate to locate a second reference block in a second reference image, wherein the second reference image is from a second reference image list; and determining a bilateral matching error between the first reference block and the second reference block.

31. The computer-readable storage medium according to claim 29, wherein, in order to determine the second motion vector for the current block from the second candidate motion vector predictor list based on the bilateral matching, the instructions cause one or more processors to perform the following operations: using the first motion vector to locate a first reference block in a first reference image; determining a bilateral matching cost error between the first reference block and a second reference block, the second reference block being located using a corresponding motion vector of each of a plurality of candidates in the second candidate motion vector predictor list; identifying a candidate with a minimum bilateral matching error from the plurality of candidates; and deriving the second motion vector for the current block from the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates.

32. The computer-readable storage medium according to request 31, wherein, in order to derive the second motion vector for the current block from the motion vector of the candidate with the minimum bilateral matching error from the plurality of candidates, the instructions cause one or more processors to perform bilateral matching motion vector refinement.

33. The computer-readable storage medium according to claim 32, wherein, in order to perform the bilateral matching motion vector refinement, the instructions cause one or more processors to: identify an initial second reference block using the motion vectors of the candidate with the minimum bilateral matching error from the plurality of candidates; search for a final second reference block within a search range around the initial second reference block, the final second reference block having a lower bilateral matching error relative to the first reference block compared to the bilateral matching error generated by the initial second reference block relative to the first reference block; and determine the second motion vector for the current block based on the final second reference block.

34. The computer-readable storage medium pursuant to request item 29 stores additional instructions that, when executed, cause one or more processors to: determine whether to enable double prediction for all slices containing the current block; and in response to the decision to enable double prediction for the slice containing the current block, receive the third syntax element.

Images