Improved merging using motion vector differences.

Abstract

In some examples, a video decoding device includes a receiving circuit and a processing circuit. The processing circuit receives coded information for a current block in a current picture from a coded video bitstream. The coded information includes a syntax element having a value indicating that a merge using motion vector differential (MMVD) mode is applied to the current block. The processing circuit determines whether a merge candidate for the current block satisfies requirements for applying bi-predictive motion refinement in response to a motion vector predictor of the merge candidate being a bi-predictor. The processing circuit can constrain the merge candidate for use in the MMVD mode if the merge candidate for the current block satisfies the requirements. The processing circuit then reconstructs the current block in the MMVD mode using the merge candidate constrained for use in the MMVD mode.

Description

[Technical Field]

[0001]

[0001] Incorporation by Reference This application claims the benefit of priority to U.S. patent application Ser. No. 63 / 413,494, entitled "Improved Merging Using Motion Vector Differentials," filed on October 5, 2022. The disclosure of the prior application is incorporated herein by reference in its entirety.

[0002]

[0002] Technical Field This disclosure describes embodiments generally related to video coding. [Background technology]

[0003]

[0003] background The background discussion provided herein is intended to generally present the context of the present disclosure. Work under the names of the current inventors is not admitted, expressly or impliedly, as prior art to the present disclosure to the extent that that work is described in this background section or in a descriptive manner that might not otherwise qualify as prior art as of the filing date.

[0004] Image / video compression can help transmit image / video files between different devices, storage devices, and networks with minimal quality loss. In some cases, video codec technologies can compress video based on spatial and temporal redundancies. In one example, a video codec may use a technique called intra-prediction, which can compress an image based on spatial redundancy. For example, intra-prediction may use reference data from the current picture being reconstructed for sample prediction. In another example, video codecs can use a technique called inter-prediction, which can compress images based on temporal redundancy. For example, inter-prediction can predict samples in a current picture from a previously reconstructed picture using motion compensation, which is typically specified by a motion vector (MV). Summary of the Invention

[0005] Aspects of the present disclosure provide methods and apparatus for video encoding / decoding. In some examples, a video decoding apparatus includes a receiving circuit and a processing circuit. The processing circuit receives coded information for a current block in a current picture from a coded video bitstream. The coded information includes a syntax element having a value indicating that a merge with motion vector difference (MMVD) mode is applied to the current block. The processing circuit determines whether a merge candidate for the current block satisfies a requirement for applying bi-prediction motion refinement in response to a motion vector predictor of the merge candidate being a bi-predictor. If the merge candidate for the current block satisfies the requirement, the processing circuit can constrain the merge candidate for use in the MMVD mode. Then, the processing circuit reconstructs the current block in the MMVD mode using the merge candidate constrained for use in the MMVD mode.

[0006]

[0006] In some examples, the requirements for bi-predictive motion refinement include at least one of a first set of conditions for decoder side motion vector refinement (DMVR) and a second set of conditions for bi-directional optical flow (BDOF) refinement.

[0007] To constrain merge candidates for use in the MMVD mode, in some examples, the processing circuitry leaves a merge candidate outside of a base candidate list for the MMVD mode if the merge candidate meets requirements. The base candidate list for the MMVD mode is used to determine a starting point for the MMVD mode.

[0008] In one example, the processing circuit adds a merge candidate to a base candidate list for an MMVD mode if the merge candidate fails to meet the requirements. In one example, the processing circuit adds a merge candidate to a base candidate list for an MMVD mode as a uni-prediction MMVD candidate if the merge candidate is a uni-predictor. In one example, the processing circuit adds a merge candidate to a base candidate list for an MMVD mode as one bi-prediction MMVD candidate if the merge candidate is a bi-predictor and meets the requirements.

[0009]

[0009] In one example, the merge candidate is a regular merge candidate from a regular merge candidate list for the merge mode, which includes multiple merge candidate lists. The processing circuit tests each of the multiple regular merge candidates for inclusion in a base candidate list for the MMVD mode. In one example, the processing circuit determines, after testing, that the base candidate list has fewer than a predetermined number of candidates, and adds zero motion vectors to the base candidate list as both (or both) bi-predictive MMVD candidates. In another example, the processing circuit determines, after testing, that the base candidate list has fewer than a predetermined number of candidates, and adds merge candidates that are bi-predictors and meet the above requirements to the base candidate list. In one example, the processing circuit adds the merge candidate as both bi-predictive MMVD candidates. In another example, the processing circuit adds the merge candidate as one bi-predictive MMVD candidate.

[0010] In one example, the processing circuitry adds the merge candidate after other candidates in the base candidate list.

[0011] To reconstruct the current block, in some examples, the processing circuit determines a selection base from a base candidate list for the MMVD mode; determines a first MV offset depending on whether the selection base is a bi-predictive MMVD candidate; and determines a second MV offset depending on whether the selection base is a bi-predictive MMVD candidate or a uni-predictive MMVD candidate, where the second MV offset is different from the first MV offset.

[0012] To reconstruct the current block, in some examples, the processing circuit determines a selected base from a base candidate list for the MMVD mode; and, depending on whether the selected base is a bi-predictive MMVD candidate, determines a first MV offset according to a first table, the first table storing the first MV offset associated with a distance index. Depending on whether the selected base is one of the bi-predictive MMVD candidates or one of the uni-predictive MMVD candidates, the processing circuit determines a second MV offset according to a second table, the second table storing the second MV offset associated with a distance index.

[0013] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a video decoding method for video decoding. [Brief explanation of the drawings]

[0013]

[0014] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Figure 1]

[0015] FIG. 1 is a schematic diagram of an exemplary block diagram of a communication system. [Figure 2]

[0016] FIG. 2 is a schematic diagram of an exemplary block diagram of a decoder. [Figure 3]

[0017] FIG. 3 is a schematic diagram of an exemplary block diagram of an encoder. [Figure 4]

[0018] FIG. 4 illustrates the location of spatial merge candidates according to an embodiment of the present disclosure. [Figure 5]

[0019] FIG. 5 illustrates candidate pairs considered for redundancy checking of spatial merge candidates according to an embodiment of the present disclosure. [Figure 6]

[0020] FIG. 6 illustrates an example motion vector scaling for temporal merge candidates. [Figure 7]

[0021] FIG. 7 shows exemplary candidate positions for temporal merge candidates for the current CU. [Figure 8]

[0022] FIG. 8 shows an example of a search process in the merge using motion vector difference (MMVD) mode. [Figure 9]

[0023] FIG. 9 shows an example of search points in the merge using motion vector difference (MMVD) mode. [Figure 10]

[0024] FIG. 10 shows an exemplary schematic diagram of decoder-side motion vector refinement. [Figure 11]

[0025] FIG. 11 shows a flowchart outlining a process according to some embodiments of the present disclosure. [Figure 12]

[0026] FIG. 12 shows a flowchart outlining another process according to some embodiments of the present disclosure. [Figure 13]

[0027] FIG. 13 is a schematic diagram of a computer system according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014]

[0028] 1 illustrates a block diagram of a video processing system 100 according to some examples. The video processing system 100 is an example of a video encoder and video decoder in a streaming environment, an application of the disclosed subject matter. The disclosed subject matter is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, streaming services, and storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.).

[0015]

[0029] The video processing system (100) includes a capture subsystem (113), which may include a video source (101), such as a digital camera, that generates a stream of uncompressed video pictures (102). In one example, the video picture stream (102) includes samples captured by the digital camera. The video picture stream (102), depicted as a thick line to emphasize the amount of data compared to the encoded video data (104) (or coded video bitstream), may be processed by an electronic device (120) that includes a video encoder (103) coupled to the video source (101). The video encoder (103) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in detail below. The encoded video data (104) (or encoded video bitstream), depicted as a thin line to emphasize its smaller amount of data when compared to the stream of video pictures (102), may be stored on a streaming server (105) for future use. One or more streaming client subsystems, such as the client subsystems (106) and (108) of FIG. 1, may access the streaming server (105) to retrieve copies (107) and (109) of the encoded video data (101). The client subsystem (106) may include a video decoder (110), for example, within an electronic device (130). A video decoder (110) decodes an incoming copy of the coded video data (107) and generates an output stream of video pictures (111) that can be rendered on a display (112) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the coded video data (104), (107), and (109) (e.g., a video bitstream) may be coded according to a particular video coding / compression standard. Examples of these standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0016]

[0030] It should be noted that electronic devices 120 and 130 may include other components (not shown). For example, electronic device 120 may include a video decoder (not shown), and electronic device 130 may include a video encoder (not shown).

[0017]

[0031] 2 shows an example block diagram of a video decoder (210). The video decoder (210) can be included in an electronic device (230). The electronic device (230) can include a receiver (231) (e.g., a receiving circuit). The video decoder (210) can be used in place of the video decoder (110) in the example of FIG. 1.

[0018]

[0032] The receiver (231) can receive one or more coded video sequences to be decoded by the video decoder (210). In some embodiments, the receiver can receive one coded video sequence at a time, where the decoding of each coded video sequence is independent of the decoding of other coded video sequences. The coded video sequences can be received from a channel (201), which can be a hardware or software link to a storage device that stores the coded video data. The receiver (231) can receive the coded video data along with other data, such as coded audio data and / or auxiliary data streams, which can be transferred using respective entities (not shown). The receiver (231) can separate the coded video sequences from other data. To address network jitter, a buffer memory (212) can be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter referred to as the "parser (220)"). In certain applications, the buffer memory (215) is part of the video decoder (210). In other cases, it may be external to the video decoder (210) (not shown). In yet another example, there may be a buffer memory (not shown) external to the video decoder (210), for example, to deal with network jitter, and there may even be another buffer memory (212) internal to the video decoder (210), for example, to handle playback timing. If the receiver (231) is receiving data from a store-and-forward device with sufficient bandwidth and controllability or from a synchronous network, the buffer memory (212) may not be needed or may be small.For use in a best-effort packet network such as the Internet, a buffer memory (215) may be required, which may be relatively large and may advantageously be adaptively sized, and may be implemented at least in part in an operating system or similar element (not shown) outside the video decoder (210).

[0019]

[0033] The video decoder (210) may include a parser (220) for reconstructing symbols (221) from the coded video sequence. These symbol categories include information used to manage the operation of the video decoder (210) and, possibly, information for controlling a rendering device, such as a rendering device (212) (e.g., a display screen) that is not an integral part of the electronic device (230) but may be coupled to the electronic device (230), as shown in FIG. 2. The rendering device control information may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (220) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard, including variable length coding, Huffman coding, arithmetic coding with or without context effects, and various other principles. The parser (220) can extract from the coded video sequence a set of subgroup parameters for at least one subgroup of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroup can include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (220) can also extract from the coded video sequence information such as transform coefficients, quantization parameter values, motion vectors (MVs), etc.

[0020]

[0034] The parser (220) is capable of performing an entropy decoding / parsing process on the video sequence received from the buffer memory (215) to generate symbols (221).

[0021]

[0035] The reconstruction of the symbols (221) may include several different units depending on the type of coded video picture or part thereof (e.g., inter and intra pictures, inter and intra blocks) and other factors. Which units are included and how can be controlled by subgroup control information parsed by the parser (220) from the coded video sequence. The flow of such subgroup control information between the parser (220) and subsequent units is not depicted for clarity.

[0022]

[0036] Beyond the functional blocks already described, the video decoder (210) may be conceptually subdivided into a number of functional units, as described below. In a practical implementation operating within commercial constraints, many of these units will interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate.

[0023]

[0037] The first unit is a scalar / inverse transform unit (251), which receives quantized transform coefficients as well as control information (including the transform to use, block size, quantization factor, quantization scaling matrix, etc.) from the parser (220) as symbols (221). The scalar / inverse transform unit (251) can output blocks containing sample values ​​that can be input to the aggregator (222).

[0024]

[0038] In some cases, the output samples of the scalar / inverse transform unit (251) may relate to intra-coded blocks. Intra-coded blocks are blocks that do not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by the intra-picture prediction unit (252). In some cases, the intra-picture prediction unit (252) generates a block of the same size and shape as the block being reconstructed using already reconstructed surrounding information retrieved from the current picture buffer (258). The current picture buffer (258), for example, buffers the partially reconstructed and / or fully reconstructed current picture. The aggregator (255) optionally adds, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (252) to the output sample information as provided by the scalar / inverse transform unit (251).

[0025]

[0039] In other cases, the output samples of the scalar / inverse transform unit (251) may relate to a block that may be inter-coded and motion-compensated. In such cases, the motion-compensated prediction unit (253) may access the reference picture memory (257) to retrieve samples used for prediction. After motion-compensating the retrieved samples according to the symbols (221) associated with the block, these samples are added by the aggregator (255) to the output of the scalar / inverse transform unit (251) (in this case, referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (257) from which the motion-compensated prediction unit (253) retrieves prediction samples may be controlled by motion vectors available to the motion-compensated prediction unit (253), for example, in the form of symbols (251), which may have X, Y, and reference picture components. Motion compensation can also include interpolation of sample values ​​taken from a reference picture memory (257), motion vector prediction mechanisms, etc., where sub-sample accurate motion vectors are used.

[0026]

[0040] The output samples of the aggregator (255) may be subjected to various loop filtering techniques in the loop filter unit (256). The video compression techniques may include in-loop filtering techniques controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (226) as symbols (221) from the parser (220). The video compression may also depend on meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence, as well as on previously reconstructed loop-filtered sample values.

[0027]

[0041] The output of the loop filter unit (256) may be a stream of samples that can be output to a rendering device (212) or stored in a reference picture memory (257) for use in future inter-picture prediction.

[0028]

[0042] Once a given coded picture is fully reconstructed, it can be used as a reference picture for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) can become part of the reference picture memory (257), and a fresh current picture buffer can be reallocated before starting the reconstruction of a subsequent coded picture.

[0029]

[0043] The video decoder (210) may perform decoding operations according to a standard or predetermined video compression technology, such as ITU-T Rec. H.262. A coded video sequence may conform to the syntax specified by the video compression technology or standard in use, in the sense that the coded video sequence conforms to both the syntax of the video compression technology or standard and the profile, as documented in the video compression technology or standard. Specifically, a profile may select certain tools from all tools available in the video compression technology or standard as the only tools that can be used under that profile. Compliance also requires that the complexity of the coded video sequence fall within a range defined by the level of the video compression technology or standard. In some cases, the level may limit the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the levels may in some cases be further constrained by Hypothetical Reference Decoder (HRD) specifications and metadata for HRD buffer management signaled in the coded video sequence.

[0030]

[0044] In an embodiment, the receiver (231) may receive additional (redundant) data along with the coded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0031]

[0045] 3 shows an example block diagram of a video encoder (303). The video encoder (303) is included in an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmission circuit). The video encoder (303) can be used in place of the video encoder (103) in the example of FIG. 1.

[0032]

[0046] The video encoder (303) can receive video samples from a video source (301) (which in the example of FIG. 3 is not part of the electronic device (320)) that can capture video images to be coded by the video encoder (303). In another example, the video source (301) is part of the electronic device (320).

[0033]

[0047] The video source (301) may provide a source video sequence to be coded by the video encoder (303) in the form of a digital video sample stream, which may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCB, RGB, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (301) may be a storage device that stores pre-prepared video. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. Video data may be provided as multiple individual pictures that convey motion when viewed sequentially. The picture itself may be organized as a spatial array of pixels, each of which may contain one or more samples, depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following discussion focuses on examples.

[0034]

[0048] According to an embodiment, the video encoder (303) is capable of coding and compressing pictures of a source video sequence into a coded video sequence (343) in real time or under any other required time constraints. Enforcing an appropriate coding rate is one function of the controller (350). In some embodiments, the controller (350) controls and is functionally coupled to other functional units, as described below, which are not depicted for clarity. Parameters set by the controller (350) may include rate control-related parameters (picture skip, quantizer, lambda value for rate-distortion optimization techniques, etc.), picture size, group-of-picture (GOP) layout, maximum motion vector search range, etc. The controller (350) may be configured to have other appropriate functions associated with the video encoder (303) optimized for a particular system design.

[0035]

[0049] In some embodiments, the video encoder (303) is configured to operate in a coding loop. As a simplified explanation, in one example, the coding loop may include a source coder (330) (e.g., responsible for generating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture) and a (local) decoder (333) embedded in the video encoder (303). The decoder (333) reconstructs the symbols to generate sample data in a manner similar to that generated by the (remote) decoder. The reconstructed sample stream (sample data) is input to a reference picture memory (334). Because decoding of the symbol stream produces bit-exact results independent of the location (local or remote) of the decoder, the contents of the reference picture memory (334) are also bit-exact between the local and remote encoders. In other words, the predictor in the encoder "sees" as reference picture samples exactly the same samples that the decoder would "see" if it were using prediction during decoding. This basic principle of reference picture synchronization (and the resulting drift if synchronization cannot be maintained, e.g., due to channel errors) is used in several related technologies as well.

[0036]

[0050] The operation of the "local" decoder (333) may be the same as that of a "remote" decoder, such as the video decoder (210) already described in detail above in connection with Figure 2. However, briefly referring also to Figure 2, because symbols are available and the encoding / decoding of the symbols into a coded video sequence by the entropy coder (345) and parser (220) may be lossless, the entropy decoding portion of the video decoder (210), including the buffer memory (215) and parser (220), may not be fully implemented in the local decoder (333).

[0037]

[0051] In embodiments, decoder techniques other than analysis / entropy decoding present in a decoder are also present in the corresponding encoder, ideally or in substantially the same functional form. Therefore, the disclosed subject matter focuses on the operation of the decoder. A description of the encoder techniques can be omitted, as they are the reverse of the decoder techniques described generically. In certain areas, more detailed descriptions are provided below.

[0038]

[0052] During operation, in some examples, the source coder (330) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (332) codes differences between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as predictive references for the input picture.

[0039]

[0053] The local video decoder (333) can decode coded video data of pictures that can be designated as reference pictures based on symbols generated by the source coder (330). The operation of the coding engine (332) can advantageously be a non-lossless process. When the coded video data can be decoded by a video decoder (not shown in FIG. 3), the reconstructed video sequence can typically be a replica of the source video sequence with some errors. The local video decoder (333) can repeat the decoding process that can be performed by the video decoder on the reference picture, causing the reconstructed reference picture to be stored in the reference picture memory (334). In this way, the video encoder (303) can locally store copies of reconstructed reference pictures that have common content with reconstructed reference pictures obtained by a far-end video decoder (assuming there are no transmission errors).

[0040]

[0054] The predictor (335) can perform a prediction search for the coding engine (332). That is, for a new picture to be coded, the predictor (335) can search the reference picture memory (334) for sample data (as candidate reference pixel blocks) or predetermined metadata (reference picture motion vectors, block shapes, etc.), which may serve as suitable prediction references for the new picture. The predictor (335) can operate on a sample-block-pixel-block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (335), an input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (334).

[0041]

[0055] The controller (350) can manage the coding operations of the source coder (330), including, for example, setting parameters and subgroup parameters used to encode the video data.

[0042]

[0056] All outputs of the aforementioned functional units can be entropy coded in an entropy coder (345), which converts the symbols produced by the various functional units into a coded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0043]

[0057] The transmitter (340) can buffer the coded video sequence, as produced by the entropy coder (345), and prepare it for transmission over the communication channel (330), which may be a hardware / software link to a storage device that stores the coded video data. The transmitter (340) can merge the coded video data from the video coder (303) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0044]

[0058] The controller (350) can manage the operation of the video encoder (303). During coding, the controller (350) can assign a particular coded picture type to each coded picture, which can affect the coding technique that can be applied to each picture. For example, pictures may often be assigned as one of the following picture types:

[0059] An intra picture (I-picture) is one that can be coded and decoded without using any other picture in the sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0045]

[0060] A predicted picture (P-picture) can be coded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0046]

[0061] Bidirectionally predicted pictures (B-pictures) can be coded and decoded using intra- or inter-prediction, which uses at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted pictures can use more than two reference pictures and associated metadata for the reconstruction of a block.

[0047]

[0062] A source picture is typically spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and can be coded block by block. Blocks can be predictively coded with reference to other (already coded) blocks, as determined by the coding assignment applied to the respective picture. For example, blocks of an I-picture can be non-predictively coded, or they can be predictively coded with reference to previously coded blocks of the same picture (spatial or intra prediction). Pixel blocks of a P-picture can be predictively coded with spatial or temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture can be predictively coded with spatial or temporal prediction with reference to one or two previously coded reference pictures.

[0048]

[0063] The video encoder (303) may perform coding operations according to a predetermined video coding technique or standard, such as ITU-T Rec. H.233. In this operation, the video encoder (303) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancies in the input video sequence. The coded video data may therefore conform to a syntax specified by the video coding technique or standard being used.

[0049]

[0064] In an embodiment, the transmitter (340) can transmit additional data along with the coded video. The source coder (330) can include such data as part of the coded video sequence. The additional data can include temporal, spatial, and SNR enhancement layers, as well as other forms of redundant data (such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.).

[0050]

[0065] Video can be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or otherwise) between pictures. In one example, a particular picture under encoding / decoding, called the current picture, is partitioned into blocks. If a block in the current picture is similar to a reference block in a reference picture that was previously coded and is still buffered in the video, the block in the current picture can be coded by a vector called a motion vector. The motion vector points to a reference block within the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0051]

[0066] In some embodiments, bi-prediction techniques may be used for inter-picture prediction. Bi-prediction techniques use two reference pictures, such as a first reference picture and a second reference picture, that both precede the current picture in decoding order (but may be past and future, respectively, in display order) in the video. A block in the current picture may be coded with a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block may be predicted by a combination of the first and second reference blocks.

[0052]

[0067] Furthermore, to improve coding efficiency, it is possible to use merge mode techniques for inter-picture prediction.

[0053]

[0068] According to some embodiments of the present disclosure, prediction, such as inter-picture prediction and intra-picture prediction, is performed on a block-by-block basis. For example, according to the HEVC standard, pictures in a sequence of video pictures are partitioned into coding tree units (CTUs) for compression, and the CTUs within a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU includes three coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree partitioned into one or more coding units (CUs). For example, a 64x64 pixel CTU can be partitioned into one 64x64 pixel CU, four 32x32 pixel CUs, or sixteen 16x16 pixel CUs. In one example, each CU is analyzed to determine the CU's prediction type, such as an inter prediction type or an intra prediction type. A CU is divided into one or more prediction units (PUs) depending on temporal and / or spatial predictability. Generally, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of values ​​(e.g., luma values) for pixels, such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0054]

[0069] The video encoders (103, 303) and video decoders (110, 210) may be implemented using any suitable technology. In some embodiments, the video encoders (103, 303) and video decoders (110, 210) may be implemented using one or more integrated circuits. In other embodiments, the video encoders (103, 303) and video decoders (110, 210) may be implemented using one or more processors executing software instructions.

[0055]

[0070] Aspects of this disclosure provide techniques for use with an inter-prediction mode called merge with motion vector difference (MMVD) to improve coding performance.

[0056]

[0071] In video coding, various inter-prediction modes can be used. For example, in VVC, for an inter-predicted CU, the motion parameters may include MV(s), one or more reference picture indices, a reference picture list usage index, and additional information about the specific coding features used to generate the inter-predicted samples. Motion parameters can be signaled explicitly or implicitly. If a CU is coded in skip mode, the CU can be associated with a PU and cannot have significant residual coefficients, coded motion vector deltas or MV differences (e.g., MVDs), or reference picture indices. Merge mode can be specified when motion parameters for the current CU are obtained from nearby CUs, including spatial and / or temporal candidates and, optionally, additional information as introduced in VVC. Merge mode can be applied not only to skip mode but also to inter-predicted CUs. In one example, an alternative to merge mode is explicit transmission of motion parameters, where the MV(s), the corresponding reference picture index for each reference picture list, and the reference picture list usage flag, as well as other information, are explicitly signaled per CU.

[0057]

[0072] In an embodiment such as VVC, the VVC Test Model (VTM) reference software includes one or more sophisticated inter-predictive coding tools, such as enhanced merge prediction, merge motion vector difference (MMVD) mode, adaptive motion vector prediction (AMVP) mode with symmetric MVD signaling, affine motion compensation prediction, subblock-based temporal motion vector prediction (SbTMVP), adaptive motion vector resolution (AMVR), motion field storage (1 / 16 luma sample MV storage and 8x8 motion field compression), bi-prediction with CU-level weights (BCW), bi-directional optical flow (BDOF), prediction refinement using optical flow (PROF), decoder side motion vector refinement, and so on. These include differential refinement (DMVR), combined inter and intra prediction (CIIP), geometric partitioning mode (GPM), etc. Inter prediction and related methods are described in detail below.

[0058]

[0073] In some cases, enhanced merge prediction can be used. In some cases, such as in VTM4, the merge candidate list is constructed by including the following five types of candidates in order: Spatial motion vector predictor (MVP) from spatially nearby CUs, Temporal MVP from the equivalent CU, History-based MVP (HMVP) from a first-in-first-out (FIFO) table Pairwise average MVP, and Zero MV.

[0059]

[0074] The size of the merge candidate list can be signaled in the slice header. In one example, the maximum allowed size of the merge candidate list is 6 in VTM4. For each CU coded in merge mode, the index of the best merge candidate (e.g., merge index) can be coded using truncated unary binarization (TU). The first bin of the merge index can be coded with context (e.g., context-adaptive binary arithmetic coding (CABAC)), and for the other bins, bypass coding can be used.

[0060]

[0075] Some examples of the generation process for each category of merge candidates are given below: In an embodiment, spatial candidates are derived as follows: The derivation of spatial merge candidates in VVC can be the same as that in HEVC. In one example, up to four merge candidates are selected from the candidates located at the positions shown in Figure 4.

[0061]

[0076] Figure 4 illustrates spatial merge candidate positions according to an embodiment of the present disclosure. Referring to Figure 4, the derivation order is B1, A1, B0, A0, B2. Position B2 is considered only if none of the CUs in positions A0, B0, B1, or A1 are available (e.g., because the CUs belong to another slice or another tile) or are intra-coded. After the candidate in position A1 is added, redundancy checks are performed on the addition of the remaining candidates (ensuring that candidates with the same motion information are removed from the candidate list), thereby improving coding efficiency.

[0062]

[0077] To reduce computational complexity, not all possible candidate pairs are considered in the above redundancy check. Instead, only pairs connected by arrows in Figure 5 are considered, and a candidate is only added to the candidate list if the corresponding candidate used in the redundancy check does not have the same motion information.

[0063]

[0078] 5 illustrates candidate pairs considered for spatial merge candidate redundancy checking according to an embodiment of the present disclosure. Referring to FIG. 5, the pairs connected by the respective arrows include A1 and B1, A1 and A0, A1 and B2, B1 and B0, and B1 and B2. Thus, the candidate at positions B1, A0, and B2 can be compared with the candidate at position A1, and the candidate at positions B0 and / or B2 can be compared with the candidate at position B1.

[0064]

[0079] In an embodiment, temporal candidates are derived as follows: In one example, only one temporal merge candidate is added to the candidate list. Figure 6 shows exemplary motion vector scaling for temporal merge candidates. To derive a temporal merge candidate for a current CU (611) in a current picture (601), a scaled MV (621) (e.g., shown by a dotted line in Figure 6) can be derived based on a co-located CU (612) belonging to a collocated reference picture (604). The reference picture list used to derive the co-located CU (612) can be explicitly signaled in the slice header. The scaled MV (621) of the temporal merge candidate can be determined as shown by a dotted line in Figure 6. The scaled MV (621) can be scaled from the MV of the co-located CU (612) using picture order count (POC) distances tb and td. The POC distance tb may be defined to be the POC difference between the current reference picture (602) of the current picture (601) and the current picture (601). The POC distance td may be defined to be the POC difference between the equally-located reference picture (604) of the equally-located picture (603) and the equally-located picture (603). The reference picture index of the temporal merge candidate may be set to zero.

[0065]

[0080] FIG. 7 shows exemplary candidate positions (e.g., C0 and C1) for temporal merge candidates for the current CU. The position of the temporal merge candidate can be selected from candidate positions C0 and C1. Candidate position C0 is located at the bottom right corner of the equivalent position CU (710) of the current CU. Candidate position C1 is located at the center of the equivalent position CU (710) of the current CU. If the CU at candidate position C0 is unavailable, intra-coded, or outside the current row of the CTU, candidate position C1 is used to derive the temporal merge candidate. Otherwise, for example, if the CU at candidate position C0 is available, inter-coded, and within the current row of the CTU, candidate position C0 is used to derive the temporal merge candidate.

[0066]

[0081] In merge mode, in some examples, implicitly derived motion information can be used to predict samples of a CU (e.g., the current CU). In some examples, a merge with motion vector difference (MMVD) mode is used, as in VVC. The MMVD mode is used for either skip mode or merge mode using a motion vector representation method. In some examples, for example, after signaling a regular merge flag (e.g., a skip flag or a merge flag), it is possible to signal an MMVD merge flag to specify whether the MMVD mode is used for a CU.

[0067]

[0082] In MMVD, after selecting a merge candidate from a regular merge list, the merge candidate is further refined by signaled MVD information. In some examples, MMVD reuses merge candidates from the regular merge list. Among the merge candidates in the regular merge list, a candidate can be selected and further extended by a motion vector representation. MMVD provides a motion vector representation using simplified signaling. In some examples, the motion vector representation includes a starting point, a motion magnitude, and a motion direction.

[0068]

[0083] In some cases (e.g., VVC), the MMVD technique can use a merge candidate list to select starting point candidates. However, in one example, only candidates with the default merge type (MRG_TYPE_DEFAULT_N) are considered for MMVD expansion.

[0069]

[0084] In some examples, a base candidate index is used to define a starting point. The base candidate index indicates the best candidate among the candidates in a list such as that shown in Table 1. For example, the list is a merge candidate list with motion vector predictors (MVPs). The base candidate index can indicate the best candidate in the merge candidate list.

[0070] Table 1 - Example of a base candidate index (IDX)

[0071] [Table 1]

[0085] Note that in one example, the number of base candidates is equal to 1, and therefore the base candidate IDX is not signaled.

[0072]

[0086] In MMVD mode, after a merge candidate (also called an MV base or MV starting point) is selected, the merge candidate can be refined by additional information, such as signaled MVD information. The additional information can include an index used to specify the magnitude of the motion (e.g., a distance index, e.g., mmvd_distance_idx[x0][y0]) and an index used to indicate the direction of the motion (e.g., a direction index, e.g., mmvd_direction_idx[x0][y0]). In some examples, in MMVD mode, one of the first two candidates in the (regular) merge list can be selected as the MV base. For example, a merge candidate flag (e.g., mmvd_cand_flag[x0][y0]) indicates one of the first two candidates in the merge list. The merge candidate flag can be signaled to indicate (e.g., specify) which of the first two candidates was selected. The additional information can indicate the MVD (or motion offset) relative to the MV base. For example, the magnitude of the motion indicates the magnitude of the MVD, and the direction of the motion indicates the direction of the MVD.

[0073]

[0087] In one example, a merge candidate selected from the merge candidate list is used to provide a starting point or MV starting point in a reference picture. The motion vector of the current block can be expressed as a starting point and a motion offset (or MVD), where the motion offset includes the magnitude and direction of motion relative to the starting point. On the encoder side, the selection of the merge candidate and the determination of the motion offset can be based on a search process (evaluation process) as shown in FIG. 8. On the decoder side, the selected merge candidate and the motion offset can be determined based on signaling from the encoder side.

[0074]

[0088] Figure 8 shows an example of a search process (800) in MMVD mode. Figure 9 shows example search points in MMVD mode. In some examples, a subset or the entire set of search points in Figure 9 are used in the search process (800) in Figure 8. For example, by performing the search process (800) on the encoder side, additional information including a merge candidate flag (e.g., mmvd_cand_flag[x0][y0]), a distance index (e.g., mmvd_distance_idx[x0][y0]), and a direction index (e.g., mmvd_direction_idx[x0][y0]) can be determined for a current block (801) of a current picture (or current frame).

[0075]

[0089] A first motion vector 811 and a second motion vector 821 belonging to a first merge candidate are shown. The first motion vector 811 and the second motion vector 821 are MV starting points used in the search process 800. The first merge candidate may be a merge candidate in the merge candidate list constructed for the current block (801). The first and second motion vectors (811) and (821) may be associated with two reference pictures (802) and (803) in the reference picture lists L0 and L1, respectively. Referring to Figures 8-9, the first and second motion vectors (811) and (821) may point to two starting points (911) and (921) in the reference pictures (802) and (803), respectively, as shown in Figure 9.

[0076]

[0090] Referring to Figure 9, two starting points (911) and (921) in Figure 9 can be determined in the reference pictures (802) and (803). In one example, based on the starting points (911) and (921), multiple predetermined points extending from the starting points (911) and (921) in the vertical direction (indicated by +Y or -Y) or the horizontal direction (indicated by +X or -X) in the reference pictures (802) and (803) can be evaluated. In one example, A pair of points that are mirror images of each other with respect to the respective starting points 911 and 921, such as points 914 and 924 (e.g., as indicated by a shift of 1S in FIG. 8) or points 915 and 925 (e.g., as indicated by a shift of 2S in FIG. 8), can be used to determine a pair of motion vectors (e.g., MVs 813 and 823 in FIG. 8) for the current block 801. Motion vectors determined based on predetermined points around the starting point 911 or starting point 921 (e.g., MVs 813 and 823 in FIG. 8) can be evaluated.

[0077]

[0091] The distance index (e.g., mmvd_distance_idx[x0][y0]) specifies motion magnitude information and may indicate a predetermined offset (e.g., 1S or 2S in FIG. 8) from the starting point indicated by the merge candidate flag. Note that the predetermined offset is also referred to as the MMVD step in one example.

[0078]

[0092] 8, an offset (e.g., MVD (812) or MVD (822)) can be applied (e.g., added) to the horizontal or vertical component of the starting MV (e.g., MV (811) or (821)). An exemplary relationship between distance index (IDX) and predefined offsets is provided in Table 2. When full-pel MMVD is off, e.g., when the full-pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is equal to 0, the MMVD predefined offset can range from 1 / 4 luma sample to 32 luma samples. When full-pel MMVD is off, the predefined offset can have a non-integer value, such as a fraction of a luma sample (e.g., 1 / 4 pixel or 1 / 2 pixel). When full-pel MMVD is on, e.g., when a full-pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is equal to 1, the MMVD predefined offset can range from 1 luma sample to 128 luma samples. In one example, when full-pel MMVD is on, the predefined offset has only integer values, such as one or more luma samples.

[0079] Table 2 - Example relationships between distance indexes and offsets (e.g., predefined offsets)

[0080] [Table 2]

[0093] The direction index can represent the direction (or motion direction) of the MVD relative to the starting point. In one example, the direction index represents one of the four directions shown in Table 3. The meaning of the MVD sign (positive or negative sign) in Table 3 may differ depending on the information of the starting MV. In one example, when the starting MV is a uni-predictive MV or a bi-predictive MV and both reference lists point to the same side of the current picture (for example, when the POCs of the two reference pictures are both greater than the POC of the current picture, or when the POCs of the two reference pictures are both less than the POC of the current picture), the MVD positive or negative sign in Table 3 specifies the positive or negative sign of the MV offset (or MVD) added to the starting MV.

[0081]

[0094] If the starting MV is a bi-predictive MV and the two MVs point to different sides of the current picture (for example, one reference picture's POC is greater than the current picture's POC and the other reference picture's POC is less than the current picture's POC), the MVD sign in Table 3 specifies the sign of the MV offset (or MVD) added to the list0 MV component of the starting MV, while the MVD sign for the list1 MV has the opposite value. Referring to Figure 8, the starting MVs (811) and (821) are bi-predictive MVs, and the two MVs (811) and (821) point to different sides of the current picture. The POC of the L1 reference picture (803) is greater than the current picture's POC, and the POC of the L0 reference picture (802) is less than the current picture's POC. The MVD sign (e.g., the sign "+" relative to the x-axis) indicated by the direction index (e.g., 00) in Table 3 indicates the sign (e.g., the sign "+" relative to the x-axis) of the MVD (e.g., MVD(812)) added to the list0 MV component of the starting MV (e.g., (811)), and the MVD sign of MVD(822) for the list1 MV component of the starting MV (e.g., (821)) has an inverse value, such as the inverse sign "-" of the sign "+" of MVD(812).

[0082]

[0095] Referring to Table 3, directional index 00 indicates the positive direction on the x-axis, directional index 01 indicates the negative direction on the x-axis, directional index 10 indicates the positive direction on the y-axis, and directional index 11 indicates the negative direction on the y-axis.

[0083] Table 3 – Example relationship between the sign of the MV offset and the direction index

[0084] [Table 3]

[0096] In some examples, the MVD can be scaled according to the POC difference in each direction. For example, if the POC difference in both lists is the same, no scaling is required. Otherwise, if the POC difference in list0 is larger than that of list1, the MVD for list1 is scaled by defining the POC difference of L0 as td and the POC difference of L1 as tb, as described with reference to FIG. 6. If the POC difference of L1 is larger than L0, the MVD for list0 is scaled in a similar manner. If the starting MV is uni-predicted, the MVD is added to the available MV.

[0085]

[0097] In some examples, one-sided (or unilateral) MMVD positions are added in the bi-predictive case. In the case of unilateral MMVD (also called affine MMVD), individual MVDs may be used independently for each list. In one example, a non-zero MVD is applied to list X (where X is 0 or 1) and a zero MVD is applied to list (1-X). In some examples, all possible MMVD candidates may be sorted in ascending order according to template matching cost.

[0086]

[0098] In some cases, the MMVD flag is signaled immediately after sending the skip and merge flags. If the skip and merge flags are true, the MMVD flag is parsed. If the MMVD flag is equal to 1, the MMVD syntax is parsed.

[0087]

[0099] To improve the accuracy of MV in merge mode, bilateral-matching (BM)-based decoder-side motion vector refinement can be applied as in VVC. In bi-prediction operation, it is possible to search for refined MVs around the initial MVs in reference picture list L0 and reference picture list L1. The BM method can calculate the distortion between two candidate blocks in reference picture list L0 and list L1.

[0088]

[0100] Figure 10 shows an example schematic diagram of BM-based decoder-side motion vector refinement. As shown in Figure 10, a current picture (1002) may include a current block (1008). The current picture may include a reference picture list L0 (1004) and a reference picture list L1 (1006). With respect to the current block (1008), an initial reference block (1012) in the reference picture list L0 (1004) may be positioned according to an initial motion vector MV0, and an initial reference block (1014) in the reference picture list L1 (1006) may be positioned according to an initial motion vector MV1. A search process may be performed around the initial MV0 in the reference picture list L0 (1004) and the initial MV1 in the reference picture list L1 (1006). For example, a first reference candidate block (1010) may be identified in reference picture list L0 (1004), and a first reference candidate block (1016) may be identified in reference picture list L1 (1006). The SAD between candidate reference blocks (e.g., 1010 and 1016) based on each MV candidate (e.g., MV0' and MV1') surrounding the initial MV (e.g., MV0 and MV1) may be calculated. The MV candidate with the lowest SAD becomes the refined MV and may be used to generate a bi-predicted signal to predict the current block (1008).

[0089]

[0101] In some cases (e.g., VVC), DMVR is implicitly applied at the block level without the need to signal side information in the bitstream. If a block meets certain conditions, the DMVR algorithm is invoked. For example, the conditions (also referred to as requirements for DMVR or sets of conditions for DMVR) can include: (1) CU-level merge mode with bi-predictive MVs. (2) For the current picture, one reference picture is in the past and another is in the future. (3) The distances (e.g., POC differences) from the two reference pictures to the current picture are the same. (4) Both reference pictures are short-term reference pictures. (5) The CU has more than 64 luma samples. (6) Both the CU height and CU width are equal to or greater than 8 luma samples. (7) The BCW Weight Index indicates equal weighting. (8) Weighted prediction (WP) is not enabled for the current block, and (9) CIIP mode shall not be used for the current block.

[0090]

[0102] In some cases (e.g., VVC), decoder-side motion vector refinement (DMVR) is applied to CUs coded in canonical merging mode. MV pairs obtained from canonical merging candidates are used as input for the DMVR process. DMVR applies bilateral matching (BM) to find the best fit between the input MV pairs {mv L0 ,mv L1} is refined to obtain the refined MV pair {mv refinedL0 ,mv refinedL1} is used for motion compensated prediction of both luma and chroma components. The output MV of the DMVR can be called a refined MV pair and can be expressed by Eq. (1):

[0091]

number

[0103] The motion vector difference Δmv can be applied to the input MV pair to obtain a refined MV pair using the MVD mirroring property, because the input MV pair points to two different reference pictures that have equal differences in picture order count (POC) to the current picture, and these two reference pictures are in different temporal directions.

[0092]

[0104] In some examples, in DMVR, a luma coded block is divided into 16x16 sub-blocks for the MV refinement process. Δmv is derived independently for each sub-block in two steps: an integer precision motion search step (also called integer sample offset search) followed by a fractional motion search step (also called fractional sample offset search). Finally, sub-block motion compensation (MC) is performed to refine the refined MV pair {mv refinedL0 ,mv refinedL1} is applied.

[0093]

[0105] According to one aspect of the present disclosure, in the integer precision motion search step of DMVR, the search space includes 25 MV candidate pairs that can be expressed by Eq. (2):

[0094]

number

[0095]

number

[0106] In Eq. (3), W and H represent the weight and height of the sub-block, and P0 i,j [m+i,2n+j] and P1 i,j[mi,2n-j] denotes corresponding samples in the two reference pictures. If the SAD of the initial MV pair is less than a threshold, the integer precision motion search step of DMVR terminates. Otherwise, the SADs of the remaining 24 points are calculated and examined, for example, in raster scan order. The point with the smallest SAD is selected as the output of the integer precision motion search step. In some examples, to reduce the uncertainty penalty of DMVR refinement, the original MV may be prioritized during the DMVR process. In one example, the SAD between the reference blocks referenced by the initial MV candidate may be reduced by ¼ of its SAD value, thus prioritizing the initial MV candidate.

[0096]

[0107] In the fractional motion search step (fractional sample offset search) in DMVR, the candidate MV pairs selected in the integer precision motion search step (integer sample offset search step) are further refined. In some examples, to reduce computational complexity, the fractional sample refinement is derived using a parametric error surface equation instead of comparing with the SAD of additional searches. The fractional sample refinement is conditionally triggered based on the output of the integer sample offset search step. If the integer sample offset search step ends at the center with the smallest SAD in either the first or second iteration search, the fractional sample refinement is further applied. In the sub-pixel offset estimation based on the parametric error surface, the center position cost and the costs at the four neighboring positions from the center (e.g., in the first step of the integer precision motion search) are used to fit a 2D parabolic error surface equation such as Eq. (4):

[0097]

number

[0098]

number

[0108] In Eq.(5) and Eq.(6), E(-1,0), E(1,0), E(0,1), E(0,-1), and E(0,0) represent the cost values ​​at the five points. min and y min The value of is automatically constrained to be between (-8) and (8) because all cost values ​​are positive and the minimum is E(0,0). The constraint corresponds to a half-pel offset when using 1 / 16-pel MV precision in VVC. The calculated fraction (x min ,y min ) can be added to the integer distance refinement MV to obtain the sub-pixel accuracy refinement delta MV.

[0099]

[0109] A technique called bi-directional optical flow (BDOF) can be used in VVC, for example. BDOF was previously called BIO in JEM. Compared to the JEM version, BDOF in VVC can be a simpler version requiring fewer operations, especially in terms of the number of multiplications and the size of the multipliers.

[0100]

[0110] BDOF can be used to refine the bi-predictive signal of a CU at the 4x4 sub-block level. BDOF can be applied to a CU if the CU satisfies the following conditions (also called requirements for BDOF or a set of conditions for BDOF): (1) The CU is coded using the “true” bi-prediction mode, i.e., one of the two reference pictures is before the current picture in display order, and the other is after the current picture in display order. (2) The distances (e.g., POC differences) from the two reference pictures to the current picture are the same. (3) Both reference pictures are short-term reference pictures. (4) The CU is not coded using affine mode or SbTMVP merge mode. (5) The CU has more than 64 luma samples. (6) Both the CU height and CU width are equal to or greater than 8 luma samples. (7) The BCW Weight Index indicates equal weighting. (8) Weighted prediction (WP) is not enabled for the current CU, and (9) CIIP mode shall not be used for the current CU.

[0101]

[0111] In some cases, BDOF is applied only to the luma component. As the name BDOF suggests, BDOF modes can be based on the concept of optical flow, which assumes that object motion is smooth. For each 4x4 sub-block, motion refinement (v x ,v y ) can be calculated by minimizing the difference between the L0 and L1 predicted samples. Motion refinement can then be used to adjust the bi-predicted sample values ​​within the 4x4 sub-block. BDOF can include the following steps:

[0102]

[0112] First, the horizontal and vertical gradients of the two prediction signals from reference list L0 and reference list L1 are

[0103]

number

[0104]

number

[0105]

[0113] The gradient auto- and cross-correlations S1, S2, S3, S5, and S6 can then be calculated according to Eqs. (9)-(13) as follows:

[0106]

number

[0107]

number

[0108]

[0114] Motion refinement (v x ,v y ) can be derived using Eqs. (17) and (18) in terms of autocorrelation and cross-correlation as follows:

[0109]

number

[0110]

number

[0111]

number

[0115] Finally, the BDOF samples of the CU can be calculated by adjusting the bi-predictive samples in Eq. (20) as follows:

[0112]

number

[0113]

[0116] In some cases, it may be possible to use sample-based BDOF instead of block-based BDOF.

[0114]

[0117] In some examples, multi-pass DMVR can be used. In one example, in the first pass, bilateral matching (BM) is applied to the coding block. In the second pass, BM is applied to each 16x16 sub-block within the coding block. In the third pass, the motion vectors within each 8x8 sub-block are refined by applying bi-directional optical flow (BDOF). The refined motion vectors are preserved for both spatial and temporal motion vector prediction.

[0115]

[0118] Note that in some examples, the MMVD flag is derived to be implicitly false and cannot be signaled if at least one of DMVR and BDOF is available for the coded picture. Thus, MMVD is no longer available for any CU in that example, and a coding loss exists because MMVD is prohibited.

[0116]

[0119] Aspects of the present disclosure provide techniques that allow MMVD to be used when coding a picture even when at least one of DMVR and BDOF is available for coding the picture. Furthermore, in some examples, this technique can diversify candidates for various coding schemes. For example, if a bi-predictor merge candidate qualifies for DMVR or BDOF, the merge candidate is constrained from MMVD. Thus, coding performance can be improved using more diversified coding schemes. In some examples, MMVD signaling can be improved by checking the availability of DMVR and BDOF for the base index of the MMVD.

[0117]

[0120] It should be noted that the techniques may be used separately or combined in any order. In the following description, MMVD may refer to the merged motion vector differential technique in the above description or any variation thereof. Similarly, in the following description, DMVR, BDOF, MP-DMVR, and sample-based BDOF may refer to the techniques in the above description or any variation thereof. Furthermore, the techniques may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0118]

[0121] According to some aspects of the present disclosure, MMVD is allowed when DMVR or BDOF is available for coding a picture, but base candidate selection for the MMVD mode is constrained, and therefore the coding schemes available for coding a picture may be more diversified. In some examples, the base candidate for the MMVD mode is selected from a regular merge candidate list. With respect to the selection of a merge candidate from the regular merge candidate list for the MMVD mode, the merge candidate is not used for the MMVD mode (if the MV predictor of the merge candidate is a bi-predictor and at least one bi-predictive motion refinement method, such as BDOF or DMVR, may be applied to the merge candidate). In other words, if the merge candidate meets the requirements for applying bi-predictive motion refinement, the merge candidate is not selected for the MMVD mode.

[0119]

[0122] In some embodiments, the base candidate list of the MMVD may be formed based on the merge candidates in the regular merge candidate list by selecting one or more merge candidates in the regular merge candidate list. The candidates in the base candidate list may be used to determine the starting point of the MMVD. The number of candidates in the base candidate list of the MMVD is set to be N, where N is a positive integer. In one example (e.g., VVC), N is 2. To form the base candidate list of the MMVD, in one example, if a merge candidate in the regular merge candidate list meets the requirements for applying bi-predictive motion refinement (e.g., is bi-predictor and at least one of the bi-predictive motion refinement methods such as BDOF or DMVR may be applied), the merge candidate is not added to the base candidate list of the MMVD.

[0120]

[0123] In some embodiments, if the number of available base candidates for MMVD derivation is less than N, zero MV for both reference lists (bi-predictive) may be added to the list as a base candidate for MMVD derivation.

[0121]

[0124] In some embodiments, if the number of available base candidates for MMVD derivation is less than N, a MV predictor (a bi-predictor to which at least one bi-predictive motion refinement method, such as BDOF or DMVR, may be applied) may be added to the list as a base candidate for MMVD derivation. In one example, merge candidates in a regular merge candidate list may be checked. In the first round, if a merge candidate in the regular merge candidate list meets the requirements for applying bi-predictive motion refinement (e.g., if it is a bi-predictor and to which at least one bi-predictive motion refinement method, such as BDOF or DMVR, may be applied), the merge candidate is not added to the base candidate list of the MMVD. After the first round, if the number of available base candidates in the base candidate list of the MMVD is less than N, a merge candidate in the regular merge candidate list that meets the requirements for applying bi-predictive motion refinement may be added to the base candidate list of the MMVD.

[0122]

[0125] According to another aspect of the present disclosure, construction of an MMVD candidate list for an MMVD mode is constrained. In one example, a bi-predictive MMVD candidate with an MMVD offset added to both reference lists (e.g., a bi-sided bi-predictive MMVD candidate) may not be used as an MMVD candidate if at least one bi-predictive motion refinement method, such as BDOF or DMVR, may be applied to the coding block.

[0123]

[0126] According to another aspect of the present disclosure, different definitions of MV offsets can be used for different types of MMVD candidates during MMVD candidate list construction. In some examples, two definitions of MV offsets can be used. A first definition of MV offset is used for both bi-predictive MMVD candidates for both reference lists when at least one of bi-predictive motion refinement methods, such as BDOF or DMVR, may be applied to the coding block, and a second definition of MV offset is used for one bi-predictive MMVD candidate (the MV offset is only added to one of the reference lists) or uni-predictive MMVD candidate. In some examples, the MV offsets in the first definition (e.g., the MV offsets of both MMVD bi-predictive candidates) can start at a larger value than the MV offsets in the second definition (e.g., the MV offsets of one bi-predictive MMVD candidate or uni-predictive MMVD). In one example, a relatively small MV offset can be achieved by using DMVR and / or BDOF; and a relatively large MV offset can be achieved by using MMVD.

[0124]

[0127] In one example, a first definition of the MV offsets for bilateral bi-predictive MMVD candidates can use Table 4.

[0125] Table 4

[0126] [Table 4]

[0128] In Table 4, p is a constant value, and its typical value can be 4 or 8 depending on the search range of motion refinement. A second definition of the MV offset for one bi-predictive MMVD candidate or uni-predictive MMVD can use Table 2. The MV offset according to Table 4 is larger than the MV offset according to Table 2.

[0127]

[0129] FIG. 11 shows a flowchart outlining a process (1100) according to an embodiment of the present disclosure. The process (1100) can be used in a video encoder. In various embodiments, the process (1100) is performed by a processing circuit, such as a processing circuit that performs the functions of the video encoder (103), a processing circuit that performs the functions of the video encoder (303), etc. In some embodiments, the process (1100) is implemented by software instructions, and thus, the processing circuit performs the process (1100) when it executes the software instructions. The process begins at (S1101) and proceeds to (S1110).

[0128]

[0130] At (S1110), inter prediction is determined for coding the current block in the picture.

[0129]

[0131] In (S1120), the motion vector predictor of the merging candidate of the current block is a bi-predictor.

[0130]

[0132] At (S1130), it is determined whether a merge candidate for the current block satisfies a requirement for applying bi-predictive motion refinement. In some examples, the encoder needs to determine an inter-prediction technique for use with the current block. In some examples, the encoder diversifies the inter-prediction techniques and determines the best one that can achieve the lowest rate-distortion cost. For example, if bi-predictive motion refinement may be used for a merge candidate, the merge candidate may be constrained (e.g., excluded) from MMVD, such that other merge candidates, e.g., merge candidates to which bi-predictive motion refinement may not be applied, may be used in MMVD. The requirement for applying bi-predictive motion refinement may be at least one of a DMVR requirement and a BDOF requirement.

[0131]

[0133] At (S1140), if the merge candidate for the current block is bi-predictor and meets the requirements, the merge candidate is constrained for use in merge using motion vector differential (MMVD) mode.

[0132]

[0134] At (S1150), a base candidate list for MMVD mode is determined in which merge candidates are constrained for use in MMVD mode.

[0133]

[0135] At (S1160), a selection base is determined from the motion vector differential (MVD) information and base candidate list for the MMVD mode. In some examples, the encoder can determine the MBD information and selection base that can provide the lowest rate-distortion cost among various inter-prediction techniques.

[0134]

[0136] At (S1170), a bitstream is generated having MVD information, a selected base from the base candidate list, and coding information specifying the MMVD mode.

[0135]

[0137] In some examples, the requirements for applying bi-predictive motion refinement include at least one of a first set of conditions for applying decoder-side motion vector refinement (DMVR) and a second set of conditions for applying bi-directional optical flow (BDOF) refinement.

[0136]

[0138] In some examples, to constrain merge candidates for use in MMVD mode, if a merge candidate meets requirements, the merge candidate is left outside (excluded from) the base candidate list for MMVD mode. The base candidate list for MMVD mode is used to determine the starting point for MMVD mode.

[0137]

[0139] In some examples, if a merge candidate does not meet the requirements, the merge candidate is added to the base candidate list in the MMVD mode.

[0138]

[0140] In some examples, if the merge candidate is a uni-predictor, the merge candidate is added to the base candidate list for the MMVD mode as a uni-predictive MMVD candidate.

[0139]

[0141] In some examples, if the merge candidate is bi-predictive and meets the requirements, the merge candidate is added to the base candidate list of the MMVD mode as a half-bi-predictive MMVD candidate, for which MVD is applied to the MV of one of the reference pictures.

[0140]

[0142] In some examples, the merge candidate is a canonical merge candidate from a canonical merge candidate list for a merge mode, the canonical merge candidate list including multiple canonical merge candidates. The multiple canonical merge candidates are each tested for addition to a base candidate list for an MMVD mode. If, after testing the multiple canonical merge candidates, the base candidate list has fewer candidates than a preset number, zero motion vectors may be added to the base candidate list as bilateral bi-predictive MMVD candidates.

[0141]

[0143] In some examples, if, after testing multiple regular merge candidates, the base candidate list has fewer candidates than a preset number, merge candidates that are bi-predictors and meet the requirements are added to the base candidate list. In one example, the merge candidate is added as a bi-sided bi-predictive MMVD candidate. In another example, the merge candidate is added as a single bi-predictive MMVD candidate. In one example, the merge candidate is added after the other candidates in the base candidate list.

[0142]

[0144] In some examples, the selection base (starting point) is determined from the base candidate list of the MMVD mode. In some examples, the first MV offset is determined in response to the selection base being both bi-predictive MMVD candidates. In other examples, the second MV offset is determined in response to the selection base being one bi-predictive MMVD candidate or one-predictive MMVD candidate. The second MV offset is different from the first MV offset. In one example, the first MV offset is greater than the second MV offset.

[0143]

[0145] In one example, the first MV offset is determined according to a first table in response to the selection base being a bi-predictive MMVD candidate, the first table storing first MV offsets associated with distance indices such as Table 4. The second MV offset is determined according to a second table in response to the selection base being one bi-predictive MMVD candidate or one uni-predictive MMVD candidate, the second table storing second MV offsets associated with distance indices such as Table 2.

[0144]

[0146] The process then proceeds to (S1199) and ends.

[0145]

[0147] The process 1100 may be adapted as appropriate. Steps in the process 1100 may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0146]

[0148] FIG. 12 shows a flowchart outlining a process (1200) according to an embodiment of the present disclosure. The process (1200) can be used in a video decoder. In various embodiments, the process (1200) is performed by a processing circuit, such as a processing circuit that performs the functions of the video decoder (110), a processing circuit that performs the functions of the video decoder (210), etc. In some embodiments, the process (1200) is implemented by software instructions, and thus, the processing circuit performs the process (1200) when it executes the software instructions. The process begins at (S1201) and proceeds to (S1210).

[0147]

[0149] At (S1210), coded information for a current block of a current picture is received from a coded video bitstream, the coded information including a syntax element having a value indicating that a merge using motion vector differential (MMVD) mode is applied to the current block.

[0148]

[0150] At (S1220), in response to the motion vector predictor of the merge candidate being a bi-predictor, it is determined whether the merge candidate for the current block satisfies a requirement for applying bi-predictive motion refinement. In one example, the requirement is a requirement for DMVR. In another example, the requirement is a requirement for BDOF.

[0149]

[0151] In (S1230), if the merge candidate for the current block meets the requirements, the merge candidate is constrained for use in the MMVD mode.

[0150]

[0152] At (S1240), the current block is reconstructed according to the MMVD mode using merge candidates constrained for use in the MMVD mode.

[0151]

[0153] In some examples, the requirements for applying bi-predictive motion refinement include at least one of a first set of conditions for applying decoder-side motion vector refinement (DMVR) and a second set of conditions for applying bi-directional optical flow (BDOF) refinement.

[0152]

[0154] In some examples, to constrain merge candidates for use in MMVD mode, a merge candidate is excluded from the MMVD mode base candidate list if the merge candidate meets requirements. The MMVD mode base candidate list is used to determine the starting point for MMVD mode.

[0153]

[0155] In some cases, if a merge candidate fails to meet the requirements, the merge candidate is added to the base candidate list in the MMVD mode.

[0154]

[0156] In some examples, if the merge candidate is a uni-predictor, the merge candidate is added as a uni-predictive MMVD candidate into the base candidate list for the MMVD mode.

[0155]

[0157] In some examples, if a merge candidate is bi-predictive and meets the requirements, the merge candidate is added to the base candidate list for the MMVD mode as a bi-predictive MMVD candidate, where MVD is applied to the MV of one reference picture.

[0156]

[0158] In some examples, the merge candidate is a canonical merge candidate from a canonical merge candidate list for a merge mode, where the canonical merge candidate list includes multiple canonical merge candidates. Each of the multiple canonical merge candidates is tested for inclusion in the base candidate list for the MMVD mode. If, after testing the multiple canonical merge candidates, the base candidate list has fewer than a predetermined number of candidates, a zero motion vector can be added to the base candidate list as a bilateral bi-predictive MMVD candidate.

[0157]

[0159] In some examples, if, after testing multiple regular merge candidates, the base candidate list has fewer candidates than a predetermined number, a merge candidate that is bi-predictive and meets the requirements is added to the base candidate list. In one example, the merge candidate is added as a bi-sided bi-predictive MMVD candidate. In another example, the merge candidate is added as a single bi-predictive MMVD candidate. In one example, the merge candidate is added after the other candidates in the base candidate list.

[0158]

[0160] In some examples, a selection base (starting point) is determined from a base candidate list for an MMVD mode to reconstruct a current block. In one example, a first MV offset is determined depending on whether the selection base is a bi-predictive MMVD candidate on both sides. In another example, a second MV offset is determined depending on whether the selection base is a bi-predictive MMVD candidate or a uni-predictive MMVD candidate. The second MV offset is different from the first MV offset. In one example, the first MV offset is greater than the second MV offset.

[0159]

[0161] In one example, the first MV offset is determined according to a first table in response to the selection base being a bi-predictive MMVD candidate, the first table storing first MV offsets associated with distance indices such as Table 4. The second MV offset is determined according to a second table in response to the selection base being one bi-predictive MMVD candidate or one uni-predictive MMVD candidate, the second table storing second MV offsets associated with distance indices such as Table 2.

[0160]

[0162] The process then proceeds to (S1299) and ends.

[0161]

[0163] The process 1200 may be adapted as appropriate. Steps in the process 1200 may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0162]

[0164] The techniques described above may be implemented as computer software using computer-readable instructions and may be physically stored on one or more computer-readable media. For example, Figure 13 illustrates a computer system (1300) suitable for implementing certain embodiments of the disclosed subject matter.

[0163]

[0165] Computer software may be coded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms to create code that contains instructions that may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or that may go through interpretation, microcode execution, etc.

[0164]

[0166] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0165]

[0167] 13 for computer system 1300 are exemplary in nature and are not intended to suggest any limitation on the scope of functionality or application of the computer software implementing embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement related to any one or combination of components illustrated in the exemplary embodiment of computer system 1300.

[0166]

[0168] The computer system (1300) may include certain human interface input devices that may respond to input by one or more human users, for example, via tactile input (e.g., keystrokes, swipes, data glove movements), auditory input (e.g., voice, claps), visual input (e.g., gestures), or olfactory input (not shown). Human interface devices may also be used to capture certain media that do not necessarily involve direct conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still-image cameras), and video (e.g., two-dimensional video, three-dimensional video, including stereoscopic pictures).

[0167]

[0169] The input human interface devices may include one or more of (only one of each is depicted) a keyboard (1301), a mouse (1302), a trackpad (1303), a touch screen (1310), a data glove (not shown), a joystick (1305), a microphone (1306), a scanner (1307), and a camera (1308).

[0168]

[0170] The computer system (1300) may also include certain human interface output devices that may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices can include haptic output devices (e.g., haptic feedback via a touch screen (1310), data gloves (not shown), joystick (1305), although there can be haptic feedback devices that do not serve as input devices), auditory output devices (e.g., speakers (1309), headphones (not shown)), visual output devices (e.g., screens (1310), including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output, three-dimensional or higher output by means such as stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0169]

[0171] The computer system (1300) may also include human-accessible storage devices and associated media, such as optical media including CD / DVD ROM / RW (1320) using media such as CD / DVD (1321), thumb drives (1322), removable hard drives or solid state drives (1323), legacy magnetic media (not shown) such as tape and floppy disks (not shown), and specialized ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0170]

[0172] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not encompass transmission media, carrier waves, or other transitional signals.

[0171]

[0173] The computer system (1300) may also include interfaces to one or more communications networks (1355). Networks may be, for example, wireless, wired, or optical. Networks may further be local, wide area, metropolitan, automotive, real-time, delay-tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), TV wired or wireless wide area digital networks (including cable TV, satellite TV, and terrestrial TV), automotive networks including CANBus, etc. Particular networks typically require an external network interface adapter attached to a particular general-purpose data port or peripheral bus (1349) (e.g., a USB port on the computer system (1300)); others are commonly integrated into the core of the computer system (1300) by attaching to a system bus, as described below (e.g., an Ethernet interface is integrated in a PC computer system, and a cellular network interface is integrated in a smartphone computer system). Using any of these networks, the computer system (1300) can communicate with other entities. Such communication can be one-way receive-only (e.g., broadcast TV), one-way transmit-only (e.g., CANbus to certain CANbus devices), or bidirectional, such as with other computer systems using local or wide-area digital networks. Specific protocols and protocol stacks, as described above, can be used with each of these networks and network interfaces.

[0172]

[0174] The aforementioned human interface devices, human accessible storage devices, and network interfaces can be attached to the core (1340) of the computer system (1300).

[0173]

[0175] The core (1340) may include one or more central processing units (CPUs) (1341), graphics processing units (GPUs) (1342), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (1343), task-specific hardware accelerators (1344), graphics adapters (1350), etc. These devices, along with read-only memory (ROM) (1345), random access memory (1346), and internal mass storage devices (e.g., internal non-user-accessible hard drives, SSDs, etc.) (1347), may be connected via a system bus (1348). In some computer systems, the system bus (1348) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1348) or via a peripheral bus (1349). In one example, a screen 1310 can be connected to a graphics adapter 1350. Peripheral bus architectures include PCI, USB, etc.

[0174]

[0176] The CPU (1341), GPU (1342), FPGA (1343), and accelerator (1344) may combine to execute specific instructions that may constitute the aforementioned computer code. The computer code may be stored in ROM (1345) or RAM (1346). Temporary data may be stored in RAM (1346), while persistent data may be stored, for example, in internal mass storage (1347). Rapid storage and retrieval from any memory device may be enabled through the use of cache memory, which may be closely associated with one or more of the CPU (1341), GPU (1342), mass storage (1347), ROM (1345), RAM (1346), etc.

[0175]

[0177] The computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those having skill in the computer software arts.

[0176]

[0178] By way of example, and not limitation, a computer system having the architecture (1300), and in particular the core (1340), can provide functionality as a result of operations by a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, as well as specific storage of the core (1340) that is non-transitory in nature, such as the core's internal mass storage (1347) or ROM (1345). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (1340). The computer-readable media can include one or more memory devices or chips, depending on particular needs. The software can cause the core (1340), and in particular the processor (including a CPU, GPU, FPGA, etc.) therein, to perform certain processes or portions of certain processes described herein, including defining data structures stored in RAM (1346) and modifying such data structures according to processes defined by the software. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embedded in circuitry (e.g., accelerator (1344)), which may execute in place of or in conjunction with software to perform a particular process or portion of a particular process described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry embodying logic for execution, or both, as appropriate. The present disclosure encompasses any appropriate combination of hardware and software.

[0177]

[0179] The use of "at least one" in this disclosure is intended to include any one or combination of the listed elements. For example, reference to at least one of A, B, or C refers to at least one of A, B, and C; at least one of A, B, and / or C; and at least one of A through C; is intended to include A only, B only, C only, or any combination thereof.

[0178]

[0180] While this disclosure describes a number of exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. It will thus be understood that those skilled in the art will be able to devise many systems and methods that, while not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope.

[0179]

[0181] Additional notes (Appendix 1) 1. A video decoding method comprising: receiving coded information for a current block in a current picture from a coded video bitstream, the coded information including a syntax element having a value indicating that a merge using motion vector differential (MMVD) mode is applied to the current block; determining whether the merge candidate for the current block satisfies a requirement for applying bi-predictive motion refinement according to whether the motion vector predictor of the merge candidate is a bi-predictor; constraining a merge candidate for the current block for use in the MMVD mode if the merge candidate satisfies the requirements; and reconstructing the current block in the MMVD mode using the merge candidates constrained for use in the MMVD mode; A method comprising:

[0180] (Appendix 2) 2. The method of claim 1, wherein the requirements for applying bi-predictive motion refinement include at least one of a first set of conditions for applying decoder-side motion vector refinement (DMVR) and a second set of conditions for applying bi-directional optical flow (BDOF) refinement.

[0181] (Appendix 3) 12. The method of claim 1, wherein the step of constraining the merging candidates for use in the MMVD mode further comprises: removing the merge candidate from the base candidate list for the MMVD mode if the merge candidate satisfies the requirement; wherein the base candidate list for the MMVD mode is used to determine a starting point for the MMVD mode.

[0182] (Appendix 4) 3. The method of claim 2, further comprising: if the merge candidate does not satisfy the requirement, adding the merge candidate to a base candidate list of the MMVD mode; If the merge candidate is a uni-predictor, adding the merge candidate as a uni-predictive MMVD candidate into a base candidate list for the MMVD mode; and adding the merge candidate into a base candidate list of the MMVD mode as one bi-predictive MMVD candidate if the merge candidate is bi-predictive and satisfies the requirements; A method comprising at least one of the following:

[0183] (Appendix 5) 4. The method of claim 3, wherein the merge candidate is a canonical merge candidate from a canonical merge candidate list for a merge mode, the canonical merge candidate list including a plurality of canonical merge candidates, the method comprising: testing each of the plurality of regular merge candidates for inclusion in the base candidate list for the MMVD mode; making a determination after said testing that said base candidate list has fewer than a predetermined number of candidates; and adding the zero motion vector as a bilateral bi-predictive MMVD candidate into the base candidate list; A method comprising:

[0184] (Appendix 6) 4. The method of claim 3, wherein the merge candidate is a canonical merge candidate from a canonical merge candidate list for a merge mode, the canonical merge candidate list including a plurality of canonical merge candidates, the method comprising: testing each of the plurality of regular merge candidates for inclusion in the base candidate list for the MMVD mode; making a determination after said testing that said base candidate list has fewer than a predetermined number of candidates; and adding merge candidates that are bi-predictors and satisfy the requirements into the base candidate list; A method comprising:

[0185] (Appendix 7) 7. The method of claim 6, wherein adding the merge candidate to the base candidate list further comprises: adding the merge candidate as both bi-predictive MMVD candidates; and adding the merge candidate as one of the bi-predictive MMVD candidates; The method includes at least one of:

[0186] (Appendix 8) 7. The method of claim 6, wherein adding the merge candidate to the base candidate list further comprises: adding the merge candidate after other candidates in the base candidate list; A method comprising: (Appendix 9) 10. The method of claim 1, wherein the step of reconstructing the current block further comprises: determining a selected base from the base candidate list of the MMVD mode; determining a first MV offset in response to the selection base being both bi-predictive MMVD candidates; and determining a second MV offset different from the first MV offset according to whether the selection base is one of the bi-predictive MMVD candidates or one of the uni-predictive MMVD candidates; A method comprising:

[0187] (Appendix 10) 10. The method of claim 1, wherein the step of reconstructing the current block further comprises: determining a selected base from the base candidate list of the MMVD mode; determining a first MV offset according to a first table in response to the selection base being both bi-predictive MMVD candidates, the first table storing first MV offsets associated with distance indexes; and determining a second MV offset according to whether the selection base is one of the bi-predictive MMVD candidates or one-predictive MMVD candidates according to a second table, the second table storing second MV offsets associated with distance indexes; A method comprising:

[0188] (Appendix 11) 1. A video decoding device comprising: receiving coded information for a current block in a current picture from a coded video bitstream, the coded information including a syntax element having a value indicating that a merge using motion vector differential (MMVD) mode is applied to the current block; determining whether the merge candidate for the current block satisfies a requirement for applying bi-predictive motion refinement according to whether the motion vector predictor of the merge candidate is a bi-predictor; constraining a merge candidate for the current block for use in the MMVD mode if the merge candidate satisfies the requirements; and reconstructing the current block in the MMVD mode using the merge candidates constrained for use in the MMVD mode; 1. An apparatus comprising: a processing circuit configured to:

[0189] (Appendix 12) 12. The apparatus of claim 11, wherein the requirements for applying the bi-predictive motion refinement include at least one of a first set of conditions for decoder-side motion vector refinement (DMVR) and a second set of conditions for bi-directional optical flow (BDOF) refinement.

[0190] (Appendix 13) 12. The apparatus of claim 11, wherein the processing circuitry: leaving the merge candidate outside the base candidate list of the MMVD mode if the merge candidate satisfies the requirements; wherein a base candidate list for the MMVD mode is used to determine a starting point for the MMVD mode.

[0191] (Appendix 14) 14. The apparatus of claim 13, wherein the processing circuitry: if the merge candidate fails to meet the requirements, adding the merge candidate into a base candidate list of the MMVD mode; If the merge candidate is a uni-predictor, adding the merge candidate as a uni-predictive MMVD candidate into a base candidate list for the MMVD mode; and adding the merge candidate into a base candidate list of the MMVD mode as one bi-predictive MMVD candidate if the merge candidate is bi-predictive and satisfies the requirements; 10. An apparatus configured to perform at least one of:

[0192] (Appendix 15) 14. The apparatus of claim 13, wherein the merge candidate is a regular merge candidate from a regular merge candidate list for a merge mode, the regular merge candidate list including a plurality of regular merge candidates, and the processing circuitry: testing each of the plurality of regular merge candidates for inclusion in the base candidate list for the MMVD mode; making a determination after said testing that said base candidate list has fewer than a predetermined number of candidates; and adding the zero motion vector as a bilateral bi-predictive MMVD candidate into the base candidate list; The apparatus is configured to:

[0193] (Appendix 16) 14. The apparatus of claim 13, wherein the merge candidate is a regular merge candidate from a regular merge candidate list for a merge mode, the regular merge candidate list including a plurality of regular merge candidates, and the processing circuitry: testing each of the plurality of regular merge candidates for inclusion in the base candidate list for the MMVD mode; making a determination after said testing that said base candidate list has fewer than a predetermined number of candidates; and adding merge candidates that are bi-predictors and satisfy the requirements into the base candidate list; The apparatus is configured to: (Appendix 17) 17. The apparatus of claim 16, wherein the processing circuitry: adding the merge candidate as at least one of a double-sided bi-predictive MMVD candidate and a single-sided bi-predictive MMVD candidate into the base candidate list; The apparatus is configured to: (Appendix 18) 17. The apparatus of claim 16, wherein the processing circuitry: adding the merge candidate after other candidates in the base candidate list; The apparatus is configured to: (Appendix 19) 12. The apparatus of claim 11, wherein the processing circuitry: determining a selected base from the base candidate list of the MMVD mode; determining a first MV offset in response to the selection base being both bi-predictive MMVD candidates; and determining a second MV offset different from the first MV offset according to whether the selection base is one of the bi-predictive MMVD candidates or one of the uni-predictive MMVD candidates; The apparatus is configured to: (Appendix 20) 12. The apparatus of claim 11, wherein the processing circuitry: determining a selected base from the base candidate list of the MMVD mode; determining a first MV offset according to a first table in response to the selection base being both bi-predictive MMVD candidates, the first table storing first MV offsets associated with distance indexes; and determining a second MV offset according to whether the selection base is one of the bi-predictive MMVD candidates or one-predictive MMVD candidates according to a second table, the second table storing second MV offsets associated with distance indexes; The apparatus is configured to:

Claims

1. 1. A video decoding method comprising: receiving coded information for a current block in a current picture from a coded video bitstream, the coded information including a syntax element having a value indicating that a merge using motion vector differential (MMVD) mode is applied to the current block; determining whether the merge candidate for the current block satisfies a requirement for applying bi-predictive motion refinement in response to the motion vector predictor of the merge candidate being a bi-predictor; constraining a merge candidate for the current block for use in the MMVD mode if the merge candidate satisfies the requirements; and reconstructing the current block in the MMVD mode using the merge candidates constrained for use in the MMVD mode; A method comprising:

2. 2. The method of claim 1, wherein the requirements for applying bi-predictive motion refinement include at least one of a first set of conditions for applying decoder-side motion vector refinement (DMVR) and a second set of conditions for applying bi-directional optical flow (BDOF) refinement.

3. 10. The method of claim 1, wherein constraining the merge candidates for use in the MMVD mode further comprises: removing the merge candidate from the base candidate list for the MMVD mode if the merge candidate satisfies the requirement; wherein the base candidate list for the MMVD mode is used to determine a starting point for the MMVD mode.

4. 4. The method of claim 3, further comprising: if the merge candidate does not satisfy the requirement, adding the merge candidate to a base candidate list for the MMVD mode; If the merge candidate is a uni-predictor, adding the merge candidate as a uni-predictive MMVD candidate into a base candidate list for the MMVD mode; and adding the merge candidate into a base candidate list for the MMVD mode as one bi-predictive MMVD candidate if the merge candidate is bi-predictive and satisfies the requirements; A method comprising at least one of the following:

5. 4. The method of claim 3, wherein the merge candidate is a regular merge candidate from a regular merge candidate list for a merge mode, the regular merge candidate list including a plurality of regular merge candidates, the method comprising: testing each of the plurality of regular merge candidates for inclusion in the base candidate list for the MMVD mode; making a determination after said testing that said base candidate list has fewer than a predetermined number of candidates; and adding the zero motion vector as a bilateral bi-predictive MMVD candidate into the base candidate list; A method comprising:

6. 4. The method of claim 3, wherein the merge candidate is a regular merge candidate from a regular merge candidate list for a merge mode, the regular merge candidate list including a plurality of regular merge candidates, the method comprising: testing each of the plurality of regular merge candidates for inclusion in the base candidate list for the MMVD mode; making a determination after said testing that said base candidate list has fewer than a predetermined number of candidates; and adding merge candidates that are bi-predictors and meet the requirements into the base candidate list; A method comprising:

7. 7. The method of claim 6, wherein adding the merge candidate to the base candidate list further comprises: adding the merge candidate as both bi-predictive MMVD candidates; and adding the merge candidate as one of the bi-predictive MMVD candidates; The method includes at least one of:

8. 7. The method of claim 6, wherein adding the merge candidate to the base candidate list further comprises: adding the merge candidate after other candidates in the base candidate list; A method comprising:

9. 10. The method of claim 1, wherein the step of reconstructing the current block further comprises: determining a selected base from the base candidate list of the MMVD mode; determining a first MV offset in response to the selection base being both bi-predictive MMVD candidates; and determining a second MV offset different from the first MV offset in response to the selection base being one of the bi-predictive MMVD candidates or one of the uni-predictive MMVD candidates; A method comprising:

10. 10. The method of claim 1, wherein the step of reconstructing the current block further comprises: determining a selected base from the base candidate list of the MMVD mode; determining a first MV offset according to a first table in response to the selection base being both bi-predictive MMVD candidates, the first table storing first MV offsets associated with distance indexes; and determining a second MV offset according to a second table according to whether the selection base is one of the bi-predictive MMVD candidates or one of the uni-predictive MMVD candidates, the second table storing second MV offsets associated with distance indexes; A method comprising:

11. A computer program product that causes a computer to carry out the method according to any one of claims 1 to 10.

12. A video decoding device comprising a processing circuit configured to perform the method of any one of claims 1 to 10.

13. 1. A video encoding method comprising: determining to use inter prediction for coding a current block in a picture; determining whether a merge candidate for the current block satisfies requirements for applying bi-predictive motion refinement; determining a base candidate list for a merge using motion vector differential (MMVD) mode by constraining a merge candidate for the current block to not be used in the MMVD mode if the merge candidate is bi-predictor and satisfies the requirement; determining motion vector differential (MVD) information for the MMVD mode and a selected base from the base candidate list; generating an encoded video bitstream having coding information specifying the MMVD mode and the selection base; A method comprising:

Images