Method, apparatus and program for enhanced decoder-side motion vector refinement

Abstract

The processing circuit receives a coded video bitstream, the coded video bitstream including coding information for a first block, the coding information indicating the use of decoder-side motion refinement (DMVR) for the first block. The first block has a first sample size. The processing circuit applies an enhanced DMVR to the first block, the enhanced DMVR using a second sample size for a two-sided matching cost calculation, the second sample size being different from the first sample size of the first block. The processing circuit calculates a plurality of two-sided matching costs respectively associated with a plurality of motion vector adjustments, and at least a first two-sided matching cost of the plurality of two-sided matching costs is calculated based on the second sample size, which is different from the first sample size of the first block. The processing circuit determines a first refined motion vector and performs reconstruction accordingly.

Description

[Technical Field]

[0001] Incorporation by Reference This application claims the benefit of priority to U.S. Patent Application No. 18 / 378,085, entitled "Method and Apparatus for Enhanced Decoder-Side Motion Vector Refinement," filed October 9, 2023, which in turn claims the benefit of priority to U.S. Provisional Application No. 63 / 414,776, entitled "Method and Apparatus for Enhanced Decoder-Side Motion Vector Refinement," filed October 10, 2022. The disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002] Technical Field This disclosure describes embodiments generally related to video coding. [Background technology]

[0003] The background discussion provided herein is intended to generally set forth the context of the present disclosure. The work of the inventors identified in this application, to the extent that their work is described in this background section, as well as aspects of this specification that may not qualify as prior art as of the filing date, are not admitted expressly or impliedly as prior art to the present disclosure.

[0004] Image / video compression can help transmit image / video data between various devices, storage, and networks with minimal quality degradation. In some examples, video codec technology can compress video based on spatial and temporal redundancy. In one example, a video codec can use a technique called intra-prediction, which can compress images based on spatial redundancy. For example, intra-prediction can use reference data from the current picture being reconstructed for sample prediction. In another example, a video codec can use a technique called inter-prediction, which can compress images based on temporal redundancy. For example, inter-prediction can predict samples of a current picture from a previously reconstructed picture using motion compensation. Motion compensation can be indicated by a motion vector (MV). Summary of the Invention [Means for solving the problem]

[0005] Aspects of the present disclosure include methods and apparatus for video encoding / decoding. In some examples, the apparatus for video decoding includes a processing circuit. The processing circuit receives an encoded video bitstream, the encoded video bitstream including coding information for a first block, the coding information indicating use of decoder side motion refinement (DMVR) for the first block. The first block has a first sample size. The processing circuit applies an extended DMVR to the first block, the extended DMVR using a second sample size for two-sided matching cost calculation, the second sample size being different from the first sample size of the first block. The processing circuit calculates multiple two-sided matching costs respectively associated with the multiple motion vector adjustments, and at least a first two-sided matching cost of the multiple two-sided matching costs is calculated based on the second sample size, different from the first sample size of the first block. The processing circuit determines a first refined motion vector for the first block based on the multiple bilateral matching costs associated with the multiple motion vector adjustments, and reconstructs the first block having a first sample size based on the first refined motion vector.

[0006] According to an aspect of the present disclosure, the first sample size is larger than the second sample size.

[0007] According to an aspect of the present disclosure, the first block is a sub-block within a larger block, and the first sample size is smaller than the second sample size.

[0008] In some examples, the first block has a first width of N luma samples and a first height of M luma samples, the first sample size corresponds to the number of samples in the first block, the second sample size corresponds to the number of samples in a second block having a second width and a second height, the second width is 2×S1 luma samples larger than the first width, and the second height is 2×S2 luma samples larger than the first height, and N, M, S1, and S2 are positive integers.

[0009] In some examples, the first block has a first width of N luma samples and a first height of M luma samples, the first sample size corresponding to the number of samples in the first block, the second sample size corresponding to the number of samples in a second block having a second width and a second height, the second width being a multiple of the first width, and the second height being a multiple of the first height.

[0010] In some examples, the processing circuit determines an enhanced DMVR block having a center aligned with the first block, the enhanced DMVR block having a second sample size. The processing circuit determines a first enhanced DMVR reference block in the first reference picture and a second enhanced DMVR reference block in the second reference picture corresponding to the enhanced DMVR block according to the first motion vector adjustment, the first enhanced DMVR reference block and the second enhanced DMVR reference block having the second sample size. The processing circuit calculates a first bilateral matching cost between the first enhanced DMVR reference block and the second enhanced DMVR reference block.

[0011] In some examples, the processing circuit determines a first DMVR reference block in the first reference picture and a second DMVR reference block in the second reference picture corresponding to the first block according to the first motion vector adjustment, where the first DMVR reference block and the second DMVR reference block have a first sample size. The processing circuit determines a first extended DMVR reference block having a center aligned with the first DMVR reference block, where the first extended DMVR reference block has a second sample size. The processing circuit determines a second extended DMVR reference block having a center aligned with the second DMVR reference block, where the second extended DMVR reference block has a second sample size. The processing circuit calculates a first bilateral matching cost between the first extended DMVR reference block and the second extended DMVR reference block.

[0012] In some examples, the processing circuitry determines to apply the enhanced DMVR in response to the first sample size being less than a threshold value.

[0013] In some examples, the processing circuitry determines that a boundary of the first block is at a boundary of a higher-level processing unit and constrains the boundary from expansion for additional samples.

[0014] According to another aspect of the present disclosure, a processing circuit determines a second refined motion vector for a second block by applying a DMVR to the second block, the second block and the first block having an overlap area, and the processing circuit reconstructs the overlap area based on the first refined motion vector and the second refined motion vector.

[0015] It should be noted that the position difference between the first block and the second block is in at least one of the horizontal and / or vertical directions.

[0016] In some examples, the processing circuitry selects a refined motion vector associated with a lower bilateral matching cost from the first refined motion vector and the second refined motion vector, and reconstructs the overlap area according to the refined motion vector.

[0017] In some examples, the processing circuit calculates an average motion vector of the first refined motion vector and the second refined motion vector, and reconstructs the overlap area according to the average motion vector.

[0018] In some examples, the processing circuit calculates a weighted average motion vector of the first refined motion vector and the second refined motion vector, where the weight for the refined motion vector is proportional to the inverse of a bilateral matching cost of the refined motion vector, and the processing circuit reconstructs the overlap area according to the weighted average motion vector.

[0019] In some examples, the processing circuit generates a first predicted sample for a sample in the overlap area based on a first refined motion vector, generates a second predicted sample for the sample in the overlap area based on a second refined motion vector, and calculates a weighted average of the first predicted sample and the second predicted sample as a reconstructed sample for the sample in the overlap area.

[0020] In some examples, the processing circuit decodes at least one syntax element at a sequence level, a picture level, a slice level, a tile group level, and a tile level, the syntax element having a value indicating a reconstruction of the overlap area based on the first refined motion vector and the second refined motion vector.

[0021] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for video decoding / encoding. [Brief explanation of the drawings]

[0022] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings.

[0023] [Figure 1] FIG. 1 is a schematic diagram of an exemplary block diagram of a communication system (100).

[0024] [Figure 2] FIG. 2 is a schematic diagram of an exemplary block diagram of a decoder.

[0025] [Figure 3] FIG. 2 is a schematic diagram of an exemplary block diagram of an encoder.

[0026] [Figure 4] 1 shows an exemplary schematic diagram of decoder-side motion vector refinement based on two-sided matching in one example.

[0027] [Figure 5] 1 illustrates an illustration of an expanded region of a block in bidirectional optical flow (BDOF) in one example.

[0028] [Figure 6] 1 shows the search area in some examples.

[0029] [Figure 7] 10 illustrates a sub-block diagram for decoder-side motion vector refinement in one example.

[0030] [Figure 8] 10A-10C show diagrams of blocks with overlapping sub-blocks in some examples.

[0031] [Figure 9] 10A-10C show diagrams of blocks with overlapping sub-blocks in some examples.

[0032] [Figure 10] 10A-10C show diagrams of blocks with overlapping sub-blocks in some examples.

[0033] [Figure 11] 10A-10C show diagrams of blocks with overlapping sub-blocks in some examples.

[0034] [Figure 12] 1 shows a flowchart outlining another process according to some embodiments of the present disclosure.

[0035] [Figure 13] 1 shows a flowchart outlining a process according to some embodiments of the present disclosure.

[0036] [Figure 14] 1 is a schematic diagram of a computer system according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0037] 1 illustrates a block diagram of a video processing system (100) in some examples. The video processing system (100) is an example of an application of the disclosed subject matter, a video encoder and video decoder in a streaming environment. The disclosed subject matter is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, streaming services, and storing compressed video on digital media, including CDs, DVDs, memory sticks, etc.

[0038] The video processing system (100) includes a capture subsystem (113) that may include a video source (101), such as a digital camera, and generates a stream of uncompressed video pictures (102). In one example, the stream of video pictures (102) includes samples captured by the digital camera. The stream of video pictures (102), shown with a thick line to emphasize its large amount of data compared to the encoded video data (104) (or encoded video bitstream), can be processed by an electronic device (120) that includes a video encoder (103) coupled to the video source (101). The video encoder (103) can include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (104) (or encoded video bitstream), shown with a thin line to emphasize its small amount of data compared to the stream of video pictures (102), can be stored on a streaming server (105) for future use. One or more streaming client subsystems, such as the client subsystems (106) and (108) of FIG. 1, can access the streaming server (105) to retrieve copies (107) and (109) of the encoded video data (104). The client subsystem (106) can include a video decoder (110), for example, within an electronic device (130). The video decoder (110) decodes the incoming copy (107) of the encoded video data and generates an outgoing stream of video pictures (111) that can be rendered on a display (112) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (104), (107), and (109) (e.g., a video bitstream) can be encoded according to some video encoding / compression standard.Examples of these standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0039] It should be noted that the electronic devices (120) and (130) may include other components (not shown). For example, the electronic device (120) may include a video decoder (not shown), and the electronic device (130) may include a video encoder (not shown).

[0040] 2 shows an exemplary block diagram of a video decoder (210). The video decoder (210) can be included in an electronic device (230). The electronic device (230) can include a receiver (231) (e.g., a receiving circuit). The video decoder (210) can be used in place of the video decoder (110) in the example of FIG. 1.

[0041] The receiver (231) can receive one or more coded video sequences, e.g., included in a bitstream, to be decoded by the video decoder (210). In some embodiments, one coded video sequence is received at a time, and the decoding of each coded video sequence is independent of the decoding of the other coded video sequences. The coded video sequences can be received from a channel (201), which can be a hardware / software link to a storage device that stores the encoded video data. The receiver (231) can receive the encoded video data along with other data, e.g., coded audio data and / or auxiliary data streams, which can be forwarded to respective usage entities (not shown). The receiver (231) can separate the coded video sequences from other data. To address network jitter, a buffer memory (215) can be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter, "parser (220)"). In some applications, the buffer memory (215) is part of the video decoder (210). In other applications, the buffer memory may be external to the video decoder (210) (not shown). In still other applications, there may be a buffer memory (not shown) external to the video decoder (210), for example, to deal with network jitter, and another buffer memory (215) internal to the video decoder (210), for example, to handle playback timing. If the receiver (231) receives data from a storage / forwarding device with sufficient bandwidth and controllability or from an isosynchronous network, the buffer memory (215) may be unnecessary or may be small. For use with best-effort packet networks such as the Internet, the buffer memory (215) may be necessary, may be relatively large, may be advantageously adaptively sized, and may be implemented, at least in part, in an operating system or similar element (not shown) external to the video decoder (210).

[0042] The video decoder (210) may include a parser (220) that reconstructs symbols (221) from the coded video sequence. These symbol categories, as shown in FIG. 2, include information used to manage the operation of the video decoder (210) and, potentially, information for controlling a rendering device, such as a rendering device (212) (e.g., a display screen) that is not an integral part of the electronic device (230) but may be coupled to the electronic device (230). The control information for the rendering device may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (220) can parse and entropy decode the received coded video sequence. The coding of the coded video sequence may follow a variety of video coding techniques or standards, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser (220) can extract from the coded video sequence a set of subgroup parameters for at least one subgroup of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroup can include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (220) can also extract information from the coded video sequence, such as transform coefficients, quantizer parameter values, and motion vectors.

[0043] The parser (220) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (215) to generate symbols (221).

[0044] The reconstruction of the symbols (221) can involve several different units, depending on the type of coded video picture or part thereof (e.g., inter / intra picture, inter / intra block, etc.) and other factors. Which units are involved and how can be controlled by subgroup control information parsed from the coded video sequence by the parser (220). The flow of such subgroup control information between the parser (220) and the following units is not shown for clarity.

[0045] In addition to the functional blocks already mentioned, the video decoder (210) can be conceptually subdivided into several functional units, as described below. In a practical implementation operating under commercial constraints, many of these units will interact closely with each other and may be, at least partially, integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate.

[0046] The first unit is a scaler / inverse transform unit (251), which receives quantized transform coefficients and control information from the parser (220) as symbols (221), including which transform to use, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit (251) can output blocks containing sample values ​​that can be input to an aggregator (255).

[0047] In some cases, the output samples of the scaler / inverse transform unit (251) may belong to intra-coded blocks. Intra-coded blocks are blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (252). In some cases, the intra-picture prediction unit (252) generates a block of the same size and shape as the block being reconstructed using surrounding, already reconstructed information fetched from the current picture buffer (258). The current picture buffer (258), for example, buffers a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (255) may add, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (252) to the output sample information provided by the scaler / inverse transform unit (251).

[0048] In other cases, the output samples of the scaler / inverse transform unit (251) may relate to an inter-coded, potentially motion-compensated, block. In such cases, the motion-compensated prediction unit (253) may access a reference picture memory (257) to fetch samples used for prediction. After motion-compensating the fetched samples according to the symbols (221) associated with the block, these samples may be added by an aggregator (255) to the output of the scaler / inverse transform unit (251) (in this case, referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (257) from which the motion-compensated prediction unit (253) fetches the prediction samples may be controlled by motion vectors, which are available to the motion-compensated prediction unit (253) in the form of symbols (221), which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory (257) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, and the like.

[0049] The output samples of the aggregator (255) can be subjected to various loop filtering techniques in a loop filtering unit (256). Video compression techniques can include in-loop filtering techniques controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (256) as symbols (221) from the parser (220). Video compression can also respond to meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence, and can also respond to previously reconstructed, loop-filtered sample values.

[0050] The output of the loop filter unit (256) can be a sample stream that can be output to a rendering device (212) and can also be stored in a reference picture memory (257) for use in future inter-picture prediction.

[0051] Certain coded pictures, once fully reconstructed, can be used as reference pictures for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) can become part of the reference picture memory (257), and a new current picture buffer can be reallocated before starting reconstruction of the next coded picture.

[0052] The video decoder (210) can perform decoding operations according to a given video compression technology or standard, such as ITU-T Recommendation H.265. The coded video sequence can conform to the syntax specified by the video compression technology or standard being used. This means that the coded video sequence conforms to both the syntax of the video compression technology or standard and the profile described in the video compression technology or standard. Specifically, the profile can select certain tools from all tools available in the video compression technology or standard as the only tools available under that profile. Also required for compliance is that the complexity of the coded video sequence must be within a range defined by the level of the video compression technology or standard. In some cases, the level constrains the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level can, in some cases, be further constrained through a hypothetical reference decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0053] In some embodiments, the receiver (231) can receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence(s). The additional data may be used by the video decoder (210) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) improvement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0054] 3 shows an example block diagram of a video encoder (303). The video encoder (303) is included in an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmission circuit). The video encoder (303) can be used in place of the video encoder (103) in the example of FIG. 1.

[0055] The video encoder (303) can receive video samples from a video source (301) (which is not part of the electronic device (320) in the example of Figure 3) that can capture video images to be encoded by the video encoder (303). In another example, the video source (301) is part of the electronic device (320).

[0056] The video source (301) can provide a source video sequence to be encoded by the video encoder (303) in the form of a digital video sample stream, which can be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCB, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (301) can be a storage device that stores prepared video. In a video conferencing system, the video source (301) can be a camera that captures local image information as a video sequence. The video data can be provided as multiple individual pictures that, when viewed in sequence, give the impression of motion. The pictures themselves can be organized as a spatial array of pixels, each of which can contain one or more samples, depending on the sampling structure, color space, etc., in use. The following discussion focuses on samples.

[0057] According to some embodiments, the video encoder (303) may encode and compress pictures of a source video sequence into a coded video sequence (343) in real time, or under any other time constraints as needed. Enforcing an appropriate coding rate is one function of the controller (350). In some embodiments, the controller (350) controls and is operatively coupled to other functional units, as described below. Coupling is not shown for clarity. Parameters set by the controller (350) may include rate control-related parameters (picture skip, quantizer, lambda value for rate-distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (350) may be configured with other appropriate functions for optimizing the video encoder (303) for a particular system design.

[0058] In some embodiments, the video encoder (303) is configured to operate in an encoding loop. As a very simplified explanation, in one example, the encoding loop can include a source coder (330) (e.g., responsible for generating symbols, such as a symbol stream, based on an input picture to be encoded and reference picture(s)) and a (local) decoder (333) embedded in the video encoder (303). The decoder (333) reconstructs the symbols to generate sample data in a manner similar to that generated by a (remote) decoder. The reconstructed sample stream (sample data) is input to a reference picture memory (334). Because decoding of the symbol stream produces bit-accurate results independent of the decoder location (local or remote), the contents of the reference picture memory (334) are also bit-accurate between the local and remote encoders. In other words, the predictive portion of the encoder "sees" the exact same sample values ​​as the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchronism (and the resulting drift if synchronism cannot be maintained, eg, due to channel errors) is also used in several related techniques.

[0059] The operation of the "local" decoder (333) may be the same as a "remote" decoder, such as the video decoder (210) already described in detail in connection with Figure 2. However, briefly referring also to Figure 2, because symbols are available and the encoding / decoding of symbols into an encoded video sequence by the entropy coder (345) and parser (220) may be lossless, the entropy decoding portion of the video decoder (210), including the buffer memory (215) and parser (220), may not be fully implemented in the local decoder (333).

[0060] In some embodiments, decoder technology, with the exception of parsing / entropy decoding, present in a decoder is present in the same or substantially identical functional form in the corresponding encoder. Thus, the disclosed subject matter focuses on the operation of the decoder. A description of the encoder technology can be omitted, as it is the inverse of the decoder technology, which is comprehensively described. In certain areas, more detailed descriptions are provided below.

[0061] In operation, in some examples, the source coder (330) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (332) codes differences between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as predictive references for the input picture.

[0062] The local video decoder (333) can decode the coded video data of a picture that may be designated as a reference picture based on the symbols generated by the source coder (330). The operation of the coding engine (332) can advantageously be a lossy process. When the coded video data is decoded by a video decoder (not shown in FIG. 3), the reconstructed video sequence may be a replica of the source video sequence, typically with some errors. The local video decoder (333) can replicate the decoding process that may be performed on the reference picture by the video decoder and store the reconstructed reference picture in the reference picture memory (334). In this way, the video encoder (303) can locally store a copy of the reconstructed reference picture that has common content with the reconstructed reference picture that would be obtained by the far-end video decoder (in the absence of transmission errors).

[0063] The predictor (335) can perform the prediction search for the coding engine (332). That is, for a new picture to be coded, the predictor (335) can search the reference picture memory (334) for sample data (as candidate reference pixel blocks) or types of metadata, such as reference picture motion vectors, block shapes, etc., that may serve as suitable prediction references for the new picture. The predictor (335) can operate on a sample block or pixel block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (335), the input picture can have prediction references drawn from multiple reference pictures stored in the reference picture memory (334).

[0064] The controller (350) can manage the coding operations of the source coder (330), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0065] The output of all the functional units described above may undergo entropy coding in an entropy coder (345), which converts the symbols produced by the various functional units into a coded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable length coding, or arithmetic coding.

[0066] The transmitter (340) can buffer the coded video sequence produced by the entropy coder (345) and prepare it for transmission over a communication channel (360), which can be a hardware or software link to a storage device that stores the encoded video data. The transmitter (340) can merge the coded video data from the video encoder (303) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0067] The controller (350) can manage the operation of the video encoder (303). During encoding, the controller (350) can assign a certain coding picture type to each coded picture, which can affect the coding technique that can be applied to each picture. For example, pictures may often be assigned as one of the following picture types:

[0068] Intra pictures (I pictures) can be coded and decoded without using other pictures in the sequence as a source of prediction. Some video coders allow various types of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures.

[0069] Predictive pictures (P pictures) may be encoded and decoded using intra prediction or inter prediction, using motion vectors and reference indices to predict the sample values ​​of each block.

[0070] Bidirectionally predicted pictures (B pictures) may be coded and decoded using intra- or inter-prediction, using two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-predictive pictures may use three or more reference pictures and associated metadata for the reconstruction of a single block.

[0071] A source picture is typically spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and may be coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks, as determined by the coding assignment applied to the block's respective picture. For example, blocks of an I picture may be non-predictively coded or predictively coded with reference to previously coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of a P picture may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. Blocks of a B picture may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.

[0072] The video encoder (303) may perform encoding operations in accordance with a predetermined video encoding technique or standard, such as ITU-T Recommendation H.265. In its operations, the video encoder (303) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancies in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard being used.

[0073] In some embodiments, the transmitter (340) can transmit additional data along with the encoded video. The source coder (330) can include such data as part of the coded video sequence. The additional data may include other forms of redundant data, such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0074] Video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or otherwise) between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is divided into blocks. If a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture can be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0075] In some embodiments, inter-picture prediction may use a bi-prediction technique. According to the bi-prediction technique, two reference pictures, such as a first reference picture and a second reference picture, are used, both of which precede a current picture in a video in decoding order (but may be past and future in display order, respectively). A block in the current picture may be coded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. A block may be predicted by a combination of the first and second reference blocks.

[0076] Furthermore, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0077] According to some embodiments of the present disclosure, prediction, such as inter-picture prediction and intra-picture prediction, is performed block-by-block. For example, according to the HEVC standard, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and the CTUs within a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree-decomposed into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one CU of 64x64 pixels, four CUs of 32x32 pixels, or 16 CUs of 16x16 pixels. In one example, each CU is analyzed to determine a prediction type for the CU (e.g., inter prediction type or intra prediction type). The CU is divided into one or more prediction units (PUs) depending on temporal and / or spatial predictability. Generally, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of values ​​(e.g., luma values) for pixels of 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0078] It should be noted that the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using any suitable technology. In one embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more processors executing software instructions.

[0079] In inter-picture prediction, a merge mode can be used to improve coding efficiency. In merge mode, motion vectors are derived from neighboring blocks and directly used for motion compensation. To improve the accuracy of the MV in merge mode, bilateral-matching (BM)-based decoder-side motion vector refinement can be applied, such as in VVC (VMV). In bi-predictive operation, a refined MV can be searched around the initial MV in reference picture list L0 and reference picture list L1. The BM method can calculate the distortion between two candidate blocks in reference picture list L0 and list L1.

[0080] FIG. 4 shows an exemplary schematic diagram of BM-based decoder-side motion vector refinement. As shown in FIG. 4, a current picture (402) may include a current block (408). The current picture may include a reference picture list L0 (404) and a reference picture list L1 (406). For the current block (408), an initial reference block 412 may be located in the reference picture list L0 (404) based on an initial motion vector MV0, and an initial reference block (414) may be located in the reference picture list L1 (406) based on an initial motion vector MV1. A search process may be performed around the initial MV0 in the reference picture list L0 (404) and the initial MV1 in the reference picture list L1 (406). For example, a first candidate reference block (410) may be identified in the reference picture list L0 (404), and a second candidate reference block (416) may be identified in the reference picture list L1 (406). The sum of absolute differences (SAD) between candidate reference blocks (e.g., (410) and (416)) based on each MV candidate (e.g., MV0′ and MV1′) around the initial MV (e.g., MV0 and MV1) can be calculated. The MV candidate with the lowest SAD becomes the refined MV and can be used to generate a bi-predictive signal for predicting the current block (408).

[0081] In some instances (e.g., VVC), DMVR is implicitly applied at the block level without the need to signal side information in the bitstream. If a block satisfies certain conditions, the DMVR algorithm is invoked. For example, the conditions (also called requirements for DMVR or a set of conditions for DMVR) can include the following: (1) CU-level merge mode with bi-predictive MV (2) For a current picture, one reference picture is in the past and another reference picture is in the future. (3) The distances (e.g., POC differences) from the two reference pictures to the current picture are the same. (4) Both reference pictures are short-term reference pictures. (5) The CU has more than 64 luma samples. (6) Both CU height and CU width are equal to or greater than 8 luma samples. (7) The weight index for bi-prediction with CU level weights (BCW) indicates equal weights. (8) Weighted prediction (WP) is not enabled for the current block. (9) The combined inter and intra prediction (CIIP) mode is not used for the current block.

[0082] Note that the refined MVs derived by the DMVR process are used to generate inter prediction samples and may be used in temporal motion vector prediction for future picture coding. In some instances, the original MVs are used in the deblocking process and are also used in spatial motion vector prediction for future CU coding.

[0083] In some cases (e.g., VVC), decoder-side motion vector refinement (DMVR) is applied to CUs coded in normal merge mode. MV pairs obtained from normal merge candidates are used as input for the DMVR process. DMVR is performed by refining the input MV pair {mv L0 ,mv L1}, and refine the refined MV pair {mv refinedL0 ,mv refinedL1 The output MV of the DMVR may be called a refined MV pair and can be expressed by equation (1): MV refinedL0 =mv L0 +Δmv MV refinedL1 =mv L1 -Δmv Equation (1)

[0084] By using the MVD mirroring property, a motion vector difference Δmv is applied to the input MV pair to obtain a refined MV pair, because the input MV pair refers to two different reference pictures that have equal differences in picture order count (POC) relative to the current picture, and these two reference pictures are in different temporal directions.

[0085] In some examples, in DMVR, a luma coding block is divided into 16x16 sub-blocks for the MV refinement process. The refinement search range is two integer luma samples from the initial MV. Δmv is derived independently for each sub-block in two steps: an integer precision motion search step (also called integer sample offset search) followed by a fractional motion search step (also called fractional sample offset search). Finally, sub-block motion compensation (MC) is performed to refine the refined MV pair {mv refinedL0 ,mv refinedL1}.

[0086] According to an aspect of the present disclosure, in the integer precision motion search step of DMVR, the search space includes 25 MV candidate pairs that can be expressed by equation (2): MV L0(i,j) =mv L0(0,0) +(i,j) MV L1(i,j) =mv L1(0,0) -(i,j) Equation (2) where (i, j) represent the coordinates of the search point around the initial MV pair, and i and j are integer values ​​between −2 and 2 inclusive in one example. In some examples, the SAD of the initial MV pair is calculated first, and the SAD can be calculated according to equation (3).

number

[0087] In equation (3), W and H denote the weight and height of the sub-block. i,j [m+i,2n+j] and P1 i,j [mi,2n-j] indicates corresponding samples in two reference pictures. If the SAD of the initial MV pair is less than a threshold, the integer precision motion search step of DMVR ends. Otherwise, the SADs of the remaining 24 points are calculated and checked, for example, in raster scan order. The point with the smallest SAD is selected as the output of the integer precision motion search step. In some examples, to reduce the uncertainty penalty of DMVR refinement, the original MV may be favored during the DMVR process. In one example, the SAD between the reference blocks referenced by the initial MV candidate is reduced by ¼ of the SAD value, thereby favoring the initial MV candidate.

[0088] In the fractional motion search step (fractional sample offset search) in DMVR, the candidate MV pairs selected in the integer precision motion search step (integer sample offset search) are further refined. In some examples, to save computational complexity, the fractional sample refinement is derived by using a parametric error surface formula instead of an additional search using SAD comparison. The fractional sample refinement is conditionally invoked based on the output of the integer sample offset search step. If the integer sample offset search step ends with the center having the smallest SAD in either the first or second iteration of the search, the fractional sample refinement is further applied. In parametric error surface-based sub-pixel offset estimation, the center location cost and the costs at four neighboring locations from the center (e.g., in the first step of integer precision motion search) are used to fit a 2D parabolic error surface formula, such as Equation (4): E(x,y)=α(xx min ) 2 +β(yy min ) 2 +γ Equation (4) where (x min ,y min ) corresponds to the fractional position with the minimum cost, and γ corresponds to the minimum cost value. There are five unknown parameters, namely α, β, γ, x min , and y min To determine (x), equation (4) is fitted to five of the 25 SAD costs calculated in the first step. By solving equation (4) using the cost values ​​of the five search points, (x) is obtained, for example, according to equations (5) and (6). min ,y min ) is calculated. x min =(E(-1,0)-E(1,0)) / (2(E(-1,0)+E(1,0)-2E(0,0))) Equation (5) y min =(E(0,-1)-E(0,1)) / (2((E(0,-1)+E(0,1)-2E(0,0))) Equation (6)

[0089] In equations (5) and (6), E(-1,0), E(1,0), E(0,1), E(0,-1), and E(0,0) represent the cost values ​​at the five points. All the cost values ​​are positive, and the minimum value is E(0,0), so x min and y min The value of is automatically constrained to be between -8 and 8. This constraint corresponds to a half pixel offset with 1 / 16 pixel MV precision in VVC. The calculated fraction (x min ,y min ) can be added to the integer distance refined MV to obtain a refined delta MV with sub-pixel accuracy.

[0090] In VVC and other techniques, bilinear interpolation and sample padding may be applied. The resolution of the MV may be, for example, 1 / 16 luma sample. Samples at fractional positions may be interpolated using an 8-tap interpolation filter. In DMVR, the search point may surround the initial fractional pixel MV with an integer sample offset, and therefore, samples at fractional positions need to be interpolated for the DMVR search process. To reduce computational complexity, a bilinear interpolation filter may be used to generate fractional samples for the search process in DMVR. Another important advantage is that by using a bilinear filter with a search range of two samples, DVMR does not access more reference samples than a conventional motion compensation process. After the refined MV is achieved using the DMVR search process, a conventional 8-tap interpolation filter may be applied to generate the final prediction. Due to the lack of access to more reference samples compared to a conventional MC process, samples that may not be needed for the original MV-based interpolation process but may be needed for the refined MV-based interpolation process can be padded from available samples.

[0091] When the width and / or height of a CU is greater than 16 luma samples, the CU may be further divided into sub-blocks with width and / or height equal to 16 luma samples. The maximum unit size of the DMVR search process may be limited to 16x16.

[0092] For example, VVC can use a technique called bi-directional optical flow (BDOF), which was previously called BIO in JEM. Compared to the JEM version, BDOF in VVC can be a simpler version that requires less computation, especially in terms of the number of multiplications and the size of the multipliers.

[0093] BDOF may be used to refine the bi-predictive signal of a CU at the 4x4 sub-block level. BDOF may be applied to a CU if the CU satisfies the following conditions (also called BDOF requirements or a set of BDOF conditions): (1) The CU is coded using a "true" bi-predictive mode, i.e., one of the two reference pictures is before the current picture in display order, and the other is after the current picture in display order. (2) The distances (e.g., POC differences) from the two reference pictures to the current picture are the same. (3) Both reference pictures are short-term reference pictures. (4) The CU is not coded using affine mode or SbTMVP merge mode. (5) The CU has more than 64 luma samples. (6) Both CU height and CU width are equal to or greater than 8 luma samples. (7) The weight index of BCW indicates equal weight. (8) Weighted prediction (WP) is not enabled for the current CU. (9) CIIP mode is not used for the current CU.

[0094] In some examples, BDOF is applied only to the luma component. As the name BDOF suggests, BDOF modes can be based on the concept of optical flow, which assumes that object motion is smooth. For each 4x4 sub-block, motion refinement (v x ,v y ) may be calculated by minimizing the difference between the L0 predicted samples and the L1 predicted samples. The motion refinement may then be used to adjust the bi-predicted sample values ​​within a 4x4 sub-block. BDOF may include steps such as:

[0095] First, the horizontal and vertical gradients of the two predicted signals from reference list L0 and reference list L1

number

number

[0096] Then, the gradient autocorrelations and cross-correlations S1, S2, S3, S5, S6 can be calculated according to equations (9) to (13) as follows:

number

number

[0097] Movement refinement (v x ,v y ) can be derived using the cross-correlation and autocorrelation terms using equations (17) and (18) as follows:

number

number

[0098] Finally, the BDOF samples of the CU may be calculated by adjusting the bi-predictive samples in equation (20), as follows:

number

[0099] To derive the gradient value, we select some predicted samples I in list k (k=0,1) outside the current CU boundary. (k)(i,j) needs to be generated. As shown in FIG. 5, BDOF in VVC can use one extended row / column (502) around the boundary (506) of the CU (504). To control the computational complexity of generating prediction samples outside the boundary, prediction samples in the extended area (e.g., the non-shaded area in FIG. 5) can be generated by taking reference samples at nearby integer positions directly (e.g., using a floor() operation on the coordinates) without interpolation, and a normal 8-tap motion compensation interpolation filter can be used to generate prediction samples in the CU (e.g., the shaded area in FIG. 5). The extended sample values ​​can be used only for gradient calculation. For the remaining steps in the BDOF process, if any sample and gradient values ​​outside the CU boundary are needed, the sample and gradient values ​​can be padded (e.g., repeated) from the nearest neighbors of the sample and gradient values.

[0100] In some instances, sample-based BDOF can be used instead of block-based BDOF. Sample-based BDOF uses block-based motion refinement (v x ,v y ), it is performed sample by sample. The coding block is divided into 8x8 sub-blocks. For each sub-block, whether to apply BDOF is decided by checking the SAD between two reference sub-blocks against a threshold. If it is decided to apply BDOF to a sub-block, a sliding 5x5 window is used for all samples in the sub-block, and the existing BDOF process is applied for all sliding windows to obtain v x and v y The derived motion refinement (v x ,v y ) is applied to adjust the bi-predicted sample value for the center sample of the window.

[0101] In some examples, multi-pass DMVR may be used. In one example, in the first pass, bilateral matching (BM) is applied to the coding block. In the second pass, BM is applied to each 16x16 sub-block in the coding block. In the third pass, the MVs in each 8x8 sub-block are refined by applying bidirectional optical flow (BDOF). The refined MVs are stored for both spatial and temporal motion vector prediction.

[0102] Specifically, the first pass performs block-based bilateral matching MV refinement. In the first pass, a refined MV is derived by applying the BM to the coding block. Similar to decoder-side motion vector refinement (DMVR), in bipredictive operations, a refined MV is searched around two initial MVs (MV0 and MV1) in the reference picture lists L0 and L1. The refined MVs (MV0_pass1 and MV1_pass1) are derived around the initial MV based on the minimum bilateral matching cost between the two reference blocks in L0 and L1. The bilateral matching cost may be calculated by any appropriate error measurement metric that measures the error between the two reference blocks in L0 and L1. In one example, the bilateral matching cost includes a term that is the sum of absolute differences (SAD) between corresponding samples in the two reference blocks in L0 and L1.

[0103] The BM can derive intDeltaMV to integer sample precision by performing a local search, which applies a 3x3 square search pattern to loop through the search range [-sHor,sHor] horizontally and [-sVer,sVer] vertically, where the values ​​of sHor and sVer are determined by the block dimensions and the maximum value of sHor and sVer is 8.

[0104] The bilateral matching cost is calculated as bilCost = mvDistanceCost + sadCost. If the block size cbW × cbH is greater than 64, the mean removed SAD (MRSAD) cost function is applied to remove the DC effect of distortion between reference blocks. If bilCost at the center point of the 3x3 search pattern has the minimum cost, the intDeltaMV local search ends. Otherwise, the current minimum-cost search point becomes the new center point of the 3x3 search pattern, and the search continues for the minimum cost until the end of the search range is reached.

[0105] The existing fractional sample refinement is further applied to derive the final deltaMV. The refined MV after the first pass is derived as follows: MV0_pass1=MV0+deltaMV Equation (21) MV1_pass1=MV1-deltaMV Equation (22)

[0106] In the second pass, subblock-based bilateral matching MV refinement is performed. Specifically, in the second pass, refined MVs are derived by applying BM to 16x16 grid subblocks. For each subblock, refined MVs are searched around the two MVs (MV0_pass1 and MV1_pass1) obtained in the first pass in the reference picture lists L0 and L1. Refined MVs (MV0_pass2(sbIdx2) and MV1_pass2(sbIdx2)) are derived based on the minimum bilateral matching cost between the two reference subblocks in L0 and L1.

[0107] For each sub-block, BM performs a full search to derive the integer sample precision intDeltaMV. The full search has a search range of [-sHor, sHor] in the horizontal direction and [-sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block dimensions and the maximum value of sHor and sVer is 8.

[0108] The bilateral matching cost is calculated by applying a cost factor to the sum of absolute transformed differences (SATD) cost between two reference sub-blocks as bilCost = satdCost × costFactor. In some examples, the search area (2 × sHor+1) × (2 × sVer+1) is divided into up to five diamond-shaped search regions.

[0109] Figure 6 shows some examples of a search area 600. The search area 600 is divided into five search regions 601 to 605. The shapes of the search regions resemble diamonds.

[0110] In some examples, each search region is assigned a costFactor determined by the distance between each search point and the starting MV (intDeltaMV), and each diamond region is processed in order starting from the center of the search area. Within each region, the search points are processed in raster scan order starting from the top left corner of the region and moving toward the bottom right corner. If the minimum bilCost within the current search region is less than a threshold equal to sbW × sbH, the integer-pixel full search terminates; otherwise, the integer-pixel full search continues with the next search region until all search points have been examined. Furthermore, if the difference between the previous minimum cost and the current minimum cost in a successive iteration is less than a threshold equal to the area of ​​the block, the search process terminates.

[0111] In some examples, fractional sample refinement, such as DMVR fractional sample refinement in VVC, is further applied to derive the final deltaMV (sbIdx2). The refined MV in the second pass is derived as follows: MV0_pass2(sbIdx2)=MV0_pass1+deltaMV(sbIdx2) Formula (23) MV1_pass2(sbIdx2)=MV1_pass1-deltaMV(sbIdx2) Formula (24)

[0112] In the third pass, sub-block-based bidirectional optical flow MV refinement may be performed. In particular, in the third pass, refined MVs are derived by applying BDOF to 8x8 grid sub-blocks. For each 8x8 sub-block, BDOF refinement is applied to derive scaled Vx and Vy without clipping, starting from the refined MV of the parent sub-block in the second pass. The derived bioMv(Vx,Vy) is rounded to 1 / 16 sample precision and clipped between -32 and 32. The refined MVs in the third pass (MV0_pass3(sbIdx3) and MV1_pass3(sbIdx3)) are derived as follows: MV0_pass3(sbIdx3)=MV0_pass2(sbIdx2)+bioMv Formula (25) MV1_pass3(sbIdx3)=MV0_pass2(sbIdx2)-bioMv Formula (26)

[0113] In some examples, affine merging using DMVR on a translational model may be performed. In one example, refinement of the base MVs (translational part of the affine model) of the affine model of a coding block coded in affine merging mode may be performed by applying the first step of multi-pass DMVR. For example, when a candidate satisfies the DMVR condition, a translational MV offset is added to all CPMVs of the candidate in the affine merging list. The MV offset is derived by minimizing the cost of two-sided matching, as in DMVR. Note that the DMVR condition is not changed.

[0114] Note that the MV offset search process may be the same as the first pass of the multi-pass DMVR in some examples, such as in the ECM. Specifically, a 3x3 square search pattern is used to loop through a search range of [-8, +8] horizontally and [-8, +8] vertically to find the best integer MV offset. A half-pixel search is then performed around the best integer position, and error surface estimation is performed to find the MV offset with 1 / 16 accuracy. The refined CPMV is stored as the multi-pass DMVR in the ECM for both spatial and temporal motion vector prediction.

[0115] According to one aspect of the present disclosure, the existing DMVR refinement process for sub-blocks is based on a bilateral matching (BM) cost of the same sub-block size. When the granularity of the refined MV becomes finer (smaller sub-block size), the bilateral matching cost at the sub-block level may become less accurate due to the reduced number of pixels.

[0116] Aspects of this disclosure provide techniques that can improve the coding gain of DMVR. In some examples, techniques are used to extend the sample area for BM cost calculation beyond the sub-block size, which can increase cost calculation accuracy. In some examples, techniques are used to perform DMVR refinement in an overlapping manner.

[0117] According to certain aspects of the present disclosure, extended samples are used for two-sided cost calculation in the DMVR process. In some examples, an encoder / decoder may determine to apply extended DMVR to a current block having a first sample size. The extended DMVR uses a second sample size for the two-sided matching cost calculation, where the second sample size is different from the first sample size of the current block. The first sample size is the number of samples in the current block. The second sample size is the number of samples in each of the extended reference blocks used for the two-sided matching cost calculation. Furthermore, multiple two-sided matching costs, each associated with multiple motion vector adjustments, are calculated. At least a first two-sided matching cost of the multiple two-sided matching costs is calculated based on the second sample size, which is different from the first sample size of the current block. For example, the first two-sided matching cost is calculated as the SAD between two extended reference blocks, where the two extended reference blocks each have the second sample size. A first refined motion vector for the current block is determined based on the multiple two-sided matching costs associated with the multiple motion vector adjustments. For example, the motion vector adjustment with the lowest two-sided matching cost is selected. A first refined motion vector is calculated as a combination of the initial motion vector (e.g., from the merge list) and the selected motion vector adjustment. A current block having a first sample size is reconstructed based on the first refined motion vector. For example, the current block is predicted based on one or more reference blocks indicated by the first refined motion vector, where the one or more reference blocks have the same number of samples as the current block.

[0118] In some embodiments, when refining the MV for a current block or sub-block with size N×M (where N is the width in luma samples, M is the height in luma samples, and M is an integer value in luma samples), an additional or fewer number of samples (e.g., S1 samples on each side of the width and S2 samples on each side of the height) extended outward from each boundary of the block / sub-block are used in calculating the two-sided matching cost. For example, when additional samples are used, the size for cost calculation is (N+2S1)×(M+2S2), or when fewer samples are used, the size is (N-2S1)×(M-2S2). The refined MV still applies for blocks of size N×M.

[0119] In some examples, the DMVR sub-block size for MV refinement is 4x4 luma samples, and the bilateral matching cost is calculated based on the samples in an 8x8 block (extended by 2 samples on each side). The center position of the 8x8 block is properly aligned with the DMVR sub-block. In some examples, the center position of the DMVR block / sub-block for obtaining the refined MV is aligned with the center position of the block for cost calculation.

[0120] Figure 7 shows a diagram of a sub-block (701) for DMVR refinement in one example. In the example of Figure 7, larger blocks (702) are formed by extending the sub-block (701) from each boundary. For example, S1 samples are extended from each boundary in the width direction of the sub-block (701), and S2 samples are extended from each boundary in the height direction of the sub-block (701). The larger blocks (702) are referred to in some examples as extended DMVR blocks.

[0121] In some embodiments, when refining an MV for a current block or sub-block with size NxM (where N is the width in luma samples, M is the height in luma samples, and M is an integer value in luma samples), the two-sided matching cost may be calculated using samples of the block expanded to a multiple (e.g., twice) of the original size. For example, the size of the block for the two-sided matching cost calculation is 2Nx2M. The refined MV is still applied for blocks of size NxM.

[0122] In some examples, the DMVR sub-block size for MV refinement is 8x8 luma samples, and the bilateral matching cost is calculated based on samples in a 16x16 block (extended by 2x both horizontally and vertically). The center position of the 16x16 block is properly aligned with the DMVR sub-block. In some examples, the center position of the DMVR block / sub-block for obtaining the refined MV is aligned with the center position of the extended block for bilateral cost calculation.

[0123] In some embodiments, when a block / sub-block boundary for refinement is at the boundary of a higher level processing unit such as a picture / slice / tile / tile group, this side boundary is not extended for the two-sided matching cost calculation.

[0124] In some embodiments, the extended samples for the two-sided matching cost calculation are applied only to certain block sizes, rather than to all block sizes. For example, when the block size of a block is smaller than a threshold, the extended samples for the two-sided matching cost calculation may be applied for the block. When the block size of a block is equal to or greater than a threshold, the extended samples for the two-sided matching cost calculation are not applied for the block. In some embodiments, the extended samples for the two-sided matching cost calculation are applied only to certain DMVR sub-block sizes, rather than to all sub-block sizes. For example, when the block size of a sub-block is smaller than a threshold, the extended samples for the two-sided matching cost calculation may be applied to the sub-block. When the block size of a sub-block is equal to or greater than the threshold, the extended samples for the two-sided matching cost calculation are not applied to the sub-block.

[0125] It should be noted that, in general, the DMVR process may be performed on non-overlapping sub-blocks within a block. According to certain aspects of the present disclosure, the DMVR process may be performed on two or more overlapping sub-blocks. When the DMVR process is applied to overlapping sub-blocks, the DMVR process may be referred to as an overlapping DMVR refinement process. In some examples, the encoder / decoder may determine a first refined motion vector for a first block by applying the DMVR process to the first block, and determine a second refined motion vector for the second block by applying the DMVR process to the second block. The second block and the first block have an overlapping area. The encoder / decoder may reconstruct the overlapping area based on the first refined motion vector and the second refined motion vector.

[0126] In some embodiments, for a DMVR process for subblocks of subblock size N x M within a block, the subblocks being refined can overlap, and the subblocks to which the DMVR process is applied can be referred to as DMVR subblocks.

[0127] In some examples, sub-blocks of size N×M may be arranged every N / 2 samples in the horizontal direction.

[0128] FIG. 8 shows a diagram of a block (800) in some examples. Sub-blocks of size N×M are arranged in the block (800). For sub-blocks to which the DMVR process is applied, the sub-blocks are arranged every N / 2 samples in the horizontal direction. For example, the sub-blocks for each column are arranged from left to right as follows: column (801), column (811), column (802), column (812), column (803), column (813), and column (804). The columns (801) to (804) of sub-blocks do not overlap and are shown as solid grids. The columns (811) to (813) of sub-blocks do not overlap and are shown as dashed grids. The column (811) overlaps with the columns (801) and (802). The column (812) overlaps with the columns (802) and (803). Column (813) overlaps with column (804) and column (804).

[0129] In some examples, sub-blocks of size NxM may be arranged every M / 2 samples in the vertical direction.

[0130] FIG. 9 shows a diagram of a block (900) in some examples. Sub-blocks of size N×M are arranged in the block (900). For sub-blocks to which the DMVR process is applied, the sub-blocks are arranged every M / 2 samples in the vertical direction. For example, the sub-blocks for each row are arranged from top to bottom as follows: row (921), row (931), row (922), row (932), row (923), row (933), and row (924). Sub-block rows (921) to (924) do not overlap and are shown as solid grids. Sub-block rows (931) to (933) do not overlap and are shown as dashed grids. Row (931) overlaps with row (921) and row (922). Row (932) overlaps with row (922) and row (923). Row (933) overlaps with row (924) and row (924).

[0131] In some examples, sub-blocks of size NxM may be arranged in both the horizontal and vertical directions with a position difference, such as every N / 2 samples in the horizontal direction and every M / 2 samples in the vertical direction.

[0132] FIG. 10 shows a diagram of a block (1000) in some examples. N×M sub-blocks are arranged in the block (1000). For the sub-blocks to which the DMVR process is applied, the sub-blocks are arranged with overlapping sub-blocks every N / 2 samples in the horizontal direction and every M / 2 samples in the vertical direction. For example, sub-blocks (1001) to (1016) do not overlap and are shown as solid grids. Sub-blocks (1021) to (1029) do not overlap and are shown as dashed grids. Each of sub-blocks (1021) to (1029) overlaps with four sub-blocks in sub-blocks (1001) to (1016).

[0133] In some examples, both the horizontal and vertical directions may have duplicated DMVR sub-blocks, e.g., every N / 2 samples in the horizontal direction and every M / 2 samples in the vertical direction. For example, the duplication patterns of Figures 8, 9, and 10 may be applied.

[0134] Figure 11 shows a diagram of a block (1100) with overlapping sub-blocks in some examples, where the overlapping patterns in Figures 8, 9 and 10 apply.

[0135] In some embodiments, each overlap area may have a refined delta MV value derived from all overlapping DMVR sub-blocks according to a predefined rule.

[0136] In the example of FIG. 11, subblocks A, B, C, and D overlap. Specifically, subblock B is located N / 2 samples horizontally from subblock A, subblock C is located M / 2 samples vertically from subblock A, and subblock D is located N / 2 samples horizontally and M / 2 samples vertically from subblock A. In FIG. 11, each overlapping area is labeled with an overlapping block name. If an area is only marked with one name, that area is a non-overlapping area. For example, each subblock has a size of 8×8 luma samples, the non-overlapping area for only subblock A has a size of 4×4 luma samples, and the overlapping areas for all of subblocks A, B, C, and D have a size of 4×4 luma samples.

[0137] In some embodiments, for the overlap area, a delta MV for DMVR refinement is determined from one of the overlapping blocks based on the lowest two-sided matching cost among all the overlapping blocks. For example, the overlapping area (1101) is the overlapping area of ​​all subblocks A, B, C, and D. When DMVR refinement is applied to subblock A, the result is a first delta MV. The first delta MV for subblock A can achieve a first two-sided matching cost associated with the first delta MV. When DMVR refinement is applied to subblock B, the result is a second delta MV. The second delta MV for subblock B can achieve a second two-sided matching cost associated with the second delta MV. When DMVR refinement is applied to subblock C, the result is a third delta MV. The third delta MV for subblock C can achieve a third two-sided matching cost associated with the third delta MV. When DMVR refinement is applied to subblock D, the result is a fourth delta MV. The fourth delta MV for sub-block D can achieve a fourth two-sided matching cost associated with the fourth delta MV. The delta MV of the DMVR refinement for the overlap area (1101) is selected as the one having the lowest two-sided matching cost among the first delta MV, the second delta MV, the third delta MV, and the fourth delta MV.

[0138] In some embodiments, for the overlap area, the delta MV of the DMVR refinement is calculated by averaging the delta MVs of the overlap blocks. For example, the delta MV of the DMVR refinement for the overlap area (1101) is the average of the first delta MV, the second delta MV, the third delta MV, and the fourth delta MV.

[0139] In some embodiments, for the overlap area, the delta MV of the DMVR refinement is calculated by a weighted average of the delta MVs of the overlap blocks. For example, the delta MV of the DMVR refinement for the overlap area (1101) is a weighted average of the first delta MV, the second delta MV, the third delta MV, and the fourth delta MV.

[0140] In some examples, the weight of each overlapping block may be calculated from the two-sided matching cost of that overlapping block. In one example, the weight is proportional to the inverse of the two-sided matching cost of the overlapping block. For example, to calculate the delta MV of the DMVR refinement for the overlapping area (1101), the first weight for the first delta MV is the inverse of the first two-sided matching cost, the second weight for the second delta MV is the inverse of the second two-sided matching cost, the third weight for the third delta MV is the inverse of the third two-sided matching cost, and the fourth weight for the fourth delta MV is the inverse of the fourth two-sided matching cost.

[0141] In some embodiments, for the overlap area, a prediction sample is derived by a weighted average of multiple prediction samples, where each prediction sample is calculated using an MV derived from the DMVR applied to the overlapping block. In one example, to derive a prediction sample at a position within the overlap area (1101), a first prediction for the position is determined based on a first delta MV, a second prediction for the position is determined based on a second delta MV, a third prediction for the position is determined based on a third delta MV, and a fourth prediction for the position is determined based on a fourth delta MV. The prediction sample at the position is calculated as a weighted average of the first prediction, the second prediction, the third prediction, and the fourth prediction.

[0142] In some embodiments, a high-level syntax is signaled to indicate which of the above-described refined deltaMV derivation methods is used. The high-level syntax may be signaled at the sequence level, picture level, or slice / tile / tile group level.

[0143] 12 shows a flowchart outlining a process (1200) according to one embodiment of the present disclosure. The process (1200) may be used in a video decoder. In various embodiments, the process (1200) is performed by a processing circuit, such as a processing circuit that performs the functions of the video decoder (110), a processing circuit that performs the functions of the video decoder (210), or the like. In some embodiments, the process (1200) is implemented with software instructions, such that the processing circuit performs the process (1200) when it executes the software instructions. The process begins at (S1201) and proceeds to (S1210).

[0144] At (S1210), an encoded video bitstream is received. The encoded video bitstream includes coding information for a first block. The coding information indicates use of decoder-side motion refinement (DMVR) for the first block. The first block has a first sample size.

[0145] In (S1220), an extended DMVR is applied to the first block. The extended DMVR uses a second sample size for the two-sided matching cost calculation, and the second sample size is different from the first sample size of the first block. The first sample size is the number of samples in the first block. The second sample size is the number of samples in each of the extended reference blocks used for the two-sided matching cost calculation.

[0146] At step S1230, a plurality of bilateral matching costs respectively associated with the plurality of motion vector adjustments are calculated. At least a first bilateral matching cost of the plurality of bilateral matching costs is calculated based on a second sample size different from the first sample size of the first block. For example, the first bilateral matching cost is calculated as a sum of adjacent deviations (SAD) between two extended reference blocks, each having a second sample size.

[0147] At (S1240), a first refined motion vector for the first block is determined based on multiple bilateral matching costs associated with the multiple motion vector adjustments. For example, the motion vector adjustment with the lowest bilateral matching cost is selected. The first refined motion vector is calculated as a combination of the initial motion vector (e.g., from the merge list) and the selected motion vector adjustment.

[0148] At (S1250), a first block having a first sample size is reconstructed based on a first refined motion vector. For example, the first block is predicted based on one or more reference blocks indicated by the first refined motion vector, the one or more reference blocks having the same number of samples as the first block.

[0149] According to certain aspects of the present disclosure, the first sample size is greater than the second sample size. For example, if the first sample size is greater than a threshold, a smaller sample size may be used in the two-sided matching cost calculation to reduce computational complexity.

[0150] According to another aspect of the present disclosure, the first block is a sub-block of a larger block, and the first sample size is smaller than the second sample size. For example, when the first block is a sub-block, a larger sample size may be used in the two-sided matching cost calculation to improve cost calculation accuracy.

[0151] In some examples, the first block has a first width of N luma samples and a first height of M luma samples, the first sample size corresponds to the number of luma samples in the first block, the second sample size corresponds to the number of luma samples in a second block having a second width and a second height, the second width is 2×S1 luma samples larger than the first width, and the second height is 2×S2 luma samples larger than the first height, and N, M, S1, and S2 are positive integers.

[0152] In some examples, the first block has a first width of N luma samples and a first height of M luma samples, the first sample size corresponding to the number of luma samples in the first block, the second sample size corresponding to the number of luma samples in a second block having a second width and a second height, the second width being a multiple of the first width, and the second height being a multiple of the first height.

[0153] In some examples, an extended DMVR block having a center aligned with the first block is determined to calculate multiple bilateral matching costs respectively associated with multiple motion vector adjustments. The extended DMVR block has a second sample size. A first extended DMVR reference block in the first reference picture and a second extended DMVR reference block in the second reference picture are determined corresponding to the extended DMVR block according to the first motion vector adjustment. The first extended DMVR reference block and the second extended DMVR reference block have the second sample size. A first bilateral matching cost, such as a sum of average deviations (SAD) between the first extended DMVR reference block and the second extended DMVR reference block, is calculated.

[0154] In some examples, to calculate multiple bilateral matching costs respectively associated with the multiple motion vector adjustments, a first DMVR reference block in a first reference picture and a second DMVR reference block in a second reference picture corresponding to a first block are determined according to the first motion vector adjustment. The first DMVR reference block and the second DMVR reference block have a first sample size. Then, a first extended DMVR reference block centered with the first DMVR reference block is determined, the first extended DMVR reference block having a second sample size. Further, a second extended DMVR reference block centered with the second DMVR reference block is determined. The second extended DMVR reference block has a second sample size. A first bilateral matching cost, such as a sum of average deviations (SAD) between the first extended DMVR reference block and the second extended DMVR reference block, is calculated.

[0155] According to an aspect of the present invention, an overlapping DMVR process may be applied. In some examples, a second refined motion vector for a second block is determined by applying DMVR to the second block. The second block and the first block have an overlapping area. The overlapping area is reconstructed based on the first refined motion vector and the second refined motion vector.

[0156] It should be noted that the position difference between the first block and the second block is in at least one of the horizontal and / or vertical directions. In one example, the first block and the second block have a position difference (e.g., a position shift) in the horizontal direction and overlap in the horizontal direction. In another example, the first block and the second block have a position difference (e.g., a position shift) in the vertical direction and overlap in the vertical direction. In another example, the first block and the second block have a position difference (e.g., a position shift) in both the horizontal and vertical directions and overlap in the horizontal and vertical directions.

[0157] To reconstruct the overlap area, in some examples, a refined motion vector associated with a lower bilateral matching cost is selected from the first refined motion vector and the second refined motion vector, and the overlap area is reconstructed according to the refined motion vector.

[0158] To reconstruct the overlap area, in some examples, an average motion vector between the first refined motion vector and the second refined motion vector is calculated, and the overlap area is reconstructed according to the average motion vector.

[0159] To reconstruct the overlap area, in some examples, a weighted average motion vector of the first refined motion vector and the second refined motion vector is calculated. The weight of the refined motion vector is proportional to the inverse of the bilateral matching cost of the refined motion vector. The overlap area is reconstructed according to the weighted average motion vector.

[0160] To reconstruct the overlap area, in some examples, a first predicted sample for a sample in the overlap area is generated based on a first refined motion vector, and a second predicted sample for the sample in the overlap area is generated based on a second refined motion vector, and a weighted average of the first predicted sample and the second predicted sample is calculated as a reconstructed sample for the sample in the overlap area.

[0161] In some examples, at least one syntax element at a sequence level, a picture level, a slice level, a tile group level, and a tile level is decoded, and the syntax element has a value indicating which reconstruction method is used for the overlap area.

[0162] Then, the process proceeds to (S1299) and ends.

[0163] Process 1200 may be adapted as appropriate. Steps of process 1200 may be modified and / or omitted. Additional steps may be added. Any suitable order of performance may be used.

[0164] FIG. 13 shows a flowchart outlining a process (1300) according to one embodiment of the present disclosure. The process (1300) may be used in a video encoder. In various embodiments, the process (1300) is performed by a processing circuit, such as a processing circuit that performs the functions of the video encoder (103), a processing circuit that performs the functions of the video encoder (303), or the like. In some embodiments, the process (1300) is implemented by software instructions, such that the processing circuit performs the process (1300) when it executes the software instructions. The process begins at (S1301) and proceeds to (S1310).

[0165] At (S1310), it is determined to apply enhanced decoder-side motion refinement (DMVR) to a first block having a first sample size. The enhanced DMVR uses a second sample size for a two-sided matching cost calculation, the second sample size being different from the first sample size of the first block. The first sample size is the number of samples in the first block. In some examples, the second sample size is the number of samples in each of the enhanced reference blocks used for the two-sided matching cost calculation.

[0166] In (S1320), multiple bilateral matching costs respectively associated with the multiple motion vector adjustments are calculated. At least a first bilateral matching cost of the multiple bilateral matching costs is calculated based on a second sample size different from the first sample size of the first block. For example, the first bilateral matching cost is calculated as a SAD between two extended reference blocks, each of which has a second sample size.

[0167] At (S1330), a first refined motion vector for the first block is determined based on multiple bilateral matching costs associated with the multiple motion vector adjustments. For example, the motion vector adjustment with the lowest bilateral matching cost is selected. The first refined motion vector is calculated as a combination of the initial motion vector (e.g., from the merge list) and the selected motion vector adjustment.

[0168] At (S1340), a first block having a first sample size is reconstructed based on a first refined motion vector. For example, the first block is predicted based on one or more reference blocks indicated by the first refined motion vector, the one or more reference blocks having the same number of samples as the first block.

[0169] According to certain aspects of the present disclosure, the first sample size is greater than the second sample size. For example, if the first sample size is greater than a threshold, a smaller sample size may be used in the two-sided matching cost calculation to reduce computational complexity.

[0170] According to another aspect of the present disclosure, the first block is a sub-block of a larger block, and the first sample size is smaller than the second sample size. For example, when the first block is a sub-block, a larger sample size may be used in the two-sided matching cost calculation to improve cost calculation accuracy.

[0171] In some examples, the first block has a first width of N luma samples and a first height of M luma samples, the first sample size corresponds to the number of luma samples in the first block, the second sample size corresponds to the number of luma samples in a second block having a second width and a second height, the second width is 2×S1 luma samples larger than the first width, and the second height is 2×S2 luma samples larger than the first height, and N, M, S1, and S2 are positive integers.

[0172] In some examples, the first block has a first width of N luma samples and a first height of M luma samples, the first sample size corresponding to the number of luma samples in the first block, the second sample size corresponding to the number of luma samples in a second block having a second width and a second height, the second width being a multiple of the first width, and the second height being a multiple of the first height.

[0173] In some examples, an extended DMVR block having a center aligned with the first block is determined to calculate multiple bilateral matching costs respectively associated with multiple motion vector adjustments. The extended DMVR block has a second sample size. A first extended DMVR reference block in the first reference picture and a second extended DMVR reference block in the second reference picture are determined corresponding to the extended DMVR block according to the first motion vector adjustment. The first extended DMVR reference block and the second extended DMVR reference block have the second sample size. A first bilateral matching cost, such as a sum of average deviations (SAD) between the first extended DMVR reference block and the second extended DMVR reference block, is calculated.

[0174] In some examples, to calculate multiple bilateral matching costs respectively associated with the multiple motion vector adjustments, a first DMVR reference block in the first reference picture and a second DMVR reference block in the second reference picture are determined corresponding to the first block according to the first motion vector adjustment. The first DMVR reference block and the second DMVR reference block have a first sample size. Then, a first extended DMVR reference block is determined having a center aligned with the first DMVR reference block, and the first extended DMVR reference block has a second sample size. Further, a second extended DMVR reference block is determined having a center aligned with the second DMVR reference block. The second extended DMVR reference block has a second sample size. A first bilateral matching cost, such as a sum of average deviation (SAD) between the first extended DMVR reference block and the second extended DMVR reference block, is calculated.

[0175] According to an aspect of the present disclosure, an overlapping DMVR process may be applied. In some examples, a second refined motion vector for a second block is determined by applying DMVR to the second block. The second block and the first block have an overlapping area. The overlapping area is reconstructed based on the first refined motion vector and the second refined motion vector.

[0176] It should be noted that the position difference between the first block and the second block is in at least one of the horizontal and / or vertical directions. In one example, the first block and the second block have a position difference (e.g., a position shift) in the horizontal direction and overlap in the horizontal direction. In another example, the first block and the second block have a position difference (e.g., a position shift) in the vertical direction and overlap in the vertical direction. In another example, the first block and the second block have a position difference (e.g., a position shift) in both the horizontal and vertical directions and overlap in the horizontal and vertical directions.

[0177] To reconstruct the overlap area, in some examples, a refined motion vector associated with a lower bilateral matching cost is selected from the first refined motion vector and the second refined motion vector, and the overlap area is reconstructed according to the refined motion vector.

[0178] To reconstruct the overlap area, in some examples, an average motion vector between the first refined motion vector and the second refined motion vector is calculated, and the overlap area is reconstructed according to the average motion vector.

[0179] To reconstruct the overlap area, in some examples, a weighted average motion vector of the first refined motion vector and the second refined motion vector is calculated, where the weight of the refined motion vector is proportional to the inverse of the bilateral matching cost of the refined motion vector, and the overlap area is reconstructed according to the weighted average motion vector.

[0180] To reconstruct the overlap area, in some examples, a first predicted sample for a sample in the overlap area is generated based on a first refined motion vector, and a second predicted sample for the sample in the overlap area is generated based on a second refined motion vector, and a weighted average of the first predicted sample and the second predicted sample is calculated as a reconstructed sample for the sample in the overlap area.

[0181] In some examples, at least one of a sequence level, a picture level, a slice level, a tile group level, and a tile level syntax element is encoded in the coded video bitstream, the syntax element having a value indicating which reconstruction method should be used for the overlap area.

[0182] Then, the process proceeds to (S1399) and ends.

[0183] The process 1300 may be adapted as appropriate. Steps of the process 1300 may be modified and / or omitted. Additional steps may be added. Any suitable order of performance may be used.

[0184] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 14 illustrates a computer system (1400) suitable for implementing certain embodiments of the disclosed subject matter.

[0185] Computer software may be coded using any suitable machine code or computer language and may apply assembly, compilation, linking, or similar mechanisms to create code containing instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc. directly, or through interpretation, microcode execution, etc.

[0186] The instructions may be executed on various types of computers or components thereof including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0187] 14 for computer system 1400 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. Neither the arrangement of components should be interpreted as having any dependency or requirement regarding any one or combination of components shown in the exemplary embodiment of computer system 1400.

[0188] The computer system (1400) may include certain human interface input devices that can respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, claps), visual input (e.g., gestures), or olfactory input (not shown). Human interface devices may also be used to capture certain media that do not necessarily involve direct human conscious input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still-image cameras), and video (e.g., two-dimensional video, three-dimensional video, including stereoscopic video).

[0189] The input human interface devices may include one or more (only one of each is shown) of a keyboard (1401), a mouse (1402), a trackpad (1403), a touchscreen (1410), a data glove (not shown), a joystick (1405), a microphone (1406), a scanner (1407), and a camera (1408).

[0190] The computer system (1400) may also include some type of human interface output device. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (1410), data gloves (not shown), or joystick (1405); however, haptic feedback devices may also function as input devices), audio output devices (e.g., speakers (1409), headphones (not shown)), visual output devices (e.g., a screen (1410), including a CRT screen, LCD screen, plasma screen, or OLED screen; each may or may not have touchscreen input capability, each may or may not have haptic feedback capability, some of which may output two-dimensional visual output or output in greater than three dimensions through means such as stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0191] The computer system (1400) may also include human-accessible storage and associated media, such as optical media including CD / DVD ROM / RW (1420) along with CD / DVD or similar media (1421), thumb drives (1422), removable hard drives or solid state drives (1423), legacy magnetic media such as tape and floppy disks (not shown), specialized ROM / ASIC / PLD-based devices (not shown) such as security dongles, etc.

[0192] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0193] The computer system (1400) may also include an interface (1454) to one or more communication networks (1455). Networks may be, for example, wireless, wired, or optical. Networks may further be local, wide-area, metropolitan, in-vehicle, and industrial, real-time, delay-tolerant, and the like. Examples of networks include Ethernet, WLAN, cellular networks including GSM, 3G, 4G, 5G, LTE, and the like; TV wired or wireless wide-area digital networks including cable, satellite, and terrestrial broadcast television; and in-vehicle and industrial networks including CAN Bus. Some networks typically require an external network interface adapter attached to some kind of general-purpose data port or peripheral bus (1449) (e.g., a USB port on the computer system (1400)). Others are typically integrated into the core of the computer system (1400) by attachment to a system bus, as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1400) can communicate with other entities. Such communication may be unidirectional, receive-only (e.g., broadcast television), unidirectional transmit-only (e.g., CANbus to certain CANbus devices), or bidirectional, for example, to other computer systems using local or wide-area digital networks. Each of these networks and network interfaces, as described above, may use certain protocols and protocol stacks.

[0194] The aforementioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core (1440) of the computer system (1400).

[0195] A core (1440) may include one or more central processing units (CPUs) (1441), graphics processing units (GPUs) (1442), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (1443), hardware accelerators for certain tasks (1444), graphics adapters (1450), etc. These devices may be connected through a system bus (1448), along with read-only memory (ROM) (1445), random access memory (1446), and internal mass storage devices (1447) such as internal non-user-accessible hard drives or solid-state drives (SSDs). In some computer systems, the system bus (1448) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1448) or through a peripheral bus (1449). In one example, a screen 1410 can be connected to a graphics adapter 1450. Architectures for peripheral buses include PCI, USB, and the like.

[0196] The CPU (1441), GPU (1442), FPGA (1443), and accelerator (1444) may execute certain instructions that, in combination, may constitute the above-mentioned computer code. The computer code may be stored in ROM (1445) or RAM (1446). Temporary data may also be stored in RAM (1446), while persistent data may be stored, for example, in internal mass storage device (1447). Rapid storage and retrieval to any of the memory devices may be enabled through the use of cache memory, which may be closely associated with one or more of the CPU (1441), GPU (1442), mass storage device (1447), ROM (1445), RAM (1446), etc.

[0197] The computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those having skill in the computer software arts.

[0198] By way of example and not limitation, the architecture (1400), and in particular a computer system having a core (1440), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage, as discussed above, as well as media associated with some type of storage of the core (1440) that is non-transitory, such as the core's internal mass storage (1447) or ROM (1445). Software implementing various embodiments of the present disclosure can be stored on such devices and executed by the core (1440). The computer-readable media can include one or more memory devices or chips, depending on particular needs. The software can cause the core (1440) and, in particular, the processor (including a CPU, GPU, FPGA, etc.) therein to perform certain processes or certain portions thereof described herein, including defining data structures stored in RAM (1446) and modifying such data structures according to software-defined processes. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (1444)), which may operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. Reference to software includes logic, and vice versa, as appropriate. Reference to a computer-readable medium may encompass circuitry (e.g., an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both, as appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0199] The use of "at least one" or "one of" in this disclosure is intended to include any one or combination of the listed elements. For example, reference to at least one of A, B, or C; at least one of A, B, and C; at least one of A, B, and / or C; and at least one of A through C is intended to include A only, B only, C only, or any combination thereof. Reference to one of A or B, and one of A and B is intended to include A or B or (A and B). The use of "one of" does not exclude any combination of the listed elements, where applicable, such as when the elements are not mutually exclusive.

[0200] While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. Thus, those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are thus within its spirit and scope.

Claims

1. 1. A method of video decoding comprising: receiving a coded video bitstream including coding information for a first block, the coding information indicating that decoder-side motion refinement (DMVR) should be used for the first block, the first block having a first sample size; applying an enhanced DMVR to the first block, the enhanced DMVR using a second sample size to calculate a two-sided matching cost, the second sample size being different from the first sample size of the first block; calculating a plurality of bilateral matching costs respectively associated with a plurality of motion vector adjustments, wherein at least a first bilateral matching cost of the plurality of bilateral matching costs is calculated based on the second sample size different from the first sample size of the first block; determining a first refined motion vector for the first block based on the plurality of bilateral matching costs associated with the plurality of motion vector adjustments; and reconstructing the first block having the first sample size based on the first refined motion vector. method.

2. The method of claim 1 , wherein the first sample size is greater than the second sample size.

3. The method of claim 1 , wherein the first block is a sub-block within a larger block, and the first sample size is smaller than the second sample size.

4. 4. The method of claim 3, wherein the first block has a first width of N luma samples and a first height of M luma samples, the first sample size corresponding to the number of samples in the first block, the second sample size corresponding to the number of samples in a second block having a second width and a second height, the second width being 2×S luma samples greater than the first width, and the second height being 2×S luma samples greater than the first height, and N, M, S, and S are positive integers.

5. 4. The method of claim 3, wherein the first block has a first width of N luma samples and a first height of M luma samples, the first sample size corresponding to the number of samples in the first block, the second sample size corresponding to the number of samples in a second block having a second width and a second height, the second width being a multiple of the first width, and the second height being a multiple of the first height.

6. Calculating the plurality of bilateral matching costs respectively associated with the plurality of motion vector adjustments includes: determining an extended DMVR block having a center aligned with the first block, the extended DMVR block having the second sample size; determining a first extended DMVR reference block in a first reference picture and a second extended DMVR reference block in a second reference picture corresponding to the extended DMVR block according to the first motion vector adjustment, wherein the first extended DMVR reference block and the second extended DMVR reference block have the second sample size; calculating the first bilateral matching cost between the first extended DMVR reference block and the second extended DMVR reference block. The method of claim 3.

7. Calculating the plurality of bilateral matching costs respectively associated with the plurality of motion vector adjustments includes: determining a first DMVR reference block in a first reference picture and a second DMVR reference block in a second reference picture corresponding to the first block according to the first motion vector adjustment, wherein the first DMVR reference block and the second DMVR reference block have the first sample size; determining a first extended DMVR reference block having a center aligned with the first DMVR reference block, the first extended DMVR reference block having the second sample size; determining a second extended DMVR reference block having a center aligned with the second DMVR reference block, the second extended DMVR reference block having the second sample size; calculating the first bilateral matching cost between the first extended DMVR reference block and the second extended DMVR reference block. The method of claim 3.

8. Determining to apply the extended DMVR to the first block having the first sample size includes: and determining to apply the enhanced DMVR in response to the first sample size being less than a threshold. The method of claim 1.

9. determining that a boundary of the first block is at a boundary of a higher-level processing unit; further comprising constraining the boundary to prevent expansion for additional samples. The method of claim 1.

10. determining a second refined motion vector for a second block by applying the DMVR to the second block, the second block and the first block having an overlapping area; and reconstructing the overlap area based on the first refined motion vector and the second refined motion vector. The method of claim 1.

11. The method of claim 10 , wherein the position difference between the first block and the second block is in at least one of a horizontal direction and / or a vertical direction.

12. Reconstructing the overlap area comprises: selecting a refined motion vector from the first refined motion vector and the second refined motion vector that is associated with a lower bilateral matching cost; and reconstructing the overlap area according to the refined motion vector. The method of claim 10.

13. Reconstructing the overlap area comprises: calculating an average motion vector of the first refined motion vector and the second refined motion vector; and reconstructing the overlap area according to the average motion vector. The method of claim 10.

14. Reconstructing the overlap area comprises: calculating a weighted average motion vector of the first refined motion vector and the second refined motion vector, wherein a weight for a refined motion vector is proportional to the inverse of a bilateral matching cost of that refined motion vector; and reconstructing the overlap area according to the weighted average motion vector. The method of claim 10.

15. Reconstructing the overlap area comprises: generating a first predicted sample for a sample in the overlap area based on the first refined motion vector; generating second predicted samples for the samples in the overlap area based on the second refined motion vector; calculating a weighted average of the first predicted sample and the second predicted sample as a reconstructed sample for the samples in the overlap area. The method of claim 10.

16. and further comprising decoding a syntax element at at least one of a sequence level, a picture level, a slice level, a tile group level, and a tile level, the syntax element having a value indicating a reconstruction of the overlap area based on the first refined motion vector and the second refined motion vector. The method of claim 10.

17. 17. Video decoding device, comprising processing circuitry configured to perform a method according to any one of claims 1 to 16.

18. A computer program product for causing a computer to carry out the method according to any one of claims 1 to 16.

Images