Method and apparatus for improving affine motion

Abstract

The processing circuit determines an initial affine model using bi-prediction for a current block in an affine motion inter-prediction mode, where the initial affine model predicts the current block based on a first reference picture and a second reference picture. The processing circuit performs multi-stage bilateral matching motion refinement on the initial affine model, having at least a first stage and a second stage, to derive an improved affine model, where one of the first stage and the second stage applies the first bilateral matching motion refinement to a translational component of the initial affine model, and the other of the first stage and the second stage applies the second bilateral matching motion refinement to a non-translational component of the initial affine model. The processing circuit reconstructs the current block using the improved affine model based on the first reference picture and the second reference picture.

Description

Technical Field

[0001] [Cross-Reference to Related Applications] This application claims the benefit of priority of U.S. Patent Application No. 18 / 380,523, filed October 16, 2023, entitled "METHOD AND APPARATUS FOR AFFINE MOTION REFINEMENT", which claims the benefit of priority of U.S. Provisional Application No. 63 / 417,277, filed October 18, 2022, entitled "Method and Apparatus for Affine Motion Refinement". The entire disclosure of the prior application is incorporated herein by reference.

[0002] [Technical Field] The present disclosure generally describes embodiments related to video coding.

Background Art

[0003] The description of the background art provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the inventors, to the extent described in this Background section, as well as aspects of the description that would not otherwise be recognized as prior art at the time of filing, are neither expressly nor implicitly admitted as prior art against the present disclosure.

[0004] Image / video compression can help transmit image / video data across different devices, storage, and networks with minimal quality degradation. In some cases, video codec techniques can compress video based on spatial and temporal redundancy. In one example, a video codec can use a technique called intra-prediction, which can compress images based on spatial redundancy. For example, intra-prediction can use reference data from the currently reconstructed picture for sample prediction. In another example, a video codec can use a technique called inter-prediction, which can compress images based on temporal redundancy. For example, inter-prediction can predict samples in the current picture from previously reconstructed pictures using motion compensation. Motion compensation can generally be represented by motion vectors (MV). [Overview of the project]

[0005] Aspects of this disclosure include methods and apparatus for video coding / decoding. In some examples, the apparatus for video decoding includes a processing circuit. The processing circuit receives coding information for a current block in a current picture, the coding information indicating that the current block is coded in affine motion interprediction mode. The processing circuit determines an initial affine model for the current block using biprediction, the initial affine model predicts the current block based on a first reference picture and a second reference picture. The processing circuit derives an improved affine model by performing a multi-stage bilateral matching motion improvement on the initial affine model, the first of the first and second stages applying the first bilateral matching motion improvement to the translational component of the initial affine model, and the other of the first and second stages applying the second bilateral matching motion improvement to the non-translational component of the initial affine model. The processing circuit reconstructs the current block using the improved affine model based on the first and second reference pictures.

[0006] In some examples, the processing circuit applies a first bilateral matching motion improvement to the translational component in a first stage before the second stage, and applies a second bilateral matching motion improvement to the non-translational component in a second stage after the first stage.

[0007] In some examples, the processing circuit applies a second bilateral matching motion improvement to the non-translational component in the first stage before the second stage, and applies a first bilateral matching motion improvement to the translational component in the second stage after the first stage.

[0008] In some examples, for a first bilateral matching motion improvement, the processing circuit determines motion vector offsets to apply to each of the control point motion vectors of the initial affine model in order to minimize the bilateral matching cost value between the first affine prediction in the first reference picture and the second affine prediction in the second reference picture.

[0009] In some examples, for a second bilateral matching motion improvement, the processing circuit determines the predicted output of the current block according to the current affine model in a gradient-based improvement iteration related to the current affine model, calculates the gradient of the predicted samples in the predicted output, solves the linear form formed by the gradient to determine the delta values ​​of the control point motion vectors, determines whether one or more stopping conditions are met, determines the next affine model based on the delta values ​​of the control point motion vectors if none of the one or more stopping conditions are met, inputs the next affine model into the next gradient-based improvement iteration, and stops the second bilateral matching motion improvement if one or more stopping conditions are met.

[0010] In some examples, depending on whether one or more stopping conditions are met, the processing circuit determines the delta control point motion vectors of each control point motion vector based on the last affine model in the last gradient-based improvement iteration of the second bilateral matching motion improvement.

[0011] In some examples, the processing circuit stores multiple bilateral matching costs associated with each affine model used in one of several gradient-based improvement iterations of the second bilateral matching motion improvement, determines the lowest bilateral matching cost from the multiple bilateral matching costs, and determines the delta control point motion vector of each control point motion vector based on the affine model associated with the lowest bilateral matching cost.

[0012] In some examples, the improved affine model is formed by improved control point motion vectors, which include motion vector offsets derived by a first bilateral matching motion improvement and applied to each of the improved control point motion vectors, and delta control point motion vectors derived by a second bilateral matching motion improvement and applied to each of the control point motion vectors.

[0013] In some examples, a multi-stage bilateral matching motion improvement for an initial affine model includes at least a third stage in which a third bilateral matching motion improvement is applied to the control point motion vector at the control point of the current block.

[0014] In some cases, the third stage is performed before the first and second stages, or between the first and second stages, or after the first and second stages.

[0015] In some examples, the third stage applies a third bilateral matching motion improvement with one or more iterations for each of the control points of the current block. The processing circuit determines a pair of improved control point motion vectors at the control point that minimizes the bilateral matching cost between a first affine prediction in the first reference picture and a second affine prediction in the second reference picture, in the iterations for the control point.

[0016] In some examples, the third stage applies a third bilateral matching motion improvement with one or more iterations for each combination of N control points, where N is a positive integer greater than 1 and less than the total number of control points. The processing circuit determines, in iterations for the N control point combinations, a motion vector offset to apply to each of the control point motion vectors at the N control points in the combination that minimizes the bilateral matching cost value between the first affine prediction in the first reference picture and the second affine prediction in the second reference picture.

[0017] In some examples, the processing circuit determines a first motion vector for a subblock of the current block coded in affine motion interprediction mode according to an initial affine model, and applies decoder-side motion vector refinement (DMVR) to determine a second motion vector for the subblock using the refinement to the first motion vector. The second motion vector using the refinement to the first motion vector minimizes the bilateral matching cost between the first reference subblock in the first reference picture and the second reference subblock in the second reference picture. The processing circuit reconstructs the subblock according to the second motion vector for the subblock.

[0018] In some examples, the processing circuit stores a first motion vector in a subblock buffer for a subblock, and the first motion vector in the subblock buffer can be used to predict the motion vector for another block.

[0019] In some examples, the processing circuit stores a second motion vector in a subblock buffer for a subblock, and the second motion vector in the subblock buffer can be used to predict the motion vector for another block.

[0020] In some examples, the processing circuit determines improved motion vectors for each of the subblocks of the current block, determines a regression model based on the improved motion vectors for the subblocks, derives improved control point motion vectors for the improved affine model of the current block according to the regression model, and reconstructs the current block according to the improved control point motion vectors.

[0021] In some examples, the processing circuit stores the control point motion vectors of the initial affine model in a buffer associated with the current block, and these control point motion vectors can be used for affine inheritance.

[0022] In some examples, the processing circuit stores the improved control point motion vectors of the improved affine model in a buffer associated with the current block, and the improved control point motion vectors can be used for affine inheritance.

[0023] In some examples, the processing circuit stores the control point motion vectors of the initial affine model in a buffer associated with the current block, and these control point motion vectors can be used for affine inheritance.

[0024] A part of the present disclosure also provides a non-temporary computer-readable medium that stores instructions causing a computer to perform a method for video decoding / encoding when executed by the computer. [Brief explanation of the drawing]

[0025] Further features, properties, and various advantages of the subject matter of the disclosure will become clearer from the detailed description and accompanying drawings below. [Figure 1] This is a schematic diagram of an exemplary block diagram of a communication system (100). [Figure 2] This is a schematic diagram of an exemplary block diagram of a decoder. [Figure 3] This is a schematic diagram of an exemplary block diagram of an encoder. [Figure 4A] Affine models are shown in several examples. [Figure 4B]Affine models are shown in several examples. [Figure 5] The affine motion vector field for each subblock in one example is shown. [Figure 6] The positions of candidate spatial merges in several examples are shown. [Figure 7] This demonstrates the inheritance of control point motion vectors in several examples. [Figure 8] The locations of candidate positions for the constructive affine merge mode in several examples are shown. [Figure 9] The difference between the sample motion vector and the subblock motion vector in several examples is shown. [Figure 10] A schematic diagram of affine motion estimation in several examples is shown. [Figure 11] This section describes a search process for affine motion estimation in several examples. [Figure 12] A schematic diagram of bilateral matching-based decoder-side motion vector improvements in several examples is shown. [Figure 13] The search domain in several examples is shown. [Figure 14] A flowchart outlining the process according to several embodiments of this disclosure is shown. [Figure 15] A flowchart outlining another process according to several embodiments of this disclosure is shown. [Figure 16] This is a schematic diagram of a computer system according to one embodiment. [Modes for carrying out the invention]

[0026] Figure 1 shows a block diagram of a video processing system (100) in several examples. The video processing system (100) is an example of an application of the subject of disclosure, and is a video encoder and video decoder in a streaming environment. The subject of disclosure is similarly applicable to other video-enabled applications, such as video conferencing, digital TV, streaming services, and storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.).

[0027] The video processing system (100) includes a capture subsystem (113) which may include a video source (101), such as a digital camera, that creates a stream of uncompressed video pictures (102). In one example, the stream of video pictures (102) includes samples captured by the digital camera. The stream of video pictures (102), shown in thick lines to highlight the higher data volume compared to encoded video data (104) (or encoded video bitstream), can be processed by an electronic device (120) which includes a video encoder (103) coupled to the video source (101). The video encoder (103) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject, as will be described in more detail below. The encoded video data (104) (or encoded video bitstream), shown as a thin line to highlight the lower data volume compared to the stream of video pictures (102), can be stored in a streaming server (105) for future use. One or more streaming client subsystems, such as client subsystems (106) and (108) in Figure 1, can access a streaming server (105) to retrieve copies (107) and (109) of encoded video data (104). Client subsystem (106) may, for example, include a video decoder (110) within an electronic device (130). The video decoder (110) decodes the input copy (107) of the encoded video data and creates an output stream (111) of a video picture that can be rendered on a display (112) (e.g., a display screen) or another rendering device (not shown). In some streaming systems, the encoded video data (104), (107), and (109) (e.g., video bitstreams) can be encoded according to several video coding / compression standards. Examples of these standards include ITU-T Recommendation H.265. For example, a video coding standard under development is informally known as VVC (Versatile Video Coding).The disclosed subject matter may be used in the context of VVC.

[0028] It should be noted that electronic devices (120) and (130) may include other components (not shown). For example, electronic device (120) may include a video decoder (not shown), and electronic device (130) may include a video encoder (not shown).

[0029] Figure 2 shows an exemplary block diagram of a video decoder (210). The video decoder (210) may be included in an electronic device (230). The electronic device (230) may include a receiver (231) (e.g., a receiving circuit). The video decoder (210) can be used in place of the video decoder (110) in the example of Figure 1.

[0030] The receiver (231) may receive, for example, one or more encoded video sequences contained in a bitstream to be decoded by a video decoder (210). In one embodiment, one encoded video sequence may be received at a time, and the decoding of each encoded video sequence is independent of the decoding of other encoded video sequences. The encoded video sequences may be received from a channel (201), which may be a hardware / software link to a storage device that stores encoded video data. The receiver (231) may receive the encoded video data together with other data (e.g., encoded audio data and / or auxiliary data streams), which may be forwarded to their respective usage entities (not shown). The receiver (231) may separate the encoded video sequences from other data. To prevent network jitter, a buffer memory (215) may be coupled between the receiver (231) and an entropy decoder / parser (220) (hereinafter referred to as the "parser" (220)). In certain applications, the buffer memory (215) is part of the video decoder (210). In other cases, it may be external to the video decoder (not shown). In yet other cases, for example, to prevent network jitter, there may be a buffer memory (not shown) outside the video decoder (210), and for example, to deal with playback timing, there may be another buffer memory (215) inside the video decoder (210). If the receiver (231) is receiving data from a storage / transfer device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (215) may not be necessary or can be small. For use in best-effort packet networks such as the Internet, the buffer memory (215) may be necessary, can be relatively large, and can be advantageously adaptive in size, and may be at least partially implemented in an operating system and similar elements (not shown) outside the video decoder (210).

[0031] The video decoder (210) may include a parser (220) for reconstructing symbols (221) from the encoded video sequence. These symbol categories include information used to manage the operation of the video decoder (210) and potentially include information for controlling a rendering device, such as a rendering device (212) (e.g., a display screen) that is not an integral part of the electronic device (230) but can be coupled to the electronic device (230), as shown in Figure 2. The rendering device control information may be in the form of Supplementary Enhancement Information (SEI) (SEI messages) or Video Usability Information (VUI) parameter set fragments (not shown). The parser (220) may parse / entropy decode the received encoded video sequence. The encoding of the encoded video sequence may follow video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, and arithmetic coding with or without context sensitivity. The parser (220) may extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder, based on at least one parameter corresponding to a group. The subgroups may include groups of pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transformation units (TU), prediction units (PU), etc. The parser (220) may also extract information from the encoded video sequence such as transformation coefficients, quantization parameter values, motion vectors, etc.

[0032] The parser (220) may perform an entropy decoding / analysis operation on the video sequence received from the buffer memory (215) in order to generate a symbol (221).

[0033] The reconstruction of the symbol (221) may involve multiple different units, depending on the type of encoded video picture or part thereof (e.g., interpicture and intrapicture, interblock and intrablock) and other factors. Which units are involved and how they are involved can be controlled by subgroup control information analyzed from the encoded video sequence by the parser (220). The flow of such subgroup control information between the parser (220) and the following multiple units is not illustrated for clarity.

[0034] In addition to the functional blocks described above, the video decoder (210) can be conceptually subdivided into several functional units, as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and can be integrated at least partially. However, for the purpose of illustrating the subject of disclosure, it is appropriate to conceptually subdivide it into the following functional units.

[0035] The first unit is the scaler / inverse unit (251). The scaler / inverse unit (251) receives quantized transformation coefficients as symbols (221) from the parser (220), along with control information (including which transformation to use, block size, quantization coefficients, quantization scaling matrix, etc.). The scaler / inverse unit (251) can output a block containing sample values ​​that can be input to the aggregator (255).

[0036] In some cases, the output samples of the scaler / inverse unit (251) may relate to intra-coded blocks. Intra-coded blocks may relate to blocks that do not use prediction information from previously reconstructed pictures but can use prediction information from portions of the current picture that were reconstructed before it. Such prediction information can be provided by the intra-picture prediction unit (252). In some cases, the intra-picture prediction unit (252) generates blocks of the same size and shape as the block being reconstructed, using the surrounding already reconstructed information taken from the current picture buffer (258). The current picture buffer (258) buffers, for example, partially reconstructed current pictures and / or fully reconstructed current pictures. In some cases, the aggregator (255) adds the prediction information generated by the intra-prediction unit (252) to the output sample information provided by the scaler / inverse unit (251) for each sample.

[0037] In other cases, the output samples of the scaler / inverse unit (251) may be associated with intercoded and potentially motion-compensated blocks. In such cases, the motion-compensated prediction unit (253) may access the reference picture memory (257) to retrieve samples to be used for prediction. After motion-compensating the retrieved samples according to the symbols (221) associated with the blocks, these samples can be added by the aggregator (255) to the output of the scaler / inverse unit (251) (in this case, called residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (257) from which the motion-compensated prediction unit (253) retrieves prediction samples can be controlled by motion vectors available to the motion-compensated prediction unit (253) in the form of symbols (221) which may have X, Y and reference picture components, for example. Motion compensation may also include interpolation of sample values ​​retrieved from the reference picture memory (257) when the precise motion vectors of subsamples are used, motion vector prediction mechanisms, etc.

[0038] The output samples of the aggregator (255) can undergo various loop filtering techniques within the loop filtering unit (256). The video compression technique may include an in-loop filtering technique, which is controlled by parameters contained in the encoded video sequence (also called the encoded video bitstream) and made available to the loop filtering unit (256) as symbols (221) from the parser (220). Video compression may also respond to metadata obtained during the decoding of earlier parts (in decoding order) of the encoded picture or encoded video sequence, as well as to previously reconstructed and loop-filtered sample values.

[0039] The output of the loop filter unit (256) can be a sample stream, which can be output to the rendering device (212) and stored in the reference picture memory (257) for future interpicture prediction.

[0040] A particular encoded picture, once fully reconstructed, can be used as a reference picture for future predictions. For example, once the encoded picture corresponding to the current picture is fully reconstructed and the encoded picture is identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) can become part of the reference picture memory (257), and a new current picture buffer can be reallocated before the reconstruction of subsequent encoded pictures begins.

[0041] The video decoder (210) may perform the decoding operation according to a given video compression technique or standard such as ITU-T Rec. H.265. The encoded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that it conforms to both the syntax and profile of the video compression technique or standard as documented in the video compression technique or standard. Specifically, a profile may select specific tools from all the tools available in the video compression technique or standard, as tools to be used only under that profile. Also, for compliance, the complexity of the encoded video sequence may need to be within the range defined by the level of the video compression technique or standard. In some cases, the level may limit the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limitations set by the level may be further limited through the Hypothetical Reference Decoder (HRD) specification and metadata about HRD buffer management transmitted in the encoded video sequence.

[0042] In one embodiment, the receiver (231) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, time, space, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0043] Figure 3 shows an exemplary block diagram of a video encoder (303). The video encoder (303) is contained within an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmitting circuit). The video encoder (303) can be used in place of the video encoder (103) in the example of Figure 1.

[0044] The video encoder (303) may receive video samples from a video source (301) (not part of the electronic device (320) in the example in Figure 3) that can capture video images coded by the video encoder (303). In another example, the video source (301) is part of the electronic device (320).

[0045] The video source (301) may provide a source video sequence to be encoded by a video encoder (303) in the form of a digital video sample stream, the digital video sample stream can be any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 Y CrCB, RGB, etc.), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media delivery system, the video source (301) may be a storage device that stores pre-prepared video. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. The video data may be provided as a series of individual pictures that convey motion when viewed sequentially. The picture itself may be composed of a spatial array of pixels, and each pixel may contain one or more samples, depending on the sampling structure, color space, etc., in use. The following description will focus on samples.

[0046] According to one embodiment, the video encoder (303) may encode and compress the pictures of a source video sequence into an encoded video sequence (343) in real time or under any other required time constraints. Achieving an appropriate encoding speed is one function of the controller (350). In some embodiments, the controller (350) can control and be functionally coupled to other functional units, as described below. The couplings are not shown for clarity. Parameters set by the controller (350) may include rate control-related parameters (such as picture skipping, quantization, lambda values ​​for rate distortion optimization techniques), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (350) may be configured to have other appropriate functions related to the video encoder (303) optimized for a particular system design.

[0047] In some embodiments, the video encoder (303) is configured to operate in an encoding loop. In a very simplified description, in one example, the encoding loop may include a source coder (330) (responsible for generating symbols, such as a symbol stream, based on, for example, the input picture and reference picture to be encoded) and a (local) decoder (333) embedded in the video encoder (303). The decoder (333) reconstructs the symbols to generate sample data, similar to what a (remote) decoder would generate. The reconstructed sample stream (sample data) is fed into the reference picture memory (334). Since decoding the symbol stream yields bitwise-accurate results independent of the decoder's location (local or remote), the contents of the reference picture memory (334) are also bitwise-accurate between the local and remote encoders. In other words, the predictive portion of the encoder "sees" the exact same sample values ​​as the reference picture samples that the decoder "sees" when using predictions during decoding. This fundamental principle of reference picture synchronization (including drift resulting from, for example, the inability to maintain synchronization due to channel errors) is also used in several related technologies.

[0048] The operation of the “local” decoder (333) can be the same as that of the “remote” decoder (210), such as the video decoder (210), which has already been explained in detail above in relation to Figure 2. However, referring briefly to Figure 2, since symbols are available and the encoding / decoding of symbols to the encoded video sequence by the entropy coder (345) and parser (220) can be reversible, the entropy decoding portion of the video decoder (210), including the buffer memory (215) and parser (220), does not have to be fully implemented in the local decoder (433).

[0049] In one embodiment, the decoder techniques, excluding analysis / entropy decoding present within the decoder, are present in the corresponding encoder in the same or substantially the same functional form. Therefore, the subject of the disclosure focuses on the decoder operation. A description of the encoder techniques can be omitted, as it is the inverse of the comprehensively described decoder techniques. More detailed descriptions are provided below in specific areas.

[0050] During operation, in some examples, the source coder (330) may perform motion-compensated predictive coding, which predictively codes the input picture by referencing one or more previously coded pictures from a video sequence designated as “reference pictures”. In this way, the coding engine (332) codes the difference between the pixel blocks of the input picture and the pixel blocks of the reference picture that may be selected as predictive references for the input picture.

[0051] A local video decoder (333) may decode the encoded video data of a picture that may be designated as a reference picture based on symbols generated by the source coder (330). The operation of the encoding engine (332) may, advantageously, be lossy. If the encoded video data can be decoded by a video decoder (not shown in Figure 3), the reconstructed video sequence may typically be a replica of the source video sequence with some errors. The local video decoder (333) may replicate the decoding process that may be performed by the video decoder on the reference picture and store the reconstructed reference picture in the reference picture memory (334). Thus, the video encoder (303) may locally store a copy of the reconstructed reference picture having common content as the reconstructed reference picture (without transmission errors) to be acquired by the far-end video decoder.

[0052] The predictor (335) may perform a predictive search for the coding engine (332). That is, for a new picture to be coded, the predictor (335) may search the reference picture memory (334) for sample data (as candidate reference pixel blocks) or specific metadata (reference picture motion vector, block shape, etc.). These may function as appropriate predictive references for the new picture. The predictor (335) may operate sample block-by-pixel block to find appropriate predictive references. In some cases, the input picture determined by the search results obtained by the predictor (335) may have predictive references drawn from multiple reference pictures stored in the reference picture memory (334).

[0053] The controller (350) may manage the encoding operation of the source coder (330), including, for example, setting parameters and subgroup parameters used to encode video data.

[0054] The outputs of all the above functional units may undergo entropy coding in the entropy coder (345). The entropy coder (345) converts the symbols generated by the various functional units into coded video sequences by applying lossless compression to the symbols according to techniques such as Huffman coding, variable-length coding, arithmetic coding, etc.

[0055] The transmitter (340) may buffer the encoded video sequence generated by the entropy coder (345) and prepare it for transmission over a communication channel (360), which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (340) may merge the encoded video data from the video encoder (330) with other data to be transmitted (e.g., encoded audio data and / or auxiliary data streams (not shown)).

[0056] The controller (350) may manage the operation of the video encoder (303). During encoding, the controller (350) may assign a specific encoding picture type to each encoding picture. This encoding picture type may affect the encoding technique that can be applied to each picture. For example, a picture may be assigned as one of the following picture types:

[0057] An intra-picture (I-picture) may be encoded and decoded without using other pictures in the sequence as a source for prediction. Some video codecs enable different types of intra-pictures, including, for example, Independent Decoder Refresh (IDR) pictures.

[0058] The prediction picture (P-picture) may be encoded and decoded using intra-prediction or inter-prediction, which uses motion vectors and reference indices to predict the sample values ​​of each block.

[0059] A bidirectional predictive picture (B-picture) may be encoded and decoded using intra-prediction or inter-prediction, which uses two motion vectors and a reference index to predict the sample values ​​for each block. Similarly, a multiple predictive picture may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0060] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each), and each block may be coded. Blocks may be predictively coded by referencing other (already coded) blocks, as determined by the coding assignment applied to each picture in the block. For example, blocks of picture I may be coded non-predictively, or they may be coded predictively by referencing already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of picture P may be coded predictively by referencing one previously coded reference picture, via spatial prediction or temporal prediction. Blocks of picture B may be coded predictively by referencing one or two previously coded reference pictures, via spatial prediction or temporal prediction.

[0061] The video encoder (303) may perform coding operations in accordance with a predetermined video coding technique or standard, such as ITU-T Rec. H.265. In such operations, the video encoder (303) may perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to the syntax specified by the video coding technique or standard being used.

[0062] In one embodiment, the transmitter (340) may transmit additional data along with the encoded video. The source coder (330) may include such data as part of the coded video sequence. The additional data may include time / space / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0063] Video may be captured as multiple source pictures (video pictures) in a time series. Intra-picture prediction (often abbreviated as intra-prediction) utilizes spatial correlations in a given picture, while inter-picture prediction utilizes (temporal or other) correlations between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is partitioned into blocks. When a block in the current picture is analogous to a reference block in a reference picture that has been previously coded in the video and is still buffered, the block in the current picture can be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension to identify the reference picture if multiple reference pictures are used.

[0064] In some embodiments, a dual prediction technique can be used in interpicture prediction. According to the dual prediction technique, two reference pictures are used, such as a first reference picture and a second reference picture, both of which are earlier in the decoding order than the current picture in the video (but may be earlier and later in the display order, respectively). Blocks in the current picture can be coded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. Blocks can be predicted by combinations of the first and second reference blocks.

[0065] Furthermore, merge mode technology can be used in interpicture prediction to improve coding efficiency.

[0066] According to some embodiments of this disclosure, predictions such as interpicture prediction and intrapicture prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are partitioned into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU contains three coding tree blocks (CTBs), which are one lumen CTB and two chroma CTBs. Each CTU can be recursively quadtree-partitioned into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one 64x64 pixel CU, or four 32x32 pixel CUs, or sixteen 16x16 pixel CUs. In one example, each CU is analyzed to determine the prediction type of the CU, such as interpicture type or intrapicture type. A CU is divided into one or more prediction units (PUs) depending on its temporal and / or spatial predictability. Generally, each PU contains a chroma prediction block (PB) and two chroma PBs. In one embodiment, the coding (encoding / decoding) prediction operation is performed in units of prediction blocks. Using a chroma prediction block as an example of a prediction block, the prediction block contains a matrix of pixel values ​​(e.g., chroma values) such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0067] It should be noted that the video encoders (103) and (303) and the video decoders (110) and (210) can be implemented using any suitable technique. In one embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) can be implemented using one or more integrated circuits. In another embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) can be implemented using one or more processors that execute software instructions.

[0068] Aspects of this disclosure provide techniques for improving affine motion. In some examples, these techniques can be applied to blocks in an affine motion interprediction mode using biprediction.

[0069] In VVC, various interprediction modes are available. For interprediction CUs, motion parameters can include the MV, one or more reference picture indices, a reference picture list usage index, and further information about specific coding features to be used for interprediction sample generation. Motion parameters can be explicitly or implicitly transmitted. When a CU is coded in skip mode, the CU can be associated with a PU and cannot have significant residual coefficients, coded motion vector differences or MV differences (e.g., MVD), or reference picture indices. A merge mode can be specified, in which case the motion parameters for the current CU are obtained from adjacent CUs, including spatial and / or temporal candidates, and further information as optionally introduced in VVC. The merge mode can be applied not only to skip mode but also to interprediction CUs. In one example, an alternative to the merge mode is the explicit transmission of motion parameters, where the MV, the corresponding reference picture indices for each reference picture list, and the reference picture list usage flag, as well as other information, are explicitly transmitted for each CU.

[0070] In one embodiment, such as VVC, the VVC Test Model (VTM) reference software supports augmented merge prediction, merge motion vector difference (MMVD) mode, adaptive motion vector (AMVP) mode using symmetric MVD signal transfer, affine motion compensation prediction, subblock-based temporal motion vector prediction (SbTMVP), adaptive motion vector resolution (AMVR), motion field storage (1 / 16 lumasample MV storage and 8x8 motion field compression), bi-prediction with CU-level weights (BCW), bi-directional optical flow (BDOF), prediction refinement using optical flow (PROF), decoder side motion vector refinement (DMVR), combined inter and intra prediction (CIIP), and geometric partitioning mode (GPM). Includes one or more interprediction coding tools, including mode, etc. Interpretation and related methods are described in detail below.

[0071] In HEVC, a translational motion model is applied for motion compensation prediction (MCP). Translational motion is the movement of all points of a moving object uniformly along the same line or in the same direction, and therefore, a motion vector can be used to define the motion of the moving object. In the real world, many types of motion can exist, such as zooming in / out, rotation, perspective motion, and other irregular motions. Non-translational motion describes the movement of points of a moving object that does not move uniformly. Some motion models can include a translational component for translational motion and a non-translational component for non-translational motion.

[0072] Block-based affine transform motion compensation prediction can be applied to VTM and other systems. Figure 4A shows the affine motion field of block (402) described by motion information of two control points (4 parameters). Figure 4B shows the affine motion field of block (404) described by motion vectors of three control points (6 parameters).

[0073] As shown in Figure 4A, in the four-parameter affine motion model, the motion vector at the sample position (x,y) within block (402) can be derived by the following equation (1).

number

number

[0074] As shown in Figure 4B, in the 6-parameter affine motion model, the motion vector at the sample position (x,y) within block (404) can be derived using the following equation (3).

number

number

[0075] As shown in Figure 5, block-based affine transformation prediction can be applied to simplify motion-compensated prediction. To derive the motion vector for each 4x4 luma subblock, the motion vector of the center sample (e.g., (502)) of each subblock (e.g., (504)) within the current block (500) can be calculated according to equations (1) to (4) and rounded to a fractional precision of 1 / 16. Then, a motion-compensated interpolation filter can be applied to generate predictions for each subblock using the derived motion vectors. The subblock size of the chroma component can also be set as 4x4. The MV of a 4x4 chroma subblock can be calculated as the average of the MVs of four corresponding 4x4 luma subblocks.

[0076] In affine merge prediction, the affine merge (AF_MERGE) mode can be applied to CUs that have both a width and height of 8 or more. The current CU's CPMV can be generated based on the motion information of the spatially adjacent CUs. Up to five CPMVP candidates can be applied to affine merge prediction, and an index can be signaled to indicate which of the five CPMVP candidates is available for the current CU. In affine merge prediction, three types of CPMV candidates can be used to form the affine merge candidate list: (1) inherited affine merge candidates extrapolated from the CPMV of the adjacent CU, (2) constructed affine merge candidates having a CPMVP derived using the translational MV of the adjacent CU, and (3) zero MV.

[0077] In VTM3, a maximum of two inherited affine candidates can be applied. The two inherited affine candidates can be derived from the affine motion models of adjacent blocks. For example, one inherited affine candidate can be derived from the left adjacent CU, and the other inherited affine candidate can be derived from the upper adjacent CU. An exemplary candidate block can be shown in Figure 6. As shown in Figure 6, for the left predictor (or left inherited affine candidate), the scan order can be A0->A1, and for the upper predictor (or upper inherited affine candidate), the scan order can be B0->B1->B2. Therefore, only the first available inherited candidate from each side can be selected. No pruning check is performed between the two inherited candidates. When an adjacent affine CU is identified, the control point motion vector of the adjacent affine CU can be used to derive the CPMVP candidate in the affine merge list of the current CU. As shown in Figure 7, when the adjacent lower-left block A of the current block (704) is coded in affine mode, the motion vectors v2, v3, and v4 of the upper-left, upper-right, and lower-left corners of CU(702) containing block A can be obtained. When block A is coded in a 4-parameter affine model, the two CPMVs of the current CU(704) can be calculated according to v2 and v3 of CU(702). When block A is coded in a 6-parameter affine model, the three CPMVs of the current CU(704) can be calculated according to v2, v3, and v4 of CU(702).

[0078] The constructed affine candidate for the current block may be a candidate constructed by combining adjacent translational motion information of each control point of the current block. The motion information of the control points can be derived from the specified spatial neighbors and temporal neighbors that can be shown in FIG. 8. As shown in FIG. 8, CPMV k (k=1, 2, 3, 4) represents the k-th control point of the current block (802). For CPMV1, the blocks B2->B3->A2 can be checked, and the MV of the first available block can be used. For CPMV2, the blocks B1->B0 can be checked. For CPMV3, the blocks A1->A0 can be checked. If CPMV4 is not available, TMVP can be used as CPMV4.

[0079] After the MVs of the four control points are obtained, an affine merge candidate can be constructed for the current block (802) based on the motion information of the four control points. For example, the affine merge candidate can be constructed based on combinations of the MVs of the four control points in the order of {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, and {CPMV1, CPMV3}.

[0080] A combination of three CPMVs can construct a 6-parameter affine merge candidate, and a combination of two CPMVs can construct a 4-parameter affine merge candidate. To avoid the motion scaling process, if the reference indices of the control points are different, the associated combination of the control point MVs can be discarded.

[0081] After the inherited affine merge candidates and the constructed affine merge candidates are checked, if the list is not yet full, a zero MV can be inserted at the end of the list.

[0082] In affine AMVP prediction, the affine AMVP mode can be applied to CUs having both width and height of 16 or more. To indicate whether the affine AMVP mode is used, an affine flag at the CU level can be signaled in a bitstream, and then another flag can be signaled to indicate whether a 4-parameter affine or a 6-parameter affine is applied. In affine AMVP prediction, the difference between the current CU's CPMV and the current CU's CPMVP predictor can be signaled in a bitstream. The size of the affine AMVP candidate list can be 2, and the affine AMVP candidate list can be generated by using four types of CPMV candidates in the following order: (1) Inherited affine AMVP candidates extrapolated from adjacent CU's CPMV (2) Constructed affine AMVP candidates using CPMVP derived using translational MV of adjacent CUs (3) Translational MV from adjacent CU (4) Zero MV

[0083] The order of checking inherited affine AMVP candidates can be the same as the order of checking inherited affine merge candidates. Only affine CUs with the same reference picture as the current block can be considered to determine AVMP candidates. When inherited affine motion predictors are inserted into the candidate list, the pruning process may not be applied.

[0084] Candidates for constructing AMVP can be derived from specified spatial adjacencies. As shown in Figure 8, the same check order can be applied as in the construction of affine merge candidates. Furthermore, the reference picture indices of adjacent blocks can also be checked. The first block in the check order can be interconnected and have the same reference picture as CU(802). When CU(802) is currently coded in 4-parameter affine mode and both mv0 and mv1 are available, one candidate for constructing AMVP can be determined. Candidates for constructing AMVP can be further added to the affine AMVP list. When CU(802) is currently coded in 6-parameter affine mode and all three CPMVs are available, a candidate for constructing AMVP can be added as one of the candidates in the affine AMVP list. Otherwise, the candidate for constructing AMVP can be set as unavailable.

[0085] After the inherited affine AMVP candidates and constructed AMVP candidates have been examined, if the number of candidates in the affine AMVP list is still less than 2, mv0, mv1, and mv2 can be added in order. mv0, mv1, and mv2 can act as translational MVs to predict all control point MVs of the current CU (e.g., (802)) when available. Finally, if the affine AMVP list is not yet full, zero MVs can be used to fill the affine AMVP list.

[0086] Subblock-based affine motion compensation can save memory access bandwidth and reduce computational complexity compared to pixel-based motion compensation, at the expense of prediction accuracy penalties. To achieve finer granularity in motion compensation, prediction refinement with optical flow (PROF) can be used to improve subblock-based affine motion compensation predictions without increasing memory access bandwidth for motion compensation. In VVC, after subblock-based affine motion compensation has been performed, the Luma prediction samples can be refined by adding the difference derived by the optical flow formula. PROF can be written in four steps as follows:

[0087] Step (1): Subblock-based affine motion compensation can be performed to generate subblock predictions I(i,j).

[0088] Step (2): Spatial gradient g of subblock prediction x (i,j) and g y (i,j) can be calculated at each sample position using a 3-tap filter [-1,0,1]. The gradient calculation can be the same as the gradient calculation in BDOF. For example, the spatial gradient g x (i,j) and g y (i,j) can be calculated based on equations (5) and (6), respectively.

number

number

[0089] Step (3): The improvement in Luma prediction can be calculated using the optical flow equation shown in equation (7).

number

[0090] Since the affine model parameters and the sample position relative to the subblock center may not change from subblock to subblock, Δv(i,j) can be calculated for the first subblock (e.g., (902)) and reused for other subblocks (e.g., (910)) within the same CU (e.g., (900)). dx(i,j) is calculated from the sample position (i,j) to the subblock center (x SB ,y SB If we take the horizontal offset up to ) and dy(i,j) as the vertical offset, then Δv(x,y) can be derived by the following equations (8) and (9).

number

number

[0091] To maintain accuracy, the center of the subblock (x SB ,y SB ) is ((W SB -1) / 2,(H SB It can be calculated as -1) / 2), where WSB and H SB These represent the subblock width and height, respectively.

[0092] Once Δv(x,y) is obtained, the parameters of the affine model can be obtained. For example, for a four-parameter affine model, the parameters of the affine model can be expressed by equation (10).

number

number

[0093] Step (4): Finally, the improved Luma prediction ΔI(i,j) can be added to the subblock prediction I(i,j). The final prediction I' can be generated as shown in equation (12).

number

[0094] PROF may not apply in two cases for affine-coded CUs, namely (1) when all control point MVs are the same, indicating that the CU has only translational motion, and (2) when the affine motion parameters are greater than the specified limits, because the subblock-based affine MC is degraded to a CU-based MC to avoid large memory access bandwidth requirements.

[0095] In some examples, a technique called affine motion vector difference (MVD) coding is used in affine mode. When coding in affine AMVP mode, each control point in the affine coding block has a motion vector difference (MVD). For each reference picture, the MVD of the control point is calculated from the actual CPMV value and the CPMV value of the affine AMVP predictor.

[0096] In one example, for a four-parameter affine, two MVDs (denoted by MVD0 and MVD1) are coded for each reference list according to equations (13) and (14).

number

number

[0097] In another example, for a six-parameter affine, three MVDs (indicated by MVD0, MVD1, and MVD2) are coded for each reference list according to equations (15), (16), and (17).

number

number

number

[0098] Affine motion estimation (ME) in VVC reference software such as VTM can operate for both single and dual predictions. Single prediction can be performed on either reference list L0 or reference list L1, while dual prediction can be performed on both reference list L0 and reference list L1.

[0099] Figure 10 shows a schematic diagram of Affine ME(1000). As shown in Figure 10, in Affine ME(1000), an affine single prediction (S1002) is performed on the reference list L0 to obtain a prediction P0 of the current block (also called the reference prediction P0 in reference list L0 or the affine prediction P0) based on the initial reference block in reference list L0. Also, an affine single prediction (S1004) is performed on the reference list L1 to obtain a prediction P1 of the current block (also called the reference prediction P1 in reference list L1 or the affine prediction P1) based on the initial reference block in reference list L1. In (S1006), an affine biprediction can be performed. The affine biprediction (S1006) can start with the initial prediction residual (2I-P0)-P1, where I can be the initial value of the current block. Affine biprediction (S1006) can search for candidates in reference list L1 around an initial reference predictor in reference list L1 to find the best (or selected) reference predictor with the minimum predictive residual (2I-P0)-Px, where Px is the predictor of the current block based on the selected reference predictor.

[0100] Using a reference picture, for the current coding block, the affine ME process can first select a set of control point motion vectors (CPMVs) as the base. An iterative method can be used to generate the predicted output of the current affine model corresponding to the set of CPMVs, calculate the gradient of the predicted samples, and then solve the linear equation to determine the delta CPMV to optimize the affine prediction. The iterations can be stopped when all delta CPMVs are 0 or when the maximum number of iterations has been reached. The CPMVs obtained from the iterations can be the final CPMVs for the reference picture.

[0101] After the best affine CPVMs for affine single prediction have been determined in both reference lists L0 and L1, an affine biprediction search can be performed using the best single prediction CPMV and one of the reference lists to search for the best CPMV in the other reference list to optimize the affine biprediction output. The affine biprediction search can be performed iteratively on the two reference lists to obtain the optimal result.

[0102] Figure 11 shows an exemplary affine ME process (1100) in which the final CPMV associated with a reference picture can be calculated. The affine ME process (1100) can be started from (S1102). In (S1102), the base CPMV of the current block can be determined. The base CPMV can be determined based on one of the following: the merge index, the advanced motion vector prediction (AMVP) predictor index, the affine merge index, etc.

[0103] In (S1104), the initial affine prediction for the current block can be obtained based on the base CPMV. For example, according to the base CPMV, a 4-parameter affine motion model of a 6-parameter affine motion model can be applied to generate the initial affine prediction.

[0104] In (S1106), the gradient of the initial affine prediction can be obtained. For example, the gradient of the initial affine prediction can be obtained based on equations (5) and (6).

[0105] In (S1108), the delta CPMV can be determined. In some embodiments, the delta CPMV can be associated with the displacement between the initial affine prediction and subsequent affine predictions, such as the first affine prediction. Based on the gradient of the initial affine prediction and the delta CPMV, the first affine prediction can be obtained. The first affine prediction can correspond to the first CPMV.

[0106] In (S1110), a decision can be made to check whether the delta CPMV is zero or whether the number of iterations is greater than or equal to a threshold. If the delta CPMV is zero or the number of iterations is greater than or equal to a threshold, in (S1112), the final (or selected) CPMV can be determined. The final (or selected) CPMV may be a first CPMV determined based on the gradient of the initial affine prediction and the delta CPMV.

[0107] Continuing with (S1110), a new iteration can be started if the delta CPMV is not zero or the number of iterations is less than the threshold. In the new iteration, the updated CPMV (e.g., the first CPMV) is provided to (S1104) to generate an updated affine prediction. The affine ME process (1100) can then proceed to (S1106) to calculate the gradient of the updated affine prediction. The affine ME process (1100) can then proceed to (S1108) to continue with a new iteration.

[0108] In affine motion models, the four-parameter affine motion model can be further described by equations that include rotational and zooming motions. For example, the four-parameter affine motion model can be rewritten as equation (18) below.

number

number

number

[0109] In interpicture prediction, merge mode can be used to improve coding efficiency. In merge mode, motion vectors are derived from adjacent blocks and used directly for motion compensation. To improve the accuracy of the MV in merge mode, a bilateral matching (BM) based decoder-side motion vector improvement can be applied in VVCs, etc. In bi-prediction operation, an improved MV can be searched around the initial MV in reference picture lists L0 and L1. The BM method can calculate the distortion between two candidate blocks in reference picture lists L0 and L1.

[0110] Figure 12 shows an exemplary schematic diagram of a BM-based decoder-side motion vector improvement. As shown in Figure 12, the current picture (1202) can contain the current block (1208). The current picture can contain reference picture list L0 (1204) and reference picture list L1 (1206). For the current block 1208, the initial reference block (1212) in reference picture list L0 (1204) can be positioned according to the initial motion vector MV0, and the initial reference block (1214) in reference picture list L1 (1206) can be positioned according to the initial motion vector MV1. A search process can be performed around the initial MV0 in reference picture list L0 (1204) and the initial MV1 in reference picture list L1 (1206). For example, the first candidate reference block (1210) can be identified in reference picture list L0 (1204), and the first candidate reference block (1216) can be identified in reference picture list L1 (1206). The sum of absolute differences (SAD) between candidate reference blocks (e.g., (1210) and (1216)) based on each MV candidate (e.g., MV0' and MV1') around the initial MV (e.g., MV0 and MV1) can be calculated. The MV candidate with the lowest SAD becomes the improved MV and can be used to generate a dual prediction signal to predict the current block (1208).

[0111] In some examples (e.g., VVC), the DMVR is implicitly applied at the block level without the need to signal side information within the bitstream. The DMVR algorithm is invoked when a block satisfies certain conditions. For example, conditions (also called DMVR requirements or a set of DMVR conditions) may include: (1) CU level merge mode using dual prediction MV (2) For the current picture, there is one reference picture in the past and another reference picture in the future. (3) The distance from the two reference pictures to the current picture (e.g., the POC difference) is the same. (4) Both reference pictures are short-term reference pictures. (5) CU has more than 64 luma samples. (6) Both the CU height and CU width must be at least 8 luma samples. (7) The bi-prediction using the weight index of bi-prediction with CU level weights (BCW) shows equal weights. (8) Weighted prediction (WP) is not currently enabled for the block. (9) The combined inter and intra prediction (CIIP) mode is not currently used in the block.

[0112] It should be noted that the improved MV derived by the DMVR process is used to generate interprediction samples and can be used in time motion vector prediction for future picture coding. In some examples, the original MV is used in the deblocking process and also in spatial motion vector prediction for future CU coding.

[0113] In some examples (e.g., VVC), decoder-side motion vector refinement (DMVR) is applied to CUs coded in normal merge mode. Pairs of MVs obtained from normal merge candidates are used as input to the DMVR process. The DMVR processes the input MV pair {mv L0 ,mv L1 To improve}, bilateral matching (BM) is applied, and improved MV pairs {mv refinedL0 ,mv refinedL1 Use}. The output MV of the DMVR is called the improved MV pair and can be expressed by equation (21).

number

[0114] An input MV pair refers to two different reference pictures that have an equal difference in picture order count (POC) relative to the current picture, and since these two reference pictures are in different time directions, a motion vector difference Δmv is applied to the input MV pair in order to obtain an improved MV pair by using the MVD mirroring property.

[0115] In some examples, in DMVR, the luma coding block is divided into 16x16 subblocks for the MV improvement process. The improved search range is two integer luma samples from the initial MV. Δmv is derived independently for each subblock in two steps: an integer-precision motion search step (also called an integer sample offset search) followed by a fractional motion search step (also called a fractional sample offset search). Finally, subblock motion compensation (MC) is applied to the improved MV pair {mv refinedL0 ,mv refinedL1 It is applied using}.

[0116] According to one aspect of the present invention, in the integer-precision motion search step of the DMVR, the search space includes 25 candidate MV pairs that can be represented by equation (22).

number

number

[0117] In equation (23), W and H represent the weight and height of the subblock. P0 i,j[m+i,2n+j] and P1 i,j [mi,2n-j] indicates the corresponding samples in the two reference pictures. The integer-precision motion search step of the DMVR terminates when the SAD of the initial MV pair is less than the threshold. Otherwise, the SADs of the remaining 24 points are calculated and examined, for example, in raster scan order. The point with the smallest SAD is selected as the output of the integer-precision motion search step. In some examples, the original MV may be treated favorably during the DMVR process to reduce the uncertainty penalty of the DMVR refinement. In one example, the SAD between reference blocks referenced by the initial MV candidate is reduced by 1 / 4 of the SAD value, and thus the initial MV candidate may be favored.

[0118] In DMVR, the fractional motion search step (fractional sample offset search) further refines the candidate MV pairs selected in the integer-precision motion search step (integer sample offset search). In some examples, to save computational complexity, the fractional sample refinement is derived by using the parametric error surface equation instead of further searching with SAD comparison. The fractional sample refinement is conditionally invoked based on the output of the integer sample offset search step. If the integer sample offset search step terminates at a center with the smallest SAD in either the first or second iteration of the search, the fractional sample refinement is further applied. In parametric error surface-based subpixel offset estimation, the center position cost and the costs at four adjacent positions from the center (e.g., those in the first step of the integer-precision motion search) are used to fit a 2-D parabolic error surface equation such as equation (24).

number

number

number

[0119] In equations (25) and (26), E(-1,0), E(1,0), E(1,0), E(0,-1), and E(0,0) represent the cost values ​​of the five points. All cost values ​​are positive, and the minimum value is E(0,0), so x min and y min The value of is automatically constrained between -8 and 8. The constraint corresponds to a half-pel offset with 1 / 16 pel-MV precision in VVC. The calculated fraction (x min ,y min This is added to the integer distance-improved MV to obtain the subpixel-precision-improved delta MV.

[0120] In VVC and similar systems, bilinear interpolation and sample padding can be applied. The resolution of the MV can be, for example, 1 / 16 of a sample. Samples at fractional positions can be interpolated using an 8-tap interpolation filter. In DMVR, the search points can enclose an initial fractional periMV with integer sample offsets; therefore, samples at fractional positions need to be interpolated for the DMVR search process. To reduce computational complexity, a bilinear interpolation filter can be used to generate fractional samples for the search process in DMVR. Another important effect is that by using a bilinear filter with a 2-sample search range, DVMR accesses fewer reference samples compared to a normal motion compensation process. After an improved MV is obtained using the DMVR search process, a normal 8-tap interpolation filter can be used to generate the final prediction. Samples that may not be required for the interpolation process based on the original MV, but may be required for the interpolation process based on the improved MV, can be padded from available samples because they access fewer reference samples compared to a normal MC process.

[0121] When the width and / or height of a CU is greater than 16 luma samples, the CU can be further divided into subblocks having widths and / or heights equal to 16 luma samples. The maximum unit size for the DMVR search process can be limited to 16 × 16.

[0122] In some examples, multi-path DMVR can be used. In one example, bilateral matching (BM) is applied to the coding block in the first pass. In the second pass, BM is applied to each 16x16 subblock within the coding block. In the third pass, the MV within each 8x8 subblock is improved by applying bidirectional optical flow (BDOF). The improved MV is stored for both spatial motion vector prediction and temporal motion vector prediction.

[0123] Specifically, the first pass performs block-based bilateral matching MV refinement. In the first pass, the refined MV is derived by applying the BM to the coding block. Similar to decoder-side motion vector refinement (DMVR), in bipredictive operation, the refined MV is searched around two initial MVs (MV0 and MV1) in reference picture lists L0 and L1. The refined MVs (MV0_pass1 and MV1_pass1) are derived around the initial MVs based on the minimum bilateral matching cost between the two reference blocks in L0 and L1.

[0124] BM can perform a local search to derive integer sample-precision intDeltaMV. The local search applies a 3x3 square search pattern, looping through the search ranges [-sHor,sHor] horizontally and [-sVer,sVer] vertically, where the values ​​of sHor and sVer are determined by the block dimensions, and the maximum values ​​of sHor and sVer are 8.

[0125] The bilateral matching cost is calculated as bilCost = mvDistanceCost + sadCost. When the block size cbW × cbH is greater than 64, the mean removed SAD (MRSAD) cost function is applied to remove the DC effect of distortion between reference blocks. The intDeltaMV local search terminates when the bilCost of the center point of the 3x3 search pattern has the minimum cost. Otherwise, the current minimum cost search point becomes the new center point of the 3x3 search pattern, and the search for the minimum cost continues until the end of the search range is reached.

[0126] To derive the final deltaMV, the existing fractional sample improvements are further applied. The improved MV after the first pass is derived as follows:

number

number

[0127] In the second pass, a subblock-based bilateral matching MV improvement is performed. Specifically, in the second pass, the improved MV is derived by applying the BM to a 16x16 grid subblock. For each subblock, the improved MV is searched for in the reference picture lists L0 and L1 around the two MVs (MV0_pass1 and MV1_pass1) obtained in the first pass. The improved MVs (MV0_pass2(sbIdx2) and MV1_pass2(sbIdx2)) are derived based on the minimum bilateral matching cost between the two reference subblocks in L0 and L1.

[0128] For each subblock, BM performs a brute-force search to derive integer sample-precision intDeltaMV. The brute-force search has a horizontal search range of [-sHor, sHor] and a vertical search range of [-sVer, sVer], where the values ​​of sHor and sVer are determined by the block dimensions, and the maximum values ​​of sHor and sVer are 8.

[0129] The bilateral matching cost is calculated by applying a cost factor to the cost of the sum of absolute transformed differences (SATD) between two reference subblocks, where bilCost = satdCost × costFactor. In some examples, the search region (2 × sHor + 1) × (2 × sVer + 1) is divided into up to five diamond-shaped search regions.

[0130] Figure 13 shows the search region (1300) in several examples. The search region (1300) is divided into five search regions (1301) to (1305). The shape of the search region is similar to a diamond (rhombus).

[0131] In some examples, each search region is assigned a costFactor determined by the distance (intDeltaMV) between each search point and the starting MV, and each diamond region is processed in an order starting from the center of the search region. Within each region, search points are processed in a raster scan order starting from the top left of the region and moving toward the bottom right corner. When the smallest bilCost in the current search region is less than a threshold equal to sbW × sbH, the integer Pell brute-force search terminates; otherwise, the integer Pell brute-force search continues to the next search region until all search points have been examined. Furthermore, if the difference between the previous minimum cost in an iteration and the current minimum cost is less than a threshold equal to the region of the block, the search process terminates.

[0132] In some examples, fractional sample improvements, such as the DMVR fractional sample improvement in VVC, are further applied to derive the final deltaMV(sbIdx2). The improved MV in the second pass is derived as follows:

number

number

[0133] In the third pass, a subblock-based bidirectional optical flow MV improvement can be performed. Specifically, in the third pass, the improved MV is derived by applying BDOF to an 8x8 grid of subblocks. For each 8x8 subblock, the BDOF improvement is applied to derive scaled Vx and Vy without clipping, starting from the improved MV of the parent subblock in the second pass. The derived bioMv(Vx,Vy) is rounded to 1 / 16 sample precision and clipped between -32 and 32. The improved MV in the third pass (MV0_pass3(sbIdx3) and MV1_pass3(sbIdx3)) is derived as follows:

number

number

[0134] In some cases, affine merging using DMVR can be performed on translational models. For example, the improvement of the base MV (the translational part of the affine model) of a coding block coded in affine merging mode can be achieved by applying the first step of multipath DMVR. For instance, when a candidate satisfies the DMVR condition, a translational MV offset is added to all CPMVs of the candidate in the affine merging list. The translational MV offset is derived by minimizing the cost of bilateral matching, such as DMVR. Note that the DMVR condition remains unchanged in some cases.

[0135] It should be noted that the translational MV offset search process can be the same as the first pass of a multipath DMVR in some examples, such as ECM. Specifically, a 3x3 square search pattern is used to find the best integer MV offset by looping through the search ranges of [-8, +8] horizontally and [-8, +8] vertically. A half-pel search is then performed around the best integer position, and error surface estimation is performed to find the MV offset with an accuracy of 1 / 16. The improved CPMV can be stored as a multipath DMVR in ECM for both spatial motion vector prediction and time motion vector prediction.

[0136] Some aspects of this disclosure provide techniques for application to affine model-based DMVR methods or CPMV-based DMVR methods that can improve the prediction accuracy of affine bipredictions.

[0137] According to one aspect of this disclosure, a multi-step bilateral matching motion improvement process (e.g., DMVR or similar method) can be applied to an affine block (a block coded using affine motion compensation prediction) using biprediction. The multi-step bilateral matching motion improvement process describes a motion improvement process having multiple steps (also called steps) that determine motion improvement to an initial motion model so as to minimize the distortion error of corresponding portions in two reference pictures (according to motion improvement in addition to the initial motion model) for a current block in a current picture, where one of the two reference pictures is before the current picture in the picture sequence, and the other of the two reference pictures is after the current picture in the picture sequence. For affine biprediction, an affine model can be used to predict a block based on a first affine prediction (e.g., also called a first reference sample) for the block in a first reference picture and a second affine prediction (e.g., also called a second reference sample) for the block in a second reference picture. In some examples, an encoder / decoder can start with an initial affine model using biprediction for the current block. The initial affine model can be determined based on any appropriate technique, such as the affine merge mode or the affine AMVP mode. The encoder / decoder can derive an improved affine model by performing a multi-stage bilateral matching motion improvement on the initial affine model, having at least a first and a second stage. One of the first and second stages applies the first bilateral matching motion improvement to the translational component of the initial affine model, and the other of the first and second stages applies the second bilateral matching motion improvement to the non-translational component of the initial affine model. The current block is then reconstructed using the improved affine model based on the first and second reference pictures.

[0138] In some embodiments, in the first stage of multi-stage BM motion improvement, an affine DMVR is performed on the translational component to find the best translational MV offset that minimizes the bilateral matching cost between each pair of affine predictions in two reference pictures (e.g., a first reference sample in the first reference picture and a second reference sample in the second reference picture). In the second stage, an affine model-based bilateral matching improvement method is performed to further improve the affine model.

[0139] In one embodiment, the affine model-based bilateral matching improvement process can be performed by searching for the minimum matching error within a specific search range. In one example, the search can be performed in a manner similar to that of DMVR. Unlike DMVR, the search range can include not only the translational portion but also the rotational and zoom coefficients.

[0140] In some examples of affine model-based bilateral matching improvement processes, multiple candidate affine prediction pairs can be determined based on the search range according to the affine motion model. Each candidate affine prediction pair in the multiple candidate affine prediction pairs may include each candidate affine prediction in the first reference frame and each candidate affine prediction in the second reference frame. For each candidate affine prediction pair in the multiple candidate affine prediction pairs, a bilateral matching cost value can be calculated. The bilateral matching cost value can represent the difference between the candidate affine prediction in the first reference frame and the candidate affine prediction in the second reference frame. The bilateral matching cost value can be determined based on one of the following: mean square error (MSE), mean absolute difference (MAD), SAD, sum of absolute transformed difference (SATD), etc. In one example, an affine prediction pair from among multiple candidate affine prediction pairs related to the minimum cost value can be selected to predict the current block. For example, an affine motion model (also called an affine model) can be determined based on the CPMV associated with the selected affine prediction pair. Based on the determined affine motion model, further predictions can be made at the block level, subblock level, or pixel level of the current block.

[0141] It should be noted that any suitable affine model can be used. The affine model may include affine parameters for translational motion, and / or rotational motion, and / or zoom motion, and / or other motions. For example, the affine model may include a zoom parameter that characterizes the zoom motion. In the second step, adjustments to the zoom parameter can be evaluated based on the bilateral matching costs, such as SAD, SSE, etc., between the affine predictions associated with the adjustments.

[0142] In some cases, affine model-based bilateral matching improvements can be performed by minimizing the distortion model of the current block's affine prediction. The distortion model can be similar to the affine ME described with reference to Figures 10-11, can be based on a first-order Taylor expansion, and in some cases is called a gradient-based improvement iterative process.

[0143] In some examples, to perform a gradient-based refined iteration process, the strain model (or strain value) between two affine predictions in the first iteration of the bilateral matching process can be provided by equations (33) to (35) as follows:

number

number

number

[0144] In some examples, P 1,L0 (i,j) and P 1,L1(i,j) can be determined according to the affine ME search process shown in Figure 11. As shown in equation (34), P 0,L0 (i,j) can be the initial predictor of the current block based on the initial affine prediction (or base CPMV) in the reference list L0. x0,L0 (i,j) is the initial predictor P in the x direction. 0,L0 This can be defined as the gradient of (i,j). y0,L0 (i,j) is the initial predictor P in the y direction. 0,L0 The gradient can be defined as (i,j). Δv x0,L0 (i,j) can be the difference or displacement of two affine predictions (or subblocks) in the reference list L0 along the x-direction, such as the first iterative affine prediction and the initial affine prediction. Δv y0,L0 (i,j) can be the difference or displacement of two affine predictions (or subblocks) in the reference list L0 along the y-direction, such as the first iterative affine prediction and the initial affine prediction.

[0145] Similarly, as shown in equation (35), P 0,L1 (i,j) can be the initial predictor of the current block based on the initial affine prediction (or base CPMV) in the reference list L1. x0,L1 (i,j) is the initial predictor P in the x direction. 0,L1 This can be defined as the gradient of (i,j). y0,L1 (i,j) is the initial predictor P in the y direction. 0,L1 The gradient can be defined as (i,j). Δv x0,L1 (i,j) can be the difference or displacement of two affine predictions (or subblocks) in the reference list L1 along the x-direction, such as the first iterative affine prediction and the initial affine prediction. Δv y0,L1 (i,j) can be the difference or displacement of two reference blocks (or subblocks) in the reference list L1 along the y-direction, such as the first iterative affine prediction and the initial affine prediction.

[0146] Δv x0,L0 (i,j), Δv y0,L0 (i,j), Δv x0,L1(i,j) and Δv y0,L1 Depending on whether at least one of (i,j) is non-zero, the bilateral matching can proceed to the second iteration. In the second iteration, the second predictor P of the current block is determined based on the second iteration affine prediction in the reference list L0. 2,L0 (i,j) and the second predictor P of the current block based on the second iterative affine prediction in the reference list L1. 2,L1 (i,j) can be determined according to equations (36) to (37) as follows.

number

number

[0147] The second iterative affine prediction in the reference list L0 can be indicated by affine motion 0' (e.g., affMV0') from the current block to the second iterative affine prediction in the reference list L0. The second iterative affine prediction in the reference list L1 can be indicated by affine motion 1' (e.g., affMV1') from the current block to the second iterative affine prediction in the reference list L1.

[0148] Furthermore, P 2,L0 (i,j) and P 2,L1 The pixel difference (or cost value) between (i,j) can be calculated according to the following equation (38).

Equation

[0149] The iteration of the bilateral matching process ends when the number of iterations N is greater than or equal to a threshold, or the displacement Δv between the N-th iterative affine prediction in the reference list L0 and the N+1-th iterative affine prediction in the reference list L0 N+1,L0 (i,j) is zero, or the displacement Δv between the N-th iterative affine prediction in the reference list L1 and the N+1-th iterative affine prediction in the reference list L1 N+1,L1 (i,j) is zero. Accordingly, N iterative affine prediction pairs can be generated based on the bilateral matching process. Each of the N iterative affine prediction pairs may include a respective iterative affine prediction in the reference list L0 and a respective iterative affine prediction in the reference list L1. Each of the N iterative affine prediction pairs may also include a respective distortion value indicating the difference between the corresponding iterative affine prediction in the reference list L0 and the corresponding iterative affine prediction in the reference list L1.

[0150] The current block can be predicted based on the best (or selected) iterative affine prediction pair with the minimum distortion value, according to the distortion values ​​(or cost values) of N iterative affine prediction pairs. For example, an affine motion model can be determined based on the CPMV associated with the selected iterative affine prediction pair. Based on the determined affine motion model, further predictions can be made at the block level, subblock level, or pixel level of the current block.

[0151] In one embodiment, the gradient-based improvement iteration process stops during the second stage of affine model-based bilateral matching improvement when the maximum number of iterations is reached or when the delta-affine parameters are not changed. The final delta-affine parameters are used to derive the final affine CPMV in addition to the improved CPMV values ​​from the first stage of translational improvement.

[0152] In another embodiment, the gradient-based improvement iteration process stops during the second stage of affine model-based bilateral matching improvement when the maximum number of iterations is reached or when the delta-affine parameters are not changed. At each iteration, the bilateral matching cost between pairs of affine predictions (e.g., for reference samples in a reference picture) is stored in a list along with the improved CPMV value. At the end of the iteration, the CPMV value with the lowest bilateral matching cost is used as the final improved CPMV value.

[0153] In some embodiments, in the first step, an affine model-based bilateral matching improvement is applied to improve the affine model. Then, in the second step, the translational MV offset is improved to generate the final CPMV.

[0154] In one embodiment, the gradient-based improvement iteration process stops when the maximum number of iterations is reached during the first stage of affine model-based bilateral matching improvement, or when the delta-affine parameters are not changed. The final delta-affine parameters are used to derive the final affine CPMV in addition to the improved CPMV values ​​from the first stage of translational improvement.

[0155] In another embodiment, the gradient-based improvement iteration process stops during the first stage of affine model-based bilateral matching improvement when the maximum number of iterations is reached or when the delta-affine parameters are not changed. At each iteration, the bilateral matching cost between pairs of affine predictions (e.g., reference samples in a reference picture) is stored in a list along with the improved CPMV value. At the end of the iteration, the CPMV value with the lowest bilateral matching cost is used as the final improved CPMV value.

[0156] In some embodiments, in addition to the affine DMVR steps for the translational component and the affine model-based bilateral matching improvement steps, the multi-step bilateral matching motion improvement process may include another step (referred to as the third step) for performing improvements on one or more of the CPMVs. The third step may be performed before or after either the first step and / or the second step.

[0157] In one embodiment, the third step may include multiple iterations for refining a control point. In one example, each iteration is associated with an MV offset within a specific range. In each iteration, the control point is refined sequentially. In the iteration, the refinement is applied to the CPMV of the control point to minimize the bilateral matching cost between pairs of affine predictions in two reference pictures. For example, for the refinement, pairs of refined CPMV values ​​are generated for reference picture L0 and reference picture L1, the corresponding affine predictions (also called reference samples for the current block) in reference picture L0 and reference picture L1 are determined, and the bilateral matching cost is calculated, for example, as the SAD between the two affine predictions in reference picture L0 and reference picture L1. In the iteration, the refined CPMV value of the control point that achieves the lowest bilateral matching cost is determined for the control point and used as the basis for the next iteration, or as the final CPMV value of the control point if the iteration is the last iteration.

[0158] In another embodiment, the third stage may include multiple iterations for refining a control point. In one example, each iteration may be associated with a control point and apply various MV offsets to refine the control point's CPMV value. In the iteration of the control point, the refinement is applied to the control point's CPMV to minimize the bilateral matching cost between pairs of affine predictions in two reference pictures. For example, for the refinement, pairs of refined CPMV values ​​for reference picture L0 and reference picture L1 are generated, the corresponding affine predictions (also called reference samples for the current block) in reference picture L0 and reference picture L1 are determined, and the bilateral matching cost is calculated, for example, as the SAD between pairs of affine predictions in reference picture L0 and reference picture L1. In the iteration, the refined CPMV value of the control point that achieves the lowest bilateral matching cost is used for the control point and is used as the basis for the next iteration, or as the final CPMV value of the control point if the iteration is the last iteration.

[0159] In another embodiment, N combinations of CPMVs can be improved together in each iteration. In an iteration, multiple MV offsets may be tested as improvement steps, with each improvement having the same pair of MV offsets (with opposite signs in L0 and L1) applied to the CPMVs in the N combinations of CPMVs being improved. In an iteration, the set of results for the N CPMVs with the best (e.g., lowest) bilateral matching cost is retained for these control points and used as the basis for the next iteration or as the final CPMV value when the iteration ends. Each possible set of N CPMVs can then be improved in separate iterations. In one example, N is equal to 2. In one example, the block has three control points called the first control point, the second control point, and the third control point. In the first iteration, the combination of CPMVs for the first and second control points is improved. In the second iteration, the combination of CPMVs for the first and third control points is improved. In the third iteration, the combination of CPMVs for the second and third control points is improved.

[0160] According to one aspect of this disclosure, the DMVR improvement can be applied to subblocks of an affine block (e.g., a block with affine motion compensation prediction). A subblock of an affine block is also called an affine subblock.

[0161] In one embodiment, DMVR improvements applied to subblocks of an affine block are performed in the same manner as DMVR improvements for a normal bipredictive interblock. For example, each subblock of an affine block is treated as a normal bipredictive interblock, and the DMVR improvements are applied to the subblock.

[0162] In some examples, each affine subblock (e.g., a subblock within an affine block) is individually improved, and the final improved MV of the affine subblock is used in motion compensation for the affine subblock.

[0163] In some examples, the improved MV of a subblock is used solely for motion compensation and is not stored in the MV buffer for the subblock (e.g., a 4x4 inter-MV buffer in a VTM, and / or a time MV buffer). The subblock's MV buffer stores the subblock MV before the DMVR improvement for the subblock.

[0164] In some cases, the improved MV of a subblock is stored in the subblock's MV buffer.

[0165] In some cases, the improved MV of a subblock is used solely for motion compensation and is not used as an MVP candidate for other blocks.

[0166] In some examples, all improved subblock MVs (e.g., improved subblock MVs) are used as inputs to a regression model for deriving the affine block CPMV.

[0167] In one example, the CPMV derived by the regression model is used for affine motion compensation of an affine block. In another example, the original CPMV is stored for the affine block (and / or affine history candidate buffer) to be used for affine inheritance by other blocks.

[0168] In another example, the CPMV derived by the regression model is used for affine motion compensation of affine blocks and stored for affine blocks (and / or affine history buffers) used for affine inheritance by other blocks.

[0169] In some embodiments, the original CPMV is stored for later use as an affine block (and / or affine history candidate buffer), such as inherited affine merge or inherited affine MVP candidate derivation by other blocks.

[0170] Figure 14 shows a flowchart outlining process (1400) according to one embodiment of the present disclosure. Process (1400) can be used in a video decoder. In various embodiments, process (1400) is performed by processing circuits, such as a processing circuit that performs the functions of a video decoder (110), a processing circuit that performs the functions of a video decoder (210), and so on. In some embodiments, process (1400) is implemented with software instructions, and therefore, when a processing circuit executes a software instruction, the processing circuit executes process (1400). The process begins at (S1401) and proceeds to (S1410).

[0171] In (S1410), coding information for the current block in the current picture is received. The coding information indicates that the current block is coded in affine motion interprediction mode.

[0172] In (S1420), an initial affine model using biprediction is determined for the current block. The initial affine model predicts the current block based on the first and second reference pictures. The initial affine model can be determined based on any appropriate technique, such as the affine merge mode or the affine AMVP mode.

[0173] In (S1430), a multi-stage bilateral matching motion improvement having at least a first and a second stage is performed on the initial affine model to derive an improved affine model. In one of the first and second stages, the first bilateral matching motion improvement is applied to the translational component of the initial affine model, and in the other of the first and second stages, the second bilateral matching motion improvement is applied to the non-translational component of the initial affine model.

[0174] In (S1440), the current block is reconstructed using the improved affine model based on the first and second reference pictures.

[0175] In some embodiments, to perform multi-stage bilateral matching motion improvement, a first bilateral matching motion improvement is performed on the translational component in a first stage prior to a second stage, and a second bilateral matching motion improvement is performed on the non-translational component in a second stage after the first stage.

[0176] In some embodiments, to perform multi-stage bilateral matching motion improvement, a second bilateral matching motion improvement is performed on the non-translational component in a first stage prior to the second stage, and a first bilateral matching motion improvement is performed on the translational component in a second stage after the first stage.

[0177] In some examples, for a first bilateral matching motion improvement, a motion vector offset is determined to be applied to each of the control point motion vectors of the initial affine model in order to minimize the bilateral matching cost value between the first affine prediction in the first reference picture and the second affine prediction in the second reference picture. In one example, each control point has a pair of control point motion vectors, one of which is for the prediction by the first reference picture and the other for the prediction by the second reference picture. The motion vector offset can be applied to pairs of control point motion vectors in opposite directions. In one example, when the initial affine model has two control points, the motion vector offset is applied to four control point motion vectors. In another example, when the initial affine model has three control points, the motion vector offset is applied to six control point motion vectors.

[0178] In some examples, for a second bilateral matching motion improvement, a gradient-based improvement iteration related to the current affine model generates a predicted output for the current block according to the current affine model. The gradient of the predicted samples in the predicted output is then calculated. The linear form formed by the gradient is solved to determine the delta values ​​of the control point motion vectors of the block. Furthermore, it is determined whether one or more stopping conditions are met. Depending on whether none of the one or more stopping conditions are met, the next affine model is determined based on the delta values ​​of the control point motion vectors. The next affine model is input to the next gradient-based improvement iteration. Depending on whether one or more stopping conditions are met, the second bilateral matching motion improvement stops.

[0179] In one example, depending on whether one or more stopping conditions are met, the delta control point motion vectors of each control point motion vector are determined based on the final affine model in the last gradient-based improvement iteration of the second bilateral matching motion improvement.

[0180] In one example, multiple bilateral matching costs are stored, and each of these bilateral matching costs relates to an affine model used in one of several gradient-based improvement iterations of a second bilateral matching motion improvement. For example, the bilateral matching cost related to the affine model in a gradient-based improvement iteration is calculated as the SAD between the first affine prediction in the first reference picture and the second affine prediction in the second reference picture. The lowest bilateral matching cost among the multiple bilateral matching costs is determined. Then, based on the affine model associated with the lowest bilateral matching cost, the delta control point motion vectors of each control point motion vector are determined.

[0181] In some examples, the improved affine model is formed by improved control point motion vectors, which include motion vector offsets derived by a first bilateral matching motion improvement and applied to each of the improved control point motion vectors, and delta control point motion vectors derived by a second bilateral matching motion improvement and applied to each of the control point motion vectors.

[0182] In some examples, a multi-stage bilateral matching motion improvement for an affine model includes at least a third stage in which a third bilateral matching motion improvement is applied to the control point motion vectors of the affine model. In one example, the third stage is performed before the first and second stages. In another example, the third stage is performed between the first and second stages. In yet another example, the third stage is performed after the first and second stages.

[0183] In some examples, the third stage applies a third bilateral matching motion improvement with one or more iterations for each of the control points in the current block. For example, in the iterations for a control point, a pair of improved control point motion vectors is determined at the control point, and this pair of improved control point motion vectors minimizes the bilateral matching cost between a first affine prediction in the first reference picture and a second affine prediction in the second reference picture.

[0184] In some examples, the third stage applies a third bilateral matching motion improvement with one or more iterations for each combination of N control points, where N is a positive integer greater than 1 and less than the total number of control points. In one example, N is 2. In some examples, the iterations for the combinations of N control points determine the same pair of motion vector offsets to apply to each pair of control point motion vectors for the N control points in the combination. The pair of motion vector offsets minimizes the bilateral matching cost value between the first affine prediction in the first reference picture and the second affine prediction in the second reference picture.

[0185] According to one aspect of this disclosure, a DMVR refinement can be applied to a subblock of a current block in affine motion interprediction mode. In some examples, a first motion vector is determined for a subblock of a current block coded in affine motion interprediction mode according to an affine model. A decoder-side motion vector refinement (DMVR) is determined for the subblock to determine a second motion vector using an improvement to the first motion vector. The second motion vector using the improvement to the first motion vector minimizes the bilateral matching cost between a first reference subblock in a first reference picture and a second reference subblock in a second reference picture. The subblock is then reconfigured according to the second motion vector for the subblock.

[0186] In some examples, the first motion vector is stored in a subblock buffer for a subblock, and the first motion vector in the subblock buffer can be used to predict the motion vector for another block. In some examples, the second motion vector is stored in a subblock buffer for a subblock, and the second motion vector in the subblock buffer can be used to predict the motion vector for another block.

[0187] In some examples, improved motion vectors are determined for each subblock of the current block. A regression model is determined based on the improved motion vectors for the subblocks. The improved control point motion vectors for the improved affine model of the current block are derived according to the regression model. The current block is reconstructed according to the improved control point motion vectors.

[0188] In some examples, the control point motion vectors of the initial affine model (the initial affine model before multi-stage bilateral matching refinement) are stored in a buffer associated with the current block, and the control point motion vectors can be used for affine inheritance by other blocks.

[0189] In some examples, the refined control point motion vectors of the refined affine model (the refined affine model after multi-stage bilateral matching refinement) are stored in a buffer associated with the current block, and the refined control point motion vectors can be used for affine inheritance by other blocks.

[0190] Then, the process proceeds to (S1499) and ends.

[0191] The process (1400) can be appropriately adapted. The steps of the process (1400) may be modified and / or omitted. Further steps may be added. Any suitable implementation order may be used.

[0192] Figure 15 shows a flowchart outlining a process (1500) according to an embodiment of the present disclosure. The process (1500) can be used in a video encoder. In various embodiments, the process (1400) is executed by processing circuitry, such as processing circuitry that performs the functions of the video encoder (103), processing circuitry that performs the functions of the video encoder (303), or the like. In some embodiments, the process (1500) is implemented in software instructions, such that when the processing circuitry executes the software instructions, the processing circuitry performs the process (1500). The process starts at (S1501) and proceeds to (S1510).

[0193] At (S1510), it is determined to code the current block in the current picture in an affine motion inter prediction mode.

[0194] In (S1520), an initial affine model using biprediction is determined for the current block. The initial affine model using biprediction predicts the current block based on the first and second reference pictures.

[0195] In (S1530), a multi-stage bilateral matching motion improvement having at least a first stage and a second stage is performed on the initial affine model to derive an improved affine model, wherein one of the first and second stages applies the first bilateral matching motion improvement to the translational component of the initial affine model, and the other of the first and second stages applies the second bilateral matching motion improvement to the non-translational component of the initial affine model.

[0196] In (S1540), the current block is reconstructed using an improved affine model based on the first and second reference pictures.

[0197] In (S1550), coding information for the current block is generated within the coding video bitstream.

[0198] In some embodiments, to perform multi-stage bilateral matching motion improvement, a first bilateral matching motion improvement is performed on the translational component in a first stage prior to a second stage, and a second bilateral matching motion improvement is performed on the non-translational component in a second stage after the first stage.

[0199] In some embodiments, to perform multi-stage bilateral matching motion improvement, a second bilateral matching motion improvement is performed on the non-translational component in a first stage prior to the second stage, and a first bilateral matching motion improvement is performed on the translational component in a second stage after the first stage.

[0200] In some examples, for a first bilateral matching motion improvement, a motion vector offset is determined to be applied to each of the control point motion vectors of the initial affine model in order to minimize the bilateral matching cost value between the first affine prediction in the first reference picture and the second affine prediction in the second reference picture. In one example, each control point has a pair of control point motion vectors, one of which is for the prediction by the first reference picture and the other for the prediction by the second reference picture. The motion vector offset can be applied to pairs of control point motion vectors in opposite directions. In one example, when the initial affine model has two control points, the motion vector offset is applied to four control point motion vectors. In another example, when the initial affine model has three control points, the motion vector offset is applied to six control point motion vectors.

[0201] In some examples, for a second bilateral matching motion improvement, a gradient-based improvement iteration related to the current affine model generates a predicted output for the current block according to the current affine model. The gradient of the predicted samples in the predicted output is then calculated. The linear form formed by the gradient is solved to determine the delta values ​​of the control point motion vectors of the block. Furthermore, it is determined whether one or more stopping conditions are met. Depending on whether none of the one or more stopping conditions are met, the next affine model is determined based on the delta values ​​of the control point motion vectors. The next affine model is input to the next gradient-based improvement iteration. Depending on whether one or more stopping conditions are met, the second bilateral matching motion improvement stops.

[0202] In one example, depending on whether one or more stopping conditions are met, the delta control point motion vectors of each control point motion vector are determined based on the final affine model in the last gradient-based improvement iteration of the second bilateral matching motion improvement.

[0203] In one example, multiple bilateral matching costs are stored, and each of these bilateral matching costs relates to an affine model used in one of several gradient-based improvement iterations of a second bilateral matching motion improvement. For example, the bilateral matching cost related to the affine model in a gradient-based improvement iteration is calculated as the SAD between the first affine prediction of the current block in the first reference picture and the second affine prediction of the current block in the second reference picture. The lowest bilateral matching cost is determined from the multiple bilateral matching costs. Then, the delta control point motion vectors of each control point motion vector are determined based on the affine model associated with the lowest bilateral matching cost.

[0204] In some examples, the improved affine model is formed by improved control point motion vectors, which include motion vector offsets derived by a first bilateral matching motion improvement and applied to each of the improved control point motion vectors, and delta control point motion vectors derived by a second bilateral matching motion improvement and applied to each of the control point motion vectors.

[0205] In some examples, a multi-stage bilateral matching motion improvement for an affine model includes at least a third stage in which a third bilateral matching motion improvement is applied to the control point motion vectors of the affine model. In one example, the third stage is performed before the first and second stages. In another example, the third stage is performed between the first and second stages. In yet another example, the third stage is performed after the first and second stages.

[0206] In some examples, the third stage applies a third bilateral matching motion improvement with one or more iterations for each of the control points in the current block. For example, in the iterations for a control point, a pair of improved control point motion vectors is determined at the control point, and this pair of improved control point motion vectors minimizes the bilateral matching cost between a first affine prediction in the first reference picture and a second affine prediction in the second reference picture.

[0207] In some examples, the third stage applies a third bilateral matching motion improvement with one or more iterations for each combination of N control points, where N is a positive integer greater than 1 and less than the total number of control points. In one example, N is 2. In some examples, the iterations for the combinations of N control points determine the same pair of motion vector offsets to apply to each pair of control point motion vectors for the N control points in the combination. The pair of motion vector offsets minimizes the bilateral matching cost value between the first affine prediction in the first reference picture and the second affine prediction in the second reference picture.

[0208] According to one aspect of this disclosure, a DMVR refinement can be applied to a subblock of a current block in affine motion interprediction mode. In some examples, a first motion vector is determined for a subblock of a current block coded in affine motion interprediction mode according to an affine model. A decoder-side motion vector refinement (DMVR) is determined for the subblock to determine a second motion vector using an improvement to the first motion vector. The second motion vector using the improvement to the first motion vector minimizes the bilateral matching cost between a first reference subblock in a first reference picture and a second reference subblock in a second reference picture. The subblock is then reconfigured according to the second motion vector for the subblock.

[0209] In some examples, the first motion vector is stored in a sub-block buffer for a sub-block, and the first motion vector in the sub-block buffer can be used for motion vector prediction for another block. In some examples, the second motion vector is stored in a sub-block buffer for a sub-block, and the second motion vector in the sub-block buffer can be used for motion vector prediction for another block.

[0210] In some examples, a refined motion vector is determined for each of the sub-blocks of the current block. A regression model is determined based on the refined motion vectors for the sub-blocks. Refined control point motion vectors of a refined affine model for the current block are derived according to the regression model. The current block is reconstructed according to the refined control point motion vectors.

[0211] In some examples, the control point motion vectors of an initial affine model (the initial affine model before multi-stage bilateral matching refinement) are stored in a buffer associated with the current block, and the control point motion vectors can be used for affine inheritance by other blocks.

[0212] In some examples, the refined control point motion vectors of a refined affine model (the refined affine model after multi-stage bilateral matching refinement) are stored in a buffer associated with the current block, and the refined control point motion vectors can be used for affine inheritance by other blocks.

[0213] Next, the process proceeds to (S1599) and ends.

[0214] The process (1500) can be appropriately adapted. The steps of the process (1500) may be modified and / or omitted. Further steps may be added. Any suitable implementation order may be used.

[0215] The above technology is implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, Figure 16 shows a computer system (1600) suitable for implementing a particular embodiment of the subject matter disclosed.

[0216] Computer software can be coded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms, and can contain instructions that can be executed directly or through interpretation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0217] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, game devices, and Internet of Things devices.

[0218] The components shown in Figure 16 for the computer system (1600) are essentially illustrative and are not intended to imply any limitations on the scope or functionality of computer software implementing embodiments of the present disclosure. Furthermore, the configuration of the components should not be construed as having any dependence or requirement on any one or combination of components shown in the exemplary embodiments of the computer system (1600).

[0219] The computer system (1600) may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users, for example, through haptic input (keystrokes, swipes, data glove movements, etc.), audio input (voices, applause, etc.), visual input (gestures, etc.), or olfactory input (not shown). The human interface input devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (voices, music, ambient sounds, etc.), images (scanned images, photographic images obtained from still image cameras, etc.), or video (2D video, 3D video including stereoscopic images, etc.).

[0220] Human interface input devices may include one or more of the following: keyboard (1601), mouse (1602), trackpad (1603), touch screen (1610), data glove (not shown), joystick (1605), microphone (1606), scanner (1607), and camera (1608) (only one of each is shown).

[0221] The computer system (1600) may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback via a touch screen (1610), data glove (not shown), or joystick (1605), although there may also be tactile feedback devices that do not function as input devices), audio output devices (e.g., speakers (1609), headphones (not shown)), visual output devices (screens (1610), including CRT screens, LCD screens, plasma screens, OLED screens, etc., regardless of whether each has touch screen input functionality or tactile feedback functionality, some of which can output two-dimensional visual output or output beyond three dimensions through means such as stereoscopic image output or virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0222] The computer system (1600) may also include optical media or similar media (1621) including CD / DVD ROM / RW (1620) having CD / DVD, thumb drives (1622), removable hard drives or solid-state drives (163), legacy magnetic media such as tapes and floppy disks (not shown), and special ROM / ASIC / PLD-based devices such as security dongles (not shown), as well as their associated media.

[0223] Those skilled in the art should also understand that, when used in connection with the subject matter now disclosed, the term “computer-readable medium” does not encompass a transmission medium, carrier wave, or other transient signal.

[0224] The computer system (1600) may also include an interface (1654) to one or more communication networks (1655). The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide-area, metropolitan, vehicle and industrial, real-time, latency-tolerant, etc. Examples of networks include local area networks such as Ethernet® and wireless LAN; cellular networks including GSM®, 3G, 4G, 5G, LTE, etc.; wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial TV; and vehicle and industrial networks including CANBus, etc. Certain networks generally require external network interface adapters (e.g., USB ports on the computer system (1600)) attached to specific general-purpose data ports or peripheral buses (1649), while others are generally integrated into the core of the computer system (1600) by attachment to system buses described later (e.g., an Ethernet® interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1600) can communicate with other entities. Such communication can be unidirectional and receivable only (e.g., broadcast television), unidirectional and receivable only (e.g., from a specific CANbus to a specific CANbus device), or bidirectional, for example, using a local or wide-area digital network, to other computer systems. As described above, specific protocols and protocol stacks can be used in each of these networks and network interfaces.

[0225] The above-mentioned human interface device, human-accessible storage device, and network interface can be mounted on the core (1640) of the computer system (1600).

[0226] The core (1640) may include one or more central processing units (CPUs) (1641), graphics processing units (GPUs) (1642), dedicated programmable processing units in the form of field-programmable gate arrays (FPGAs) (1643), hardware accelerators for specific tasks (1644), graphics adapters (1650), etc. These devices may be connected via a system bus (1648) along with read-only memory (ROM) (1645), random access memory (RAM) (1646), internal non-user-accessible hard drives, SSDs, and other internal mass storage (1647). In some computer systems, the system bus (1648) is accessible in the form of one or more physical plugs to allow for expansion with further CPUs, GPUs, etc. Peripheral devices can be attached directly to the core's system bus (1648) or via a peripheral bus (1649). For example, a screen (1610) can be connected to a graphics adapter (1650). The peripheral bus architecture includes PCI, USB, etc.

[0227] The CPU (1641), GPU (1642), FPGA (1643), and accelerator (1644) can execute specific instructions that, when combined, constitute the computer code described above. This computer code can be stored in ROM (1645) or RAM (1646). Temporary data can be stored in RAM (1646), while permanent data can be stored, for example, in internal mass storage (1647). High-speed storage and retrieval of any of the memory devices can be enabled through the use of cache memory that can be closely associated with one or more CPUs (1641), GPUs (1642), mass storage (1647), ROMs (1645), RAM (1646), etc.

[0228] A computer-readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of a type that is well known and available to those skilled in the computer software technology.

[0229] For example, and not an limitation, a computer system having the architecture (1600) and, in particular, the core (1640) can provide functionality as a result of a processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage as described above, as well as media associated with specific storage of the core (1640) of a non-transient nature, such as core internal mass storage (1647) or ROM (1645). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (1640). The computer-readable media can include one or more memory devices or chips according to specific needs. The software can cause the core (1640) and, in particular, the processor (including CPU, GPU, FPGA, etc.) within it to execute specific processes or specific parts of specific processes as described herein, including defining data structures to be stored in RAM (1646) and modifying such data structures according to processes defined by the software. Furthermore, or alternatively, a computer system may provide functionality as a result of being embodied in a circuit (e.g., an accelerator (1644)) in logic hardwired or otherwise, which circuit may, in place of or in conjunction with software, execute a particular process or a particular part of a particular process described herein. References to software include logic, and vice versa, as appropriate. References to computer-readable media may include circuits that store software for execution (e.g., integrated circuits (ICs)), circuits that embody logic for execution, or both, where appropriate. This disclosure encompasses any appropriate combination of hardware and software.

[0230] The use of “at least one” or “one” in this disclosure is intended to include any one or a combination of the elements described. For example, references to at least one of A, B, or C, at least one of A, B, and C, at least one of A, B, and / or C, and at least one of A through C are intended to include A only, B only, C only, or any combination thereof. References to one of A or B and one of A and B are intended to include A or B or (A and B). The use of “one” does not exclude any combination of the elements described, where applicable, such as when the elements are not mutually exclusive.

[0231] While this disclosure describes several exemplary embodiments, there are various modifications, substitutions, and equivalent alternatives that fall within the scope of this disclosure. Therefore, those skilled in the art will understand that various systems and methods, not expressly shown or described herein, embody the principles of this disclosure and thus fall within the spirit and scope of this disclosure.

Claims

1. A method for video decoding, The steps include: receiving coding information for the current block in the current picture, wherein the coding information indicates that the current block is coded in affine motion interprediction mode; A step of determining an initial affine model for the current block using biprediction, wherein the initial affine model predicts the current block based on a first reference picture and a second reference picture. A step of deriving an improved affine model by performing multi-stage bilateral matching motion improvement having at least a first stage and a second stage on the initial affine model, wherein the step of performing the multi-stage bilateral matching motion improvement is In one of the first and second stages, a step is to apply a first bilateral matching motion improvement to the translational component of the initial affine model, In the first and the other of the second steps, the step of applying a second bilateral matching motion improvement to the non-translational component of the initial affine model. Steps including, The steps include: Reconstructing the current block using the improved affine model based on the first reference picture and the second reference picture; Includes, The multi-stage bilateral matching motion improvement for the initial affine model includes at least a third stage of applying a third bilateral matching motion improvement to the control point motion vector at the control point of the current block, The third step involves applying a third bilateral matching motion improvement having one or more iterations for each combination of N control points, where N is a positive integer greater than 1 and less than the total number of control points, and the method is as follows: A method further comprising the step of determining a motion vector offset to apply to each of the control point motion vectors at the N control points in a combination that minimizes the bilateral matching cost value between a first affine prediction in the first reference picture and a second affine prediction in the second reference picture, in an iteration over a combination of N control points.

2. The step of performing the aforementioned multi-stage bilateral matching motion improvement is: In the first step preceding the second step, the steps include applying the first bilateral matching motion improvement to the translational component, In the second step following the first step, the second bilateral matching motion improvement is applied to the non-translational component. The method according to claim 1, further comprising:

3. The step of performing the aforementioned multi-stage bilateral matching motion improvement is: In the first step preceding the second step, the steps include applying the second bilateral matching motion improvement to the non-translational component, In the second step following the first step, the first bilateral matching motion improvement is applied to the translational component. The method according to claim 1, further comprising:

4. The step of performing the aforementioned multi-stage bilateral matching motion improvement is: The method according to claim 1, further comprising the step of determining a motion vector offset to be applied to each of the control point motion vectors of the initial affine model in order to minimize the bilateral matching cost value between the first affine prediction in the first reference picture and the second affine prediction in the second reference picture.

5. The step of performing the aforementioned multi-stage bilateral matching motion improvement is: For the improvement of the second bilateral matching motion, In a gradient-based improvement iteration related to the current affine model, the steps include determining the predicted output of the current block according to the current affine model, The steps include: calculating the gradient of the predicted sample in the prediction output; The steps include solving the linear form formed by the aforementioned gradient to determine the delta values ​​of the control point motion vectors, A step of determining whether one or more stopping conditions are met, A step of determining the next affine model based on the delta value of the control point motion vector in response to the failure of any of the one or more of the aforementioned stopping conditions being met, wherein the next affine model is input to the next gradient-based improvement iteration, The steps include stopping the second bilateral matching motion improvement when one or more of the aforementioned stopping conditions are met, and The method according to claim 1, further comprising:

6. The method according to claim 5, further comprising the step of determining the delta control point motion vector of each control point motion vector based on the last affine model in the last gradient-based improvement iteration of the second bilateral matching motion improvement, in response to the one or more stopping conditions being met.

7. A step of storing multiple bilateral matching costs associated with each affine model used in one of the multiple gradient-based improvement iterations of the second bilateral matching motion improvement, The steps include determining the lowest bilateral matching cost from the aforementioned multiple bilateral matching costs, The steps include determining the delta control point motion vector of each control point motion vector based on the affine model associated with the minimum bilateral matching cost, and The method according to claim 5, further comprising:

8. The method according to claim 1, wherein the improved affine model is formed by improved control point motion vectors, the improved control point motion vectors include a motion vector offset derived by the first bilateral matching motion improvement and applied to each of the improved control point motion vectors, and a delta control point motion vector derived by the second bilateral matching motion improvement and applied to each of the control point motion vectors.

9. The method according to claim 1, wherein the third step is performed before the first and second steps, or the third step is performed between the first and second steps, or the third step is performed after the first and second steps.

10. The step of performing the aforementioned multi-stage bilateral matching motion improvement is: A step of determining a first motion vector for a subblock of the current block coded in the affine motion interprediction mode according to the initial affine model, A step of applying a decoder-side motion vector improvement (DMVR) to determine a second motion vector for the subblock using the improvement to the first motion vector, wherein the second motion vector using the improvement to the first motion vector minimizes the bilateral matching cost between the first reference subblock in the first reference picture and the second reference subblock in the second reference picture. A step of reconfiguring the subblock according to the second motion vector for the subblock. The method according to claim 1, further comprising:

11. The method according to claim 10, further comprising the step of storing the first motion vector in a subblock buffer for the subblock, wherein the first motion vector in the subblock buffer is used for motion vector prediction for another block.

12. The method according to claim 10, further comprising the step of storing the second motion vector in a subblock buffer for the subblock, wherein the second motion vector in the subblock buffer is used for motion vector prediction for another block.

13. The steps include determining an improved motion vector for each of the subblocks of the current block, The steps include determining a regression model based on the improved motion vector for the subblock, The steps include: deriving the improved control point motion vector of the improved affine model of the current block according to the regression model; The steps include: Reconfiguring the current block according to the improved control point motion vector; The method according to claim 10, further comprising:

14. The method according to claim 13, further comprising the step of storing the control point motion vectors of the initial affine model in a buffer associated with the current block, wherein the control point motion vectors are used for affine inheritance.

15. The method according to claim 13, further comprising the step of storing the improved control point motion vector of the improved affine model in a buffer associated with the current block, wherein the improved control point motion vector is used for affine inheritance.

16. The method according to claim 1, further comprising the step of storing the control point motion vectors of the initial affine model in a buffer associated with the current block, wherein the control point motion vectors are used for affine inheritance.

17. A device for video decoding, including a processing circuit, An apparatus wherein the processing circuit is configured to perform the method according to any one of claims 1 to 16.

18. A computer program that causes a computer to perform the method described in any one of claims 1 to 16.

19. A video encoding method, The step of determining that the current block in the current picture is coded in affine motion interpredictive mode, A step of determining an initial affine model for the current block using biprediction, wherein the initial affine model predicts the current block based on a first reference picture and a second reference picture. A step of determining an improved affine model by performing multi-stage bilateral matching motion improvement having at least a first stage and a second stage on the initial affine model, wherein the step of performing multi-stage bilateral matching motion improvement is In one of the first and second stages, a step is to apply a first bilateral matching motion improvement to the translational component of the initial affine model, In the first and the other of the second steps, the step of applying a second bilateral matching motion improvement to the non-translational component of the initial affine model. Steps including, The steps include: encoding the current block using the improved affine model based on the first reference picture and the second reference picture; The steps include: encoding coding information into video data indicating that the current block is coded in affine motion interprediction mode; Includes, The multi-stage bilateral matching motion improvement for the initial affine model includes at least a third stage of applying a third bilateral matching motion improvement to the control point motion vector at the control point of the current block, The third step involves applying a third bilateral matching motion improvement having one or more iterations for each combination of N control points, where N is a positive integer greater than 1 and less than the total number of control points, and the method is as follows: A method further comprising the step of determining a motion vector offset to apply to each of the control point motion vectors at the N control points in a combination that minimizes the bilateral matching cost value between a first affine prediction in the first reference picture and a second affine prediction in the second reference picture, in an iteration over a combination of N control points.

20. A method for video encoding, The step of determining that the current block in the current picture is coded in affine motion interpredictive mode, A step of determining an initial affine model for the current block using biprediction, wherein the initial affine model predicts the current block based on a first reference picture and a second reference picture. A step of determining an improved affine model by performing multi-stage bilateral matching motion improvement having at least a first stage and a second stage on the initial affine model, wherein the step of performing multi-stage bilateral matching motion improvement is In one of the first and second stages, a step is to apply a first bilateral matching motion improvement to the translational component of the initial affine model, In the first and the other of the second steps, the step of applying a second bilateral matching motion improvement to the non-translational component of the initial affine model. Steps including, The steps include: encoding the current block using the improved affine model based on the first reference picture and the second reference picture; The steps include encoding coding information into a video bitstream that indicates the current block is coded in affine motion interprediction mode, The steps of transmitting the video bitstream and Includes, The multi-stage bilateral matching motion improvement for the initial affine model includes at least a third stage of applying a third bilateral matching motion improvement to the control point motion vector at the control point of the current block, The third step involves applying a third bilateral matching motion improvement having one or more iterations for each combination of N control points, where N is a positive integer greater than 1 and less than the total number of control points, and the method is as follows: A method further comprising the step of determining a motion vector offset to apply to each of the control point motion vectors at the N control points in a combination that minimizes the bilateral matching cost value between a first affine prediction in the first reference picture and a second affine prediction in the second reference picture, in an iteration over a combination of N control points.

Images