Method for Affine Motion Refinement

Abstract

The disclosed aspects provide methods and apparatus for video encoding / decoding. In some examples, the apparatus for video decoding includes a processing circuit. The processing circuit receives a coded video bitstream including a current picture. The current picture includes a current block. The processing circuit determines, based on decoded information from the coded video bitstream, that the current block in the current picture is coded in an affine dual prediction mode. The current block is predicted by a plurality of control point motion vectors. The processing circuit refines the plurality of control point motion vectors by applying bilateral matching motion refinement and reconstructs the current block according to the refined plurality of control point motion vectors associated with the current block.

Description

Technical Field

[0001] [Related Applications] This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 390,112, "Method for Affine Motion Refinement," filed on July 18, 2022, and U.S. Patent Application No. 17 / 983,376, "METHOD FOR AFFINE MOTION REFINEMENT," filed on November 8, 2022. The disclosures of the foregoing applications are hereby incorporated herein by reference in their entirety.

[0002] [Technical Field] This disclosure generally describes embodiments related to video coding.

Background Art

[0003] The background description provided herein is for the purpose of generally presenting the background of the present disclosure. The research of the presently named inventors, to the extent it is not described in this background section, is not admitted to be prior art to the present disclosure, whether explicitly or implicitly, in the same manner as aspects of the description that may not be considered prior art at the time of filing.

[0004] Uncompressed digital images and / or videos can include a series of pictures, each picture having spatial dimensions of, for example, 1920×1080 luminance samples and associated chrominance samples. A series of pictures can have, for example, a fixed or variable picture rate of 60 pictures per second or 60 Hz (also known as the frame rate in short form). Uncompressed images and / or videos have specific bitrate requirements. For example, an 8-bit / sample 1080p60 4:2:0 video (1920×1080 luminance sample resolution at a 60 Hz frame rate) requires a bandwidth close to 1.5 Gbit / s. One hour of such a video requires more than 600 GByte of storage space.

[0005] One purpose of image and / or video coding and decoding can be to reduce redundancy in an input image and / or video signal through compression. Compression can, in some cases, help reduce the bandwidth and / or storage space requirements by more than an order of magnitude. The description herein uses video encoding / decoding as an example for illustration, but the same techniques can be applied to image encoding / decoding in a similar manner without departing from the spirit of the disclosure. Both lossless compression and lossy compression, and combinations thereof, can be utilized. Lossless compression represents techniques where an exact copy of the original signal can be reconstructed from the compressed original signal. When using lossy compression, the reconstructed signal is not identical to the original signal, but the distortion between the original signal and the reconstructed signal is small enough to produce a useful reconstructed signal for the intended application. In the case of video, lossy compression is widely utilized. The amount of tolerable distortion depends on the application, and users of a particular consumer streaming application may be able to tolerate higher distortion than users of a television distribution application. The achievable compression ratio can reflect that the higher the acceptable / tolerable distortion, the higher the achievable compression ratio.

[0006] Video encoders and decoders can utilize techniques from several broad classifications, including, for example, motion compensation, transform processing, quantization, and entropy coding.

[0007] Video codec technology can include techniques known as intracoding. In intracoding, sample values are represented without reference to samples from previously reconstructed reference pictures or other data. In some video codecs, a picture is spatially subdivided into blocks of samples. When all blocks of samples are coded in an intra mode, that picture can be an intra picture. Intra pictures, and their derivatives such as independent decoder refresh pictures, can be used to reset the decoder state and thus can be used as the first picture in a coded video bitstream and video session or as a still image. Samples of an intra block can be transformed, and the transform coefficients can be quantized prior to entropy coding. Intra prediction can be a technique that minimizes sample values in the domain before transformation. In some cases, the smaller the DC value after transformation and the smaller the AC coefficients, the fewer bits are required at a given quantization step size to represent the block after entropy coding.

[0008] For example, traditional intracoding used in MPEG-2 generation coding technology does not use intra prediction. However, some new video compression technologies attempt to perform prediction based on, for example, surrounding sample data and / or metadata obtained during the encoding and / or decoding of data blocks. Such techniques are hereinafter referred to as "intra prediction" techniques. In at least some cases, intra prediction uses only reference data from the current picture being reconstructed rather than from a reference picture.

[0009] There can be many different forms of intra prediction. When more than one such technique can be used in a given video coding technique, the particular technique used can be coded as a particular intra prediction mode that uses the particular technique. In a particular case, an intra prediction mode can have sub-modes and / or parameters, and the sub-modes and / or parameters can be coded either individually or included in a mode codeword that defines the prediction mode being used. Which codeword should be used for a given combination of mode, sub-mode, and / or parameter can affect the improvement of coding efficiency through intra prediction, and thus entropy coding techniques can be used to convert the codeword into a bitstream.

[0010] A particular intra prediction mode was introduced by H.264, improved in H.265, and further improved in more recent coding techniques such as the joint exploration model (JEM), versatile video coding (VVC), and benchmark set (BMS). A prediction block can be formed using neighboring sample values of already available samples. The sample values of the neighboring samples are copied into the prediction block according to a direction. The reference to the direction in use can be coded within the bitstream or can itself be predicted.

[0011] Referring to FIG. 1A, a subset of nine prediction directions can be seen shown in the lower right, corresponding to 33 of the 35 intra modes defined in H.265. The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction in which the sample is being predicted. For example, arrow (102) indicates that sample (101) is predicted from one or more samples at an angle of 45 degrees from horizontal and upward to the right. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples at an angle of 22.5 degrees from horizontal and downward to the left of sample (101).

[0012] Referring further to FIG. 1A, in the upper left, a square block (104) of 4×4 samples (shown by the thick dashed line) is shown. The square block (104) contains 16 samples, and each sample is labeled with "S", its Y - dimensional position (e.g., row index), and its X - dimensional position (e.g., column index). For example, sample S21 is the second sample from the top in the Y - dimension and the first sample from the left in the X - dimension. Similarly, sample S44 is the fourth sample within block (104) in both the Y and X dimensions. When the block is of size 4×4 samples, S44 is in the lower right. Further, reference samples following a similar numbering scheme are shown. The reference samples are labeled with "R", its Y - position (e.g., row index) and X - position (column index) with respect to block (104). In both H.264 and H.265, the predicted samples are in the neighborhood of the block being reconstructed, and thus negative values need not be used.

[0013] Intra picture prediction can operate by copying the reference sample value from neighboring samples when indicated by the signaled prediction direction. For example, a coded video bitstream includes signaling indicating a prediction direction that coincides with arrow (102) for this block. That is, the sample is predicted at an angle of 45 degrees from horizontal and upward to the right. In this case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Sample S44 is then predicted from reference sample R08.

[0014] In certain cases, in order to calculate the reference sample, when the direction cannot be evenly divided by 45 degrees in particular, the values of multiple reference samples may be combined, for example, through interpolation.

[0015] The number of possible directions has been increasing as video coding technology has evolved. In H.264 (2003), there were possibilities of presenting 9 different directions. That increased to 33 in H.265 (2013). Currently, JEM / VVC / BMS can support up to 65 directions. Experiments are conducted to identify the most likely directions, and specific techniques in entropy coding are used to represent these likely directions with a small number of bits while accepting a specific penalty for less likely directions. Further, the direction itself may be predicted from neighboring directions in neighboring already decoded blocks.

[0016] FIG. 1B shows a diagram (110) showing 65 intra prediction directions according to JEM to illustrate the increase in the number of prediction directions over time.

[0017] The mapping of intra prediction direction bits representing directions within a coded video bitstream may vary depending on the video coding technology. Such mapping can range from, for example, a simple direct mapping to complex adaptive schemes including codewords, most likely modes, and similar techniques. However, in most cases, in video content, there may exist certain directions that are not statistically likely to occur much higher than certain other directions. Since the goal of video compression is redundancy reduction, these less likely directions will be represented by a larger number of bits than the more likely directions in well - operating video coding technologies.

[0018] Image and / or video coding and decoding can be performed using inter - picture prediction with motion compensation. Motion compensation is a lossy compression technique where a block of sample data from a previously reconstructed picture or a portion thereof (reference picture) is spatially shifted in the direction indicated by a motion vector (hereinafter, MV) and then used for the prediction of a newly reconstructed picture or picture portion. In some cases, the reference picture can be the same as the picture currently being reconstructed. The MV can have two dimensions X and Y, or three dimensions where the third dimension is an indication of the reference picture in use (the latter can be indirectly the temporal dimension).

[0019] In some video compression techniques, the motion vectors (MVs) applicable to a particular region of sample data can be predicted from other MVs, for example, from MVs related to another region of sample data that is spatially adjacent to the region being reconstructed and that precedes the MV in decoding order. Doing so can, as a result, reduce the amount of data required to code the MVs, thereby removing redundancy and improving compression. MV prediction can, for example, be statistically possible when coding an input video signal obtained from a camera (known as natural video) because regions larger than the region to which a single MV is applicable move in a similar direction and thus, in some cases, can be predicted using similar motion vectors derived from the MVs of neighboring regions. This results in an MV found for a given region that is similar or identical to the MV predicted from surrounding MVs. Also, this can be presented in fewer bits than would be used if the MV were directly coded after entropy coding. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., an MV) obtained from the original signal (i.e., the sample stream). In other cases, MV prediction itself can be lossy, for example, when rounding errors occur when calculating predictors from some surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). From among the many MV prediction mechanisms provided by H.265, the technique called "spatial merge" will be described below with reference to FIG. 2.

[0021] Referring to FIG. 2, the current block (201) includes samples found by the encoder as being predictable from a previous block of the same size that has been spatially shifted during motion search processing. Instead of directly coding the MV, the MV can be derived from metadata associated with one or more reference pictures, e.g., from the nearest reference picture (in decoding order), using an MV associated with any one of five surrounding samples A0, A1, and B0, B1, B2 (202 - 206 respectively). In H.265, MV prediction can use a predictor from the same reference picture that neighboring blocks are using. SUMMARY OF THE INVENTION

[0022] Disclosed aspects provide a method and apparatus for video encoding / decoding. In some examples, an apparatus for video decoding includes a processing circuit. The processing circuit receives a coded video bitstream including a current picture. The current picture includes a current block. The processing circuit determines, based on decoded information from the coded video bitstream, that the current block within the current picture is coded in an affine dual prediction mode. The current block is predicted by a plurality of control point motion vectors. The processing circuit refines the plurality of control point motion vectors by applying bilateral matching motion refinement and reconstructs the current block according to the refined plurality of control point motion vectors associated with the current block.

[0023] In some examples, the processing circuit determines a first set of motion vector refinement offset values for a first plurality of control point motion vectors of a current block associated with a first reference picture. The first plurality of motion vector refinement offset values are not the same value. Further, the processing circuit determines a second set of motion vector refinement offset values for a second plurality of control point motion vectors of a second reference picture according to the first set of motion vector refinement offset values. The second set of motion vector refinement offset values has a sign opposite to that of the first set of motion vector refinement offset values. In one example, the first set of motion vector refinement offset values includes a value of 0 for application to a first control point motion vector of a first control point at the upper right corner of the current block.

[0024] In some examples, the processing circuit determines a first set of affine parameter refinement offset values for first affine parameters of a first affine model of a current block associated with a first reference picture. Further, in one example, the processing circuit generates a second set of affine parameter refinement offset values for second affine parameters of a second affine model of the current block associated with a second reference picture according to the first set of affine parameter refinement offset values, and the second set of affine parameter refinement offset values has a sign opposite to that of the first set of affine parameter refinement offset values. The first affine model and the second affine model are four-parameter affine models or six-parameter affine models.

[0025] In one example, the motion vectors of the plurality of control points are based on at least one of an affine merge candidate and an affine merge motion vector difference (MMVD) candidate.

[0026] In some examples, the processing circuit determines a first refinement offset value for applying a first refinement to a first parameter. The first refinement updates a plurality of control point motion vectors to a first intermediate plurality of refined control point motion vectors that minimize a bilateral matching cost based on the first refinement to the first parameter. The processing circuit determines a second refinement offset value for applying a second refinement to a second parameter. The second refinement updates the first intermediate plurality of refined control point motion vectors to a second intermediate plurality of refined control point motion vectors that minimize a bilateral matching cost based on the second refinement to the second parameter. In one example, the first parameter and the second parameter for refinement correspond to a first control point motion vector of a first control point and a second control point motion vector of a second control point.

[0027] In another example, the first parameter and the second parameter for refinement correspond to a first affine parameter and a second affine parameter of an affine model.

[0028] In one example, the processing circuit determines that a decrease in the bilateral matching cost based on the first refinement is less than a threshold and skips determining a second refinement offset value for applying the second refinement to the second parameter. The processing circuit reconstructs the current block based on the first intermediate plurality of refined control point motion vectors.

[0029] In some examples, in response to refinement of a plurality of control point motion vectors, the processing circuit applies affine motion compensation to sub-blocks of the current block and determines a first reference sub-block in a first reference picture and a second reference sub-block in a second reference picture corresponding to the sub-blocks of the current block. The processing circuit calculates a bilateral matching cost based on a distortion between the first reference sub-block and the second reference sub-block.

[0030] In some examples, the processing circuit, in response to the refinement of a plurality of control point motion vectors, applies affine motion compensation to a subset of sub-blocks of a current block, and determines a first subset of reference sub-blocks in a first reference picture corresponding to the subset of sub-blocks of the current block and a second subset of reference sub-blocks in a second reference picture. The processing circuit calculates a bilateral matching cost based on the distortion between the first subset of reference sub-blocks and the second subset of reference sub-blocks.

[0031] In one example, the subset of sub-blocks within the current block includes the sub-block at the center position of the current block.

[0032] In another example, the subset of sub-blocks within the current block includes one or more sub-blocks at one or more corners of the current block.

[0033] In another example, the subset of sub-blocks within the current block includes sub-blocks of a checkerboard pattern within the current block.

[0034] In another example, the subset of sub-blocks within the current block includes sub-sampled affine sub-blocks.

[0035] In another example, the subset of sub-blocks within the current block includes N×N affine sub-blocks (e.g., the affine sub-blocks include 4×4 luminance samples), the center of the N×N affine sub-blocks coincides with the center of the current block, and N is a positive integer.

[0036] In some examples, a first affine motion compensation sub-block size is used during bilateral matching motion refinement processing. The current block is reconstructed using a second affine motion compensation sub-block size. The first affine motion compensation sub-block size is larger than the second affine motion compensation sub-block size.

[0037] In some examples, a simple interpolation filter, such as a bilinear interpolation filter, is used during bilateral matching motion refinement processing.

[0038] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding, cause the computer to perform a method for video decoding.

Brief Description of the Drawings

[0039] Further features, characteristics, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

[0040]

Figure 1A

[0041]

Figure 1B

[0042]

Figure 2

[0043]

Figure 3

[0044]

Figure 4

[0045]

Figure 5

[0046]

Figure 6

[0047]

Figure 7

[0048]

Figure 8

[0049]

Figure 9A

[0050]

Figure 9B

[0051]

Figure 10

[0052]

Figure 11

[0053]

Figure 12

[0054]

Figure 13

[0055]

Figure 14

[0056]

Figure 15

[0057]

Figure 16

[0058]

Figure 17

[0059]

Figure 18

[0060]

Figure 19

[0061]

Figure 20

[0062]

Figure 21

[0063]

Figure 22

[0064]

Figure 23

[0065]

Figure 24

[0066]

Figure 25

[0067]

Figure 26

[0068]

Figure 27

[0069]

Figure 28A

Figure 28B

[0070]

Figure 29

[0071]

Figure 30

[0072]

Figure 31

[0073]

Figure 32

[0074]

Figure 33

[0075]

Figure 34

[0076]

Figure 35

[0077]

Figure 36

[0078]

Figure 37

[0079]

Figure 38

[0080]

Figure 39

[0081]

Figure 40

Mode for Carrying Out the Invention

[0082] FIG. 3 shows an exemplary block diagram of a communication system (300). The communication system (300) includes a plurality of terminal devices that can communicate with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via a network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) perform unidirectional data transmission. For example, the terminal device (310) can code video data (a stream of video pictures captured by the terminal device (310)) for transmission to another terminal device (320) via the network (350). The coded video data can be transmitted in the form of one or more coded video bitstreams. The terminal device (320) can receive the coded video data from the network (350), decode the coded video data to restore the video pictures, and display the video pictures according to the restored video data. Unidirectional data transmission may be common in media serving applications and the like.

[0083] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of coded video data, for example, during a video conference. In bidirectional data transmission, the terminal devices (330) and (340) can code video data (e.g., a stream of video pictures captured by the terminal device) for transmission to the terminal devices (330) and (340) via the network (350). Each of the terminal devices (330) and (340) can receive the coded video data transmitted by the other of the terminal devices (330) and (340), decode the coded video data to restore the video pictures, and display the video pictures on an accessible display device according to the restored video data.

[0084] In the example of FIG. 3, the terminal devices (310), (320), (330), and (340) are each shown as a server, a personal computer, and a smartphone, but the principles of the present disclosure are not limited thereto. Embodiments of the present disclosure have applications in laptop computers, tablet computers, media players, and / or dedicated video conferencing facilities. The network (350) represents any number of networks that carry the coded video data among the terminal devices (310), (320), (330), and (340), and includes, for example, wired (wired) and / or wireless communication networks. The communication network (350) may exchange data over circuit-switched and / or packet-switched channels. Representative networks include electronic communication networks, local area networks, wide area networks, and / or the Internet. For the purposes of the discussion of the present invention, the architecture and topology of the network (350) may not be important for the operation of the present disclosure, unless otherwise specifically noted hereinafter.

[0085] FIG. 4 shows a video encoder and a video decoder in a streaming environment as an example of the application of the disclosed subject matter. The disclosed subject matter is equally applicable to, for example, video conferencing, digital TV, streaming services, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc., and other video-enabled applications, etc.

[0086] A streaming system may include a capture subsystem (413) that can include, for example, a video source (401) that generates an uncompressed video picture stream (402). In one example, the video picture stream (402) includes samples captured by a digital camera. The video picture stream (402) is shown in bold lines to emphasize its high data capacity when compared to the encoded video data (404) (or coded video bitstream), and can be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) includes hardware, software, or a combination thereof and can enable or implement aspects of the disclosed subject matter as detailed below. The encoded video data (404) (or encoded video bitstream) is shown in thin lines to emphasize its low data capacity when compared to the video picture stream (402) and can be stored in a streaming server (405) for future use. One or more streaming client subsystems, such as client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to read copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include, for example, a video decoder (410) within an electronic device (430). The video decoder (410) decodes an input copy (407) of the encoded video data and generates an output video picture stream (411) that can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., video bitstreams) can be encoded according to a particular video coding / compression standard. Examples of these standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is known as VVC (Versatile Video Coding) in short. The disclosed subject matter may be used in the context of VVC.

[0087] Note that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) can include a video decoder (not shown), and the electronic device (430) can also include a video encoder (not shown).

[0088] FIG. 5 shows an exemplary block diagram of a video decoder (510). The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). In the example of FIG. 4, the video decoder (510) can be used instead of the video decoder (410).

[0089] The receiver (531) can receive one or more coded video sequences to be decoded by the video decoder (510). In an embodiment, one coded video sequence is received at a time, and the decoding of each coded video sequence is independent of the decoding of other coded video sequences. The coded video sequence may be received from a channel (501) which may be a hardware / software link to a storage device storing the encoded video data. The receiver (531) may receive the encoded video data along with other data, such as coded audio data and / or auxiliary data streams that may be transferred to respective usage entities (not shown). The receiver (531) may separate the coded video sequence from the other data. To remove network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter, “parser (520)”). In certain applications, the buffer memory (515) is part of the video decoder (510). Alternatively, it may be external to the video decoder (510) (not shown). Still alternatively, for example, in addition to a buffer memory (515) that may be external to the video decoder (510) to remove network jitter or internal to the video decoder (510) to process playout timing, another buffer memory (not shown) may exist. When the receiver (531) is receiving data controllably from a storage / transfer device with sufficient bandwidth or from an isosynchronous network, the buffer memory (515) may not be necessary or can be small. For use in a best-effort packet network such as the Internet, the buffer memory (515) may be required, may be relatively large, and advantageously be of an adaptable size and at least partially implemented in an operating system or similar elements external to the video decoder (510) (not shown).

[0090] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the coded video sequence. The categories of these symbols include information used to manage the operation of the video decoder (510) and, in some cases, information for controlling a rendering device (512) (e.g., a display screen) that is not an integrated part of the electronic device (530) but can be coupled to the electronic device (530) as shown in FIG. 5. The control information for the rendering device may be in the form of an SEI (Supplemental Enhancement Information) message or a VUI (Video Usability Information) parameter set fragment (not shown). The parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence can follow a video coding technology or standard and can follow various principles including variable length coding, Huffman coding, arithmetic coding with or without context dependence, etc. The parser (520) may extract a set of subgroup parameters from the coded video sequence based on at least one parameter corresponding to at least one subgroup of pixels in the video decoder for at least one of the subgroups. The subgroups may include GOP (Groups of Picture), picture, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), prediction unit (PU), etc. The parser (520) may also extract information such as transform coefficients, quantization parameter values, motion vectors, etc. from the coded video sequence.

[0091] The parser (520) may perform an entropy decoding / parsing operation on the video sequence received from the buffer memory (515) to generate symbols (521).

[0092] The reconstruction of symbol (521) may include a plurality of different units depending on the type of the coded video picture or a portion thereof (e.g., inter and intra pictures, inter and intra blocks) and other factors. How each unit is included can be controlled by group control information parsed by a parser (520) from the coded video sequence. Such a flow of subgroup control information between the parser (520) and the following plurality of units is not shown for clarity.

[0093] Beyond the function blocks already mentioned, the video decoder (510) can be conceptually subdivided into a number of functional units as described below. In an actual implementation operating under commercial constraints, many of these units can interact closely with each other and can be at least partially integrated with each other. However, for the purpose of explaining the disclosed subject matter, the following conceptual subdivision into functional units is appropriate.

[0094] The first unit is a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives, as symbol (521) from the parser (520), quantized transform coefficients and control information including which transform to use, block size, quantization coefficients, quantization scaling matrix, etc. The scaler / inverse transform unit (551) can output a block including sample values that can be input to an aggregator (555).

[0095] In some cases, the output samples of the scaler / inverse transform unit (551) may be related to intra-coded blocks. An intra-coded block is a block that does not use prediction information from a previously reconstructed picture but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information can be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates a block of the same size and shape as the block being reconstructed, using the surrounding already-reconstructed information fetched from the current picture buffer (558). The current picture buffer (558) buffers, for example, the partially reconstructed current picture and / or the fully reconstructed current picture. The aggregator (555) adds, in some cases, for each sample, the prediction information generated by the intra prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551).

[0096] In other cases, the output samples of the scaler / inverse transform unit (551) may be related to inter-coded, and possibly motion-compensated, blocks. In such cases, the motion compensation prediction unit (553) can access the reference picture memory (557) to fetch the samples used for prediction. After motion-compensating the samples fetched according to the symbol (521) associated with the block, these samples can be added by the aggregator (555) to the output of the scaler / inverse transform unit (551) to generate the output sample information (in this case, called the residual samples or the residual signal). The address in the reference picture memory (557) from which the motion compensation prediction unit (553) fetches the prediction samples can be controlled by the available motion vectors of the motion compensation prediction unit (553) in the form of a symbol (521) that may have, for example, X, Y, and reference picture components. Motion compensation can include interpolation of the sample values fetched from the reference picture memory (557) when an exact sub-sample motion vector is in use, a motion vector prediction mechanism, etc.

[0097] The output samples of the aggregator (555) can undergo various loop filtering techniques in the loop filter unit (556). Video compression techniques can include in-loop filter techniques that are controlled by parameters included in the coded video sequence (also referred to as the coded video bitstream), enabling the loop filter unit (556) to use the symbols (521) from the parser (520). Video compression can respond not only to previously reconstructed and loop-filtered sample values but also to meta information obtained during the decoding of the previous part (in decoding order) of the coded picture or coded video sequence.

[0098] The output of the loop filter unit (556) can be a sample stream that can be output to the renderer device (512) and stored in the reference picture memory (557) for use in future inter-picture prediction.

[0099] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. For example, when the coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and a fresh current picture buffer can be reallocated before starting the reconstruction of subsequent coded pictures.

[0100] The video decoder (510) may perform a decoding operation according to a standard such as ITU-T Rec.H.265 or a predetermined video compression technique. In the sense that the coded video sequence conforms to both the video compression technique or standard and the profile documented in the video compression technique or standard, the coded video sequence may conform to the syntax specified by the video compression technique or standard in use. Specifically, the profile can select specific tools from all the tools available in the video compression technique or standard as tools that can only be used under the profile. Also, what is necessary for compliance may be that the complexity of the coded video sequence is within the limits determined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, the maximum frame rate, the maximum reconstruction sample rate (measured, for example, in megasamples per second), the maximum reference picture size, etc. The limits set by the level may, in some cases, be further restricted through the HRD (Hypothetical Reference Decoder) specification and the metadata for HRD buffer management signaled in the coded video sequence.

[0101] In an embodiment, the receiver (531) may receive additional (redundant) data together with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (510) to correctly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0102] FIG. 6 shows an exemplary block diagram of a video encoder (603). The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used in place of the video encoder (403) in the example of FIG. 4.

[0103] Video encoder (603) may receive video samples from a video source (601) (not part of the electronic device (620) in the example of FIG. 6) that can capture a video image to be coded by the video encoder (603). In another example, the video source (601) is part of the electronic device (620).

[0104] Video source (601) may provide a source video sequence to be coded by video encoder (603) in the form of a digital video sample stream of any suitable bit depth (e.g., 8 bits, 10 bits, 12 bits,...), any color space (e.g., BT.601 Y CrCB, RGB,...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media providing system, video source (601) may be a storage device storing previously prepared video. In a video conferencing system, video source (601) may be a camera that captures local image information as a video sequence. Video data may be provided as a plurality of individual pictures that give motion when viewed in succession. Each picture itself may be organized as a spatial array of pixels. Each pixel may contain one or more samples depending on the sampling structure, color space, etc. in use. A person skilled in the art can immediately understand the relationship between pixels and samples. The following description focuses on samples.

[0105] According to an embodiment, the video encoder (603) may code and compress pictures of a source video sequence into a coded video sequence (643) in real time or under any other time constraints required. Implementing an appropriate coding speed is one function of the control unit (650). In some embodiments, the control unit (650) controls other functional units described below and is functionally coupled to the other functional units. The coupling is not shown for clarity. The parameters set by the control unit (650) may include rate control related parameters (picture skip, quantizer, lambda value of rate distortion optimization technique,...), picture size, GOP (group of pictures) layout, maximum motion vector search range, etc. The control unit (650) may be configured to have other appropriate functions related to the video encoder (603) optimized for a specific system design.

[0106] In some embodiments, the video encoder (603) is configured to operate within a coding loop. As a very simplified explanation, in one example, the coding loop may include a source coder (630) (which is responsible for generating symbols such as a symbol stream based on the input picture and reference pictures to be coded), and a (local) decoder (633) built into the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in the same way as a (remote) decoder does. The reconstructed sample stream (sample data) is input into the reference picture memory (634). When the decoding of the symbol stream results in a bit-exact result independent of the decoder location (local or remote), the content of the reference picture memory (634) is also bit-exact between the local encoder and the remote encoder. In other words, the prediction part of the encoder "sees" the exact same sample values as the decoder "sees" when using prediction during decoding, as reference picture samples. This basic principle of reference picture simultaneity (and the resulting drift if simultaneity cannot be maintained, for example due to channel errors) is used similarly in some related technologies.

[0107] The operation of the "local" decoder (633) may be the same as that of a "remote" decoder such as the video decoder (510) detailed above in relation to FIG. 5. However, referring briefly to FIG. 5 as well, since symbols are available and the encoding / decoding of symbols into the coded video sequence by the entropy coder (645) and the parser (520) can be lossless, the entropy decoding part of the video decoder (510) including the buffer memory (515) and the parser (520) may not be fully implemented in the local decoder (633).

[0108] In an embodiment, decoder techniques other than parsing / entropy decoding present in a decoder exist in a corresponding encoder in the same or substantially the same functional form. Accordingly, the described subject matter focuses on decoder operations. The description of encoder techniques can be omitted since they are the reverse of the decoder techniques that are comprehensively described. In certain areas, more detailed descriptions are provided below.

[0109] During operation, in some examples, the source coder (630) may perform motion-compensated predictive coding. This predictively codes an input picture by referring to one or more previous coded pictures from a video sequence designated as a "reference picture". In this method, the coding engine (632) codes the difference between a pixel block of the input picture and a pixel block of a reference picture that may be selected as a prediction reference for the input picture.

[0110] The local video decoder (633) may decode the coded video data of a picture that may be designated as a reference picture based on the symbols generated by the source coder (630). The operation of the coding engine (632) may advantageously be a lossy process. When the coded video data can be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may typically be a replica of the source video sequence with some errors. The local video decoder (633) may replicate the decoding process that may be performed by the video decoder for the reference picture, resulting in a reconstructed reference picture to be stored in the reference picture memory (634). Thus, the video encoder (603) may store a copy of the reconstructed reference picture having the same content as the reconstructed reference picture obtained by the remote video decoder (if there are no transmission errors).

[0111] Predictor (635) may perform predictive search for the coding engine (632). That is, for a new picture to be coded, predictor (635) may search the reference picture memory (634) for sample data (such as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc. that may function as appropriate prediction criteria for the new picture. Predictor (635) may operate on a sample block - pixel block basis to find appropriate prediction criteria. In some examples, the input picture may have prediction criteria drawn from a plurality of reference pictures stored in the reference picture memory (634) as determined by the search results obtained by predictor (635).

[0112] Control unit (650) may manage the coding operations of source coder (630), including for example the setting of parameters and subgroup parameters used for the coding of video data.

[0113] The outputs of all the aforementioned functional units may undergo entropy coding in entropy coder (645). Entropy coder (645) converts the symbols generated by the various functional units into a coded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable - length coding, arithmetic coding, etc.

[0114] Transmitter (640) may buffer the coded video sequence generated by entropy coder (645) for transmission via a communication channel (660) which may be a hardware / software link to a storage device that may store the coded video data. Transmitter (640) may merge the coded video data from video encoder (603) with other data to be transmitted, such as coded audio data and / or an auxiliary data stream (source not shown).

[0115] The control unit (650) may manage the operation of the video encoder (603). During coding, the control unit (650) may assign to each coded picture a type of the coded picture that may affect the coding technique applicable to each picture. For example, a picture may often be assigned as one of the following picture types.

[0116] An intra picture (I picture) may be a picture that can be coded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, for example, IDR (Independent Decoder Refresh) pictures. Those skilled in the art recognize the variations of I pictures and their individual applications and characteristics.

[0117] A predictive picture (P picture) may be a picture that can be coded and decoded using intra prediction or inter prediction, typically using one motion vector and a reference index to predict the sample values of each block.

[0118] A bi-directionally predictive picture (B picture, Bi-directionally Predictive Picture (B Picture)) may be a picture that can be coded and decoded using intra prediction or inter prediction, typically using up to two motion vectors and reference indexes to predict the sample values of each block. Similarly, a multi-predictive picture can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0119] The source picture may generally be spatially subdivided into a plurality of sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples each) and coded block by block. The blocks may be coded predictively by reference to other (already coded) blocks determined by the coding assignment applied to each picture of the block. For example, blocks of an I picture may be coded non-predictively, or they may be coded predictively by reference to already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of a P picture may be coded predictively via spatial prediction or via temporal prediction by reference to one previously coded reference picture. Blocks of a B picture may be coded predictively via spatial prediction or via temporal prediction by reference to one or two previously coded reference pictures.

[0120] The video encoder (603) may perform coding operations in accordance with a predetermined video coding technology or standard such as ITU-T Rec.H.265. In that operation, the video encoder (603) may perform various compression operations including predictive coding operations that utilize the temporal and spatial redundancy in the input video sequence. The coded video data may thus conform to the syntax specified by the video coding technology or standard being used.

[0121] In one embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include such data as part of the coded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0122] Video may be captured as a plurality of source pictures (video pictures) in a time series. Intra-picture prediction (which may be abbreviated as intra-prediction) utilizes the spatial correlation within a given picture, and inter-picture prediction utilizes the (temporal or other) correlation between pictures. In one example, a particular picture during encoding / decoding is referred to as the current picture and is partitioned into blocks. When a block in the current picture is similar to a reference block in a reference picture that has been previously coded and is still buffered in the video, the block in the current picture can be coded by a vector called a motion vector. The motion vector points to the reference block within the reference picture and can have a third dimension that identifies the reference picture when multiple reference pictures are in use.

[0123] In some embodiments, bi-prediction techniques can be used in inter-picture prediction. According to the bi-prediction technique, two reference pictures such as a first reference picture and a second reference picture that are both before the current picture in the video in decoding order (but may be in the past and future in display order respectively) are used. A block within the current picture can be coded by a first motion vector pointing to a first reference block within the first reference picture and a second motion vector pointing to a second reference block within the second reference picture. The block can be predicted by combining the first reference block and the second reference block.

[0124] Furthermore, in order to improve coding efficiency, merge mode techniques can be used in inter-picture prediction.

[0125] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed within a unit of blocks. For example, according to the HEVC standard, pictures in a video picture sequence are partitioned into coding tree units (CTUs) for compression. CTUs within a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU contains three coding tree blocks (CTBs), namely one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree partitioned into one or more coding units (CUs). For example, a 64×64 pixel CTU can be partitioned into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one example, each CU is analyzed to determine a prediction type of the CU, such as an inter-prediction type or an intra-prediction type. The CU is partitioned into one or more prediction units (PUs) depending on temporal and / or spatial predictability. Typically, each PU contains a luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed within a unit of prediction blocks. Using the luma prediction block as an example of a prediction block, the prediction block contains a matrix of values (e.g., luma values) for pixels such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0126] FIG. 7 shows an exemplary diagram of a video encoder (703). The video encoder (703) is configured to receive a processing block (e.g., a prediction block) of sample values within a current video picture in a video picture sequence and encode the processing block into a coded picture that is part of a coded video sequence. In one example, the video encoder (703) is used instead of the video encoder (403) in the example of FIG. 4.

[0127] In the example of HEVC, the video encoder (703) receives a matrix of sample values of a processing block, such as a prediction block of 8×8 samples. The video encoder (703) determines, for example using rate distortion optimization, whether the processing block is optimally coded using an intra mode, an inter mode, or a bi-prediction mode. When the processing block is coded in the intra mode, the video encoder (703) may use intra prediction techniques to code the processing block into the coded picture. When the processing block is coded in the inter mode or the bi-prediction mode, the video encoder (703) may use inter prediction or bi-prediction techniques respectively to code the processing block into the coded picture. In certain video coding techniques, the merge mode can be an inter-picture prediction sub-mode in which the motion vector is derived from one or more motion vector predictors without the benefit of coded motion vector components external to the predictor. In certain other video coding techniques, there may be motion vector components applicable to the target block. In one example, the video encoder (703) includes other components, such as a mode decision module (not shown), to determine the mode of the processing block.

[0128] In the example of FIG. 7, the video encoder (703) includes an inter encoder (730), an intra encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general control unit (721), and an entropy encoder (725) that are coupled together as shown in FIG. 7.

[0129] The inter - encoder (730) is configured to receive samples of a current block (e.g., a block being processed), compare the block with one or more reference blocks (e.g., blocks in a previous picture and a subsequent picture) in a reference picture, generate inter - prediction information (e.g., an explanation of redundant information by an inter - coding technique, a motion vector, merge mode information), and calculate an inter - prediction result (e.g., a predicted block) based on the inter - prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture decoded based on the encoded video information.

[0130] The intra - encoder (722) is configured to receive samples of a current block (e.g., a block being processed), and in some cases, compare the block with already - coded blocks in a sample picture, generate quantized coefficients after transformation, and also generate intra - prediction information (e.g., intra - prediction direction information by one or more intra - coding techniques) in some cases. In one example, the intra - encoder (722) also calculates an intra - prediction result (e.g., a predicted block) based on the intra - prediction information and reference blocks within the same picture.

[0131] The general control unit (721) is configured to determine general control data and control other components of the video encoder (703) based on the general control data. In one example, the general control unit (721) determines the mode of a block and provides a control signal to a switch (726) based on the mode. For example, when the mode is the intra - mode, the general control unit (721) controls the switch (726) to select an intra - mode result for use by the residual calculator (723), and controls the entropy encoder (725) to select the intra - prediction information and include the intra - prediction information in the bitstream. When the mode is the inter - mode, the general control unit (721) controls the switch (726) to select an inter - prediction result for use by the residual calculator (723), and controls the entropy encoder (725) to select the inter - prediction information and include the inter - prediction information in the bitstream.

[0132] The residual calculator (723) is configured to calculate the difference (residual data) between the received block and the selected prediction result from the intra encoder (722) or the inter encoder (730). The residual encoder (724) is configured to operate based on the residual data to encode the residual data and generate transform coefficients. In one example, the residual encoder (724) is configured to transform the residual data from the spatial domain to the frequency domain and generate transform coefficients. The transform coefficients are then subjected to quantization processing to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform inverse transformation and generate decoded residual data. The decoded residual data can be appropriately used by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and the inter prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and the intra prediction information. The decoded block is appropriately processed to generate a decoded picture, which in some examples is buffered in a memory circuit (not shown) and can be used as a reference picture.

[0133] The entropy encoder (725) is configured to format the bitstream to include the encoded block. The entropy encoder (725) is configured to include various information in the bitstream according to an appropriate standard such as the HEVC standard. In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra prediction information or inter prediction information), residual information, and other appropriate information in the bitstream. Note that there is no residual information when coding a block in either the merge submode of the inter mode or the bi-prediction mode according to the disclosed subject matter.

[0134] FIG. 8 shows an exemplary diagram of a video decoder (810). The video decoder (810) is configured to receive a coded picture that is part of a coded video sequence and decode the coded picture to generate a reconstructed picture. In one example, the video decoder (810) is used in place of the video decoder (410) in the example of FIG. 4.

[0135] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872) that are coupled together as shown in FIG. 8.

[0136] The entropy decoder (871) may be configured to reconstruct from the coded picture specific symbols that represent the generated syntax elements of the coded picture. Such symbols may include, for example, the coded mode of a block (e.g., intra mode, inter mode, bi - directional mode, the latter two of merge sub - modes or another sub - mode), prediction information (e.g., intra prediction information or inter prediction information) that can identify specific samples or metadata used for prediction by each of the intra decoder (872) or the inter decoder (880). The symbols may also include residual information in the form of, for example, quantized transform coefficients. In one example, when the prediction mode is an inter or bi - directional prediction mode, the inter prediction information is provided to the inter decoder (880), and when the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information is inverse - quantized and provided to the residual decoder (873).

[0137] The inter decoder (880) is configured to receive the inter prediction information and generate an inter prediction result based on the inter prediction information.

[0138] The intra decoder (872) is configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0139] The residual decoder (873) is configured to perform inverse quantization to extract the inverse quantized transform coefficients, process the inverse quantized transform coefficients, and convert the residual information from frequency domain information to the spatial domain. The residual decoder (873) may also require specific control information (for including the quantizer parameter (QP)). This information may be provided by the entropy decoder (871) (since this is only low-capacity control information, the data path is not shown).

[0140] The reconstruction module (874) is configured to combine, in the spatial domain, the residual information as the output by the residual decoder (873) and the prediction result (optionally as the output by the inter or intra prediction module) to form a reconstruction block that can be part of a reconstructed picture and can also be part of a reconstructed video. Other appropriate operations such as a deblocking operation can be performed to improve visual quality.

[0141] Note that the video encoders (403), (603), and (703), and the video decoders (410), (510), and (810) can be implemented using any appropriate technology. In one embodiment, the video encoders (403), (603), and (703), and the video decoders (410), (510), and (810) can be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (603), and the video decoders (410), (510), and (810) can be implemented using one or more processors that execute software instructions.

[0142] Some aspects of the present disclosure provide techniques for an affine motion refinement method.

[0143] ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC1 / SC29 / WG11) issued the H.265 / HEVC (High Efficiency Video Coding) standard in 2013 (version 1), 2014 (version 2), 2015 (version 3), and 2016 (version 4). In 2015, the two standardization organizations formed the JVET (Joint Video Exploration Team) together to explore the possibility of developing the next-generation video coding standard after HEVC. In October 2017, the two standardization organizations announced the Joint Call for Proposals on Video Compression with Capability beyond HEVC (CfP). By February 15, 2018, 22 CfP responses regarding standard dynamic range (SDR), 12 CfP responses regarding high dynamic range (HDR), and 12 CfP responses regarding 360 video categories were submitted respectively. In April 2018, all the received CfP responses were evaluated at the 122nd MPEG / 10th JVET meeting. As a result of this meeting, JVET officially announced the standardization process for the next-generation video coding after HEVC. The new standard was named VVC (Versatile Video Coding), and JVET was renamed the Joint Video Expert Team. In 2020, ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC1 / SC29 / WG11) published the VVC video coding standard (version 1).

[0144] In inter prediction, for each inter-predicted coding unit (CU), motion parameters are necessary for coding the features of VVC, for example, used for generating the inter-predicted samples. The motion parameters can include motion vectors, reference picture indices, reference picture list use indices, and / or additional information. The motion parameters can be signaled explicitly or implicitly. When a CU is coded in skip mode, the CU can be associated with one PU, and significant residual coefficients, coded motion vector deltas, and / or reference picture indices may not be required. When a CU is coded in merge mode, the motion parameters of the CU can be obtained from neighboring CUs. The neighboring CUs can include spatial and temporal candidates, and additional schedules (or additional candidates) such as those introduced in VVC. The merge mode can be applied not only to skip mode but also to inter-predicted CUs. An alternative to the merge mode is the explicit transmission of motion parameters, where the motion vector, the corresponding reference picture index for each reference picture list, the reference picture list use flag, and / or other necessary information can be signaled explicitly for each CU.

[0145] In VVC, the VVC Test model (VTM) reference software can include a number of improved new inter-prediction coding tools, including one or more of the following. (1) Enhanced merge prediction (2) Merge motion vector difference (MMVD) (3) AMVP mode using symmetric MVD signaling (4) Affine motion compensation prediction (5) Subblock-Based Temporal Motion Vector Prediction (SbTMVP) (6) Adaptive motion vector resolution (AMVR) (7) Motion field storage: 1 / 16 luma sample MV storage and 8×8 motion field compression (8) Bi-prediction with CU-level weights (BCW) (9) Bi-directional optical flow (BDOF) (10) Decoder-side motion vector refinement (DMVR) (11) Combined Inter-Intra Prediction (CIIP) (12) Geometric partitioning mode (GPM).

[0146] In HEVC, a translational motion model is applied to motion compensation prediction (MCP). In the real world, many types of motion such as zoom in / out, rotation, perspective motion, and other irregular motions can exist. Block-based affine transform motion compensation prediction can be applied as in VTM. Figure 9A shows the affine motion field of a block (902) described by the motion information of two control points (4 parameters). Figure 9B shows the affine motion field of a block (904) described by three control point motion vectors (6 parameters).

[0147] As shown in Figure 9A, in a 4-parameter affine motion model, the motion vector at the sample position (x, y) within the block (902) can be derived as follows in Equation (1).

Equation

Number

[0148] As shown in FIG. 9B, in the six-parameter affine motion model, the motion vector at the sample position (x, y) within the block (904) can be derived as follows by Equation (3).

Number

Number

[0149] As shown in FIG. 10, in order to simplify motion compensation prediction, block-based affine transform prediction can be applied. To derive the motion vector of each 4×4 luma sub-block, the motion vector of the central sample (e.g., (1002)) of each sub-block (e.g., (1004)) within the current block (1000) can be calculated according to formulas (1) to (4) and rounded to a fractional accuracy of 1 / 16. Next, a motion compensation interpolation filter can be applied to generate a prediction for each sub-block using the derived motion vector. The sub-block size of the chroma component can also be set to 4×4. The MV of the 4×4 chroma sub-block can be calculated as the average of the MVs of the four corresponding 4×4 luma sub-blocks.

[0150] In affine merge prediction, the affine merge (AF_MERGE) mode can be applied to a CU whose width and height are both 8 or more. The CPMV of the current CU can be generated based on the motion information of spatially adjacent CUs. Up to five CPMVP candidates can be applied to affine merge prediction, and an index can be signaled to indicate which of the five CPMVP candidates can be used for the current CU. In affine merge prediction, three types of CPMV candidates can be used to form an affine merge candidate list. (1) An inherited affine merge candidate estimated from the CPMV of a neighboring CU, (2) an affine merge candidate constructed using the CPMVP derived using the translational MV of a neighboring CU, and (3) a zero MV.

[0151] In VTM3, a maximum of two inherited affine candidates can be applied. The two inherited affine candidates can be derived from the affine motion models of neighboring blocks. For example, one inherited affine candidate can be derived from the left neighboring CU, and the other inherited affine candidate can be derived from the upper neighboring CU. Exemplary candidate blocks can be shown in FIG. 11. As shown in FIG. 11, for the left predictor (or left inherited affine candidate), the scan order can be A0→A1, and for the upper predictor (or upper inherited affine candidate), the scan order can be B0→B1→B2. Therefore, only the first available inherited candidate can be selected from each side. Pruning checks should not be performed between the two inherited candidates. When the neighboring affine CU is identified, the CPMV candidates in the affine merge list of the current CU can be derived using the control point motion vectors of the neighboring affine CU. As shown in FIG. 12, when coding the neighboring lower left block A of the current block (1204) in affine mode, the motion vectors v2, v3, and v4 of the upper left corner, upper right corner, and lower left corner of the CU (1202) containing that block A can be obtained. When coding block A with a 4-parameter affine model, two CPMVs of the current CU (1204) can be calculated based on v2 and v3 of CU (1202). When coding block A with a 6-parameter affine model, three CPMVs of the current CU (1204) can be calculated according to v2, v3, and v4 of CU (1202).

[0152] The affine candidates formed by the current block can be candidates formed by combining the neighboring translational motion information of each control point of the current block. The motion information of the control point can be derived from specific spatial neighbors and temporal neighbors as shown in FIG. 13. As shown in FIG. 13, the CPMV k(k = 1, 2, 3, 4) represents the k-th control point of the current block (1302). In CPMV1, the B2→B3→A2 blocks are checked, and the MV of the first available block can be used. In CPMV2, the B1→B0 blocks can be checked. In CPMV3, the A1→A0 blocks can be checked. If CPMV4 is not available, TMVP can be used as CPMV4.

[0153] After the MVs of the four control points are achieved, the affine merge candidates of the current block (1302) can be constructed based on the movement information of the four control points. For example, based on the combinations of the MVs of the four control points, the affine merge candidates can be constructed in the following order: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, and {CPMV1, CPMV3}.

[0154] The combination of three CPMVs can form a 6-parameter affine merge candidate, and the combination of two CPMVs can form a 4-parameter affine merge candidate. To avoid motion scaling processing, if the reference indices of the control points are different, the related combinations of the control point MVs can be discarded.

[0155] After the inherited affine merge candidates and the constructed affine merge candidates are checked, if the list is not full yet, zero MVs can be inserted at the end of the list.

[0156] In affine AMVP prediction, the affine merge (AF_MERGE) mode can be applied to CUs with both width and height greater than or equal to 16. The affine flag at the CU level is signaled in the bitstream to indicate whether the affine AMVP mode is used, and another flag can be signaled to indicate whether 4-parameter affine or 6-parameter affine is applied. In affine AMVP prediction, the difference between the predictors of the current UCPMV and the CPMVP of the current CU can be signaled in the bitstream. The size of the affine AMVP candidate list can be set to 2, and four types of CPMV candidates can be used in the following order to generate the affine AMVP candidate list: (1) Inherited affine AMVP candidates estimated from the CPMV of neighboring CUs. (2) Affine AMVP candidates constructed using the CPMVP derived using the translational MV of neighboring CUs. (3) Translational MVs from neighboring CUs. (4) Zero MVs.

[0157] The checking order of the inherited affine AMVP candidates can be the same as that of the inherited affine merge candidates. To determine the AVMP candidates, only affine CUs with the same reference picture as the current block can be considered. Pruning cannot be applied when the inherited affine motion predictor is inserted into the candidate list.

[0158] The constructed AMVP candidates can be derived from the specified spatial neighbors. As shown in FIG. 13, the same check order as in the affine merge candidate configuration can be applied. Also, the reference picture indexes of neighboring blocks can be checked. The first block in the check order can be inter-coded and have the same reference picture as the current CU (1302). When the current CU (1302) is coded in the 4-parameter affine mode and both mv0 and mv1 are available, one constructed AMVP candidate can be determined. The constructed AMVP candidate can be further added to the affine AMVP list. When the current CU (1302) is coded in the 6-parameter affine mode and all three CPMVs are available, the constructed AMVP candidate can be added as one candidate to the affine AMVP list. Otherwise, the constructed AMVP candidate can be set as unavailable.

[0159] After the inherited affine AMVP candidates and the constructed AMVP candidates are checked, if there are still less than two candidates in the affine AMVP list, mv0, mv1, and mv2 can be added in order. mv0, mv1, and mv2 can function as translational MVs that predict all control point MVs of the current CU (e.g., (1302)) when available. Finally, if the affine AMVP list is not yet full, zero MVs can be used to fill the affine AMVP list.

[0160] Affine motion compensation based on sub-blocks can save memory access bandwidth at the expense of prediction accuracy penalty and reduce computational complexity compared to pixel-based motion compensation. To achieve finer granularity of motion compensation, prediction refinement with optical flow (PROF) can be used to refine the sub-block-based affine motion compensation prediction without increasing the memory access bandwidth for motion compensation. In VVC, after the sub-block-based affine motion compensation is performed, the luma prediction samples can be refined by adding the difference derived by the optical flow formula. PROF can be described in the following four steps.

[0161] Step (1): Affine motion compensation based on sub-blocks can be performed to generate a sub-block prediction I(i,j).

[0162] Step (2): Using a 3-tap filter [-1, 0, 1], the spatial gradients g x (i,j) and g y (i,j) of the sub-block prediction at each sample position can be calculated. The gradient calculation can be the same as the gradient calculation in BDOF. For example, the spatial gradients g x (i,j) and g y (i,j) can be calculated based on equations (5) and (6) respectively.

Equation

[0163] Step (3): The refinement of the luma prediction can be calculated by the optical flow equation as shown in Equation (7).

Number

[0164] Since the parameters of the affine model and the sample position relative to the center of the sub-block cannot be changed from one sub-block to another sub-block, Δv(i,j) is calculated for the first sub-block (e.g., (1402)) and can be reused for other sub-blocks (e.g., (1410)) within the same CU (e.g., (1400)). If the horizontal offset from the sample position (i,j) to the center (x SB , y SB ) of the sub-block is dx(i,j) and the vertical offset is dy(i,j), then Δv(x,y) can be derived as follows by Equations (8) and (9). [Number]

[0165] To maintain accuracy, the center (x SB , y SB ) of the sub - block can be calculated as ((W SB - 1) / 2, (H SB - 1) / 2). Here, W SB and H SB are the width and height of the sub - block respectively.

[0166] When Δv(x, y) is obtained, the parameters of the affine model can be obtained. For example, in the case of a 4 - parameter affine model, the parameters of the affine model can be shown in Equation (10). [Number] For example, in the case of a 6 - parameter affine model, the parameters of the affine model can be shown in Equation (11). [Number] Here, (v 0x , v 0y ), (v 1x , v 1y ), (v 2x , v 2y ) are the top - left control - point motion vector, the top - right control - point motion vector, and the bottom - left control - point motion vector respectively, and w and h can be the width and height of the CU respectively.

[0167] Step (4): Finally, the luma prediction refinement ΔI(i, j) can be added to the sub - block prediction I(i, j). The final prediction I’ can be generated as shown in Equation (12). [Number]

[0168] PROF shall not be applied to the affine-coded CUs in the following two cases: (1) All control point MVs are the same, which indicates that the CU has only translational motion. (2) To avoid large memory access bandwidth requirements, the affine MC based on sub-blocks degrades to the MC based on CUs, so the affine motion parameters become larger than the specified limits.

[0169] Note that when coding in the affine AMVP mode, each control point of the affine coding block has a motion vector difference (MVD). For each reference picture, the MVD of the control point is calculated from the actual CPMV value of the control point and the CPMV value of the affine AMVP predictor.

[0170] In one example, for 4-parameter affine, two MVDs (represented by MVD0 and MVD1) are coded for each reference list according to Equations (13) and (14).

Equation

[0171] In another example, for 6-parameter affine, three MVDs (represented by MVD0, MVD1, and MVD2) are coded for each reference list according to Equations (15), (16), and (17).

Equation

[0172] Affine motion estimation (Motion Estimation (ME)), such as the VVC reference software VTM, can operate for both uni-prediction and bi-prediction. Uni-prediction can be executed for one of the reference lists L0 and L1, and bi-prediction can be executed for both reference lists L0 and L1.

[0173] Figure 15 is a schematic diagram of the affine ME (1500). As shown in Figure 15, in the affine ME (1500), affine uni-prediction (S1502) can be performed for the reference list L0 to obtain the prediction P0 of the current block based on the initial reference block in the reference list L0. Also, affine uni-prediction (S1504) can be performed for the reference list L1 to obtain the prediction P1 of the current block based on the initial reference block in the reference list L1. In (S1506), affine bi-prediction can be performed. The affine bi-prediction (S1506) can start from the initial prediction residual (2I - P0) - P1, where I can be the initial value of the current block. The affine bi-prediction (S1506) can search for candidates in the reference list L1 around the initial reference block in the reference list L1 and find the best (or selected) reference block with the minimum prediction residual (2I - P0) - Px. Here, Px is the prediction of the current block based on the selected reference block.

[0174] In the reference picture, for the current coding block, the affine ME process can first select a set of control point motion vectors (CPMVs) as a basis. Using an iterative method, the prediction output of the current affine model corresponding to the set of CPMVs can be generated, the gradient of the prediction samples can be calculated, and a linear equation can be solved to determine the delta CPMV to optimize the affine prediction. The iteration can stop when all delta CPMVs are zero or when the maximum number of iterations is reached. The CPMV obtained from the iteration can be the final CPMV of the reference picture.

[0175] After the best affine CPVMs of both reference lists L0 and L1 are determined for affine single prediction, an affine dual prediction search can be performed using the best single prediction CPMV and one of the reference lists, and the best CPMV can be searched in the other reference list to optimize the affine dual prediction output. The affine dual prediction search can be repeatedly performed for the two reference lists to obtain optimal results.

[0176] FIG. 16 shows an exemplary affine ME process (1600) capable of calculating the final CPMV associated with a reference picture. The affine ME process (1600) can start at (S1602). At (S1602), the basic CPMV of the current block can be determined. The basic CPMV can be determined based on any of a merge index, an advanced motion vector prediction (AMVP) prediction index, an affine merge index, etc.

[0177] At (S1604), an initial affine prediction of the current block can be obtained based on the basic CPMV. For example, according to the basic CPMV, a 4-parameter affine motion model of a 6-parameter affine motion model can be applied to generate an initial affine prediction.

[0178] At (S1606), the gradient of the initial affine prediction can be obtained. For example, based on equations (5) and (6), the gradient of the initial affine prediction can be obtained.

[0179] At (S1608), the delta CPMV can be determined. In some embodiments, the delta CPMV can be associated with the displacement between the initial affine prediction and a subsequent affine prediction such as the first affine prediction. Based on the gradient between the initial affine prediction and the delta CPMV, the first affine prediction can be obtained. The first affine prediction can correspond to the first CPMV.

[0180] In (S1610), a determination can be made to check whether the delta CPMV is 0 or whether the number of iterations is greater than or equal to a threshold value. If the delta CPMV is 0 or the number of iterations is greater than or equal to the threshold value, in (S1612), the final (or selected) CPMV can be determined. The final (or selected) CPMV can be the first CPMV determined based on the initial affinity prediction and the gradient of the delta CPMV.

[0181] Continuing to refer to (S1610), if the delta CPMV is not 0 or the number of iterations is less than the threshold value, a new iteration can be started. In the new iteration, an updated CPMV (e.g., the first CPMV) can be provided to (S1604) to generate an updated affinity prediction. Then, the affine ME process (1600) proceeds to (S1606), where the gradient of the updated affinity prediction can be calculated. The affine ME process (1600) then proceeds to (S1608), where the new iteration can be continued.

[0182] In the affine motion model, the four-parameter affine motion model can be further described by an equation including rotation and zoom motions. For example, the four-parameter affine motion model can be rewritten as follows in Equation (18).

Equation

Equation

[0183] The bi-directional optical flow (BDOF) in VVC was previously called BIO in JEM. Compared with the JEM version, the BDOF of VVC can be made into a simpler version that requires less computation, especially regarding the number of multiplications and the size of the multipliers.

[0184] BDOF can be used to refine the dual prediction signals of the CU at the 4×4 sub-block level. BDOF can be applied to the CU when the CU meets the following conditions. (1) The CU is coded using the "true" dual prediction mode. That is, one of the two reference pictures is in front of the current picture in display order, and the other is behind the current picture in display order. (2) The distances (such as POC differences) from the two reference pictures to the current picture are the same. (3) Both reference pictures are short-term reference pictures. (4) The CU is not coded using the affine mode or the SbTMVP merge mode. (5) The CU has more than 64 luma samples. (6) Both the height and width of the CU are 8 luma samples or more. (7) The BCW weight index indicates equal weights. (8) Weighted prediction (WP) is not valid in the current CU. (9) The CIIP mode is not used in the current CU.

[0185] BDOF can only be applied to the luma component. As the name BDOF indicates, the BDOF mode can be based on the concept of optical flow on the premise that the movement of the object is smooth. For each 4×4 sub-block, the motion refinement (v x , v y ) can be calculated by minimizing the difference between the predicted samples of L0 and L1. Then, the dual-predicted sample values within the 4×4 sub-block can be adjusted using the motion refinement. BDOF can include the following steps.

[0186] First, the following horizontal and vertical gradients of the two predicted signals from reference list L0 and reference list L1 can be calculated by directly calculating the difference between two neighboring samples:

Equation

Equation

[0187] Next, the autocorrelations and cross-correlations of the gradients S1, S2, S3, S5, and S6 can be calculated according to the following equations (23) to (27).

Equation

Equation

[0188] Next, the motion refinement (v x , v y ) can be derived using the terms of cross - correlation and auto - correlation by using the following equations (31) and (32).

Equation

Equation

[0189] Finally, the BDOF samples of the CU can be calculated by adjusting the dual - prediction samples of the following Equation (34).

Equation

[0190] To derive the gradient value, some of the prediction samples I within list k (k = 0, 1) outside the current CU boundary (k)(i,j) needs to be generated. As shown in FIG. 17, for BDOF in VVC, one extended row / column (1702) around the boundary (1706) of the CU (1704) can be used. To control the complexity of the calculation for generating prediction samples outside the boundary, the prediction samples within the extended region (e.g., the non-shadowed region in FIG. 17) can be directly generated by taking reference samples at the nearest integer positions (e.g., using the floor() operation on the coordinates) without interpolation, and the prediction samples within the CU (e.g., the shadowed region in FIG. 17) can be generated using the normal 8-tap motion compensation interpolation filter. The extended sample values can be used only in gradient calculations. In the remaining steps of the BDOF process, when sample and gradient values outside the CU boundary are required, the sample and gradient values can be padded (e.g., repeated) from the nearest neighbors of the sample and gradient values.

[0191] If the width and / or height of the CU exceeds 16 luma samples, the CU can be divided into sub-blocks with a width and height equal to 16 luma samples, and the boundaries of the sub-blocks can be treated as CU boundaries in the BDOF process. The maximum unit size of the BDOF process can be limited to 16×16. For each sub-block, the BDOF process can be skipped. If the sum of absolute difference (SAD) between the initial L0 and L1 prediction samples is smaller than the threshold, the BDOF process shall not be applied to the sub-block. The threshold can be set equal to (8*W*(H>>1). Here, W represents the width of the sub-block, and H can represent the height of the sub-block. To avoid further complexity in SAD calculation, the SAD between the initial L0 and L1 prediction samples calculated in the DMVR process can be reused in the BDOF process.

[0192] If BCW is currently valid for a block, i.e., the BCW weight index indicates unequal weights, the bidirectional optical flow can be disabled. Similarly, if WP is currently valid for a block, i.e., the luma weight flag (e.g., luma_weight_lx_flag) is 1 for either of the two reference pictures, BDOF can also be disabled. If the CU is coded in symmetric MVD mode or CIIP mode, BDOF can also be disabled.

[0193] To improve the accuracy of the MV in merge mode, decoder-side motion vector refinement based on bilateral-matching (BM), as in VVC, can be applied. In the bi-prediction operation, the refined MV can search around the initial MVs in reference picture list L0 and reference picture list L1. The BM method can calculate the distortion between two candidate blocks in reference picture list L0 and list L1.

[0194] FIG. 18 is a schematic diagram showing an example of decoder-side motion vector refinement based on BM. As shown in FIG. 18, the current picture (1802) can include the current block (1808). The current picture can include a reference picture list L0 (1804) and a reference picture list L1 (1806). The current block (1808) can include an initial reference block (1812) in the reference picture list L0 (1804) according to the initial motion vector MV0 and an initial reference block (1814) in the reference picture list L1 (1806) according to the initial motion vector MV1. Search processing can be performed around the initial MV0 of the reference picture list L0 (1804) and the initial MV1 of the reference picture list L1 (1806). For example, a first candidate reference block (1810) can be identified in the reference picture list L0 (1804), and a first candidate reference block (1816) can be identified in the reference picture list L1 (1806). The SAD between candidate reference blocks (e.g., (1810) and (1816)) based on each MV candidate (e.g., MV0' and MV1') around the initial MV (e.g., MV0 and MV1) can be calculated. The MV candidate with the lowest SAD becomes the refined MV and is used to generate a dual prediction signal for predicting the current block (1808).

[0195] The application of DMVR can be restricted as follows and can be applied only to CUs coded based on modes and features like VVC. (1) CU-level merge mode using dual prediction MV. (2) For the current picture, one reference picture is in the past and another reference picture is in the future. (3) The distances (such as POC differences) from the two reference pictures to the current picture are the same. (4) Both reference pictures are short-term reference pictures. (5) The CU has 64 or more luma samples. (6) Both the height and width of the CU are 8 luma samples or more. (7) The BCW weight index indicates equal weights. (8) In the current block, weighted prediction (WP) is not effective. (9) In the current block, the CIIP mode is not used.

[0196] Using the refined MV derived by DMVR processing, an inter-prediction sample can be generated and used for temporal motion vector prediction for future picture coding. The original MV can be used in deblocking processing and for spatial motion vector prediction for future CU coding.

[0197] In DVMR, the search points can surround the initial MV, and the MV offset can follow the MV difference mirroring rule. That is, any point checked by DMVR, indicated by the candidate MV pair (MV0, MV1), can follow the MV difference mirroring rules shown in equations (35) and (36). [Number] MV_offset can represent the refinement offset between the initial MV and the refined MV in any of the reference pictures. The refinement search range can be two integer luma samples from the initial MV. The search can include an integer sample offset search stage and a fractional sample refinement stage.

[0198] For example, for integer sample offset search, a full search of 25 points can be applied. First, the SAD of the initial MV pair can be calculated. If the SAD of the initial MV pair is smaller than the threshold, the integer sample stage of DMVR can be terminated. Otherwise, the SAD of the remaining 24 points can be calculated and checked in a scan order such as raster scan order. The point with the minimum SAD can be selected as the output of the integer sample offset search stage. To reduce the penalty for the uncertainty of DMVR refinement, the original MV during DMVR processing can be preferentially selected. The SAD between the reference blocks referred to by the initial MV candidates can be reduced by 1 / 4 of the SAD value.

[0199] After integer sample search, fractional sample refinement can be performed. To reduce the computational complexity, instead of additional search by SAD comparison, parametric error surface equations can be used to derive fractional sample refinement. Fractional sample refinement can be conditionally called based on the output of the integer sample search stage. If the integer sample search stage ends at the center with the minimum SAD in either the first iterative search or the second iterative search, further application of fractional sample refinement can be performed.

[0200] In sub-pixel offset estimation based on the parametric error surface, the cost at the center position and the costs at four neighboring positions from the center can be used to fit the equation of the 2D parabolic error surface based on Equation (37).

Number

Number

[0201] Bilinear interpolation and sample padding, such as in VVC, can be applied. The resolution of the MV can be, for example, 1 / 16 of the luma samples. Samples at fractional positions can be interpolated using an 8-tap interpolation filter. In DMVR, since the search points can surround the initial fractional pel MV using integer sample offsets, it is necessary to interpolate samples at fractional positions for DMVR search processing. To reduce the computational complexity, a bilinear interpolation filter can be used to generate fractional samples for DMVR search processing. As another important effect, by using a bilinear filter within a 2-sample search range, DVMR does not access more reference samples than normal motion compensation processing. After a refined MV is obtained in DMVR search processing, a normal 8-tap interpolation filter can be applied to generate the final prediction. To avoid accessing more reference samples compared to normal MC processing, samples that are necessary for interpolation processing based on the refined MV but not necessary for interpolation processing based on the original MV can be padded from the available samples.

[0202] If the width and / or height of the CU exceeds 16 luma samples, the CU can be further divided into sub-blocks with a width and / or height equal to 16 luma samples. The maximum unit size of DMVR search processing can be limited to 16×16.

[0203] In one embodiment, a merge with motion vector difference (MMVD) mode, such as in VVC, which can use implicitly derived motion information to predict samples of a CU (e.g., the current CU), is used. The MMVD mode is used in either the skip mode or the merge mode depending on the motion vector representation method. For example, after signaling a skip flag or a merge flag, an MMVD merge flag can be signaled to specify whether to use the MMVD mode for the CU.

[0204] In some examples, MMVD reuses merge candidates. A candidate can be selected from among the merge candidates and further extended by a motion vector representation method. MMVD provides motion vector representation with simplified signaling. In some examples, the motion vector representation method includes a starting point, a magnitude of motion, and a direction of motion.

[0205] In some examples (e.g., VVC), the MMVD technique can select a candidate for the starting point using a merge candidate list. However, in the example, only candidates of the default merge type (MRG_TYPE_DEFAULT_N) are considered in the expansion of MMVD.

[0206] In some examples, the starting point can be defined by a basic candidate index. The basic candidate index indicates the best candidate among the candidates in the list, as shown in Table 1. For example, the list is a merge candidate list including a motion vector predictor (MVP). The basic candidate index can indicate the optimal candidate within the merge candidate list. [Table 1] Example of Basic Candidate Index (IDX) [Table 1]

[0207] Note that in one example, when the number of basic candidates is equal to 1, the basic candidate IDX is not signaled.

[0208] In the MMVD mode, after a merge candidate (also referred to as an MV base or an MV starting point) is selected, the merge candidate can be refined by additional information such as the signaled MVD information. The additional information can include an index used to specify the magnitude of motion (e.g., a distance index such as mmvd_distance_idx[x0][y0]) and an index used to indicate the direction of motion (e.g., a direction index such as mmvd_direction_idx[x0][y0]). In the MMVD mode, one of the 12th candidates in the merge candidate list can be selected as the MV base. For example, a merge candidate flag (e.g., mmvd_cand_flag[x0][y0]) indicates one of the 12th candidates in the merge list. The merge candidate flag can be signaled to indicate (e.g., specify) that one of the 12th candidates has been selected. The additional information can indicate the MVD (or motion offset) with respect to the MV base. For example, the magnitude of motion indicates the magnitude of the MVD, and the direction of motion indicates the direction of the MVD.

[0209] In one example, a starting point or an MV starting point is given to a reference picture using a merge candidate selected from the merge candidate list. The motion vector of the current block can be represented by the starting point and a motion offset (or MVD) including the magnitude of motion and the direction of motion with respect to the starting point. On the encoder side, the selection of the merge candidate and the determination of the motion offset can be based on a search process (evaluation process) as shown in FIG. 19. On the decoder side, the selected merge candidate and the motion offset can be determined based on the signaling from the encoder side.

[0210] FIG. 19 shows an example of a search process (1900) in the MMVD mode. FIG. 20 shows an example of search points in the MMVD mode. In some examples, a subset or the entire set of the search points in FIG. 20 is used in the search process (1900) of FIG. 19. By performing the search process (1900) on the encoder side, for example, for the current block (1901) of the current picture (or current frame), additional information including a merge candidate flag (e.g., mmvd_cand_flag[x0][y0]), a distance index (e.g., mmvd_distance_idx[x0][y0]), and a direction index (e.g., mmvd_direction_idx[x0][y0]) can be determined.

[0211] The first motion vector (1911) and the second motion vector (1921) belonging to the first merge candidate are shown. The first motion vector (1911) and the second motion vector (1921) are the MV start points used in the search process (1900). The first merge candidate can be a merge candidate in the merge candidate list configured for the current block (1901). The first and second motion vectors (1911) and (1921) can be associated with two reference pictures (1902) and (1903) in the reference picture lists L0 and L1, respectively. Referring to FIGS. 19 to 20, the first and second motion vectors (1911) and (1921) can point to two start points (2011) and (2021) in the reference pictures (1902) and (1903), respectively, as shown in FIG. 20.

[0212] Referring to FIG. 20, the two starting points (2011) and (2021) in FIG. 20 can be determined in the reference pictures (1902) and (1903). In the example, based on the starting points (2011) and (2021), a plurality of predetermined points extending vertically (represented by +Y or -Y) or horizontally backward (represented by +X and -X) from the starting points (2011) and (2021) in the reference pictures (1902) and (1903) can be evaluated. In one example, a pair of motion vectors (e.g., MV(1913) and (1923) in FIG. 19) that can form a motion vector predictor candidate for the current block (1901) can be determined using pairs of points that mirror each other with respect to each starting point (2011) or (2021), such as the pair of points (2014) and (2024) (e.g., indicated by a 1S shift in FIG. 19), or the pair of points (2015) and (2025) (e.g., indicated by a 2S shift in FIG. 19). Those motion vector predictor candidates (e.g., MV(1913) and (1923) in FIG. 19) determined based on the predetermined points around the starting point (2011) or (2021) can be evaluated.

[0213] The distance index (e.g., mmvd_distance_idx[x0][y0]) can identify the magnitude of motion information and indicate a predetermined offset (e.g., 1S or 2S in FIG. 19) from the starting point indicated by the merge candidate flag. Note that the predetermined offset is also called the MMVD step in one example.

[0214] Referring to FIG. 19, an offset (e.g., MVD(1912) or MVD(1922)) can be applied (e.g., added) to the horizontal or vertical component of the starting MV (e.g., MV(1911) or (1921)). An exemplary relationship between the distance index (IDX) and a predetermined offset is specified in Table 2. When the full-pel MMVD is off, for example, when the full-pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is equal to 0, the range of the predetermined offset of the MMVD can be from 1 / 4 luma sample to 32 luma samples. When the full-pel MMVD is off, the predefined offset can have a value that is not an integer, such as a ratio of luma samples (e.g., 1 / 4 pixel or 1 / 2 pixel). When the full-pel MMVD is on, for example, when the full-pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is equal to 1, the range of the predetermined offset of the MMVD can be from 1 luma sample to 128 luma samples. In the example, when the full-pel MMVD is on, the predefined offset has only integer values such as one or more luma samples. [Table 2] Distance Index and Offset (e.g., Exemplary Relationship of Predefined Offset) [Table 2]

[0215] The direction index can represent the direction (or movement direction) of the MVD with respect to the starting point. In the example, as shown in Table 3, the direction index represents one of four directions. The meaning of the MVD codes in Table 3 may vary depending on the information of the starting MV. In one example, when the starting MV is a single-prediction MV, or when the starting MV is a bi-prediction MV where both reference lists point to the same side of the current picture (for example, the POCs of both reference pictures are greater than the POC of the current picture, or the POCs of both reference pictures are less than the POC of the current picture), the MVD code in Table 3 specifies the sign of the MV offset (or MVD) added to the starting MV.

[0216] When the starting MV is a bi-prediction MV that points to different sides of the current picture (for example, the POC of one reference picture is greater than the POC of the current picture and the POC of the other reference picture is less than the POC of the current picture), the MVD code in Table 3 specifies the sign of the MV offset (or MVD) added to the list0 MV component of the starting MV, and the MVD code of the list1 MV has the opposite value. Referring to Figure 19, the starting MVs (1911) and (1921) are bi-prediction MVs where the two MVs (1911) and (1921) point to different sides of the current picture. The POC of the L1 reference picture (1903) is greater than the POC of the current picture, and the POC of the L0 reference picture (1902) is less than the POC of the current picture. The MVD code (for example, the sign of the x-axis "+") indicated by the direction index (for example, 00) in Table 2 specifies the sign of the MVD (for example, MVD(1912)) added to the list0 MV component of the starting MV (for example, (1911)), and the MVD code of MVD(1922) for the list1 MV component of the starting MV (for example, (1921)) has the opposite value, such as the sign "-" opposite to the sign "+" of MVD(1912).

[0217] Referring to Table 3, the direction index 00 indicates the positive direction of the x-axis, the direction index 01 indicates the negative direction of the x-axis, the direction index 10 indicates the positive direction of the y-axis, and the direction index 11 indicates the negative direction of the y-axis. [Table 3] Example of the relationship between the sign and direction index of the MMVD offset [Table 3]

[0218] The syntax element mmvd_merge_flag[x0][y0] can be used to represent the MMVD merge flag of the current CU. For example, when the MMVD merge flag (e.g., mmvd_merge_flag[x0][y0]) is equal to 1, it specifies to use the MMVD mode to generate the inter-prediction parameters of the current CU. When the MMVD merge flag (e.g., mmvd_merge_flag[x0][y0]) is equal to 0, it specifies not to use the MMVD mode to generate the inter-prediction parameters. The array indices x0 and y0 can specify the position (x0, y0) of the top-left luma sample of the coding block (e.g., the current CB) to be considered, based on the top-left luma sample of the picture (e.g., the current picture).

[0219] When the MMVD merge flag (e.g., mmvd_merge_flag[x0][y0]) does not exist for the current CU, the MMVD merge flag (e.g., mmvd_merge_flag[x0][y0]) can be estimated to be equal to 0 for the current CU.

[0220] In some examples, like in the VVC standard, a single context is used to signal the MMVD merge flag (e.g., mmvd_merge_flag). For example, a single context is used to code (e.g., encode and / or decode) the MMVD merge flag with context-adaptive binary arithmetic coding (CABAC).

[0221] The syntax element mmvd_cand_flag[x0][y0] can represent a merge candidate flag. For example, the merge candidate flag (e.g., mmvd_cand_flag[x0][y0]) specifies whether the first (0) or second (1) candidate in the merge candidate list is to be used with the MVD derived from the distance index (e.g., mmvd_distance_idx[x0][y0]) and the direction index (e.g., mmvd_direction_idx[x0][y0]). The array indices x0 and y0 can specify the position (x0, y0) of the top-left luma sample of the coding block (e.g., the current CB) being considered, with reference to the top-left luma sample of the picture (e.g., the current picture).

[0222] If the merge candidate flag (e.g., mmvd_cand_flag[x0][y0]) does not exist, it can be assumed that the merge candidate flag (e.g., mmvd_cand_flag[x0][y0]) is equal to 0.

[0223] The syntax element mmvd_distance_idx[x0][y0] can represent a distance index. In one example, the distance index (e.g., mmvd_distance_idx[x0][y0]) specifies the index used to derive MmvdDistance[x0][y0] as specified in Table 4. The array indices x0 and y0 can specify the position (x0, y0) of the top-left luma sample of the coding block (e.g., the current CB) being considered, with reference to the top-left luma sample of the picture (e.g., the current picture). [Table 4] Exemplary relationship between MmvdDistance[x0][y0] and mmvd_distance_idx[x0][y0] [Table 4]

[0224] The first column of Table 4 indicates the distance index (e.g., mmvd_distance_idx[x0][y0]). The second column of Table 4 indicates the magnitude of motion (e.g., MmvdDistance[x0][y0]) when full - pel MMVD is off, for example, when the full - pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is equal to 0. The third column of Table 4 indicates the magnitude of motion (e.g., MmvdDistance[x0][y0]) when full - pel MMVD is on, for example, when the full - pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is equal to 1.

[0225] In one example, the units of the second and third columns of Table 4 are 1 / 4 luma samples. Referring to the first row of Table 4, when the distance index (e.g., mmvd_distance_idx[x0][y0]) is 0, the magnitude of motion (e.g., MmvdDistance[x0][y0]) is 1 when full - pel MMVD is off (e.g., slice_fpel_mmvd_enabled_flag is 0). The magnitude of motion (e.g., MmvdDistance[x0][y0]) is 1×1 / 4 luma samples or 1 / 4 luma sample. When the distance index (e.g., mmvd_distance_idx[x0][y0]) is 0, the magnitude of motion (e.g., MmvdDistance[x0][y0]) is 4 when full - pel MMVD is on (e.g., slice_fpel_mmvd_enabled_flag is 1). The magnitude of motion (e.g., MmvdDistance[x0][y0]) is 4×1 / 4 luma samples or 1 luma sample.

[0226] In one example, the second column of Table 4 (in units of 1 / 4 luma samples) corresponds to the second row of Table 1 (in units of luma samples), and the third column of Table 4 (in units of 1 / 4 luma samples) corresponds to the third row of Table 2 (in units of luma samples).

[0227] The syntax element mmvd_distance_idx[x0][y0] can represent a distance index. In one example, a direction index (e.g., mmvd_direction_idx[x0][y0]) specifies an index that is used to derive the direction of motion (e.g., MmvdSign[x0][y0]) as specified in Table 5. The array indices x0 and y0 specify the position (x0, y0) of the top-left luma sample of the coding block under consideration (e.g., the current CB) with respect to the top-left luma sample of the picture (e.g., the current picture). The first column of Table 5 shows the direction index (e.g., mmvd_direction_idx[x0][y0]). The second column of Table 5 shows the first sign (e.g., MmvdSign[x0][y0][0]) of the first component of the MVD (e.g., MVD x or MmvdOffset[x0][y0][0]). The third column of Table 5 shows the second sign (e.g., MmvdSign[x0][y0][1]) of the second component of the MVD (e.g., MVD y or MmvdOffset[x0][y0][1]). [Table 5] Exemplary relationship between MmvdSign[x0][y0] and mmvd_direction_idx[x0][y0]

Table 5

[0228] The first component (e.g., MmvdOffset[x0][y0][0]) and the second component (e.g., MmvdOffset[x0][y0][1]) of the MVD or the offset MmvdOffset[x0][y0] can be derived as follows.

Number

[0229] In one example, the distance index (e.g., mmvd_distance_idx[x0][y0]) is 3 and the direction index (e.g., mmvd_direction_idx[x0][y0]) is 2. Based on Table 5 and the direction index (e.g., mmvd_direction_idx[x0][y0]) being 2, the first sign (e.g., MmvdSign[x0][y0][0]) of the first component of the MVD (e.g., MVD x or MmvdOffset[x0][y0][0]) is 0, and the second sign (e.g., MmvdSign[x0][y0][1]) of the second component of the MVD (e.g., MVD y or MmvdOffset[x0][y0][1]) is “+1”. In this example, the MVD is along the positive vertical direction (+y) and there is no horizontal component.

[0230] When the full pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is equal to 0 and full pel MMVD is off, based on Table 4 and the distance index (e.g., mmvd_distance_idx[x0][y0]) being 3, the magnitude of the motion indicated by MmvdDistance[x0][y0] is 8. Based on Equations 10 - 11, the first component of the MVD (e.g., MmvdOffset[x0][y0][0]) is (8 << 2) × 0 = 0, and the second component of the MVD (e.g., MmvdOffset[x0][y0][1]) is (8 << 2) × (+1) = 2 (luma samples).

[0231] When the full pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is equal to 1 and the full pel MMVD is on, based on Table 4 and the distance index (e.g., mmvd_distance_idx[x0][y0]) being 3, the magnitude of the motion indicated by MmvdDistance[x0][y0] is 32. Based on Equation (40) and Equation (41), the first component of the MVD (e.g., MmvdOffset[x0][y0][0]) is (32 << 2) × 0 = 0, and the second component of the MVD (e.g., MmvdOffset[x0][y0][1]) is (32 << 2) × (+1) = 2 (luma samples).

[0232] According to one aspect of the present disclosure, affine merge with motion vector difference (affine MMVD) can be used in video coding. Affine MMVD selects available affine merge candidates from a merge list based on sub-blocks as basic predictors. Affine MMVD applies a motion vector offset to each control point motion vector value from the basic predictor. In one example, if no affine merge candidate is available, affine MMVD is not used. In some examples, the distance index and the offset direction index can be signaled later.

[0233] In some examples, the distance index is signaled to indicate the distance offset to be used from the offset table, as shown in Table 6. [Table 6] Example of offset table

Table 6

[0234] In some examples, the direction index can represent four directions, as shown in Table 7. Here, only the x or y direction, not both directions, can have an MV difference. [Table 7] Example of direction table

Table 7

[0235] In some examples, the inter prediction is a single prediction, and to generate a result including the MV value of each control point, the distance offset signaled in the offset direction of each control point predictor is applied.

[0236] In some examples, the inter prediction is a dual prediction, the signaled distance offset can be applied in the signaled offset direction of the L0 motion vector of the control point predictor, and the offset applied to the L1 MV can be applied to a mirrored or scaled base as in the following specific examples.

[0237] In a specific example, the inter prediction is a dual prediction, and the signaled distance offset is applied in the signaled offset direction of the L0 motion vector of the control point predictor. For the L1 CPMV, the offset is applied to a mirrored base. This means that the same amount of distance offset in the opposite direction is applied.

[0238] In another specific example, an offset mirroring method based on the POC distance is used for dual prediction. When the basic candidate is dual predicted, the offset applied to L0 is signaled, and the offset on L1 depends on the temporal positions of the reference pictures on list L0 and list L1. For example, if both reference pictures are on the same temporal side of the current picture, the same distance offset and the same offset direction are applied to the CPMVs of both L0 and L1. In another example, if the two reference pictures are on different sides of the current picture, the CPMV of L1 can have a distance offset applied in the opposite offset direction.

[0239] In another specific example, an offset scaling method based on the POC distance is used for dual prediction. When the basic candidate is dual predicted, the offset applied to L0 is signaled, and the offset of L1 can be scaled based on the temporal distance between the reference pictures in list 0 and list 1.

[0240] In some examples, the range of the distance offset value is extended. For example, three sets of distance offset values can be provided, and one set of distance offset values can be adaptively selected based on the picture resolution. In one example, the offset table is selected based on the picture resolution. Table 8 shows an example of an extended distance offset table that includes three sets of distance offset values each associated with a different picture resolution. One set of distance offset values can be selected based on the picture resolution. [Table 8] Example of an extended distance offset table [Table 8]

[0241] Template matching (TM) technology can be used for video / image coding. To further improve the compression efficiency of the VVC standard, for example, TM can be used to refine the MV. In an example, TM is used on the decoder side. In the TM mode, the MV can be refined by constructing a template (e.g., the current template) of a block (e.g., the current block) within the current picture and determining the closest match between the template of the block within the current picture and multiple possible templates (e.g., multiple possible reference templates) within the reference picture. In an embodiment, the template of the block within the current picture can include the reconstructed samples of the block's left neighboring block and the reconstructed samples of the block's upper neighboring block. TM can be used in previous video / image coding of VVC.

[0242] FIG. 21 shows an example of template matching (2100). TM can be used to derive motion information of the current CU (e.g., current block) (2101) by determining the closest match between the template of the current CU (2101) in the current picture (2110) (e.g., current template) (2121) and the templates of a plurality of possible templates in the reference picture (2111) (e.g., reference templates) (e.g., one of the plurality of possible templates is template (2125)). The template (2121) of the current CU (2101) can have any suitable shape and suitable size.

[0243] In an embodiment, the template (2121) of the current CU (2101) includes the upper template (2122) and the left template (2123). Each of the upper template (2122) and the left template (2123) can have any suitable shape and suitable size.

[0244] The upper template (2122) can include samples within one or more upper neighboring blocks of the current CU (2101). In the example, the upper template (2122) can include 4 rows of samples within one or more upper neighboring blocks of the current CU (2101). The left template (2123) can include samples within one or more left neighboring blocks of the current CU (2101). In the example, the left template (2123) can include 4 columns of samples within one or more left neighboring blocks of the current CU (2101).

[0245] Each of a plurality of possible templates of the reference picture (2111) (e.g., template (2125)) corresponds to the template (2121) of the current picture (2110). In one embodiment, the initial MV (2102) points to the reference block (2103) in the reference picture (2111) from the current CU (2101). Each of a plurality of possible templates of the reference picture (2111) (e.g., template (2125)) and the template (2121) of the current picture (2110) can have the same shape and the same size. For example, the template (2125) of the reference block (2103) includes the upper template (2126) of the reference picture (2111) and the left template (2127) of the reference picture (2111). The upper template (2126) can include samples in one or more upper neighboring blocks of the reference block (2103). The left template (2127) can include samples in one or more left neighboring blocks of the reference block (2103).

[0246] The TM cost can be determined based on a pair of templates such as a template (e.g., the current template) (2121) and a template (e.g., the reference template) (2125). The TM cost can indicate the match between template (2121) and template (2125). Based on the search around the initial MV (2102) of the current CU (2101) within the search range (2115), an optimized MV (or final MV) can be determined. The search range (2115) can have any suitable shape and any suitable number of reference samples. In an example, the search range (2115) of the reference picture (2111) includes a [-L, L]-pel range. Here, L is a positive integer such as 8 (e.g., 8 samples). For example, a difference (e.g., [0, 11]) is determined based on the search range (2115), and an intermediate MV is determined by adding the initial MV (2102) and the difference (e.g., [0, 1]). Based on the intermediate MV, an intermediate reference block and the corresponding template within the reference picture (2111) can be determined. The TM cost can be determined based on the intermediate template (1311) between the template (2121) and the reference picture (2111). The TM cost can correspond to the differences (e.g., [0, 0], [0, 1], etc. corresponding to the initial MV (2102)) determined based on the search range (2115). For example, the difference corresponding to the minimum TM cost is selected, and the sum of the difference corresponding to the minimum TM cost and the initial MV (2102) becomes the optimized MV. As described above, TM can derive the final motion information (e.g., the optimized MV) from the initial motion information (e.g., the initial MV 2102).

[0247] In the example of FIG. 21, within a search range such as [-8pel, +8pel], a better MV can be searched around the initial motion vector of the current CU.

[0248] TM can be applied in affine modes such as affine AMVP mode and affine merge mode, and can be called affine TM. FIG. 22 shows an example of TM (2200) as in the affine merge mode. The template (2221) of the current block (e.g., the current CU) (2201) can correspond to the template (e.g., the template (2121) in FIG. 21) in the TM applied to the translational motion model. The reference template (2225) of the reference block in the reference picture can include a plurality of sub-block templates (e.g., 4x4 sub-blocks) pointed by the MVs derived from the control point MVs (CPMVs) of the neighboring sub-blocks (e.g., A0 - A3 and L0 - L3 shown in FIG. 22) at the block boundary.

[0249] The search process of TM applied in the affine mode (e.g., affine merge mode) can start from CPMV0 while keeping other CPMVs (e.g., (i) CPMV1 when using the 4 - parameter model, (ii) CPMV1 and CPMV2 when using the 6 - parameter model) constant. The search can be performed in the horizontal and vertical directions. In one example, the search continues in the diagonal direction only if the zero vector is not the best difference vector found from the horizontal and vertical searches. The affine TM can repeat the same search process for CPMV1. The affine TM can repeat the same search process for CPMV2 when the 6 - parameter model is used. Based on the refined CPMV, if the zero vector is not the best difference vector from the previous iteration and the search process is repeated less than 3 times, the entire search process can resume from the refined CPMV0.

[0250] According to one aspect of the disclosure, signaling overhead can be reduced by using candidate reordering techniques based on template matching. For example, a technique called adaptive reordering of merge candidates with template matching (ARMC - TM) is used.

[0251] In some examples, ARMC-TM is used to adaptively reorder merge candidates by template matching (TM). ARMC-TM can be applied to normal merge mode, template matching (TM) merge mode, and affine merge mode (excluding SbTMVP candidates). In TM merge mode, merge candidates are reordered before refinement processing.

[0252] In some examples, after constructing a merge candidate list using ARMC-TM, the merge candidates are split into several subgroups. In the example, the subgroup size is set to 5 for normal merge mode and TM merge mode. In another example, the subgroup size is set to 3 for affine merge mode. The merge candidates of each subgroup are sorted in ascending order according to the cost value based on template matching. For simplicity, in some examples, the merge candidates of the last subgroup instead of the first subgroup are not sorted.

[0253] The template matching cost of a merge candidate is measured by the sum of absolute differences (SAD) between the samples of the template of the current block and the corresponding reference samples (also called reference template in one example) for the template. The template includes a set of reconstructed samples adjacent to the current block. The reference samples of the template are arranged according to the motion information of the merge candidate.

[0254] When a merge candidate utilizes bidirectional prediction, the reference samples of the template of the merge candidate are also generated by bidirectional prediction.

[0255] FIG. 23 is a diagram showing a reference sample of a template of a current block for a merge candidate of dual prediction. In FIG. 23, the current picture (2310) includes the current block for coding. When the merge candidate is a dual prediction merge candidate, the MV of the merge candidate can point to a first reference block in the reference picture (2320) and a second reference block in the second reference picture (2330). The template of the current block is indicated by (T), and the template includes a set of reconstructed samples in the neighborhood of the current block. The first set of reference samples of the template is in the first reference picture (2320) in the neighborhood of the first reference block, and the second set of reference samples of the template is in the second reference picture (2330) in the neighborhood of the second reference block. In one example, the template matching cost of a dual prediction merge candidate is calculated by adding a first sum of absolute differences (SAD) between the samples of the template of the current block and the first set of reference samples of the template, and a second sum of absolute differences (SAD) between the samples of the template of the current block and the second set of reference samples of the template.

[0256] In some examples, the merge candidate can be a merge candidate based on sub-blocks. In one example, for a merge candidate based on sub-blocks having a sub-block size equal to Wsub×Hsub, the above template can include several sub-templates having a size of Wsub×1, and the left template can include several sub-templates having a size of 1×Hsub. Wsub can be the width of the sub-block, and Hsub can be the height of the sub-block.

[0257] FIG. 24 shows the derivation of the template of the current block with respect to the merge candidate based on the sub-blocks and the reference samples of the template. As shown in FIG. 24, the current block (2402) may be included in the current picture (2404). The current block (2402) can include sub-blocks A to G in the first row and the first column. The current block (2402) can include a template (2406) adjacent to the upper side and the left side of the current block (2402). The same-position block (2408) of the current block (2402) is within the reference picture (2410). The same-position block (2408) can include sub-blocks A to G in the first row and the first column corresponding to the sub-blocks A to G of the current block (2402). Using the sub-block motion information (for example, corresponding to the affine motion vector) of the sub-blocks A to G in the first row and the first column of the current block (2402), the reference samples of the sub-template (or sub-reference template) of the same-position block (2408) can be derived. For example, by applying the motion information of the sub-blocks A, E, F, and G of the current block (2402), the reference samples of the sub-template arranged adjacent to the left side of the sub-blocks A, E, F, and G of the same-position block (2408) can be derived. The sub-template adjacent to the left side of the sub-blocks A, E, F, and G of the same-position block (2408) can form the left reference template of the same-position block (2408). By applying the motion information of the sub-blocks A, B, C, and D of the current block (2402), the reference samples of the sub-template arranged adjacent to the upper side of the sub-blocks A, B, C, and D of the same-position block (2408) can be derived. The sub-template adjacent to the upper side of the sub-blocks A, B, C, and D of the same-position block (2408) can further form the upper reference template of the same-position block (2408).

[0258] In some examples, ARMC based on the MV candidate type can be used. For example, merge candidates of a single candidate type such as TMVP or non - adjacent MVP (NA - MVP) are sorted based on the ARMC TM cost value. Then, the sorted candidates are added to the merge candidate list. For example, the TMVP candidate type ARMC can add more TMVP candidates with more temporal positions and different inter - prediction directions for sorting and selection. Further, the NA - MVP candidate type ARMC extends the non - adjacent MVPs with more spatial non - adjacent positions. The target reference picture of the TMVP candidate can be selected from any of the reference pictures in the list according to the scaling factor. For example, the selected reference picture is the one with the scaling factor closest to 1.

[0259] According to the disclosed aspects, candidate sorting based on template matching can be performed for MMVD and affine MMVD.

[0260] In some examples, the MMVD offset is extended to more positions for MMVD and affine MMVD modes.

[0261] Figure 25 is a diagram showing the directions in which refined positions can be added for MMVD. In Figure 25, assuming k is an integer, additional refined positions along the diagonal angle of k×π / 8 are added. The position (2501) corresponds to the basic candidate and can be used as the starting point. The positions (2511) - (2514) are in the directions of 0, π / 2, π, and 3π / 2 respectively. Even more directions are added. For example, the positions (2521) - (2524) are in the directions of π / 4, 3π / 4, 5π / 4, and 7π / 4 respectively. The positions (2531) - (2538) are in the directions of π / 8, 3π / 8, 5π / 8, 7π / 8, 9π / 8, 11π / 8, 13π / 8, and 15π / 8 respectively. Therefore, the number of directions increases from 4 to 16. Further, in one example, each direction can have 6 MMVD refined positions. The total number of possible MMVD refined positions is 16×6.

[0262] According to an aspect of the present disclosure, the SAD cost between the current template (e.g., one row above and one column to the left of the current block) and the reference template can be calculated for each refinement position. Based on the SAD cost of the refinement positions, all possible MMVD refinement positions (16×6) for each basic candidate are sorted. Next, the upper refinement positions, such as the top 1 / 8 refinement positions (e.g., 12) having the minimum template SAD cost, are retained as available positions for MMVD index coding. The MMVD index is binarized by a rice code having a parameter equal to 2.

[0263] In some examples, the refinement positions of the affine MMVD can be increased, and candidate sorting based on template matching can be applied to affine MMVD sorting. For example, the affine MMVD refinement positions are in the directions along the diagonal angles of k×π / 4, such as in 8 directions of 0, π / 4, π / 2, 3π / 4, π, 5π / 4, 3π / 2, 7π / 4. Each direction can have 6 affine MMVD refinement positions. The total number of possible affine MMVD refinement positions is 8×6. In one example, the SAD cost between the current template (e.g., one row above and one column to the left of the current block) and the reference template can be calculated for each refinement position. Based on the SAD cost of the refinement positions, all possible affine MMVD refinement positions (8×6) for each basic candidate are sorted. Next, the upper refinement positions, such as the top 1 / 2 refinement positions (e.g., 24) having the minimum template SAD cost, are retained as available positions for affine MMVD index coding.

[0264] In some examples, MVD code prediction techniques are used. In one example, combinations of possible MVD codes (e.g., various combinations of codes in the x and y directions) are sorted according to the template matching cost of the combinations of possible MVD codes, an index corresponding to the combination of true MVD codes is derived, and context coding is performed. According to the MVD code prediction technique, the combination of true MVD codes has a high probability in the previous part of the sorted order. Therefore, an appropriate signaling technique can be used to signal the index at a low signaling cost.

[0265] In one example, on the decoder side, the true MVD code can be derived. For example, the magnitude of the MVD component can be analyzed, and the context-coded MVD code prediction index can be analyzed from the bitstream that transmits the video. Further, MV candidates can be formed by creating combinations from combinations of possible MVD codes, and the magnitude of the MVD component and the MV candidates can be added to the MV prediction list. The template matching cost for the MV candidates in the MV predictor list can be calculated. The MV candidates in the MV predictor list can be sorted according to the template matching cost. Next, the combination of true MVD codes is selected from the MV prediction list using the context-coded MVD code prediction index. The MVD code prediction technique can be applied to various modes including MVD such as Inter-AMVP, Affine AMVP, MMVD, and Affine MMVD modes.

[0266] In some examples, a technique called affine model inheritance based on history parameters can be used. Specifically, in some examples, a first history-parameter table (HPT) and a second HPT are established.

[0267] FIG. 26 is a diagram (2600) showing the first HPT and the second HPT in some examples.

[0268] As shown in FIG. 26, the entry of the first HPT stores a set of affine parameters for the affine model, such as a, b, c, and d, and each affine parameter is represented by a 16-bit signed integer. The entry of the first HPT is classified by a reference list (e.g., reference picture list L0 or reference picture list L1) and a reference index. For each reference list of the first HPT, five reference indexes are supported. In the formulation method, in one example, the category of the first HPT (shown as HPTCat) is calculated as formula (42).

Number

[0269] For each category, a maximum of seven entries can be stored, and as a result, a total of 70 entries are stored in the first HPT. At the beginning of each CTU row, the number of entries in each category is initialized to 0. After decoding the CU coded affine using the reference list RefList cur and RefIdx cur , the entry in the category HPTCat(RefList cur ,RefIdx cur ) is updated using the affine parameters in the same way as the HMVP table update.

[0270] In some examples, a candidate based on the history affine parameter (history-affine-parameter-based candidate (HAPC)) is derived from one of the seven neighboring 4×4 blocks shown as A0, A1, A2, B0, B1, B2, or B3 in FIG. 26 and a set of affine parameters stored in the corresponding entry in the first HPT. The MV of the neighboring 4×4 block functions as the basic MV. In the formulation method, the MV of the current block at position (x, y) is calculated as formula (43).

Number

[0271] A second history parameter table (HPT) including basic MV information is also added. The second HPT can include nine entries, and the entries can include the basic MV, the reference index of each reference list, four affine parameters, and the basic position. In one example, an additional merge HAPC can be generated from the basic MV information and the corresponding affine model (e.g., affine parameters) stored in the entries of the second HPT.

[0272] Furthermore, in some examples, the per-pair affine merge candidates are generated by two affine merge candidates, either derived from the history or not derived from the history. In one example, the per-pair affine merge candidates are generated by averaging the CPMVs of the existing affine merge candidates in the candidate list.

[0273] In some examples, in response to the introduction of a new HAPC, the size of the sub-block-based merge candidate list increases from 5 to 15, and all of these can be included in the ARMC processing.

[0274] In the above description, the HPT (e.g., the first HPT and the second HPT) is updated online. In addition to the HPT updated line by line, in some examples, the HPT stored in the CTU upper right / above the current CTU can be used by blocks within the current CTU. After coding / decoding the CTU, the HPT can be stored in the line buffer for use in the next CTU line.

[0275] FIG. 27 is a diagram showing a history parameter table stored in a line buffer in some examples. In FIG. 27, a picture (2700) is divided into CTUs. FIG. 27 shows CTU row k and CTU row k + 1. The current CTU (2710) is in CTU row k + 1. For coding the current block (2711), in some examples, the HPT (2701) stored in the CTU above the current CTU (2710) and the HPT (2702) stored in the CTU upper right of the current CTU (2710) can be used by blocks within the current CTU. The HPT (e.g., the first HPT and the second HPT) is updated online and stored in the line buffer of the current CTU (2710) after decoding the last coding block within the current CTU.

[0276] According to one aspect of the present disclosure, non-adjacent spatial neighbors can be used in affine mode.

[0277] In the affine mode using non-adjacent spatial neighbors (NA-AFF), non-adjacent spatial neighbors can be obtained.

[0278] FIGS. 28A to 28B show patterns for obtaining non-adjacent spatial neighbors in some examples. Similar to existing non-adjacent normal merge candidates, the distance between the non-adjacent spatial neighbors in NA-AFF and the current CU is also defined based on the width and height of the current CU.

[0279] The motion information of non-adjacent spatial neighbors is used to generate additional inherited and constructed affine merge / AMVP candidates. FIG. 28A shows generating additional inherited affine merge / AMVP candidates, and FIG. 28B shows generating additional constructed affine merge / AMVP candidates.

[0280] Specifically, as shown in FIG. 28A, for the inherited candidates, the same derivation process as that of the VVC of the inherited affine merge / AMVP candidates remains unchanged except that the CPMV is inherited from non-adjacent spatial neighbors. The non-adjacent spatial neighbors are checked based on the distance to the current block, i.e., the distance from near to far. At a specific distance, only the first available neighbors (coded in affine mode) from each side (e.g., left and top) of the current block are included for the derivation of the inherited candidates. As shown by the dashed arrows in FIG. 28A, the checking order of the left and top neighbors is from bottom to top and from right to left, respectively.

[0281] For the first type of constructed candidates, as shown in FIG. 28B, first, the positions of one left and one top non-adjacent spatial neighbor are independently determined. Then, accordingly, the position of the upper-left neighbor is determined, which can enclose a rectangular virtual block together with the left and top non-adjacent neighbors.

[0282] Next, as shown in FIG. 29, the motion information of three non-adjacent neighbors is used to form the CPMVs of the upper-left (A), upper-right (B), and lower-left (C) of the virtual block, and project it onto the current CU to generate the corresponding constructed candidates.

[0283] In some examples, for the second type of configured candidates, the derivation process is similar to the configuration scheme in history-based affine model inheritance (HAMI). However, instead of using a history-based lookup table, the non-translational affine parameters are inherited from non-adjacent spatial neighbors. Specifically, the second type of affine configuration candidates is generated from a combination of 1) the translational affine parameters of adjacent neighboring 4x4 blocks and 2) the non-translational affine parameters inherited from non-adjacent spatial neighbors defined in FIG. 28A.

[0284] In some examples, the NA-AFF candidates are inserted into the existing affine merge candidate list and the affine AMVP candidate list in a specific order.

[0285] In one example, in the affine merge mode, the order includes the following: 1) SbTMVP candidates, if available, 2) inherited from adjacent neighbors, 3) inherited from non-adjacent neighbors, 4) composed from adjacent neighbors, 5) the second type of configured affine candidates from non-adjacent neighbors, 6) the first type of configured affine candidates from non-adjacent neighbors, 7) zero MV.

[0286] In another example, in the affine AMVP mode, the order includes the following: 1) inherited from adjacent neighbors, 2) composed from adjacent neighbors, 3) translational MV from adjacent neighbors, 4) translational MV from temporal neighbors, 5) inheritance from non-adjacent neighbors, 6) the first type of configured affine candidates from non-adjacent neighbors, 7) zero MV.

[0287] Since the additional candidates generated by NA-AFF are included, the size of the affine merge candidate list increases from 5 to 15. The subgroup size of the ARMC in the affine merge mode increases from 3 to 15.

[0288] In some video coders (such as ECM-5.0 software), NA-AFF is implemented without adding memory usage constraints.

[0289] In some video codecs (such as ECM), sample-based BDOF is used. In sample-based BDOF, instead of deriving motion refinement (v x , v y ) based on blocks, motion refinement is performed for each sample. For example, a coding block is divided into 8×8 sub-blocks. For each sub-block, whether to apply BDOF is determined by checking the SAD between two reference sub-blocks against a threshold. When it is determined to apply BDOF to a sub-block, for all samples within the sub-block, a sliding 5×5 window is used and the existing BDOF process is applied for each sliding window to derive v x and v y . The derived motion refinement (v x , v y ) is applied to adjust the dual-predicted sample value of the central sample of the window.

[0290] In some video codecs (such as ECM), multi-pass decoder-side motion vector refinement is used. Multi-pass decoder-side motion vector refinement can be applied to a coding block. For example, three-pass decoder-side motion vector refinement can be applied to a coding block. In the first pass, bilateral matching (BM) is applied to the coding block. In the second pass, BM is applied to each 16×16 sub-block within the coding block. In the third pass, the MV within each 8×8 sub-block is refined by applying bi-directional optical flow (BDOF). The refined MVs are stored for both spatial and temporal motion vector prediction.

[0291] In some examples, in the first pass, block-based bilateral matching MV refinement is performed. Specifically, in the first pass, refined MVs are derived by applying BM to the coding block. In the dual prediction operation, similar to decoder-side motion vector refinement (DMVR), the refined MVs are searched around two initial MVs (MV0 and MV1) in the reference picture lists L0 and L1. The refined MVs (MV0_pass1 and MV1_pass1) are derived around the initial MVs based on the minimum bilateral matching cost between two reference blocks in the reference picture lists L0 and L1.

[0292] For example, BM performs a local search to derive the integer sample accuracy intDeltaMV. The local search applies a 3×3 square search pattern to loop through a search range of [-sHor, sHor] in the horizontal direction and [-sVer, sVer] in the vertical direction. The values of sHor and sVer are determined by the block dimensions, and the maximum values of sHor and sVer are 8.

[0293] In some examples, the bilateral matching cost is calculated as bilCost = mvDistanceCost + sadCost. Here, mvDistanceCost is the cost related to the distance between the potential refined MV and the initial MV of the multi-pass decoder-side motion vector refinement, and sadCost is the cost for measuring the distortion of two reference blocks according to the potential refined MV. When the block size cbW×cbH is larger than 64, the mean-removed sum of absolute differences (MRSAD) cost function is applied to remove the DC effect of the distortion between the reference blocks. When the bilCost at the center point of the 3×3 search pattern is the minimum cost, the intDeltaMV local search ends. Otherwise, the current minimum cost search point becomes the new center point of the 3×3 search pattern, and the minimum cost search continues until the center point reaches the end of the search range.

[0294] Apply existing partial sample refinement further to derive the final deltaMV. The refined MV after the first pass is derived as follows.

Number

[0295] In some examples, in the second pass, bilateral matching MV refinement based on sub - blocks is performed. For example, in the second pass, refined MVs are derived by applying BM to 16×16 grid sub - blocks. The refined MVs are searched around the two MVs (MV0_pass1 and MV1_pass1) obtained in the first pass within the reference picture lists L0 and L1 for each sub - block. The refined MVs (MV0_pass2(sbIdx2) and MV1_pass2(sbIdx2)) are derived based on the minimum bilateral matching cost between two reference sub - blocks in L0 and L1.

[0296] For each sub - block, BM performs an exhaustive search to derive the integer sample accuracy intDeltaMV. The exhaustive search has a search range of [-sHor, sHor] in the horizontal direction and [-sVer, sVer] in the vertical direction. The values of sHor and sVer are determined by the block dimension, and the maximum values of sHor and sVer are 8.

[0297] The bilateral matching cost is calculated by applying a cost factor to the sum of absolute transformed differences (SATD) cost between two reference sub - blocks. For example, bilCost = satdCost×costFactor. The search area (2×sHor + 1)×(2×sVer + 1) is divided into five diamond - shaped search areas.

[0298] Figure 30 shows a diagram (3000) of the maximum search range divided into five rhombus search regions (3010), (3020), (3030), (3040), and (3050). In some examples, each search region is assigned a costFactor determined by the distance (intDeltaMV) between each search point and the starting MV, and each rhombus region is processed in the order starting from the center of the search region. In each search region, the search points are processed in raster scan order from the upper left to the lower right corner of the region. If the minimum bilCost within the current search region is less than a threshold (e.g., equal to the area sbW × sbH of the block), the int-pel full search ends. Otherwise, the int-pel full search continues to the next search region until all search points are examined. Further, if the difference between the previous minimum cost and the current minimum cost in an iteration is less than a threshold equal to the area of the block, the search process ends.

[0299] Apply the existing VVC DMVR sub-sample refinement further to derive the final deltaMV (sbIdx2). The refined MV in the second pass is derived as follows.

Number

[0300] In some examples, in the third pass, bidirectional optical flow MV refinement based on sub-blocks is performed. For example, in the third pass, refined MVs are derived by applying BDOF to 8×8 grid sub-blocks. For each 8×8 sub-block, starting from the refined MV of the parent sub-block in the second pass, BDOF refinement is applied to derive scaled Vx and Vy without clipping. The derived bioMv (Vx, Vy) is rounded to 1 / 16 sample accuracy and clipped between -32 and 32.

[0301] The refined MVs (MV0_pass3(sbIdx3) and MV1_pass3(sbIdx3)) in the third pass are derived as follows.

Number

[0302] In some examples, affine merge by DMVR on a translational model can be used. In some examples, the basic MV (the translational part of the affine model) of the affine model of a coding block coded in affine merge mode is refined by applying the first step of multi-pass DMVR. For example, if a candidate meets the DMVR conditions, a translational MV offset is added to all the CPMVs of the candidates in the affine merge list. And the translational MV offset is derived by minimizing the same bilateral matching cost as conventional DMVR. In some examples, the same DMVR conditions can be applied.

[0303] In some examples, the MV offset search process is the same as the first pass of multi-pass DMVR of ECM. For example, using a 3×3 square search pattern, loop through a search range of [-8, +8] horizontally and [-8, +8] vertically to find the optimal integer MV offset. Next, perform half pel search around the optimal integer position, and finally perform error surface estimation to find the MV offset with 1 / 16 precision.

[0304] In some examples, the refined CPMVs are stored for both spatial and temporal motion vector prediction as multi-pass DMVR of ECM.

[0305] According to one aspect of the present disclosure, not applying DMVR or applying only DMVR for translational MV refinement for an affine-coded block may not be optimal for coding efficiency.

[0306] Aspects of the present disclosure provide a technique for refining a coding block using affine bi-prediction by applying bilateral matching motion refinement processing (e.g., DMVR or similar techniques) to a plurality of CPMVs. For ease of explanation, the bilateral matching motion refinement processing is referred to as affine bilateral matching processing, and DMVR is used as an example of bilateral matching processing to explain the affine bilateral matching processing. Note that other suitable bilateral matching processing can be used in the affine bilateral matching processing.

[0307] In some embodiments, the affine bilateral matching processing (e.g., using DMVR processing) applies MV offsets to a first control point (e.g., CPMV0 of the upper left control point of the current block) and one or more other control points (e.g., CPMV1 and / or CPMV2), and each control point can apply a different MV offset. The set of MV offset values having the minimum bilateral matching cost is used as the final set of MV offset values applied to the CPMV to obtain the final CPMV, and the final CPMV is used to generate the affine bi-prediction signal of the current block.

[0308] In some examples, in each refinement iteration, a set of MV offset values is applied to the CPMV of one reference picture list (e.g., L0), and a set of MV offset values having opposite signs (e.g., a set of MV offset values mirrored with respect to the set of MV offset values) is applied to the corresponding CPMV of the other reference picture list (e.g., L1).

[0309] In some examples, in each refinement iteration, a set of MV offset values is applied to the CPMV of one reference picture list (e.g., L0), and the MV offset values are not applied to the corresponding CPMV of the other reference picture list (e.g., L1).

[0310] In another embodiment, an affine bilateral matching process (e.g., using DMVR processing) is applied only to the non-translational part of the affine model. For example, refinement is not applied to the first control point (e.g., CPMV0 at the upper left corner) (without MV offset), and refinement (MV offset) is applied to the second control point (CPMV1 at the upper right corner) and / or the third control point (CPMV2 at the lower left corner). Each control point can have different MV offsets applied in the refinement process.

[0311] In some examples, in each refinement iteration, one set of MV offset values is applied to the selected CPMV to be refined on one reference picture list (e.g., L0), and MV offset values with opposite signs are applied to the corresponding CPMV of the other reference picture list (e.g., L1).

[0312] In some examples, in each refinement iteration, one set of MV offset values is applied to the selected CPMV to be refined on one reference picture list (e.g., L0), and the offset values are not applied to the corresponding CPMV of the other reference picture list (e.g., L1).

[0313] In some embodiments, in the affine bilateral matching process, refinement can be applied based on the affine parameters. For example, in each refinement iteration, one set of offset values of the affine parameters is applied to the affine equation of one reference picture list (e.g., L0), the refined CPMV is calculated, and one set of offset values with opposite signs of the affine parameters is applied to the affine equation of the other reference list (e.g., L1), and the refined CPMV of the reference list is calculated. The set of offset values of the affine parameters having the minimum bilateral matching cost is used to generate the final CPMV of the current block, and the final CPMV is used to generate the affine bi-prediction signal of the current block.

[0314] In some examples, for a 6-parameter affine, the affine parameters to be refined can be one or more of a, b, c, d, e, f, as in Equation (4).

[0315] (For example, using DMVR processing) The affine bidirectional matching process searches for the best parameter offsets, e.g., Δa, Δb, Δc, Δd, Δe, Δf for the affine parameters on L0. As a result, after the refinement process, Equation (4) at L0 can be represented by Equation (50). [Number] In the reference picture list L1, the reverse offset value is applied, and the affine equation is represented by Equation (51). [Number] Equations (50) and (51) are used to generate the affine CPMV of the current block at L0 and L1, respectively.

[0316] In some examples, for a 4-parameter affine, the affine parameters to be refined can be one or more of a, b, c, f, as shown in Equation (2).

[0317] Note that when using other affine models, the refinement process can be appropriately applied to other appropriate parameters in other affine models.

[0318] In one embodiment, the affine bilateral matching process (e.g., using DMVR processing) is applied to the affine merge candidates. In another embodiment, the affine bilateral matching process (e.g., using DMVR processing) is applied to the affine MMVD candidates.

[0319] In some embodiments, in the affine bilateral matching process, if there are a plurality of offsets to be optimized by DMVR (e.g., a plurality of control points to be optimized, or a plurality of affine parameters to be optimized), the plurality of offsets can be optimized in order.

[0320] In some examples, if there are a plurality of control points to be optimized by DMVR (e.g., in a predefined order), the control points are optimized one by one using the DMVR bilateral matching cost.

[0321] In some examples, if there are a plurality of affine parameters to be optimized by DMVR (e.g., in a predefined order), the affine parameters are optimized one by one using the DMVR bilateral matching cost.

[0322] In some examples, in the predefined order of a plurality of offsets to be optimized by DMVR, if the difference in the DMVR bilateral matching costs of different candidate offsets is within a given threshold of the current offset to be optimized, the next offset to be optimized in order is skipped. For example, if the DMVR bilateral matching costs of different candidate offsets for the first CPMV are very close to each other, the DMVR optimization is skipped for other control points after the first CPMV.

[0323] Some aspects of the present disclosure also provide techniques for simplifying the affine motion compensation (MC) and / or bilateral matching cost calculation used during the affine bilateral matching process (e.g., using DMVR processing) to reduce the impact on computational complexity.

[0324] In some embodiments, during affine bilateral matching processing, the affine MC is applied only to a subset of sub-blocks. Further, the bilateral matching cost is also calculated based on the corresponding subsets of sub-blocks on the prediction output for each reference picture list.

[0325] In some examples, the subset of sub-blocks is selected based on a predetermined sub-block position.

[0326] In one example, the set of predetermined sub-blocks includes the sub-block at the central position (W / 2, H / 2) of an affine block (e.g., a block in affine mode), where W represents the width of the affine block and H represents the height of the affine block.

[0327] FIG. 31 is a diagram of a current block (3100) in affine mode during affine bilateral matching processing. In the example of FIG. 31, the set of predetermined sub-block positions includes the sub-block at the center position of the current block, as indicated by the shaded sub-blocks in FIG. 31.

[0328] In another example, the set of predetermined sub-blocks includes the four corners of the current block.

[0329] FIG. 32 is a diagram of a current block (3200) in affine mode during affine bilateral matching processing. In the example of FIG. 32, the set of predetermined sub-blocks includes four sub-blocks at each of the four corners of the current block, as indicated by the shaded sub-blocks in FIG. 32.

[0330] In another example, the set of predetermined sub-blocks includes four sub-blocks at each of the four corners of the current block and another sub-block at the center position of the current block.

[0331] FIG. 33 is a diagram of the current block (3300) in the affine mode during the affine bilateral matching process. In the example of FIG. 33, a set of predetermined sub-blocks includes four sub-blocks at each of the four corners of the current block, and another sub-block at the center position of the current block as shown by the shaded sub-block in FIG. 33.

[0332] In some examples, the set of predetermined sub-blocks used for MC and bilateral matching cost calculation can correspond to a checkerboard subsampling pattern of the sub-blocks of the affine block. In one example, the sub-blocks of the checkerboard pattern can have the size of N×N of the affine sub-blocks (e.g., the affine sub-blocks correspond to 4×4 luma sub-blocks), and N can be 1, 2, 4, etc.

[0333] FIG. 34 is a diagram of the current block (3400) in the affine mode during the affine bilateral matching process. In the example of FIG. 34, the set of predetermined sub-blocks includes sub-blocks of a checkerboard pattern such as the shaded sub-blocks or the white sub-blocks in FIG. 34.

[0334] In some examples, the affine sub-blocks (e.g., the affine sub-blocks correspond to 4×4 luma sub-blocks) are subsampled with coefficients of M and N. In an example of subsampling, the sub-blocks are selected every M sub-block columns in the horizontal direction and every N sub-block rows in the vertical direction. In one example, both M and N are equal to 1. In another example, both M and N are equal to 2.

[0335] FIG. 35 is a diagram of the current block (3500) in the affine mode during the affine bilateral matching process. In the example of FIG. 35, the set of predetermined sub-blocks is selected according to the subsampled affine sub-blocks every two sub-block columns in the horizontal direction and every two sub-block rows in the vertical direction as shown by the shaded affine sub-blocks in FIG. 35.

[0336] In some examples, N×N affine sub - blocks (e.g., the affine sub - blocks correspond to 4×4 luma sub - blocks) aligned with the center position of the current block can be used for MC and bilateral matching cost calculation. For example, N can be 2, 4, etc.

[0337] FIG. 36 is a diagram of the current block (3600) in affine mode during affine bilateral matching processing. In the example of FIG. 36, a set of predetermined sub - blocks includes 2×2 affine sub - blocks aligned with the center position of the current block, as indicated by the shaded affine sub - blocks in FIG. 36.

[0338] In some embodiments, the affine MC processing is used in the affine bilateral matching processing to determine a refinement (e.g., using DMVR processing) for optimizing the affine motion compensation for the current block, and is also used in the final affine motion compensation (e.g., optimized affine motion compensation) for reconstructing the current block. The affine MC processing can use different sub - block sizes in the affine bilateral matching processing to determine the refinement and the final affine motion compensation (MC). For example, the final affine motion compensation can use the original affine sub - block size such as 4×4 luma samples. The affine bilateral matching processing can use a larger affine sub - block size to reduce the calculation. In some examples, the affine sub - block size for motion compensation in the affine bilateral matching processing may be the original affine sub - block size multiplied by N, and N can be made equal to 2, 4, etc. Using a larger affine sub - block size can reduce the calculation.

[0339] In some embodiments, during affine bilateral matching processing (e.g., using DMVR processing), the affine MC interpolation filter can be changed to a simple interpolation filter. For example, in affine bilateral matching processing (e.g., using DMVR processing), a bilinear interpolation filter can be used for affine MC.

[0340] FIG. 37 shows a flowchart illustrating an overview of a process (3700) according to an embodiment of the present disclosure. The process (3700) can be used in a video encoder. In various embodiments, the process (3700) is executed by a processing circuit in terminal devices (310), (320), (330), and (340), a processing circuit that executes the functions of the video encoder (403), a processing circuit that executes the functions of the video decoder (603), a processing circuit that executes the functions of the video decoder (703), and the like. In some embodiments, the process (3700) is implemented by software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit executes the process (3700). The process starts at (S3701) and proceeds to (S3710).

[0341] (S3710), refinement based on affine bilateral matching used on the decoder side for affine motion compensation prediction of blocks in the affine mode is determined.

[0342] (S3720), a signal indicating the refinement based on affine bilateral matching for affine motion compensation prediction is encoded into the bitstream that carries the video.

[0343] In some examples, the signal may be a block-level signal or a high-level signal such as a sequence-level signal, a picture-level signal, a slice-level signal, a tile-level signal, a tile group-level signal, etc. In some examples, refinement based on affine bilateral matching is always performed for blocks in the affine mode. Thus, the signal is not necessary.

[0344] Next, the process proceeds to (S3799) and ends.

[0345] The process (3700) can be appropriately adapted. The steps of the process (3700) can be changed and / or omitted. Additional steps can be added. Any appropriate implementation order can be used.

[0346] FIG. 38 shows a flowchart illustrating an overview of a process (3800) according to an embodiment of the present disclosure. The process (3800) can be used in a video decoder. In various embodiments, the process (3800) is executed by a processing circuit such as a processing circuit in terminal devices (310), (320), (330), and (340), a processing circuit that executes the functions of video decoder (410), a processing circuit that executes the functions of video decoder (510), a processing circuit that executes the functions of video decoder (633), and the like. In some embodiments, the process (3800) is implemented by software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit executes the process (3800). The process starts at (S3801) and proceeds to (S3810).

[0347] (S3810), an affine refinement offset for affine motion compensation prediction for the current block in the current picture is determined. The current block is in affine mode, and the affine motion compensation prediction is associated with a first reference picture in a first reference picture list and a second reference picture in a second reference picture list.

[0348] (S3820), according to the affine refinement offset and the affine motion compensation prediction, a first reference portion in the first reference picture of the current block and a second reference portion in the second reference picture of the current block are determined.

[0349] (S3830), the distortion between the first reference portion and the second reference portion is calculated. For optimization to minimize the distortion, an affine refinement offset is determined.

[0350] In (S3840), the current block is reconstructed according to the affine refinement offset and the affine motion compensation prediction.

[0351] In some embodiments, the affine motion compensation prediction is based on a first plurality of control point motion vectors of the current block associated with a first reference picture and a second plurality of control point motion vectors associated with a second reference picture. To determine the affine refinement offset, in some examples, for the first plurality of control point motion vectors of the current block associated with the first reference picture, a first plurality of motion vector refinement offset values are determined. The first plurality of motion vector refinement offset values are not the same, and at least one motion vector refinement offset value is different from another motion vector refinement offset value.

[0352] In some examples, a second plurality of motion vector refinement offset values for the second plurality of control point motion vectors associated with the second reference picture are generated based on the first plurality of motion vector refinement offset values, and the second plurality of motion vector refinement offset values have signs opposite to those of the first plurality of motion vector refinement offset values.

[0353] In some examples, the first plurality of motion vector refinement offset values include a value of 0 for application to the first control point motion vector of the first control point at the upper right corner of the current block.

[0354] In some embodiments, the affine motion compensation prediction is based on a first affine parameter of a current block associated with a first reference picture and a second affine parameter of the current block associated with a second reference picture. In some examples, to determine an affine refinement offset, a first set of affine parameter refinement offset values for the first affine parameter of the current block associated with the first reference picture is determined. In some examples, a second set of affine parameter refinement offset values for the second affine parameter of the current block associated with the second reference picture is generated according to the first set of affine parameter refinement offset values. The second set of affine parameter refinement offset values has a sign opposite to that of the first set of affine parameter refinement offset values.

[0355] In some examples, the first affine parameter and the second affine parameter each correspond to a parameter of a four-parameter affine model. In some examples, the first affine parameter and the second affine parameter each correspond to a parameter of a six-parameter affine model.

[0356] In some examples, the affine motion compensation prediction includes an affine merge candidate. In some examples, the affine motion compensation prediction includes a merge motion vector difference (MMVD) candidate.

[0357] In some examples, affine motion compensation prediction includes at least a first parameter and a second parameter for refinement. At least the first parameter and the second parameter are optimized one by one sequentially. For example, a first refinement offset value for applying a first refinement to the first parameter is determined. The first refinement updates the affine motion compensation prediction to a first intermediate affine motion compensation prediction that minimizes distortion based on the first refinement for the first parameter. Next, a second refinement offset value for applying a second refinement to the second parameter is determined. The second refinement updates the first intermediate affine motion compensation prediction to a second intermediate affine motion compensation prediction that minimizes distortion based on the second refinement for the second parameter.

[0358] In one example, at least the first parameter and the second parameter for refinement include a first control point motion vector of a first control point and a second control point motion vector of a second control point.

[0359] In another example, at least the first parameter and the second parameter for refinement include a first affine parameter and a second affine parameter of an affine model.

[0360] In some examples, when the reduction of distortion based on the first refinement is smaller than a threshold. Optimization based on other parameters can be skipped. The first intermediate affine motion compensation prediction is set to the final affine motion compensation prediction. The current block is reconstructed according to the first intermediate affine motion compensation prediction set to the final affine motion compensation prediction.

[0361] In some examples, the first reference portion is a first reference block in a first reference picture corresponding to the current block, and the second reference portion is a second reference block in a second reference picture corresponding to the current block.

[0362] In some examples, the first reference portion includes a first subset of reference sub-blocks in a first reference picture corresponding to a subset of sub-blocks within the current block, and the second reference portion includes a second subset of reference sub-blocks in a second reference picture corresponding to a subset of sub-blocks within the current block.

[0363] In one example, the subset of sub-blocks within the current block includes the sub-block at the center position of the current block.

[0364] In another example, the subset of sub-blocks within the current block includes one or more sub-blocks at one or more corners of the current block.

[0365] In another example, the subset of sub-blocks within the current block includes sub-blocks of a checkerboard pattern within the current block.

[0366] In another example, the subset of sub-blocks within the current block includes sub-sampled affine sub-blocks.

[0367] In another example, the subset of sub-blocks within the current block includes N×N affine sub-blocks, the center of the N×N affine sub-blocks coincides with the center of the current block, and N is a positive integer.

[0368] In some examples, the first affine motion compensation sub-block size is used during optimization to minimize distortion. The current block is reconstructed using a second affine motion compensation sub-block size. The first affine motion compensation sub-block size is larger than the second affine motion compensation sub-block size.

[0369] In some examples, a simple interpolation filter such as a bilinear interpolation filter is used during optimization to minimize distortion.

[0370] Next, the process proceeds to (S3899) and ends.

[0371] The process (3800) can be appropriately adapted. The steps of the process (3800) can be changed and / or omitted. Additional steps can be added. Any suitable implementation order can be used.

[0372] FIG. 39 shows a flowchart illustrating an overview of a process (3900) according to an embodiment of the present disclosure. The process (3900) can be used in a video decoder. In various embodiments, the process (3900) is executed by a processing circuit such as a processing circuit in terminal devices (310), (320), (330), and (340), a processing circuit that executes the functions of video decoder (410), a processing circuit that executes the functions of video decoder (510), a processing circuit that executes the functions of video decoder (633), and the like. In some embodiments, the process (3900) is implemented by software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit executes the process (3900). The process starts at (S3901) and proceeds to (S3910).

[0373] (S3910), the coded video bitstream includes the current picture. The current picture includes the current block.

[0374] (S3920), based on the decoded information from the coded video bitstream, the current block in the current picture is determined in the affine bi-prediction mode. The current block is predicted by a plurality of control point motion vectors.

[0375] (S3930), bilateral matching motion refinement is applied to refine the plurality of control point motion vectors.

[0376] (S3940), the current block is reconstructed according to the refined plurality of control point motion vectors associated with the current block.

[0377] In some examples, to refine a plurality of control point motion vectors, a first set of motion vector refinement offset values for the first plurality of control point motion vectors of a current block associated with a first reference picture is determined. The first set of motion vector refinement offset values are not the same value. At least, the motion vector refinement offset values among the first set of motion vector refinement offset values are different from others. Further, a second set of motion vector refinement offset values for a second plurality of control point motion vectors associated with a second reference picture is determined according to the first set of motion vector refinement offset values. The second set of motion vector refinement offset values have signs opposite to those of the first set of motion vector refinement offset values.

[0378] In one example, the first set of motion vector refinement offset values includes a value of 0 for applying to the first control point motion vector of a first control point at the upper right corner of the current block.

[0379] In some examples, to refine a plurality of control point motion vectors, a first set of affine parameter refinement offset values for the first affine parameters of a first affine model of a current block associated with a first reference picture is determined. Further, a second set of affine parameter refinement offset values for the second affine parameters of a second affine model of the current block associated with a second reference picture is generated according to the first set of affine parameter refinement offset values, and the second set of affine parameter refinement offset values has a sign opposite to that of the first set of affine parameter refinement offset values. In one example, the first affine model and the second affine model are four-parameter affine models. In another example, the first affine model and the second affine model are six-parameter affine models.

[0380] In one example, a plurality of control point motion vectors are determined based on an affine merge candidate. In another example, the motion vectors of a plurality of control points are determined based on an affine merge motion vector difference (MMVD) candidate.

[0381] In some examples, to refine a plurality of control point motion vectors, a first refinement offset value for applying a first refinement to a first parameter is determined, and the first refinement updates the plurality of control point motion vectors to a first intermediate refined plurality of control point motion vectors that minimize a bilateral matching cost based on the first refinement with respect to the first parameter. Further, a second refinement offset value for applying a second refinement to a second parameter is determined. The second refinement updates the first intermediate refined plurality of control point motion vectors to a second intermediate refined plurality of control point motion vectors that minimize a bilateral matching cost based on the second refinement with respect to the second parameter.

[0382] In one example, the first parameter and the second parameter for refinement correspond to a first control point motion vector of a first control point and a second control point motion vector of a second control point.

[0383] In another example, the first parameter and the second parameter for refinement correspond to a first affine parameter and a second affine parameter of an affine model.

[0384] In one example, if a reduction in the bilateral matching cost based on the first refinement is less than a threshold, determination of a second refinement offset value for applying a second refinement to the second parameter is skipped. The current block is reconstructed based on the first intermediate refined plurality of control point motion vectors.

[0385] In some examples, in response to refinement of a plurality of control point motion vectors, affine motion compensation is applied to sub-blocks of a current block, and a first reference sub-block in a first reference picture and a second reference sub-block in a second reference picture corresponding to the sub-blocks of the current block are determined. Next, a bilateral matching cost is calculated based on the distortion between the first reference sub-block and the second reference sub-block.

[0386] In some embodiments, in response to refinement of a plurality of control point motion vectors, affine motion compensation is applied to a subset of sub-blocks of a current block, and a first subset of reference sub-blocks in a first reference picture and a second subset of reference sub-blocks in a second reference picture corresponding to the subset of sub-blocks of the current block are determined. Next, a bilateral matching cost is calculated based on the distortion between the first subset of reference sub-blocks and the second subset of reference sub-blocks.

[0387] In one example, the subset of sub-blocks within the current block includes the sub-block at the center position of the current block.

[0388] In another example, the subset of sub-blocks within the current block includes one or more sub-blocks at one or more corners of the current block.

[0389] In another example, the subset of sub-blocks within the current block includes sub-blocks of a checkerboard pattern within the current block.

[0390] In another example, the subset of sub-blocks within the current block includes subsampled affine sub-blocks.

[0391] In another example, the subset of sub-blocks within the current block includes N×N affine sub-blocks (for example, the affine sub-blocks include 4×4 luma samples), the center of the N×N affine sub-blocks coincides with the center of the current block, and N is a positive integer.

[0392] In some examples, a first affine motion compensation sub-block size is used during bilateral matching motion refinement processing. The current block is reconstructed using a second affine motion compensation sub-block size. The first affine motion compensation sub-block size is larger than the second affine motion compensation sub-block size.

[0393] In some examples, a simple interpolation filter, such as a bilinear interpolation filter, is used during bilateral matching motion refinement processing.

[0394] Next, the process proceeds to (S3999) and ends.

[0395] The process (3900) can be appropriately adapted. The steps of the process (3900) can be changed and / or omitted. Additional steps can be added. Any suitable implementation order can be used.

[0396] The above-described techniques can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, FIG. 40 shows a computer system (4000) suitable for implementing a particular embodiment of the subject matter of the present disclosure.

[0397] The computer software can be processed by mechanisms such as assembly, compilation, linking, etc. to generate code including instructions executable directly or through interpretation, microcode execution, etc. by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc. and can be coded using any suitable machine code or computer language.

[0398] The instructions can be executed on various computers or components thereof including, for example, personal computers, tablet computers, servers, smartphones, game devices, Internet of Things devices, etc.

[0399] The components shown in FIG. 40 of the computer system (4000) are illustrative only and do not imply any limitation as to the use or functionality of the computer software implementing the embodiments of the present disclosure. Further, the component configuration should not be construed as having any dependencies or requirements related to any one or combination of the components shown in the exemplary embodiments of the computer system (4000).

[0400] The computer system (4000) may include a specific human interface input device. Such a human interface input device may respond to input by one or more human users through, for example, sensory input (e.g., keystrokes, swipes, data glove operations), voice input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not shown). The human interface device can also be used to capture certain media that is not necessarily directly related to conscious input by a human, such as voice (e.g., conversation, music, ambient sound), images (e.g., scanned images, photographic images obtained from a digital camera), video (e.g., including two-dimensional video, three-dimensional video, stereoscopic video).

[0401] The input human interface device may include one or more of a keyboard (4001), a mouse (4002), a trackpad (4003), a touch screen (4010), a data glove (not shown), a joystick (4005), a microphone (4006), a scanner (4007), a camera (4008) (only one of which is shown).

[0402] The computer system (4000) may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through sensory output, sound, light, and smell / taste. Such human interface output devices may include sensory output devices (e.g., touch screen (4010), data glove (not shown), or joystick (4005 (sensory feedback, but there may also be sensory feedback devices that do not function as input devices)), voice output devices (e.g., speaker (4009), headphones (not shown), visual output devices (e.g., screen (4010), CRT screen, LCD screen, plasma screen, OLED screen, each with or without touch screen input capability, each with or without sensory feedback capability, some of which may output 2D visual output or output of more than three dimensions, for example, through means such as stereoscopic output, virtual reality glasses (not shown), holographic display, and smoke agent tank (not shown), and printer (not shown))).

[0403] The computer system (4000) may also include a human-accessible storage device and related media such as an optical medium including a CD / DVD ROM / RW (4020) with a medium (4021) such as a CD / DVD, a thumb drive (4022), a removable hard drive or solid state drive (4023), legacy magnetic media such as tapes and floppy disks (not shown), and devices based on dedicated ROM / ASIC / PLD such as security dongles (not shown).

[0404] One of ordinary skill in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not include transmission media, carrier waves, or other transient signals.

[0405] The computer system (4000) may also include an interface (4054) to one or more communication networks (4055). The network can be, for example, wireless, wired, or optical. The network can further be local, wide area, metropolitan area, vehicle and industrial, real-time, delay-tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, terrestrial broadcast TV, vehicle and industrial including CANBus, etc. A particular network generally requires an external network interface attached to a particular general-purpose data port or peripheral device bus (4049) (e.g., the USB port of the computer system (4000)). Others are generally integrated into the core of the computer system (4000) by attachment to a system bus as described later (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using these networks, the computer system (4000) can communicate with other entities. Such communication can be only unidirectional reception (e.g., broadcast TV), only unidirectional transmission (e.g., CANbus to a particular CANbus device), or bidirectional to other computer systems using, for example, local or wide area digital networks. Particular protocols and protocol stacks can be used for each of the networks and network interfaces described above.

[0406] The aforementioned human interface device, human-accessible storage device, and network interface can be attached to the core (4040) of the computer system (4000).

[0407] The core (4040) may include one or more central processing units (CPUs) (4041), a graphics processing unit (GPU) (4042), a dedicated programmable processing unit in the form of an FPGA (4043), a hardware accelerator for specific tasks (4044), a graphics adapter (4050), and the like. These devices may be connected through a system bus (4048) together with a built-in mass storage device (4047) such as a read-only memory (ROM) (4045), a random access memory (4046), an internal hard drive inaccessible to users, an SSD, and the like. In some computer systems, the system bus (4048) is accessible in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, and the like. Peripheral devices can be attached directly to the system bus (4048) of the core or through a peripheral device bus (4049). In the example, the screen (4010) can be connected to the graphics adapter (4050). The architecture of the peripheral device bus includes PCI, USB, and the like.

[0408] The CPU (4041), GPU (4042), FPGA (4043), and accelerator (4044) can be combined to execute specific instructions capable of generating the aforementioned computer code. The computer code can be stored in the ROM (4045) or the RAM (4046). Temporary data can also be stored in the RAM (4046), while permanent data can be stored, for example, in the built-in mass storage device (4047). Fast storage and reading to any of the memory devices can be enabled through the use of a cache memory that can be closely associated with one or more of the CPU (4041), GPU (4042), mass storage device (4047), ROM (4045), RAM (4046), and the like.

[0409] A computer-readable medium may have computer code for performing operations implemented by various computers. The medium and the computer code may be specially designed and configured for the purposes of the present disclosure or may be of the kind well known and available to those skilled in the field of computer software.

[0410] By way of example and not limitation, a computer system (4000) having an architecture, and specifically a core (4040), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied within one or more tangible computer-readable media. Such computer-readable media can be a specific storage device of the core (4040) with non-transitory characteristics such as an on-core mass storage device (4047) or ROM (4045), and media associated with a user-accessible mass storage device as described above. The software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (4040). The computer-readable media can include one or more memory devices or chips as required by a particular need. The software can cause the core (4040) and specifically the processor (including a CPU, GPU, FPGA, etc.) therein to execute a specific process or a specific part of a specific process described herein, including the definition of a data structure stored in a RAM (4046) and the modification of the data structure according to the process defined by the software. Additionally or alternatively, the computer system can provide functionality as a result of an implementation (e.g., an accelerator (4044)) within logic hardwired or other circuitry operable with or instead of the software to execute a specific process or a specific part of a specific process described herein. References to software include logic and, where appropriate, vice versa. References to computer-readable media can, where appropriate, include circuitry (such as an integrated circuit (IC)) for storing software for execution, circuitry for implementing logic for execution, or both. The present disclosure includes any suitable combination of hardware and software. Appendix A: Glossary JEM: joint exploration model VVC: versatile video coding BMS: benchmark set MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Supplementary Enhancement Information VUI: Video Usability Information GOPs: Groups of Pictures TUs: Transform Units, PUs: Prediction Units CTUs: Coding Tree Units CTBs: Coding Tree Blocks PBs: Prediction Blocks HRD: Hypothetical Reference Decoder SNR: Signal Noise Ratio CPUs: Central Processing Units GPUs: Graphics Processing Units CRT: Cathode Ray Tube LCD: Liquid-Crystal Display OLED: Organic Light-Emitting Diode CD: Compact Disc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application-Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile communications LTE: Long-Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Areas SSD: solid-state drive IC: Integrated Circuit CU: Coding Unit

[0411] Although the present disclosure has described several exemplary embodiments, there are alternatives, substitutions, and various equivalents of substitutions that are encompassed within the scope of the present disclosure. It will be apparent to those skilled in the art that, although not explicitly shown or described herein, many systems and methods can be devised that implement the principles of the present disclosure and are thus included within the spirit and scope of the present disclosure.

Claims

1. A method for video processing in a decoder, comprising: receiving a coded video bitstream including a current picture, wherein the current picture includes a current block; determining, based on decoded information from the coded video bitstream, that the current block in the current picture is coded in an affine dual prediction mode, wherein the current block is predicted by a plurality of control point motion vectors; refining the plurality of control point motion vectors by applying bilateral matching motion refinement; reconstructing the current block according to the refined plurality of control point motion vectors associated with the current block; A method comprising the steps of.

2. The step of refining the plurality of control point motion vectors comprises: determining a first set of motion vector refinement offset values for a first plurality of control point motion vectors of the current block associated with a first reference picture, wherein the first set of motion vector refinement offset values are different, further comprising the steps of, the method according to claim 1.

3. determining a second set of motion vector refinement offset values for a second plurality of control point motion vectors of a second reference picture according to the first set of motion vector refinement offset values, wherein the second set of motion vector refinement offset values have signs opposite to those of the first set of motion vector refinement offset values, The method according to claim 2, further comprising the steps of.

4. The method according to claim 2, wherein the first set of motion vector refinement offset values includes a value of 0 for application to a first control point motion vector of a first control point at the upper right corner of the current block.

5. The step of refining the plurality of control point motion vectors comprises: determining a first set of affine parameter refinement offset values for a first affine parameter of a first affine model of the current block associated with a first reference picture, the method according to claim 1.

6. Generating a second set of affine parameter refinement offset values for second affine parameters of a second affine model of the current block associated with a second reference picture according to the first set of affine parameter refinement offset values, wherein the second set of affine parameter refinement offset values has a sign opposite to that of the first set of affine parameter refinement offset values; The method according to claim 5, further comprising. **Claim 7** The method according to claim 6, wherein the first affine model and the second affine model are four-parameter affine models or six-parameter affine models. **Claim 8** The plurality of control point motion vectors are an affine merge candidate, an affine merge motion vector difference (MMVD) candidate, and are determined based on at least one of them. The method according to claim 1. **Claim 9** The step of refining the plurality of control point motion vectors includes determining a first refinement offset value for applying a first refinement to a first parameter, wherein the first refinement updates the plurality of control point motion vectors to a first intermediate refined plurality of control point motion vectors that minimizes a bilateral matching cost based on the first refinement with respect to the first parameter; determining a second refinement offset value for applying a second refinement to a second parameter, wherein the second refinement updates the first intermediate refined plurality of control point motion vectors to a second intermediate refined plurality of control point motion vectors that minimizes the bilateral matching cost based on the second refinement with respect to the second parameter; The method according to claim 1, further comprising. **Claim 10** The method according to claim 9, wherein at least the first parameter and the second parameter for refinement include a first control point motion vector of a first control point and a second control point motion vector of a second control point. **Claim 11** The method according to claim 9, wherein at least the first parameter and the second parameter for refinement include a first affine parameter and a second affine parameter of an affine model. **Claim 12** Determining that a decrease in the bilateral matching cost based on the first refinement is less than a threshold; skipping the determination of the second refinement offset value for applying the second refinement to the second parameter; reconstructing the current block based on the first intermediate refined plurality of control point motion vectors; 10. The method of claim 9 further comprising:

13. The step of refining the plurality of control point motion vectors comprises: applying affine motion compensation to sub-blocks of the current block in response to refining the plurality of control point motion vectors to determine a first reference sub-block in a first reference picture and a second reference sub-block in a second reference picture that correspond to the sub-blocks of the current block; calculating a bilateral matching cost based on the distortion between the first reference sub-block and the second reference sub-block; The method of claim 1 , comprising:

14. The step of refining the plurality of control point motion vectors comprises: applying affine motion compensation to a subset of sub-blocks of the current block in response to refining the plurality of control point motion vectors to determine a first subset of reference sub-blocks in a first reference picture and a second subset of reference sub-blocks in a second reference picture corresponding to the subset of sub-blocks of the current block; calculating a bilateral matching cost based on the distortion between the first subset of reference sub-blocks and the second subset of reference sub-blocks; The method of claim 1 , comprising:

15. The method of claim 14 , wherein the subset of sub-blocks within the current block includes a sub-block that is centrally located in the current block.

16. The method of claim 14 , wherein the subset of sub-blocks within the current block includes one or more sub-blocks at one or more corners of the current block.

17. The method of claim 14 , wherein the subset of sub-blocks within the current block comprises a checkerboard pattern of sub-blocks within the current block.

18. The method of claim 14 , wherein the subset of sub-blocks in the current block comprises sub-sampled affine sub-blocks.

19. The method according to claim 14, wherein the subset of sub-blocks within the current block includes N×N affine sub-blocks, the center of the N×N affine sub-blocks coincides with the center of the current block, and N is a positive integer.

20. Using a first affine motion compensation sub-block size during the bilateral matching motion refinement; Reconstructing the current block using a second affine motion compensation sub-block size, wherein the first affine motion compensation sub-block size is larger than the second affine motion compensation sub-block size; The method according to claim 1, further comprising.

21. A method for video processing in an encoder, comprising: Generating a coded video bitstream including a current picture, the current picture including a current block; Including, The coded video bitstream includes information indicating that the current block in the current picture is coded in an affine bi-prediction mode, the current block being predicted by a plurality of control point motion vectors, The plurality of control point motion vectors being refined by applying bilateral matching motion refinement; The current block being reconstructed according to the refined plurality of control point motion vectors associated with the current block.

Images