Method of video processing executed by decoder and method of video processing executed by encoder

Abstract

Aspects of the present disclosure provide methods and apparatuses for video encoding / decoding. In some examples, an apparatus for video decoding includes a processing circuit. The processing circuit receives a video bitstream including a current block and determines a candidate list including a plurality of motion vector (MV) candidates for predicting the current block. The processing circuit modifies a plurality of MV candidates in the candidate list according to a diversity evaluation based on motion information of the plurality of MV candidates. Modifying the plurality of MV candidates prioritizes a subset of MV candidates having a larger difference in motion information more. The processing circuit decodes an index indicating a selected MV candidate from the modified candidate list from the bitstream and reconstructs the current block based on the selected MV candidate.

Description

Technical Field

[0001] The present disclosure describes embodiments generally related to video coding.

Background Art

[0002] The description of the background provided herein is for the purpose of generally presenting the background of the present disclosure. The research of the presently named inventors, to the extent that the research is described in this background section and aspects of the description that might otherwise be eligible as prior art at the time of filing, is not admitted as prior art to the present disclosure, either explicitly or implicitly.

[0003] Uncompressed digital images and / or videos can include a sequence of pictures, each picture having, for example, spatial dimensions of 1920×1080 luminance samples and associated chrominance samples. The sequence of pictures can have, for example, a fixed or variable picture rate of 60 pictures per second, i.e., 60 Hz (also commonly known as the frame rate). Uncompressed images and / or videos have specific bitrate requirements. For example, 1080p60 4:2:0 video at 8 bits per sample (1920×1080 luminance sample resolution at a frame rate of 60 Hz) requires a bandwidth close to 1.5 Gbit / s. One hour of such video requires more than 600 Gbytes of storage space.

[0004] One purpose of the encoding and decoding of images and / or videos can be the reduction of redundancy in the input image and / or video signal by compression. Compression can help reduce the above bandwidth and / or storage space requirements by more than a factor of two in some cases. Although the description in this specification uses video encoding / decoding as an example, the same techniques can be applied to image encoding / decoding in a similar manner without departing from the spirit of the present disclosure. Both lossless compression and lossy compression, as well as combinations thereof, can be used. Lossless compression refers to a technique in which an exact copy of the original signal can be reconstructed from the compressed original signal. When using lossy compression, the reconstructed signal may not be the same as the original signal, but the distortion between the original signal and the reconstructed signal is small enough to make the reconstructed signal useful for the intended application. In the case of video, lossy compression is widely used. The amount of acceptable distortion depends on the application. For example, a user of a particular consumer streaming application may tolerate higher distortion than a user of a television distribution application. The achievable compression ratio can reflect the fact that higher acceptable / tolerable distortion can result in a higher compression ratio.

[0005] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transform processing, quantization, and entropy coding.

[0006] Video coding technology can include techniques known as intra coding. In intra coding, sample values are represented without reference to samples from previously reconstructed reference pictures or other data. In some video coders, a picture is spatially subdivided into blocks of samples. If all blocks of samples are coded in an intra mode, that picture can be an intra picture. Intra pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and thus can be used as the first picture of a coded video bitstream and video session or as a still image. Samples of an intra block can be subject to transformation, and the transform coefficients can be quantized prior to entropy coding. Intra prediction can be a technique for minimizing sample values in a pre-transformation region. In some cases, the smaller the post-transformation DC value and the smaller the AC coefficients, the fewer bits are required with a given quantization step size to represent the block after entropy coding.

[0007] For example, conventional intra coding used in MPEG-2 generation coding technology does not use intra prediction. However, some newer video compression technologies include techniques that attempt to perform prediction, for example, based on surrounding sample data and / or metadata obtained during encoding and / or decoding of data. Such techniques are hereinafter referred to as "intra prediction" techniques. Note that in at least some cases, intra prediction uses only reference data from the current picture being reconstructed and does not use reference pictures.

[0008] There can be various forms of intra prediction. When more than one such technique can be used with a given video coding technique, the particular technique in use can be coded as a particular intra prediction mode that uses that particular technique. In some cases, the intra prediction mode can have sub - modes and / or parameters, and the sub - modes and / or parameters can be coded independently or can be included in the mode codeword. This defines the prediction mode being used. Which codeword should be used for a given mode, sub - mode, and / or parameter combination can affect the coding efficiency gain through intra prediction, so entropy coding techniques can be used to convert the codeword into a bitstream.

[0009] A particular mode of intra prediction was introduced by H.264, refined in H.265, and further refined in newer coding techniques such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). The predictor block can be formed using the adjacent sample values of already available samples. The sample values of the adjacent samples are copied into the predictor block according to the direction. The reference to the direction in use can be coded in the bitstream or can itself be predicted.

[0010] Referring to Figure 1A, in the lower right, a subset of 9 predictor directions known from 33 possible predictor directions of H.265 (corresponding to 33 angular modes out of 35 intra modes) is represented. The point (101) where the arrows converge corresponds to the sample being predicted. The arrows represent the direction in which the sample is being predicted. For example, arrow (102) indicates that sample (101) is predicted from one or more samples at an angle of 45 degrees from the horizontal and to the upper right. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples at an angle of 22.5 degrees from the horizontal and to the lower left of sample (101).

[0011] Still referring to FIG. 1A, in the upper left, a square block (104) of 4×4 samples (indicated by the thick dashed line) is shown. The square block (104) contains 16 samples, and each sample is labeled using "S", its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample from the top in the Y dimension and the first sample from the left in the X dimension. Similarly, sample S44 is the fourth sample within block (104) in both the Y and X dimensions. Since the block is 4×4 samples in size, S44 is in the lower right. Further, reference samples following a similar numbering scheme are shown. The reference samples are labeled for block (104) using "R", its Y position (e.g., row index), and its X position (column index). In both H.264 and H.265, the predicted samples are adjacent to the block being reconstructed, and thus negative values need not be used.

[0012] Intra-picture prediction can work by copying the reference sample value from adjacent samples according to the prediction direction signaled by the signal. For example, assume that the coded video bitstream includes signaling indicating that for this block, the prediction direction matches arrow (102), i.e., the sample is predicted from one or more prediction samples at a 45-degree angle from the horizontal and in the upper right. In that case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0013] In some cases, the values of multiple reference samples may be combined, for example, through interpolation, to calculate the reference sample, especially when the direction is not equally divisible at 45 degrees.

[0014] The number of possible directions has been increasing as video coding technology develops. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013). Currently, JEM / VVC / BMS can support up to 65 directions. Experiments have been conducted to identify the most likely directions, and certain techniques in entropy coding are used to represent those likely directions with fewer bits while accepting some penalty for less likely directions. Furthermore, the direction itself can sometimes be predicted from adjacent directions used in adjacent, already decoded blocks.

[0015] FIG. 1B shows a schematic diagram (110) representing 65 intra prediction directions by JEM to illustrate the number of prediction directions increasing over time.

[0016] The mapping of intra prediction direction bits representing directions in the coded video bitstream can vary for each video coding technology. Such mapping can range from a simple direct mapping to complex adaptive schemes including the most probable mode, and similar techniques up to codewords. In most cases, however, there may be certain directions that occur less statistically likely in video content than certain other directions. Since the goal of video compression is to reduce redundancy, those less likely directions will be represented by more bits than the more likely directions in a well - functioning video coding technology.

[0017] Image and / or video encoding and decoding can be performed using inter-picture prediction with motion compensation. Motion compensation can be an irreversible compression technique, and a block of sample data from a previously reconstructed picture or a portion thereof (reference picture) is spatially shifted in the direction indicated by a motion vector (hereinafter MV) and then used for prediction of a newly reconstructed picture or picture portion. In some cases, the reference picture can be the same as the picture currently being reconstructed. The MV can have two dimensions X and Y, or three dimensions, where the third dimension is an indication of the reference picture in use (the latter can be, indirectly, the temporal dimension).

[0018] In some video compression techniques, the MV applicable to a particular area of sample data can be predicted from other MVs, for example, from other areas of sample data spatially adjacent to the area being reconstructed and from those preceding it in the decoding order. By doing so, the amount of data required to code the MV can be significantly reduced, thereby removing redundancy and enhancing compression. For example, when coding an input video signal obtained from a camera (known as natural video), there is a statistical likelihood that an area larger than the area to which a single MV is applicable moves in a similar direction, and thus, in some cases, the MV prediction can work effectively by being predictable using a similar motion vector derived from the MVs of adjacent areas. As a result, the MV required for a given area can be similar or the same as the MV predicted from surrounding MVs and can be represented in fewer bits than the number of bits that would be used if the MV were directly coded after entropy coding. In some cases, the MV prediction can be an example of reversible compression of a signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, the MV prediction itself can be irreversible, for example, due to rounding errors when calculating predictors from some surrounding MVs.

[0019] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms proposed by H.265, in this specification, a technique hereinafter referred to as "spatial merge" will be described with reference to FIG. 2.

[0020] Referring to FIG. 2, the current block (201) has samples recognized by the encoder during the motion search process as being predictable from a previous block of the same size that has been spatially shifted. Instead of directly coding that MV, the MV can be derived from metadata associated with one or more reference pictures, for example, from the most recent reference picture (in decoding order) using an MV associated with any one of five surrounding samples represented as A0, A1, and B0, B1, B2 (202 to 206 respectively). In H.265, MV prediction can use predictors from the same reference picture that adjacent blocks are using. SUMMARY OF THE INVENTION

[0021] Aspects of the present disclosure provide methods and apparatuses for video encoding / decoding. In some examples, an apparatus for video decoding includes a processing circuit. The processing circuit receives a video bitstream including a current block and determines a candidate list including a plurality of motion vector (MV) candidates for predicting the current block. The processing circuit modifies the plurality of MV candidates in the candidate list according to a diversity evaluation based on the motion information of the plurality of MV candidates. Modifying the plurality of MV candidates prioritizes a subset of MV candidates with a larger difference in motion information. The processing circuit decodes an index indicating a selected MV candidate from the modified candidate list from the bitstream and reconstructs the current block based on the selected MV candidate.

[0022] In some embodiments, the diversity evaluation based on motion information includes at least one of a diversity evaluation based on translational motion and / or a diversity evaluation based on affine motion.

[0023] In some embodiments, the processing circuit determines whether a first MV candidate in the candidate list satisfies the requirements, and in response to the first MV candidate satisfying the requirements, applies a diversity evaluation based on motion information to the first MV candidate and a second MV candidate preceding the first MV candidate. In some examples, the requirements include at least one of the block size associated with the first MV candidate satisfying a size requirement and / or the block coordinates of the first MV candidate satisfying a coordinate requirement.

[0024] In some embodiments, the processing circuit determines that the first MV candidate and the second MV candidate have sufficient diversity in response to one of the first MV candidate and the second MV candidate being a single prediction candidate and the other of the first MV candidate and the second MV candidate being a dual prediction candidate.

[0025] In some embodiments, the diversity evaluation based on motion information is a diversity evaluation based on translational motion. In some examples, the processing circuit calculates a first motion vector difference between the first MV candidate and the second MV candidate in the candidate list and determines whether one of the first MV candidate and the second MV candidate is redundant based on the first motion vector difference. In some examples, the processing circuit compares a first maximum component value of the first motion vector difference with a threshold value and determines that one of the first MV candidate and the second MV candidate is redundant in response to the first maximum component value being smaller than the threshold value. In one example, the threshold value can be a fixed constant value or a value that is a function of the size of the current block.

[0026] In some examples, the first MV candidate and the second MV candidate are dual prediction candidates, and the first motion vector difference is related to the first reference picture. In some examples, the processing circuit calculates a second motion vector difference between the first MV candidate and the second MV candidate, and the second motion vector difference is related to the second reference picture. Further, the processing circuit determines that one of the first MV candidate and the second MV candidate is redundant in response to both the first maximum component value of the first motion vector difference and the second maximum component value of the second motion vector difference being less than a threshold value.

[0027] In some examples, the first MV candidate and the second MV candidate are geometric partitioning mode (GPM) candidates, and the first motion vector difference is related to the first partition of each of the first MV candidate and the second MV candidate. In some examples, the processing circuit calculates a second motion vector difference between the first MV candidate and the second MV candidate, and the second motion vector difference is related to the second partition of each of the first MV candidate and the second MV candidate. Further, the processing circuit determines that one of the first MV candidate and the second MV candidate is redundant in response to both the first maximum component value of the first motion vector difference and the second maximum component value of the second motion vector difference being less than a threshold value.

[0028] In some embodiments, the diversity evaluation based on motion information is a diversity evaluation based on affine motion. In some examples, the processing circuit calculates a first control point motion vector (CPMV) difference value between a first pair of first control point motion vectors from each of a first MV candidate and a second MV candidate in a candidate list, and based on the first CPMV difference value, determines whether one of the first MV candidate and the second MV candidate is redundant. In some examples, the processing circuit compares a first maximum component value of the first CPMV difference value with a threshold value, and in response to the first maximum component value being smaller than the threshold value, determines that one of the first MV candidate and the second MV candidate is redundant. In one example, the threshold value can be a fixed constant value or a value that is a function of the size of the current block.

[0029] In some examples, the first MV candidate and the second MV candidate are dual prediction candidates, and the first CPMV difference value is between a first pair of first CPMVs associated with a first reference picture. In some examples, the processing circuit calculates a second CPMV difference value between a second pair of first CPMVs from each of the first MV candidate and the second MV candidate, and the second pair of first CPMVs is associated with a second reference picture. Further, the processing circuit determines that one of the first MV candidate and the second MV candidate is redundant in response to both a first maximum component value of the first CPMV difference value and a second maximum component value of the second CPMV difference value being smaller than the threshold value.

[0030] In some embodiments, the diversity evaluation based on motion information is a diversity evaluation based on affine motion. In some examples, the processing circuit calculates a plurality of first affine parameter difference values between a first pair of affine models from each of a first MV candidate and a second MV candidate in a candidate list, and based on the plurality of first affine parameter difference values, determines whether one of the first MV candidate and the second MV candidate is redundant. In some examples, the processing circuit compares a first maximum value among the plurality of first affine parameter difference values with a threshold value, and in response to the first maximum value being smaller than the threshold value, determines that one of the first MV candidate and the second MV candidate is redundant.

[0031] In some examples, the first MV candidate and the second MV candidate are dual prediction candidates, and the plurality of first affine parameter difference values are related to a first reference picture. In some examples, the processing circuit calculates a plurality of second affine parameter difference values between a second pair of affine models from each of the first MV candidate and the second MV candidate, and the second pair of affine models is related to a second reference picture. Further, the processing circuit determines that one of the first MV candidate and the second MV candidate is redundant in response to both a first maximum value among the plurality of first affine parameter difference values and a second maximum value among the plurality of second affine parameter difference values being smaller than the threshold value.

[0032] In some embodiments, the processing circuit determines whether one of a first MV candidate and a second MV candidate in a candidate list is redundant based on a diversity evaluation based on motion information and a diversity evaluation based on a template matching (TM) cost. In some examples, the processing circuit determines that one of the first MV candidate and the second MV candidate is redundant in response to a motion difference between the first MV candidate and the second MV candidate being smaller than a first threshold value and a template matching cost difference between the first MV candidate and the second MV candidate being smaller than a second threshold value.

[0033] In some examples, the processing circuit moves an MV candidate determined to be a redundant candidate to the end of the candidate list. In some examples, the processing circuit removes an MV candidate determined to be a redundant candidate from the candidate list.

[0034] In some examples, the processing circuit determines that one of a first MV candidate and a second MV candidate in the candidate list is a redundant candidate. The first MV candidate is associated with a first block having a first block size, and the second MV candidate is associated with a second block having a second block size. Further, the processing circuit selects the first MV candidate as the redundant candidate in response to the first block size being smaller than the second block size.

[0035] The disclosed aspects also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding, cause the computer to perform a method of video decoding.

[0036] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

Brief Description of the Drawings

[0037]

Figure 1A

Figure 1B

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9A

Figure 9B

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

DETAILED DESCRIPTION OF THE INVENTION

[0038] FIG. 3 represents an exemplary block diagram of a communication system (300). The communication system (300) includes a plurality of terminal devices that can communicate with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via a network (350). In FIG. 3, the first pair of terminal devices (310) and (320) perform unidirectional data transmission. For example, the terminal device (310) may encode video data (e.g., a stream of video data captured by the terminal device (310)) for transmission to another terminal device (320) via the network (350). The encoded video data can be transmitted in the form of one or more coded video bitstreams. The terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to recover a video picture, and display the video picture according to the recovered video data. Unidirectional data transmission can be common in media serving applications and the like.

[0039] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform two-way transmission of encoded video data, for example, during a video conference. For two-way transmission of data, in the example, each of the terminal devices (330) and (340) may encode video data (e.g., a stream of video pictures captured by that terminal device) for transmission to the other of the terminal devices (330) and (340) via the network (350). Each of the terminal devices (330) and (340) may also receive the encoded video data transmitted by the other of the terminal devices (330) and (340), may decode the encoded video data to recover the video pictures, and may display the video pictures on an accessible display device according to the recovered video data.

[0040] In the example of FIG. 3, the terminal devices (310), (320), (330), and (340) are respectively represented as a server, a personal computer, and a smartphone, but the principles of the present disclosure cannot be so limited. Embodiments of the present disclosure find use in laptop computers, tablet computers, media players, and / or dedicated video conferencing devices. The network (350) corresponds to any number of networks that transmit encoded video data between the terminal devices (310), (320), (330), and (340), including, for example, wireline (wired) and / or wireless communication networks. The communication network (350) may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of the network (350) can be irrelevant to the operation of the present disclosure unless otherwise described hereinafter.

[0041] FIG. 4 represents a video encoder and a video decoder in a streaming environment as an application example of the disclosed subject. The disclosed subject can be similarly applicable to other video-related applications including, for example, storage of compressed video on digital media including video conferencing, digital TV, streaming services, CD, DVD, memory stick, etc.

[0042] A streaming system may include a capture subsystem (413) that includes, for example, a video source (401), such as a digital camera, that generates a stream (402) of uncompressed video pictures. In the example, the stream (402) of video pictures includes samples taken by the digital camera. The stream (402) of video pictures is represented in bold lines to emphasize the high data volume as compared to the encoded video data (404) (or coded video bitstream) and can be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) can include hardware, software, or a combination thereof to enable or implement the disclosed aspects of the subject matter, as described in more detail below. The encoded video data (404) (or encoded video bitstream) is represented in thin lines to emphasize the lower data volume as compared to the stream (402) of video pictures and can be stored in a streaming server (405) for future use. One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to read copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include a video decoder (410), for example, in an electronic device (430). The video decoder (410) decodes an incoming copy (407) of the encoded video data and generates an outgoing stream (411) of video pictures that can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., video bitstream) can be encoded according to a particular video coding / compression standard. An example of such a standard is ITU-T Recommendation H.265.In an example, the video coding standard under development is commonly known as Versatile Video Coding (VVC). The subject matter disclosed may be used in relation to VVC.

[0043] Note that the electronic devices (420) and (430) can include other components (not shown). For example, the electronic device (420) can include a video decoder (not shown), and the electronic device (430) can similarly include a video encoder (not shown).

[0044] FIG. 5 shows an exemplary block diagram of a video decoder (510). The video decoder (510) can be included in an electronic device (530). The electronic device (530) can include a receiver (531) (e.g., a receiving circuit). The video decoder (510) can be used in place of the video decoder (410) of the example of FIG. 4.

[0045] The receiver (531) may receive one or more coded video sequences to be decoded by the video decoder (510). In an embodiment, one coded video sequence is received at a time, and the decoding of each coded video sequence is independent of the decoding of other coded video sequences. The coded video sequence may be received from a channel (501), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (531) may receive the encoded video data together with other data, such as coded audio data and / or auxiliary data streams, which may be transferred to their respective using entities (not shown). The receiver (531) may separate the coded video sequence from other data. To counter network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter “parser (520)”). In certain applications, the buffer memory (515) is part of the video decoder (510). Otherwise, it can be outside the video decoder (510) (not shown). Still otherwise, for example, there can be a buffer memory outside the video decoder (not shown) for countering network jitter, and in addition, another buffer memory (515) within the video decoder (510) for, for example, manipulating the playback timing. When the receiver (531) is receiving data from a storage / transfer device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (515) may not be required, or may be small. For use in a best effort packet network such as the Internet, the buffer memory (515) may be required, may be relatively large, and advantageously may be of an adaptable size and may be implemented at least in part in an operating system or similar element outside the video decoder (510) (not shown).

[0046] Video decoder (510) may include a parser (520) for reconstructing symbols (521) from the coded video sequence. The categories of those symbols include information used to manage the operation of video decoder (510) and, potentially, information for controlling a rendering device (such as a display screen) (512) that is not an essential part of electronic device (530) but can be coupled to electronic device (530) as shown in FIG. 5. The control information for the rendering device may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). Parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence can follow video coding techniques or standards and can follow various principles including variable length coding, Huffman coding, arithmetic coding with or without context dependence, etc. Parser (520) may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to that group. Subgroups can include Group of Pictures (GOP), picture, tile, slice, macroblock, Coding Unit (CU), block, Transform Unit (TU), Prediction Unit (PU), etc. Parser (520) may also extract quantization parameter values, motion vectors, etc. from the coded video sequence information such as transform coefficients.

[0047] Parser (520) may perform an entropy decoding / parsing operation on the video sequence received from buffer memory (515) to generate symbols (521).

[0048] The reconstruction of symbol (521) can have a number of different units depending on the type of the coded video picture or a portion thereof (e.g., inter and intra pictures, inter and intra blocks) and other factors. How the units are included can be controlled by subgroup control information parsed by parser (520) from the coded video sequence. The flow of such subgroup control information between parser (520) and the following plurality of units is not shown for clarity.

[0049] Beyond the function blocks already described, video decoder (510) can conceptually be subdivided into a number of functional units described below. In an actual implementation operating under commercial constraints, many of those units interact closely with each other and can at least partially be incorporated into each other. However, for the purpose of explaining the disclosed subject matter, the conceptual subdivision into functional units below is appropriate.

[0050] The first unit is a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives, as symbols (521) from parser (520), control information including which transform to use, block size, quantization coefficients, quantization scaling matrix, etc., together with the quantized transform coefficients. The scaler / inverse transform unit (551) can output a block including sample values that can be input to aggregator (555).

[0051] In some cases, the output samples of the scaler / inverse converter (551) can be related to the intra-coded blocks. An intra-coded block is a block that does not use prediction information from a previously reconstructed picture and can use prediction information from a previously reconstructed portion of the current picture. Such prediction information can be supplied by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates a block of the same size and shape as the block being reconstructed, using the surrounding already reconstructed information fetched from the current picture buffer (558). The current picture buffer (558) buffers, for example, the partially reconstructed current picture and / or the fully reconstructed current picture. The aggregator (555) adds, in some cases, for each sample, the prediction information generated by the intra prediction unit (552) to the output sample information supplied by the scaler / inverse conversion unit (551).

[0052] In other cases, the output samples of the scaler / inverse transform unit (551) can relate to inter-coded and potentially motion-compensated blocks. In such cases, the motion compensation prediction unit (553) can access the reference picture memory (557) to fetch the samples used for prediction. According to the symbols (521) related to the block, after motion-compensating the fetched samples, those samples can be added by the aggregator (555) to the output of the scaler / inverse transform unit (551) (in this case, referred to as residual samples or a residual signal) to generate output sample information. The address in the reference picture memory (557) where the motion compensation prediction unit (553) fetches the prediction samples can be controlled by the motion vector that the motion compensation prediction unit (553) can utilize in the form of, for example, symbols (521) that can have X, Y, and reference picture components. Motion compensation can also include interpolation of the sample values fetched from the reference picture memory (557) when an accurate sub-sample motion vector is used, a motion vector prediction mechanism, and the like.

[0053] The output samples of the aggregator (555) can undergo various loop filtering techniques in the loop filter unit (556). Video compression techniques can include in-loop filter techniques. This technique is included in the coded video sequence (also referred to as a coded video bitstream) and is controlled by the parameters made available to the loop filter unit (556) as symbols (521) from the parser (520). Video compression can also respond to the meta information obtained during the decoding of the previous part (in the decoding order) of the coded picture or coded video sequence, and further, can also respond to the previously configured loop filter processed sample values.

[0054] The output of the loop filter unit (556) can be a sample stream that is output to the render device (512) and further stored in the reference picture memory (557) for use in future inter-picture prediction.

[0055] Once a particular coded picture is completely reconstructed, it can be used as a reference picture for future prediction. For example, when the coded picture corresponding to the current picture is completely reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and the unused current picture buffer can be reallocated before starting the reconstruction of subsequent coded pictures.

[0056] The video decoder (510) may perform a decoding operation according to a predetermined video compression technology or standard specification such as ITU-T Recommendation H.265. The coded video sequence may conform to the syntax defined by the video compression technology or standard specification in use in the sense that the coded video sequence conforms to both the syntax of the video compression technology or standard specification and the profile documented in the video compression technology or standard specification. Specifically, the profile can select specific tools from all the tools available in the video compression technology or standard specification as the only tools available for use under that profile. Also, the complexity of the coded video sequence needs to be within the bounds defined by the level of the video compression technology or standard specification for compliance. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further restricted through the Hypothetical Reference Decoder (HRD) specification and the metadata for HRD buffer management signaled in the coded video sequence.

[0057] In an embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may also be included as part of the coded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data can take forms such as, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0058] FIG. 6 shows an exemplary block diagram of a video encoder (603). The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) may be used in place of the video encoder (403) of the example of FIG. 4.

[0059] The video encoder (603) may receive video samples from a video source (601) (not part of the electronic device (620) in the example of FIG. 6) that can capture the video image to be coded by the video encoder (603). In other examples, the video source (601) is part of the electronic device (620).

[0060] The video source (601) can supply a source video sequence to be coded by the video encoder (603) in the form of a digital video sample stream that can be of any suitable bit depth (e.g., 8 bits, 10 bits, 12 bits, etc.), any color space (e.g., BT.601 YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (601) may be a storage device storing pre-prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be supplied as a plurality of individual pictures that give motion when viewed in sequence. Each picture itself may be organized as a spatial array of pixels, and each pixel can have one or more samples depending on the sampling structure, color space, etc. in use. A person skilled in the art can easily understand the relationship between pixels and samples. This specification will hereinafter focus on samples.

[0061] According to an embodiment, a video encoder (603) can code and compress pictures of a source video sequence into a coded video sequence (643) in real time or under any other time constraints required. Enforcing an appropriate coding speed is a function of the controller (650). In some embodiments, the controller (650) controls other functional units as described below and is functionally coupled to the other functional units. The couplings are not shown for clarity. Parameters set by the controller (650) can include parameters related to rate control (picture skip, quantizer, lambda value of rate distortion optimization techniques, etc.), picture size, group-of-picture (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other appropriate functions related to the video encoder (603) optimized for a particular system design.

[0062] In some embodiments, the video encoder (603) is configured to operate in a coding loop. As an overly simplified description, by way of example, the coding loop may involve a source coder (630) (e.g., generating symbols such as a symbol stream based on an input picture to be coded and a reference picture), and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to generate sample data in the same way as a (remote) decoder would also generate. The reconstructed sample stream (sample data) is input into the reference picture memory (634). Since the decoding of the symbol stream results in a bit-exact result independent of the location of the decoder (local or remote), the content in the reference picture memory (634) is also bit-exact between the local encoder and the remote encoder. In other words, the prediction part of the encoder "sees" the same sample values as the reference picture samples that the decoder will "see" when using prediction during decoding. This basic principle of reference picture synchronization (and the resulting drift in case synchronization cannot be maintained, e.g., due to channel errors) is also used in some related technologies.

[0063] The operation of the "local" decoder (633) can be the same as that of a "remote" decoder such as the video decoder (510), which has already been described in detail above with reference to FIG. 5. Referring momentarily also to FIG. 5, however, since the symbols are available and the encoding / decoding of the symbols to the coded video sequence by the entropy encoder (645) and the parser (520) can be reversible, the entropy decoding part of the video decoder (510) including the buffer memory (515) and the parser (520) may not be fully implemented in the local decoder (633).

[0064] In an embodiment, decoder techniques, excluding parsing / entropy decoding that exists in a decoder, exist in a corresponding encoder in the same or substantially the same functional form. Accordingly, the disclosed subject matter focuses on the operation of the decoder. The description of encoder techniques may be omitted since they are the reverse of the decoder techniques described comprehensively. In certain instances, more detailed descriptions are provided below.

[0065] During operation, in some examples, source coder (630) may perform motion-compensated predictive coding. This predictively codes an input picture by referring to one or more previously coded pictures from a video sequence designated as a "reference picture". In this way, coding engine (632) codes the difference between a pixel block of a reference picture that may be selected as a predictive reference for the input picture and a pixel block of the input picture.

[0066] Local video decoder (633) may decode the coded video data of a picture that may be designated as a reference picture based on the symbols generated by source coder (630). The operation of coding engine (632) may advantageously be an irreversible process. When the coded video data may be decoded by a video decoder (not shown in FIG. 6), the reconstructed video sequence is typically a reproduction of the source video sequence with some errors. Local video decoder (633) may reproduce the decoding process that may be performed by the video decoder for the reference picture and cause the reconstructed reference picture to be stored in reference picture cache (634). In this way, video encoder (603) may locally store a copy of the reconstructed reference picture that has the same content as the reconstructed reference picture that would be obtained by a remote video decoder (without transmission errors).

[0067] Predictor (635) may perform predictive search for the coding engine (632). That is, in order for a new picture to be coded, predictor (635) may look in reference picture memory (634) for specific metadata such as reference picture motion vectors, block shapes, etc. that can serve as appropriate prediction criteria for that new picture, or sample data (as candidate reference pixel blocks). Predictor (635) may operate on a sample block-by-pixel block basis to find appropriate prediction criteria. In some cases, the input picture may have prediction criteria drawn from a plurality of reference pictures stored in reference picture memory (634), as determined by the search results obtained by predictor (635).

[0068] Controller (650) may manage the coding operations of source coder (630), including, for example, setting parameters and sub-group parameters used for encoding video data.

[0069] The outputs of all of the above functional units may undergo entropy coding in entropy coder (645). Entropy coder (645) converts symbols generated by the various functional units into a coded video sequence by reversibly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0070] The transmitter (640) may buffer the coded video sequence generated by the entropy encoder (645) for transmission via the communication channel (660). The communication channel (660) may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (640) may merge the coded video data from the video coder (603) with other data to be transmitted, such as coded audio data and / or an auxiliary data stream (source not shown).

[0071] The controller (650) may manage the operation of the video encoder (603). During coding, the controller (650) may assign to each coded picture a particular coded picture type that may affect the coding technique applicable to each picture. For example, a picture may often be assigned as one of the following picture types.

[0072] An Intra Picture (I picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow various types of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will be aware of such variations of I pictures and their respective applications and characteristics.

[0073] A Predictive Picture (P picture) may be a picture that can be encoded and decoded by intra prediction or inter prediction using at most one motion vector and a reference index to predict the sample values of each block.

[0074] A bi-directionally predictive picture (B picture) may be a picture that can be encoded and decoded using intra prediction or inter prediction that uses at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple-predictive picture(s) can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0075] A source picture may generally be spatially subdivided into a plurality of sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples respectively) and may be coded block by block. The blocks may be coded predictively with reference to other (already coded) blocks determined by the coding assignment applied to each of the blocks of the picture. For example, blocks of an I picture may be coded non-predictively, or they may be coded predictively with reference to already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of a P picture may be coded predictively by spatial prediction or temporal prediction with reference to one previously coded reference picture. Blocks of a B picture may be coded predictively by spatial prediction or temporal prediction with reference to one or two previously coded reference pictures.

[0076] The video encoder (603) may perform coding operations in accordance with a predetermined video coding technology or standard specification such as ITU-T Recommendation H.265. During its operation, the video encoder (603) may perform various compression operations including predictive coding operations that utilize the temporal and spatial redundancies in the input video sequence. Accordingly, the coded video data may conform to the syntax defined by the video coding technology or standard specification being used.

[0077] In an embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include such data as part of the coded video sequence. The additional data may have temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, and the like.

[0078] Video may be captured as a plurality of source pictures (video pictures) in a temporal sequence. Intra picture prediction (often abbreviated as intra prediction) utilizes spatial correlation within a given picture, and inter picture prediction utilizes correlation (temporal or otherwise) between pictures. In an example, a particular picture being encoded / decoded, called the current picture, is partitioned into blocks. If a block within the current picture is similar to a reference block within a reference picture that was previously coded and is still buffered within the video, that block within the current picture can be coded by a vector called a motion vector. The motion vector points to the reference block within the reference picture and can have a third dimension identifying the reference picture if multiple reference pictures are being used.

[0079] In some embodiments, dual prediction techniques may be used in inter picture prediction. According to the dual prediction technique, two reference pictures, e.g., a first reference picture and a second reference picture that both precede the current picture in decoding order within the video (however, in display order, may be in the past and future respectively), are used. A block within the current picture can be coded by a first motion vector pointing to a first reference block within the first reference picture and a second motion vector pointing to a second reference block within the second reference picture. The block is predictable by a combination of the first reference block and the second reference block.

[0080] Furthermore, the merge mode technique can be used in inter-picture prediction to improve coding efficiency.

[0081] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are partitioned into coding tree units (CTUs) for compression, and the CTUs within a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU includes three coding tree blocks (CTBs), which are one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree partitioned into one or more coding units (CUs). For example, a 64×64 pixel CTU can be divided into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In the example, each CU is analyzed to determine a prediction type for the CU, such as an inter-prediction type or an intra-prediction type. The CU is divided into one or more prediction units (PUs) according to temporal and / or spatial predictability. Generally, each PU includes one luma prediction block (PB) and two chroma PBs. In an embodiment, the prediction operation in coding (encoding / decoding) is performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values (e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0082] FIG. 7 shows an example of a video encoder (703). The video encoder (703) receives a processing block (e.g., a prediction block) of sample values within a current video picture included in a sequence of video pictures, and is configured to encode the processing block into a coded picture that is part of a coded video sequence. In the example, the video encoder (703) is used instead of the video encoder (403) in the example of FIG. 4.

[0083] In an example of HEVC, the video encoder (703) receives a matrix of sample values of a processing block such as a prediction block of 8×8 samples. The video encoder (703) determines, for example using rate distortion optimization, whether the processing block is best coded in an intra mode, an inter mode, or a bi-prediction mode. If the processing block is to be coded in the intra mode, the video encoder (703) may use intra prediction techniques to encode the processing block into the coded picture, and if the processing block is to be coded in the inter mode or the bi-prediction mode, the video encoder (703) may use inter prediction or bi-prediction techniques respectively to encode the processing block into the coded picture. In certain video coding techniques, the merge mode can be an inter-picture prediction sub-mode in which a motion vector is derived from one or more motion vector predictors without the benefit of coded motion vector components outside the predictor. In certain other video coding techniques, there may be coded motion vector components applicable to the target block. In the example, the video encoder (703) includes other components such as a mode decision module (not shown) that determines the mode of the processing block.

[0084] In the example of FIG. 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculation unit (723), a switch (726), a residual encoder (724), a general-purpose controller (721), and an entropy encoder (725) that are coupled as shown in FIG. 7.

[0085] The inter-encoder (730) receives samples of the current block (e.g., a processing block), compares the block with one or more reference blocks (e.g., blocks in the previous and subsequent pictures) in a reference picture, generates inter-prediction information (e.g., a description of redundant information according to an inter-coding technique, a motion vector, merge mode information), and is configured to calculate an inter-prediction result (e.g., a predicted block) based on the inter-prediction information using some suitable technique. In some examples, the reference picture is a decoded reference picture that has been decoded based on the coded video information.

[0086] The intra-encoder (722) receives samples of the current block (e.g., a processing block), in some cases compares the block with blocks that have already been coded in the same picture, generates the quantized coefficients after transformation, and in some cases also generates intra-prediction information (e.g., intra-prediction direction information according to one or more intra-coding techniques). In an example, the intra-encoder (722) also calculates an intra-prediction result (e.g., a predicted block) based on the intra-prediction information and reference blocks in the same picture.

[0087] The general-purpose controller (721) is configured to determine general-purpose control data and control other components of the video encoder (703) based on the general-purpose control data. In an example, the general-purpose controller (721) determines the mode of a block and supplies a control signal to the switch (726) based on the mode. For example, when the mode is the intra mode, the general-purpose controller (721) controls the switch (726) to select the intra mode result for use by the residual calculation unit (723), and selects the intra prediction information and controls the entropy encoder (725) to include the intra prediction information in the bitstream. When the mode is the inter mode, the general-purpose controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculation unit (723), and selects the inter prediction information and controls the entropy encoder (725) to include the inter prediction information in the bitstream.

[0088] The residual calculation unit (723) is configured to calculate the difference (residual data) between the received block and the prediction result selected from the intra encoder (722) or the inter encoder (730). The residual encoder (724) is configured to operate based on the residual data to encode the residual data so as to generate conversion coefficients. In an example, the residual encoder (724) is configured to convert the residual data from the spatial domain to the frequency domain and generate conversion coefficients. Next, the conversion coefficients are subjected to quantization processing to obtain quantized conversion coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform inverse conversion and generate decoded residual data. The decoded residual data can be appropriately used by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (�30) can generate a decoded block based on the decoded residual data and the inter prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and the intra prediction information. The decoded block is appropriately processed to generate a decoded picture, and the decoded picture is buffered in a memory circuit (not shown.) and can be used as a reference picture in some examples.

[0089] The entropy encoder (725) is configured to format the bitstream to include the encoded block. The entropy encoder (725) is configured to include various information in the bitstream according to an appropriate standard such as the HEVC standard. In an example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra prediction information or inter prediction information), residual information, and other appropriate information in the bitstream. Note that there is no residual information when coding a block in either the merge submode of the inter mode or the bi-prediction mode according to the disclosed subject matter.

[0090] FIG. 8 shows an example of a video decoder (810). The video decoder (810) is configured to receive a coded picture that is part of a coded video sequence, decode the coded picture, and generate a reconstructed picture. In the example, the video decoder (810) is used in place of the video decoder (410) in the example of FIG. 4.

[0091] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872) that are coupled as shown in FIG. 8.

[0092] The entropy decoder (871) may be configured to reconstruct specific symbols representing syntax elements from the coded picture, from which the coded picture is composed. Such symbols can include, for example, the mode in which a block is coded (e.g., intra mode, or inter mode or bi-prediction mode in a merge sub-mode or other sub-mode), and prediction information (e.g., intra prediction information or inter prediction information) that can identify specific samples or metadata used for prediction by the intra decoder (872) or the inter decoder (880), respectively. The symbols can also include residual information, for example, in the form of quantized transform coefficients. In the example, when the prediction mode is inter or bi-prediction mode, the inter prediction information is supplied to the inter decoder (880), and when the prediction type is intra prediction type, the intra prediction information is supplied to the intra decoder (872). The residual information can undergo inverse quantization and is supplied to the residual decoder (873).

[0093] The inter decoder (880) is configured to receive the inter prediction information and generate an inter prediction result based on the inter prediction information.

[0094] The intra decoder (872) is configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0095] The residual decoder (873) performs inverse quantization to extract the inverse quantized transform coefficients, processes the inverse quantized transform coefficients, and is configured to convert the residual information from the frequency domain to the spatial domain. The residual decoder (873) may also request specific control information (for including quantization parameters (QP)), and that information may be supplied by the entropy decoder (871) (this is only low-capacity control information, and the data path is not shown).

[0096] The reconstruction module (874) is configured to combine, in the spatial domain, the residual output by the residual decoder (873) and the prediction result (optionally output by an inter or intra prediction module) to form a reconstructed block. The reconstructed block may be a part of the reconstructed picture, and then the reconstructed picture may be a part of the reconstructed video. Note that other appropriate operations such as a deblocking operation may be performed to improve visual quality.

[0097] Note that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) can be implemented by any appropriate technology. In an embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) can be implemented using one or more integrated circuits. In other embodiments, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) can be implemented using one or more processors that execute software instructions.

[0098] Some aspects of the present disclosure provide techniques for reordering various merge candidates.

[0099] ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) published the H.265 / HEVC (High Efficiency Video Coding) standard in 2013 (Version 1), 2014 (Version 2), 2015 (Version 3), and 2016 (Version 4). In 2015, the two standard organizations jointly formed the JVET (Joint Video Exploration Team) to explore the possibility of developing the next video coding standard beyond HEVC. In October 2017, the two standard organizations jointly issued a Call for Proposals (CfP) on video compression with capabilities beyond HEVC. By February 15, 2018, 22 CfP responses regarding Standard Dynamic Range (SDR), 12 CfP responses regarding High Dynamic Range (HDR), and 12 CfP responses regarding the 360 video category were submitted respectively. In April 2018, all received CfP responses were evaluated at the 122nd MPEG / 10th JVET meeting. As a result of the meeting, JVET officially launched the standardization process for the next generation of video coding beyond HEVC. The new standard was named Versatile Video Coding (VVC), and JVET was renamed Joint Video Experts Team. In 2020, ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) published the VVC video coding standard (Version 1).

[0100] In inter prediction, for each inter-predicted coding unit (CU), motion parameters are required for the coding features of VVC, for example, to be used for the generation of inter-predicted samples. The motion parameters may include motion vectors, reference picture indexes, reference picture list use indexes, and / or additional information. The motion parameters can be signaled in an explicit or implicit way. When a CU is coded in skip mode, the CU can be associated with one PU, and significant residual coefficients, coded motion vector differences, and / or reference picture indexes may not be required. When a CU is coded in merge mode, the motion parameters of the CU can be obtained from adjacent CUs. The adjacent CUs can include spatial and temporal candidates, as well as additional schedules (or additional candidates) such as those introduced in VVC. The merge mode can be applied not only to skip mode but also to any inter-predicted CU. An alternative to the merge mode is the explicit transmission of motion parameters, and the motion vector, the corresponding reference picture index for each reference picture list, the reference picture list use flag, and / or other necessary information can be explicitly signaled for each CU.

[0101] In VVC, the VVC Test Model (VTM) reference software can include a number of newly refined inter prediction coding tools, and such tools include one or more of the following: (1) Extended merge prediction (2) Merge motion vector difference (MMVD) (3) Advanced motion vector prediction (AMVP) mode by symmetric MVD signaling (4) Affine motion compensation prediction (5) Subblock-based temporal motion vector prediction (SbTMVP) (6) Adaptive Motion Vector Resolution (AMVR) (7) Motion Field Storage: 1 / 16 Luma Sample MV Storage and 8×8 Motion Field Compression (8) Bi - prediction with CU - level Weights (BCW) (9) Bi - directional Optical Flow (BDOF) (10) Decoder - side Motion Vector Refinement (DMVR) (11) Combined Inter and Intra Prediction (CIIP) (12) Geometric Partitioning Mode (GPM)

[0102] In HEVC, the translational motion model is applied for motion compensation prediction (MCP). In the real world, there are many types of motions such as zoom - in / out, rotation, viewpoint motion, and other irregular motions. Block - based affine transform motion compensation prediction can be applied as seen in VTM. Figure 9A shows the affine motion field of a block (902) described by the motion information (4 parameters) of two control points. Figure 9B shows the affine motion field of a block (904) described by three control point motion vectors (6 parameters).

[0103] As shown in Figure 9A, in the 4 - parameter affine motion model, the motion vector at the sample position (x, y) within the block (902) can be derived by Equation (1) as follows:

Equation

[0104] As shown in FIG. 9B, in the six-parameter affine motion model, the motion vector at the sample position (x, y) within the block (904) can be derived by Equation (3) as follows: [Number] The six-parameter affine motion model can also be described by Equation (4) as follows: [Number] As shown in Equations (1) and (3), (mv 0x , mv 0y ) can be the motion vector of the control point at the upper left corner. (mv 1x , mv 1y ) can be the motion vector of the control point at the upper right corner. (mv 2x , mv 2y ) can be the motion vector of the control point at the lower left corner.

[0105] As shown in FIG. 10, to simplify motion compensation prediction, block-based affine transform prediction can be applied. To derive the motion vector for each 4×4 luma sub-block, the motion vector of the center sample (e.g., (1002)) of each sub-block (e.g., (1004)) within the current block (1000) is calculated according to Equations (1) to (4) and can be rounded to 1 / 16 fractional precision. Then, a motion compensation interpolation filter can be applied to generate a prediction for each sub-block using the derived motion vector. The sub-blocks of the chroma component can also be set to 4×4. The MV of a 4×4 chroma sub-block can be calculated as the average of the MVs of the four corresponding 4×4 luma sub-blocks.

[0106] In affine merge prediction, affine merge (AF_MERGE) can be applied to a CU whose both width and height are 8 or more. The CPMV of the current CU can be generated based on the motion information of spatially adjacent CUs. Up to five CPMVP candidates are applicable for affine merge prediction, and the index can be signaled to indicate which one of the five CPMVP candidates is available for the current CU. In affine merge prediction, three types of CPVM candidates, namely (1) an inherited affine merge candidate estimated from the CPMV of neighboring CUs, (2) a constructed affine merge candidate having a CPMVP derived using the translational motion MV of neighboring CUs, and (3) zero MV, can be used to form the affine merge candidate list.

[0107] In VTM3, up to two inherited affine candidates can be applied. The two inherited affine candidates can be derived from the affine motion models of adjacent blocks. For example, one inherited affine candidate can be derived from the left adjacent CU, and the other inherited affine candidate can be derived from the upper adjacent CU. An exemplary candidate block can be shown in FIG. 11. As shown in FIG. 11, for the left predictor (or the left inherited affine candidate), the scan order can be A0->A1, and for the upper predictor (or the upper inherited affine candidate), the scan order can be B0->B1->B2. Therefore, only the first available inherited candidate from each side can be selected. Pruning checks may not be performed between the two inherited candidates. When an adjacent affine CU is identified, the control point motion vectors of the adjacent affine CU can be used to derive CPMV candidates in the affine merge list of the current CU. As shown in FIG. 12, when the block A adjacent to the lower left of the current block (1204) is coded in affine mode, the motion vectors v2, v3, and v4 of the upper left corner, upper right corner, and lower left corner of the CU (1202) containing the block A can be obtained. When the block A is decoded with a 4-parameter affine model, the two CPMVs of the current CU (1204) can be calculated according to v2 and v3 of the CU (1202). When the block A is decoded with a 6-parameter affine model, the three CPMVs of the current CU (1204) can be calculated according to v2, v3, and v4 of the CU (1202).

[0108] The composed affine candidates of the current block can be candidates composed by combining the translational motion information in the vicinity of each control point of the current block. The motion information of the control point can be derived from the specified spatial vicinity and temporal vicinity shown in FIG. 13. As shown in FIG. 13, CPMV k(k = 1, 2, 3, 4) represents the k-th control point of the current block (1302). In the case of CPMV1, the blocks can be checked in the order of B2 -> B3 -> A2, and the MV of the first available block can be used. In the case of CPMV2, the blocks can be checked in the order of B1 -> B0. In the case of CPMV3, the blocks can be checked in the order of A1 -> A0. TMVP can be used as CPMV4 when CPMV4 is unavailable.

[0109] After obtaining the MVs of the four control points, the affine merge candidates can be configured for the current block (1302) based on the movement information of the four control points. For example, the affine merge candidates can be configured in the following order: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, and {CPMV1, CPMV3} based on the combinations of the MVs of the four control points.

[0110] Combinations of three CPMVs can form 6-parameter affine merge candidates, and combinations of two CPMVs can form 4-parameter affine merge candidates. To avoid the motion scaling process, if the reference indices of the control points are different, the related combinations of the control point MVs can be discarded.

[0111] After checking the inherited affine merge candidates and the configured affine merge candidates, if the list is still not full, zero MVs can be inserted at the end of the list.

[0112] In some examples, an affine advanced motion vector prediction (AMVP) mode may be applied to a CU where both the width and height are 16 or more. The affine flag at the CU level may be signaled in the bitstream to indicate whether the affine AMVP mode is used. Then, another flag may be signaled to indicate whether 4-parameter affine or 6-parameter affine is applied. In affine AMVP prediction, the difference of the CPMV of the current CU and the predictor of the CPMVP of the current CU may be signaled in the bitstream. The size of the affine AMVP candidate list can be 2, and the affine AMVP candidate list may use four types of CPMV candidates in the following order: (1) Inherited affine AMVP candidates estimated from the CPMV of neighboring CUs, (2) Constructed affine AMVP candidates with CPMVP derived using the translational motion MV of neighboring CUs, (3) Translational motion MVs from adjacent CUs, and (4) Zero MV and can be generated by using the four types of CPMV candidates in this order.

[0113] The check order of the inherited affine AMVP candidates can be the same as the check order of the inherited affine merge candidates. To determine the AMVP candidates, only affine CUs having the same reference picture as the current block can be considered. When the inherited affine motion predictor is inserted into the candidate list, the pruning process may not be applied.

[0114] The formed AMVP candidates can be derived from the specified spatial neighborhood. As shown in FIG. 13, the same checking order as in the affine merge candidate formation can be applied. Further, the reference picture indexes of adjacent blocks can also be checked. The first block in the checking order can be inter-coded and have the same reference picture as the current CU (1302). One formed AMVP candidate can be determined when the current CU (1302) is coded in 4-parameter affine mode and both mv0 and mv1 are available. The formed AMVP candidates can be further added to the affine AMVP list. When the current CU (1302) is coded in 6-parameter affine mode and all three CPMVs are available, the formed AMVP candidate can be added as one candidate in the affine AMVP list. Otherwise, the formed AMVP candidate can be set as unavailable.

[0115] If the number of candidates in the affine AMVP list is still less than 2 after the inherited affine AMVP candidates and the formed AMVP candidates are checked, mv0, mv1, and mv2 can be added in order. mv0, mv1, and mv2 can function as translational motion MVs to predict all control point MVs of the current CU (e.g., (1302)) if available. Finally, if the affine AMVP list is still not full, zero MVs can be used to fill the affine AMVP list.

[0116] Sub-block based affine motion compensation can save memory access bandwidth and reduce computational complexity compared to pixel-based motion compensation, at the cost of prediction accuracy penalty. To achieve finer-grained motion compensation, prediction refinement with optical flow (PROF) can be used to refine the sub-block based affine motion compensated prediction without increasing the memory access bandwidth for motion compensation. In VVC, after sub-block based affine motion compensation is performed, the luma prediction samples can be refined by adding the differences derived by the optical flow formula. PROF can be explained in the following four steps.

[0117] Step (1): Sub-block based affine motion compensation can be performed to generate a sub-block prediction I(i,j).

[0118] Step (2): The spatial gradients g x (i,j) and g y (i,j) of the sub-block prediction can be calculated at each sample position using a 3-tap filter [-1,0,1]. The gradient calculation can be the same as that in BDOF. For example, the spatial gradients g x (i,j) and g y (i,j) can be calculated based on Equations (5) and (6) respectively:

Equation

Equation

[0119] Step (3): The refinement of the luma prediction can be calculated by the optical flow formula shown in Equation (7).

Equation

[0120] ,y SB ),y SB ) of the sub-block, respectively, then Δv(x,y) can be derived by the following Equations (8) and (9):

Equation

[0121] To maintain accuracy, the center (x SB , y SB ) of the sub-block can be calculated as ((W SB - 1) / 2, (H SB - 1) / 2), where W SB and H SB are the width and height of the sub-block, respectively.

[0122] Once Δv(x, y) is obtained, the parameters of the affine model can be determined. For example, in the case of a four-parameter affine model, the parameters of the affine model can be expressed as in Equation (10). [Number] In the case of a six-parameter affine model, the parameters of the affine model can be expressed as in Equation (11). [Number] Here, (v 0x , v 0y ), (v 1x , v 1y ), (v 2x , v 2y ) can be the top-left control point motion vector, the top-right control point motion vector, and the bottom-left control point motion vector, respectively, and w and h can be the width and height of the CU, respectively.

[0123] Step (4): Finally, the luma prediction refinement Δl(i, y) can be added to the sub-block prediction I(i, j). The final prediction I’ can be generated as shown in Equation (12). [Number]

[0124] PROF cannot be applied to the affine-coded CUs in two cases. The two cases are: (1) when all control point MVs are the same, indicating that the CU has only translational motion, and (2) when the affine motion parameters are larger than the specified limits due to the degradation of sub-block-based affine MC compared to CU-based MC to avoid large memory access bandwidth requirements.

[0125] Note that when coding in affine AMVP mode, each control point of the affine coding block is noted to have a motion vector difference (MVD). For each reference picture, the MVD of the control point is calculated from the actual CPMV value of the control point and the CPMV value of the affine AMVP predictor.

[0126] In the example, for 4-parameter affine, two MVDs (represented by MVD0 and MVD). are coded for each reference list according to Equations (13) and (14):

Number

Number

[0127] In another example, for 6-parameter affine, three MVDs (represented by MVD0, MVD1, and MVD2). are coded for each reference list according to Equations (15), (16), and (17):

Number

Number

Number

[0128] Affine motion estimation (ME) can operate for both single prediction and dual prediction as seen in the VVC reference software VTM. Single prediction can be performed for one of the reference lists L0 and L1, and dual prediction can be performed for both reference lists L0 and L1.

[0129] FIG. 15 shows a schematic diagram of the affine ME (1500). As shown in FIG. 15, in the affine ME (1500), the affine single prediction (S1502) can be performed for the reference list L0 to obtain the prediction P0 of the current block based on the initial reference block in the reference list L0. The affine single prediction (S1504) can also be performed for the reference list L1 to obtain the prediction P1 of the current block based on the initial reference block in the reference list L1. The affine dual prediction can be performed at (S1506). The affine dual prediction (S1506) can start from the initial prediction residual (2I - P0) - P1, where I can be the initial value of the current block. The affine dual prediction (S1506) can search for candidates from the reference list L1 around the initial reference block in the reference list L1 to find the best (or selected) reference block having the minimum prediction residual (2I - P0) - Px, where Px is the prediction of the current block based on the selected reference block.

[0130] According to the reference picture, for the current coding block, the affine ME process can first select a set of control point motion vectors (CPMVs) as a basis. An iterative method can be used to generate the predicted output of the current affine model corresponding to the set of CPMVs, calculate the gradient of the predicted samples, and then solve a linear equation to determine the delta CPMV and optimize the affine prediction. The iteration can stop when all delta CPMVs are zero or when the maximum number of iterations is reached. The CPMV obtained from the iteration can be the final CPMV of the reference picture.

[0131] After the best affine CPMVs for both reference lists L0 and L1 are determined for the affine slice prediction, the affine dual prediction search can be performed to optimize the affine dual prediction output by using the best slice prediction CPMV and one-sided reference list and searching for the best CPMV for the other reference list. The affine dual prediction search can be repeatedly performed for the two reference lists to obtain the optimization result.

[0132] FIG. 16 shows an exemplary affine ME process (1600) by which the final CPMV related to the reference picture can be calculated. The affine ME process (1600) can start from (S1602). At (S1602), the base CPMV of the current block can be determined. The base CPMV can be determined based on one of a merge index, an advanced motion vector prediction (AMVP) predictor index, an affine merge index, etc.

[0133] At (S1604), the initial affine prediction of the current block can be obtained based on the base CPMV. For example, according to the base CPMV, a 4-parameter affine motion model or a 6-parameter affine motion model can be applied to generate the initial affine prediction.

[0134] At (S1606), the gradient of the initial affine prediction can be obtained. For example, the gradient of the initial affine prediction can be obtained based on equations (5) and (6).

[0135] (S1608), the delta CPMV can be determined. In some embodiments, the delta CPMV can be associated with the displacement between an initial prediction and subsequent affine predictions such as the first affine prediction. Based on the gradient of the initial affine prediction and the delta CPMV, the first affine prediction can be obtained. The first affine prediction can correspond to the first CPMV.

[0136] (S1610), a determination can be made to check whether the delta CPMV is zero or the number of iterations is greater than or equal to a threshold. If the delta CPMV is zero or the number of iterations is greater than or equal to the threshold, the final (or selected) CPMV can be determined at (S1612). The final (or selected) CPMV can be the first CPMV determined based on the gradient of the initial affine prediction and the delta CPMV.

[0137] Still referring to (S1610), if the delta CPMV is not zero or the number of iterations is less than the threshold, a new iteration can be started. In the new iteration, an updated CPMV (e.g., the first CPMV) can be supplied to (S1604) to generate an updated affine prediction. The affine ME process (1600) can then proceed to (S1606), where the gradient of the updated affine prediction can be calculated. The affine ME process (1600) can then proceed to (S1608) to continue the new iteration.

[0138] In the affine motion model, the four-parameter affine motion model can be further described by an equation including rotation and zoom movements. For example, the four-parameter affine motion model can be rewritten as the following equation (18):

Number

[0139] The bidirectional optical flow (BDOF) in VVC has been previously called BIO in JEM. Compared to the JEM version, the BDOF in VVC can be a simpler version that requires less computation, especially with respect to the number of multiplications and the size of the multiplier.

[0140] BDOF can be used to refine the dual-prediction signals of a CU at the 4×4 sub-block level. BDOF can be applied to a CU if the following conditions are met by the CU: (1) The CU is coded using the "true" dual-prediction mode. That is, one of the two reference pictures is before the current picture in display order, and the other is after the current picture in display order. (2) The distances (e.g., POC differences) from the two reference pictures to the current picture are the same. (3) Both reference pictures are short-term reference pictures. (4) The CU is not coded using the affine mode or the SbTMVP merge mode. (5) The CU has more than 64 luma samples. (6) Both the CU height and the CU width are greater than or equal to 8 luma samples. (7) The BCW weight index indicates equal weights. (8) Weighted prediction (WP) is not valid for the current CU. (9) The CIIP mode is not used for the current CU.

[0141] BDOF can only be applied to the luma component. As the name of BDOF indicates, the BDOF mode can be based on the concept of optical flow, which assumes that the motion of the object is smooth. For each 4×4 sub-block, the motion refinement (v x , v y ) can be calculated by minimizing the difference between the l0 prediction sample and the L1 prediction sample. The motion refinement can then be used to adjust the bi-predicted sample values within the 4×4 sub-block. BDOF can include the following steps.

[0142] First, the horizontal gradients (∂I (k) / ∂x)(i,j) and vertical gradients (∂I (k) / ∂y)(i,j) (k = 0, 1) of the two prediction signals in the reference lists L0 and L1~ can be calculated by directly calculating the difference between two adjacent samples. The horizontal gradient and the vertical gradient can be given by the following equations (21) and (22):

Equation

Equation

[0143] Next, the autocorrelations and cross-correlations of gradients S1, S2, S3, S5, and S6 can be calculated according to the following equations (23) to (27):

Equation

Equation

Equation

Equation

Equation

Equation

Equation

Equation

[0144] Next, motion refinement (v x , v y ) can be derived using the cross-correlation term and the autocorrelation term with the following equations (31) and (32):

Equation

Equation

Equation

[0145] Finally, the BDOF samples of the CU can be calculated by adjusting the dual prediction with the following Equation (34):

Equation

[0146] To derive the gradient value, several prediction samples I within list k (k = 0, 1) outside the boundary of the current CU (k)(i,j) needs to be generated. As shown in FIG. 17, for BDOF in VVC, one extended row / column (1702) around the boundary (1706) of the CU (1704) can be used. To control the computational complexity of generating prediction samples outside the boundary, the prediction samples within the extended area (the non-shadowed area in FIG. 17) can be generated by directly taking the reference samples at nearby integer positions without interpolation (for example, by using the floor() operation on the coordinates), and the normal 8-tap motion compensation interpolation filter can be used to generate the prediction samples within the CU (for example, the shadowed area in FIG. 17). The extended sample values can only be used in gradient calculations. For the remaining steps in the BDOF process, if any samples and gradient values outside the CU boundary are required, those samples and gradient values can be padded (for example, repeated) from their nearest neighbors.

[0147] If the width and / or height of the CU is greater than 16 luma samples, the CU can be divided into sub-blocks with a width and / or height equal to 16 luma samples, and the sub-block boundaries can be treated as CU boundaries in the BDOF process. The maximum unit size of the BDOF process can be limited to 16×16. For each sub-block, the BDOF process can be skipped. If the sum of absolute differences (SAD) between the initial L0 prediction samples and the initial L1 prediction samples is smaller than the threshold, the BDOF process may not be applied to that sub-block. The threshold can be set equal to 8×W×(H>>1). Here, W can indicate the width of the sub-block, and H can indicate the height of the sub-block. To avoid the additional complexity of SAD calculation, the SAD calculated in the DMVR process between the initial L0 prediction samples and the initial L1 prediction samples can be reused in the BDOF process.

[0148] If BCW is currently enabled for the current block, i.e., the BCW weight index indicates unequal weights, the bidirectional optical flow may be disabled. Similarly, if WP is currently enabled for the current block, i.e., the luma weight flag (e.g., luma_weight_lx_flag) is 1 for either of the two reference pictures, BDOF may also be disabled. If the CU is coded in symmetric MVD mode or CIIP mode, BDOF may also be disabled.

[0149] To increase the accuracy of the MV in merge mode, bilateral matching (BM)-based decoder-side motion vector refinement can be applied as seen in VVC. In the bi-prediction operation, the refined MV can be searched around the initial MV in reference picture list L0 and reference picture list L1. The BM method can calculate the distortion between two candidate blocks within reference picture list L0 and reference picture list L1.

[0150] Figure 18 shows an exemplary schematic diagram of BM-based decoder-side motion vector refinement. As shown in Figure 18, the current picture (1802) can include the current block (1808). The current picture can include a reference picture list L0 (1804) and a reference picture list L1 (1806). The current block (1808) can include an initial reference block (1812) in the reference picture list L0 (1804) according to the initial motion vector MV0, and an initial reference block (1814) in the reference picture list L1 (1806) according to the initial motion vector MV1. The search process can be executed around the initial MV0 in the reference picture list L0 (1804) and the initial MV1 in the reference picture list L1 (1806). For example, the first candidate reference block (1810) can be identified in the reference picture list L0 (1804), and the first candidate reference block (1816) can be identified in the reference picture list L1 (1806). The SAD between the candidate reference blocks (e.g., (1810) and (1816)) based on each MV candidate (e.g., MV0’ and MV1’) around the initial MV (e.g., MV0 and MV1) can be calculated. The MV candidate with the lowest SAD can become the refined MV and can be used to generate a bi-predicted signal for predicting the current block (1808).

[0151] The application of DMVR may be restricted and, as seen in VVC, may only be applied to CUs coded based on modes and characteristics as follows: (1) CU-level merge mode with bi-predicted MVs. (2) For the current picture, one reference picture is in the past and the other reference picture is in the future. (3) The distances (e.g., POC differences) from the two reference pictures to the current picture are the same. (4) Both reference pictures are short-term reference pictures. (5) The CU has more than 64 luma samples. (6) Both the CU height and the CU width are greater than or equal to 8 luma samples. (7) The BCW weight index indicates equal weights. (8) Weighted prediction (WP) is not valid for the current CU. (9) The CIIP mode is not used for the current CU.

[0152] The refined MV derived by the DMVR process is used to generate inter-prediction samples and can be used for temporal motion vector prediction for future picture coding. On the other hand, the original MV is used in the deblocking process and can be used for spatial motion vector prediction for future CU coding.

[0153] In DMVR, the search point can surround the initial MV, and the MV offset can follow the MV difference mirroring rule. In other words, any point checked by DMVR, represented by the candidate MV pair (MV0, MV1), can follow the MV difference mirroring rule shown in Eqs. (35) and (36):

Eq.

Eq.

[0154] For example, an exhaustive search of 25 points can be applied for integer sample offset search. The SAD of the initial MV pair can be calculated first. If the SAD of the initial MV pair is smaller than the threshold value, the integer sample stage of DMVR can end. Otherwise, the SADs of the remaining 24 points can be calculated and checked in a scan order such as raster scan order. The point with the minimum SAD can be selected as the output of the integer sample offset search stage. It can be made such that the original MV is preferentially selected during the DMVR process to reduce the penalty of the uncertainty of DMVR refinement. The SAD between the reference blocks referred to by the initial MV candidate can be reduced by 1 / 4 of the SAD value.

[0155] After the integer sample search, fractional sample refinement can continue. To save computational complexity, the fractional sample refinement can be derived by using a parametric error surface equation instead of an additional search involving SAD comparison. The fractional sample refinement can be conditionally called based on the output of the integer sample search stage. When the integer sample search stage ends at the center with the minimum SAD in either the first iterative search or the second iterative search, the fractional sample refinement can be further applied.

[0156] In the sub-pixel offset estimation based on the parametric error surface, the cost at the center position and the costs at four adjacent positions from the center can be used to fit a two-dimensional parabolic error surface equation based on Equation (37): [Number] Here, (x min , y min ) can correspond to the fractional position with the minimum cost, and C can correspond to the minimum cost value. By solving Equation (37) using the cost values of the five search points, (x min , y min ) can be calculated by Equations (38) and (39): [Number] [Number] x min and y min The values of, since all cost values are positive and the minimum value is E(0,0), can be automatically constrained to be between -8 and 8. For x min and y min The value constraints of can correspond to half-pel (or pixel) offsets with 1 / 16 pel MV accuracy in VVC. The calculated fractions (x min , y min ) can be added to the integer-distance refined MV to obtain a sub-pixel accuracy refined delta MV.

[0157] Bilinear interpolation and sample padding can be applied as seen in VVC. The resolution of the MV can be, for example, 1 / 16 luma samples. Samples at fractional positions can be interpolated using an 8-tap interpolation filter. In DMVR, since the search points can enclose the initial fractional pel MV with integer sample offsets, samples at fractional positions need to be interpolated for the DMVR search process. To reduce computational complexity, a bilinear interpolation filter can be used to generate fractional samples for the search process in DMVR. In another important effect, by using a bilinear filter with a 2-sample search range, DMVR does not access more reference samples compared to the normal motion compensation process. After the refined MV is obtained by the DMVR search process, a normal 8-tap interpolation filter can be applied to generate the final prediction. Samples that may not be required for the interpolation process based on the original MV but may be required for the interpolation process based on the refined MV can be padded from the available samples so as not to access more reference samples compared to the normal MC process.

[0158] If the width and / or height of a CU is greater than 16 lumasamples, the CU can be further divided into sub-blocks with width and / or height equal to 16 lumasamples. The maximum unit size of the DMVR search process can be limited to 16×16.

[0159] In an embodiment, a merge with motion vector difference (MMVD) mode is used as seen in VVC, and the implicitly derived motion information can be used to predict samples of a CU (e.g., the current CU). The MMVD mode is used for either a skip mode or a merge mode using a motion vector representation method. An MMVD merge flag can be signaled to specify whether the MMVD mode is used for a CU, for example, after signaling of a skip flag or a merge flag.

[0160] In some examples, MMVD reuses merge candidates. Among the merge candidates, a candidate is selected and further extended by a motion vector representation method. MMVD provides a motion vector representation with simplified signaling. In some examples, the motion vector representation method includes a starting point, a magnitude of motion, and a direction of motion.

[0161] In some examples (e.g., VVC), the MMVD technique can use a merge candidate list to select a candidate for the starting point. However, in one example, only candidates that are of the default merge type (MRG_TYPE_DEFAULT_N) are considered for the extension of MMVD.

[0162] In some examples, a base candidate index is used to define the starting point. The base candidate index indicates the best candidate among the candidates in the list shown in Table 1. For example, the list is a merge candidate list with a motion vector predictor (MVP). The base candidate index can indicate the best candidate within the merge candidate list.

Table 1

[0163] Note that in one example, when the number of base candidates is equal to 1, it should be noted that the base candidate IDX is not signaled.

[0164] In MMVD mode, after a merge candidate (also called an MV base or an MV starting point) is selected, the merge candidate can be refined by additional information such as the signaled MVD information. The additional information can include an index (e.g., a distance index, e.g., mmvd_distance_idx[x0][y0]) used to specify the magnitude of the motion, and an index (e.g., a direction index, e.g., mmvd_direction_idx[x0][y0]) used to indicate the direction of the motion. In MMVD mode, one of the first two candidates in the merge list can be selected as the MV base. For example, a merge candidate flag (e.g., mmvd_cand_flag[x0][y0]) indicates one of the first two candidates in the merge list. The merge candidate flag can be signaled to indicate (e.g., specify) which one of the first two candidates is selected. The additional information can indicate the MVD (or motion offset) with respect to the MV base. For example, the magnitude of the motion indicates the magnitude of the MVD, and the direction of the motion indicates the direction of the MVD.

[0165] In one example, a merge candidate selected from the merge candidate list is used to provide a starting point or an MV starting point in the reference picture. The motion vector of the current block can be represented by the starting point and a motion offset (or MVD) including the magnitude and direction of the motion with respect to the starting point. On the encoder side, the selection of the merge candidate and the determination of the motion offset can be based on a search process (evaluation process) as shown in FIG. 19. On the decoder side, the selected merge candidate and the motion offset can be determined based on the signaling from the encoder side.

[0166] FIG. 19 shows an example of a search process (1900) in the MMVD mode. FIG. 20 shows an example of search points in the MMVD mode. In some examples, some or all of the search points in FIG. 20 are used in the search process (1900) of FIG. 19. By executing the search process (1900), for example, on the encoder side, additional information including a merge candidate flag (e.g., mmvd_cand_flag[x0][y0]), a distance index (e.g., mmvd_distance_idx[x0][y0]), and a direction index (e.g., mmvd_direction_idx[x0][y0]) can be determined for the current block (1901) in the current picture (or current frame).

[0167] The first motion vector (1911) and the second motion vector (1921) belonging to the first merge candidate are shown. The first motion vector (1911) and the second motion vector (1921) are MV start points used in the search process (1900). The first merge candidate can be a merge candidate on the merge candidate list configured for the current block (1901). The first motion vector (1911) and the second motion vector (1921) can be associated with two reference pictures (1902) and (1903) in the reference picture lists L0 and L1, respectively. Referring to FIGS. 19 and 20, the first motion vector (1911) and the second motion vector (1921) can point to two start points (2011) and (2021) in the reference pictures (1902) and (1903), respectively, as shown in FIG. 20.

[0168] Referring to FIG. 20, the two starting points (2011) and (2021) in FIG. 20 can be determined by the reference pictures (1902) and (1903). In one example, based on the starting points (2011) and (2021), a plurality of predefined points extending in the vertical direction (represented by +Y or -Y) or the horizontal direction (represented by +X or -X) in the reference pictures (1902) and (1903) can be evaluated. In one example, a pair of points that mirror each other with respect to each starting point (2011) or (2021), such as the pair of points (2014) and (2024) (indicated by a 1S shift in FIG. 19) or the pair of points (2015) and (2025) (indicated by a 2S shift in FIG. 19), can be used to determine a pair of motion vectors (e.g., MV(1913) and (1923) in FIG. 19) that can form a motion vector predictor candidate for the current block (1901). The motion vector predictor candidates (e.g., MV(1913) and (1923) in FIG. 19) determined based on the predefined points surrounding the starting point (2011) or (2021) can be evaluated.

[0169] The distance index (e.g., mmvd_distance_idx[x0][y0]) can specify information about the magnitude of the motion and indicate a predefined offset (e.g., 1S or 2S in FIG. 19) from the starting point indicated by the merge candidate flag. Note that the predefined offset is also called an MMVD step in one example.

[0170] Referring to FIG. 19, an offset (e.g., MVD(1912) or MVD(1922)) can be applied (e.g., added) to the horizontal or vertical component of the starting MV (e.g., MV(1911) or (1921)). An exemplary relationship between the distance index (IDX) and the pre-defined offset is shown in Table 2. When the full pel MMVD is off, e.g., when the full pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is equal to 0, the range of the MMVD pre-defined offset can be from 1 / 4 luma samples to 32 luma samples. When the full pel MMVD is off, the pre-defined offset can have non-integer values, e.g., fractions of luma samples (e.g., 1 / 4 pixel or 1 / 2 pixel). When the full pel MMVD is on, e.g., when the full pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is equal to 1, the range of the MMVD pre-defined offset can be from 1 luma sample to 128 luma samples. In one example, when the full pel MMVD is on, the pre-defined offset has only integer values, e.g., 1 or more luma samples.

Table 2

[0171] The direction index can represent the direction of the MVD (or the direction of motion) relative to the starting point. In one example, the direction index represents one of the 4 directions shown in Table 3. The meaning of the MVD signs in Table 3 can vary according to the information of the starting MV. In one example, when the starting MV is a unidirectional prediction MV, or when the starting MV is a bidirectional prediction MV where both lists point to the same side of the current picture (e.g., the POCs of the two reference pictures are both greater than the POC of the current picture, or the POCs of the two reference pictures are both less than the POC of the current picture), the MVD signs in Table 3 specify the coding of the MV offset (or MVD) added to the starting MV.

[0172] When the starting MV is a bi-predicted MV where two MVs point to different sides of the current picture (for example, the POC of one reference picture is larger than the current POC, and the POC of the other reference picture is smaller than the current POC), the MVD code in Table 3 indicates the sign of the MV offset (or MVD) added to the MV component of list 0 of the starting MV, and the MVD coding of the MV in list 1 has the opposite value. Referring to Figure 19, the starting MVs (1911) and (1921) are bi-predicted MVs where the two MVs (1911) and (1921) point to different sides of the current picture. The POC of the L1 reference picture (1903) is larger than the POC of the current picture, and the POC of the L0 reference picture (1902) is smaller than the POC of the current picture. The MVD code (for example, the sign of the x-axis “+”) indicated by the direction index (for example, 00) in Table 2 specifies the sign of the MVD (for example, MVD(1912)) added to the MV component of list 0 of the starting MV (for example, (1911)), and the MVD code of the MVD(1922) of the MV component of list 1 of the starting MV (for example, (1921)) has an opposite value such as the sign “-” opposite to the sign “+” of MVD(1912).

[0173] Referring to Table 3, the direction index 00 indicates the positive direction on the x-axis, the direction index 01 indicates the negative direction on the x-axis, the direction index 10 indicates the positive direction on the y-axis, and the direction index 11 indicates the negative direction on the y-axis.

Table 3

[0174] The syntax element mmvd_merge_flag[x0][y0] can be used to represent the MMVD merge flag of the current CU. In one example, an MMVD merge flag (e.g., mmvd_merge_flag[x0][y0]) equal to 1 specifies that the MMVD mode is used to generate the inter prediction parameters of the current CU. An MMVD merge flag (mmvd_merge_flag[x0][y0]) equal to 0 specifies that the MMVD mode is not used to generate the inter prediction parameters. The array indices x0 and y0 can specify the position (x0, y0) of the top-left luma sample of the coding block (e.g., CB) under consideration with respect to the top-left luma sample of the picture (e.g., the current picture).

[0175] If the MMVD merge flag (e.g., mmvd_merge_flag[x0][y0]) does not exist for the current CU, the MMVD merge flag (e.g., mmvd_merge_flag[x0][y0]) can be assumed to be equal to 0 for the current CU.

[0176] In some examples, as seen in the VVC design, a single context is used to signal the MMVD merge flag (e.g., mmvd_merge_flag). For example, the single context is used to code (e.g., encode and / or decode) the MMVD merge flag with context adaptive binary arithmetic coding (CABAC).

[0177] The syntax element mmvd_cand_flag[x0][y0] can represent a merge candidate flag. In one example, the merge candidate flag (e.g., mmvd_cand_flag[x0][y0]) specifies whether the first (0) or second (1) candidate in the merge candidate list is used with an MVD derived from the distance index (e.g., mmvd_distance_idx[x0][y0]) and the direction index (e.g., mmvd_direction_idx[x0][y0]). The array indices x0 and y0 can specify the position (x0, y0) of the top-left luma sample of the coding block (e.g., CB) under consideration relative to the top-left luma sample of the picture (e.g., the current picture).

[0178] If the merge candidate flag (e.g., mmvd_cand_flag[x0][y0]) does not exist, the merge candidate flag (e.g., mmvd_cand_flag[x0][y0]) can be assumed to be equal to 0.

[0179] The syntax element mmvd_distance_idx[x0][y0] can represent a distance index. In one example, the distance index (e.g., mmvd_distance_idx[x0][y0]) specifies the index used to derive MmvdDistance[x0][y0] as shown in Table 4. The array indices x0 and y0 can specify the position (x0, y0) of the top-left luma sample of the coding block (e.g., CB) under consideration relative to the top-left luma sample of the picture (e.g., the current picture). [Table 4]

[0180] The first column of Table 4 indicates the distance index (e.g., mmvd_distance_idx[x0][y0]). The second column of Table 4 indicates the magnitude of motion (e.g., MmvdDistance[x0][y0]) when the full pel MMVD is off, i.e., when the full pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is equal to 0. The third column of Table 4 indicates the magnitude of motion (e.g., MmvdDistance[x0][y0]) when the full pel MMVD is on, i.e., when the full pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is equal to 1.

[0181] In one example, the units of the second and third columns of Table 4 are 1 / 4 luma samples. Referring to the first row of Table 4, when the distance index (e.g., mmvd_distance_idx[x0][y0]) is 0, the magnitude of motion (e.g., MmvdDistance[x0][y0]) is 1 when the full pel MMVD is off (e.g., slice_fpel_mmvd_enabled_flag is 0). The magnitude of motion (e.g., MmvdDistance[x0][y0]) is 1 × 1 / 4 luma sample, i.e., 1 / 4 luma sample. When the distance index (e.g., mmvd_distance_idx[x0][y0]) is 0, the magnitude of motion (e.g., MmvdDistance[x0][y0]) is 4 when the full pel MMVD is on (e.g., slice_fpel_mmvd_enabled_flag is 1). The magnitude of motion (e.g., MmvdDistance[x0][y0]) is 4 × 1 / 4 luma sample, i.e., 1 luma sample.

[0182] In one example, the second column of Table 4 (in units of 1 / 4 luma samples) corresponds to the second row of Table 1 (in units of 1 luma sample), and the third column of Table 4 (in units of 1 / 4 luma samples) corresponds to the third row of Table 2 (in units of 1 luma sample).

[0183] The syntax element mmvd_direction_idx[x0][y0] can represent a direction index. In one example, the direction index (e.g., mmvd_direction_idx[x0][y0]) specifies the index used to derive the motion direction (e.g., MmvdSign[x0][y0])) specified in Table 5. The array indices x0 and y0 specify the position (x0, y0) of the top-left luma sample of the coding block (e.g., CB) under consideration relative to the top-left luma sample of the picture (e.g., the current picture). The first column of Table 5 shows the direction index (e.g., mmvd_direction_idx[x0][y0]). The second column of Table 5 shows the first sign (e.g., MmvdSign[x0][y0][0]) of the first component of the MVD (e.g., MVD x or MmvdOffset[x0][y0][0]). The third column of Table 5 shows the second sign (e.g., MmvdSign[x0][y0][1]) of the second component of the MVD (e.g., MVD y or MmvdOffset[x0][y0][1]).

Table 5

[0184] The first component (e.g., MmvdOffset[x0][y0][0]) and the second component (e.g., MmvdOffset[x0][y0][1]) of the MVD or the offset MmvdOffset[x0][y0] can be derived as follows:

Equation

Equation

[0185] In one example, the distance index (e.g., mmvd_distance_idx[x0][y0]) is 3 and the direction index (e.g., mmvd_distance_idx[x0][y0]) is 2. Based on Table 5 and the fact that the direction index (e.g., mmvd_distance_idx[x0][y0]) is 2, the first sign (e.g., MmvdSign[x0][y0][0]) of the first component of the MVD (e.g., MVD x or MmvdOffset[x0][y0][0]) is 0, and the second sign (e.g., MmvdSign[x0][y0][1]) of the second component of the MVD (e.g., MVD y or MmvdOffset[x0][y0][1]) is "+1". In this example, the MVD is along the positive vertical direction (+y) and has no horizontal component.

[0186] When the full pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is equal to 0 and full pel MMVD is off, based on Table 4 and the fact that the distance index (e.g., mmvd_distance_idx[x0][y0]) is 3, the magnitude of the motion indicated by MmvdDistance[x0][y0] is 8. Based on equations (40)-(41), the first component of the MVD (e.g., MmvdOffset[x0][y0][0]) is (8 << 2) × 0 = 0, and the second component of the MVD (e.g., MmvdOffset[x0][y0][1]) is (8 << 2) × (+1) = 2 (luma samples).

[0187] When the complete per-MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is equal to 1 and the complete per-MMVD is on, based on Table 4 and the distance index (e.g., mmvd_distance_idx[x0][y0]) being 3, the magnitude of the motion indicated by MmvdDistance[x0][y0] is 32. Based on Equations (40)-(41), the first component of the MVD (e.g., MmvdOffset[x0][y0][0]) is (32<<2)×0 = 0, and the second component of the MVD (e.g., MmvdOffset[x0][y0][1]) is (32<<2)×(+1)=8 (luma samples).

[0188] According to an aspect of the present disclosure, affine merge by motion vector difference (affine MMVD) can be used in video coding. Affine MMVD selects an affine merge candidate available from a sub-block based merge list as a basic predictor. Affine MMVD applies a motion vector offset to the motion vector value of each control point from the basic predictor. In one example, when no affine merge candidate is available, affine MMVD is not used. In some examples, the distance index and the offset direction index may then be signaled.

[0189] In some examples, the distance index is signaled from an offset table to indicate which distance offset should be used, as shown in Table 6: [Table 6]

[0190] In some examples, the direction index can represent four directions, as shown in Table 7, where there is an MV difference only in the x or y direction and not in both directions. [Table 7]

[0191] In some examples, the inter prediction is a unidirectional prediction, and the signaled distance offset is applied to the offset direction of each control point predictor to generate a result including the MV value of each control point.

[0192] In some examples, the inter prediction is a bidirectional prediction, and the signaled distance offset can be applied to the signaled offset direction in the case of the L0 motion vector of the control point predictor, and the offset applied to the L1 MV can be applied based on mirroring or scaling as in the following specified examples.

[0193] In a specific example, the inter prediction is a bidirectional prediction, and the signaled distance offset is applied to the signaled offset direction in the case of the L0 motion vector of the control point predictor. In the case of the L1 CPMV, the offset is applied based on mirroring, which means that the same amount of distance offset with the opposite direction is applied.

[0194] In other specific examples, an offset mirroring method based on the POC distance is used for bidirectional prediction. When the basic candidate is bidirectionally predicted, the offset applied to L0 is as signaled, and the offset for L1 depends on the temporal position of the reference pictures in list L0 and list L1. For example, if both reference pictures are on the same temporal side of the current picture, the same distance offset and the same offset direction are applied for both CPMVs of L0 and L1. In other examples, if the two reference pictures are on different sides of the current picture, the CPMV of L1 can have a distance offset applied in the opposite offset direction.

[0195] In other specific examples, an offset scaling method based on the POC distance is used for bidirectional prediction. When the basic candidate is bidirectionally predicted, the offset applied to L0 is as signaled, and the offset for L1 can be scaled based on the temporal positions of the reference pictures in list 0 and list L1.

[0196] In some examples, the distance offset value range is extended. For example, three sets of distance offset values may be supplied, and one set of distance offset values may be adaptively selected based on the picture resolution. In one example, the offset table is selected based on the picture resolution. Table 8 shows an example of an extended distance offset table that includes three sets of distance offset values respectively related to different picture resolutions. Based on the picture resolution, one set of distance offset values may be selected.

Table 8

[0197] Template matching (TM) technology can be used in video / image coding. To further improve the compression efficiency of the VVC standard, for example, TM can be used to refine the MV. In one example, TM is used on the decoder side. According to the TM mode, the MV can be refined by constructing a template (e.g., the current template) of a block (e.g., the current block) in the current picture and determining the closest match between the template of the block in the current picture and multiple possible templates (e.g., multiple possible reference templates) in the reference picture. In an embodiment, the template of the block in the current picture can include the left adjacent reconstructed samples and the upper adjacent reconstructed samples of the block. TM can be used in video / image coding beyond VVC.

[0198] FIG. 21 shows an example of template matching (2100). TM can be used to derive motion information of the current CU (e.g., the current block) (2101) by determining the closest match between a template of the current CU (e.g., the current template) (2121) within the current picture (2110) and a template (e.g., the reference template) among a plurality of possible templates within the reference picture (2111) (e.g., one of the plurality of possible templates is template (2125)). The template (2121) of the current CU (2101) can have any suitable shape and any suitable size.

[0199] In an embodiment, the template (2121) of the current CU (2101) includes an upper template (2122) and a left template (2123). Each of the upper template (2122) and the left template (2123) can have any suitable shape and any suitable size.

[0200] The upper template (2122) can include samples within one or more upper adjacent blocks of the current CU (2101). In one example, the upper template (2122) includes four sample rows within one or more upper adjacent blocks of the current CU (2101). The left template (2123) can include samples within one or more left adjacent blocks of the current CU (2101). In one example, the left template (2123) includes four sample columns within one or more left adjacent blocks of the current CU (2101).

[0201] Each one of a plurality of possible templates (e.g., template (2125)) within the reference picture (2111) corresponds to the template (2121) within the current picture (2110). In an embodiment, the initial MV (2102) points from the current CU (2101) to the reference block (2103) within the reference picture (2111). Each one of a plurality of possible templates (e.g., template (2125)) within the reference picture (2111) and the template (2121) within the current picture (2110) can have the same shape and the same size. For example, the template (2125) of the reference block (2103) includes the upper template (2126) within the reference picture (2111) and the left template (2127) within the reference picture (2111). The upper template (2126) can include samples within one or more upper adjacent blocks of the reference block (2103). The left template (2127) can include samples within one or more left adjacent blocks of the reference block (2103).

[0202] The TM cost can be determined based on pairs of templates such as a template (e.g., the current template) (2121) and a template (e.g., the reference template) (2125). The TM cost can indicate the match between template (2121) and template (2125). The optimized MV (or the final MV) can be determined based on a search around the initial MV (2102) of the current CU (2101) within the search range (2115). The search range (2115) can have any suitable shape and any suitable number of reference samples. In one example, the search range (2115) within the reference picture (2111) includes a [-L, L] pel range, where L is a positive integer such as 8 (e.g., 8 samples). For example, a difference (e.g., [0, 1]) is determined based on the search range, and the intermediate MV is determined by the sum of the initial MV (2102) and the difference (e.g., [0, 1]). The intermediate reference block and the corresponding template within the reference picture (2111) can be determined based on the intermediate MV. The TM cost can be determined based on template (2121) and the intermediate template within the reference picture (2111). The TM cost can correspond to differences (e.g., [0, 0], [0, 1], etc. corresponding to the initial MV (2101)) determined based on the search range (2115). In one example, the difference corresponding to the minimum TM cost is selected, and the optimized MV is the sum of the difference corresponding to the minimum TM cost and the initial MV (2102). As described above, TM can derive the final motion information (e.g., the optimized MV) from the initial motion information (e.g., the initial MV 2102).

[0203] In the example of FIG. 21, a better MV can be searched around the initial motion vector of the current CU within a search range such as [-8 pels, +8 pels].

[0204] TM can be applied in affine modes such as the affine AMVP mode and the affine merge mode, and can be called affine TM. FIG. 22 shows an example of TM (2200) as seen in the affine merge mode. The template (2221) of the current block (e.g., the current CU) (2201) can correspond to the template (e.g., the template (2121) in FIG. 21) within the TM applied to the translational motion model. The reference template (2225) of the reference block in the reference picture can include a plurality of sub-block templates (e.g., 4×4 sub-blocks) indicated by the MVs derived from the control point MVs (CPMVs) of the adjacent sub-blocks (e.g., A0 to A3 and L0 to L3 shown in FIG. 22) at the block boundary.

[0205] The search process for TM applied in the affine mode (e.g., the affine merge mode) can start from CPMV0 while keeping other CPMVs (e.g., (i) CPMV1 when the 4-parameter model is used, or (ii) CPMV2 when the 6-parameter model is used) constant. The search can be performed in the horizontal and vertical directions. In one example, the diagonal direction follows after those searches only if the zero vector is not the best difference vector found from the horizontal and vertical searches. The affine TM can repeat the same search process for CPMV1. The affine TM can repeat the same search process for CPMV2 when the 6-parameter model is used. Based on the refined CPMV, if the zero vector is not the best difference vector from the previous iteration and the search process has not been repeated more than three times, the entire search process can resume from the refined CPMV0.

[0206] According to an aspect of the present disclosure, the candidate rearrangement technology based on template matching can be used to reduce signaling overhead. For example, a technology called adaptive rearrangement of merge candidates by template matching (ARMC-TM) can be used.

[0207] In some examples, using ARMC-TM, after the merge candidates are adaptively reordered by template matching (TM). ARMC-TM can be applied to regular merge mode, template matching (TM) merge mode, and affine merge mode (excluding SbTMVP candidates). In the case of TM merge mode, the merge candidates are reordered before the refinement process.

[0208] In some examples, using ARMC-TM, after the merge candidate list is constructed, the merge candidates are divided into several subgroups. In one example, the subgroup size is set to 5 in the case of regular merge mode and TM merge mode. In other examples, the subgroup size is set to 3 in the case of affine merge mode. The merge candidates within each subgroup are sorted in ascending order according to the cost value based on template matching. For simplicity, in some examples, the merge candidates in the last subgroup instead of the first are not sorted.

[0209] The template matching cost of the merge candidate is measured by the sum of absolute differences (SAD) between the samples of the template of the current block and the reference samples corresponding to that template (also called the reference template in one example). The template includes a set of reconstructed samples adjacent to the current block. The reference samples of the template are found according to the motion information of the merge candidate.

[0210] When the merge candidate utilizes bidirectional prediction, the reference samples of the template of the merge candidate are also generated by bidirectional prediction.

[0211] FIG. 23 shows a diagram representing a reference sample of a template of a current block for a merge candidate of dual prediction. In FIG. 23, the current picture (2310) includes the current block for coding. When the merge candidate is a dual prediction merge candidate, the MV of the merge candidate can point to a first reference block in the reference picture (2320) and a second reference block in the second reference picture (2330). The template of the current block is denoted by (T), and the template includes a set of reconstructed samples adjacent to the current block. The first set of reference samples of the template is in the first reference picture (2320) adjacent to the first reference block, and the second set of reference samples of the template is in the second reference picture (2330) adjacent to the second reference block. In an example, the template matching cost of the dual prediction merge candidate is calculated by adding the first sum of absolute differences (SAD) between the samples of the template of the current block and the first set of reference samples of the template and the second sum of absolute differences (SAD) between the samples of the template of the current block and the second set of reference samples of the template.

[0212] In some examples, the merge candidate can be a sub-block based merge candidate. In an example, for a sub-block based merge candidate where the sub-block size is equal to Wsub×Hsub, the upper template can include several sub-templates having a size of Wsub×1, and the left template can include several sub-templates having a size of 1×Hsub. Wsub can be the width of the sub-block, and Hsub can be the height of the sub-block.

[0213] An exemplary derivation of a template and a reference sample of the template for a current block including sub-block-based merge candidates can be shown in FIG. 24. As shown in FIG. 24, the current block (2402) can be included in the current picture (2404). The current block (2402) can include sub-blocks A to G in the first row and the first column. The current block (2402) can have a template (2406) adjacent to the upper side and the left side of the current block (2402). The collocated block (2408) of the current block (2402) is within the reference picture (2410). The collocated block (2408) can include sub-blocks A to G corresponding to the sub-blocks A to G in the current block (2402) in the first row and the first column. The sub-block motion information (e.g., corresponding to an affine motion vector) of the sub-blocks A to G in the first row and the first column of the current block (2402) can be used to derive a reference sample of a sub-template (or sub-reference template) of the collocated block (2408). For example, the motion information of the sub-blocks A, E, F, and G of the current block (2402) can be applied to derive a reference sample of a sub-template located adjacent to the left side of the sub-blocks A, E, F, and G of the collocated block (2408). The sub-template adjacent to the left side of the sub-blocks A, E, F, and G of the collocated block (2408) can form a left reference template of the collocated block (2408). The motion information of the sub-blocks A, B, C, and D of the current block (2402) can be applied to derive a reference sample of a sub-template located adjacent to the upper side of the sub-blocks A, B, C, and D of the collocated block (2408). The sub-template adjacent to the upper side of the sub-blocks A, B, C, and D of the collocated block (2408) can further form an upper reference template of the collocated block (2408).

[0214] In some examples, ARMC based on the MV candidate type may be used. For example, the merge candidates of one single candidate type, such as TMVP or non-adjacent MVP (NA-MVP), are sorted based on the ARMC TM cost value. The sorted candidates are then added to the merge candidate list. For example, the TMVP candidate type ARMC can add more TMVP candidates with more temporal positions and different inter-prediction directions to perform sorting and selection. Further, the NA-MVP candidate type ARMC extends the non-adjacent MVP with more spatial non-adjacent positions. The reference picture of the TMVP candidate can be selected from any one of the reference pictures in the list according to the scaling factor. For example, the selected reference picture is the one with the scaling factor closest to 1.

[0215] According to an aspect of the present disclosure, sorting of candidates based on template matching can be performed for MMVD and affine MMVD.

[0216] In some examples, the MMVD offset is extended to more positions for the MMVD mode and the affine MMVD mode.

[0217] FIG. 25 shows a diagram representing the direction in which refined positions are added for MMVD. In FIG. 25, additional refined positions are added along the k×π / 8 diagonal angle, where k is an integer. The position (2501) corresponds to the basic candidate and can be the starting position, and the positions (2511)-(2514) are in the directions of 0, π / 2, π, and 3π / 2, respectively. Further directions are added. For example, the positions (2521)-(2524) are in the directions of π / 4, 3π / 4, 5π / 4, and 7π / 4, respectively, and the positions (2531)-(2538) are in π / 8, 3π / 8, 5π / 8, 7π / 8, 9π / 8, 11π / 8, 13π / 8, and 15π / 8, respectively. Thus, the number of directions increases from 4 to 16. Further, in one example, each direction can have 6 MMVD refined positions. The total number of possible MMVD refined positions is 16×6.

[0218] In accordance with aspects of the present disclosure, the SAD cost between the current template (e.g., one row above and one column to the left of the current block) and the reference template can be calculated for each refinement position. Based on the SAD cost of the refinement positions, all possible MMVD refinement positions (16×6) for each basic candidate are sorted. Then, the upper portion of the refinement positions, such as the top 1 / 8 of the refinement positions (e.g., 12), having the minimum template SAD, is retained as the available positions for MMVD index coding. The MMVD index is binarized by a Rice code with a parameter equal to 2.

[0219] In some examples, the refinement positions of the affine MMVD can be increased, and the sorting of candidates based on template matching can be applied to the affine MMVD sorting. For example, the affine MMVD refinement positions are in eight directions along the k×π / 4 diagonal angles, e.g., 0, π / 4, π / 2, 3π / 4, π, 5π / 4, 3π / 2, and 7π / 4, respectively. Each direction can have six affine MMVD refinement positions. The total number of possible affine MMVD refinement positions is 8 times 6. In one example, the SAD cost between the current template (e.g., one row above and one column to the left of the current block) and the reference template can be calculated for each refinement position. Based on the SAD cost of the refinement positions, all possible affine MMVD refinement positions (8×6) for each basic candidate are sorted. Then, the upper portion of the refinement positions, such as the top 1 / 2 of the refinement positions (e.g., 24), having the minimum template SAD, is retained as the available positions for MMVD index coding.

[0220] In some examples, diversity rearrangement is applied based on the template matching cost. Diversity rearrangement can increase diversity and improve index coding efficiency. In some examples, candidates that are too redundant in terms of rate distortion (RD) are detected to create diversity within the merge candidate list. In one example, a candidate is considered redundant if the template matching cost difference between the candidate and its preceding candidate is less than the lambda value, as expressed by |D1 - D2| < λ. Here, D1 and D2 are the template matching costs obtained during the first ARMC ordering, and λ is the Lagrange parameter used in the RD criterion on the encoder side.

[0221] In some examples, an algorithm for diversity rearrangement can be executed. The algorithm can determine the minimum template matching cost difference between each candidate and its preceding candidate among all candidates in the merge candidate list. If the minimum template matching cost difference is greater than or equal to λ, the merge candidate list is considered to have sufficient diversity and the rearrangement stops. If the minimum template matching cost difference is less than λ, the candidate with the minimum template matching cost difference is considered redundant and the candidate is moved to another position in the merge candidate list. Another position is the first position where the candidate is sufficiently diverse compared to its preceding candidate.

[0222] The algorithm can stop after a finite number of iterations (when the minimum cost difference is not less than λ).

[0223] In some examples, the algorithm is applied to various merge modes such as the regular merge mode in ECM - 5.0, the merge mode by template matching, the merge mode by bilateral matching, the affine merge mode, etc. In some examples, a similar algorithm is applied to the merge MMVD method and the code MVD prediction method that also use ARMC for rearrangement.

[0224] In some examples, the value of λ is set equal to the rate distortion criterion λ used to select the best merge candidate on the encoder side for low latency settings, and to the value λ corresponding to the QP for random access settings. In some examples, the set of λ values corresponding to each signaled QP offset is supplied in the SPS, or in the slice header for QP offsets not present in the SPS.

[0225] In some examples, MVD sign prediction techniques are used. In one example, the possible MVD sign combinations (e.g., various combinations of signs in the x and y directions) are sorted according to the template matching cost of the possible MVD sign combinations, the index corresponding to the true MVD sign combination is derived, and context coding is performed. According to the MVD sign prediction technique, the true MVD sign combination is likely to be in front of the sorted order. Thus, an appropriate signaling technique can be used to signal the index with low signaling cost.

[0226] In one example, on the decoder side, the true MVD sign can be derived. For example, the magnitude of the MVD component can be parsed, and the context-coded MVD sign prediction index is parsed from the bitstream carrying the video. Further, the MV candidates can be formed by combining the possible MVD sign combinations and the magnitude of the MVD component, and the MV candidates can be added to the MV predictor list. The template matching cost of the MV candidates in the MV predictor list can be calculated. The MV candidates in the MV predictor list can be sorted according to the template matching cost. Then, the context-coded MVD sign prediction index is used to retrieve the true MVD sign combination from the MV predictor list. The MVD sign prediction technique can be applied to various modes involving MVD, such as the inter-AMVP mode, the affine AMVP mode, the MMVD mode, and the affine MMVD mode.

[0227] In some examples, a technique called history-parameter-based affine model inheritance may be used. Specifically, in some examples, a first history parameter table (HPT) and a second HPT are established.

[0228] FIG. 26 shows a diagram (2600) representing the first HPT and the second HPT in some examples.

[0229] As shown in FIG. 26, the entries of the first HPT store sets of affine parameters for an affine model, such as a, b, c, and d, and each affine parameter is represented by a 16-bit signed integer. The entries of the first HPT are classified by a reference list (e.g., reference picture list L0 or reference picture list L1) and a reference index. Five reference indexes are supported for each reference list of the first HPT. Stated formally, in one example, the category of the first HPT (denoted as HPTCat) is calculated as in Equation (42):

Equation

[0230] For each category, at most seven entries can be stored, and as a result, there are a total of 70 entries in the first HPT. At the beginning of each CTU row, the number of entries for each category is initialized to zero. After decoding the affine-coded CU using the reference list RefList cur and RefIdx cur , the affine parameters are used to update the entry within the category HPTCat(RefList cur ,RefIdx cur ).

[0231] In some examples, a history-affine-parameter-based candidate (HAPC) is derived from one of seven neighboring 4×4 blocks denoted as A0, A1, A2, B0, B1, B2, or B3 in FIG. 26 and a set of affine parameters stored in a corresponding entry within a first HPT. The MV of the neighboring 4×4 block serves as the base MV. Formulated, the MV of the current block at position (x, y) is calculated as in Equation (43): [Number] where (mv h base , mv v base ) represents the MV of the neighboring 4×4 block, and (x base , y base ) represents the center position of the neighboring 4×4 block. (x, y) can be the upper left, upper right, and lower left corners of the current block to obtain the center position MV (CPMV) of the current block, or (x, y) can be the center of the current block to obtain the normal MV of the current block.

[0232] A second history-parameter table (HPT) containing base MV information is also added. The second HPT can include nine entries, and the entries can include the base MV, reference index, and four affine parameters for each reference list, as well as the base position. In one example, an additional merged HAPC can be generated based on the base MV information stored in the entries of the second HPT and the corresponding affine model (e.g., affine parameters).

[0233] Furthermore, in some examples, pairwise affine merge candidates are generated by two affine merge candidates that are either history-derived or not history-derived. In one example, a pairwise affine merge candidate is generated by averaging the CPMVs of existing affine merge candidates within a candidate list.

[0234] In some examples, as a response to the newly introduced HAPC, the size of the sub-block-based merge candidate list is increased from 5 to 15, so that all can participate in the ARMC process.

[0235] In the above description, the HPT (e.g., the first HPT and the second HPT) is updated online. In addition to one line of the updated HPT, the HPT stored in the upper / upper-right CTU of the current CTU can, in some examples, be used by the blocks within the current CTU. After the encoding / decoding of the CTU, the HPT may be stored in the line buffer for use in the next CTU row.

[0236] FIG. 27 is a diagram that, in some examples, represents a history parameter table stored in a line buffer. In FIG. 27, a picture (2700) is partitioned into CTUs. FIG. 27 shows CTU row k and CTU row k+1. The current CTU (2710) is within CTU row k+1. For the coding of the current block (2711), the HPT (2701) stored in the CTU above the current CTU (2710) and the HPT (2702) stored in the CTU in the upper-right of the current CTU (2710) can, in some examples, be used by the blocks within the current CTU. The HPT (e.g., the first HPT and the second HPT) is updated online and stored in the line buffer of the current CTU (2710) after the decoding of the last coding block within the current CTU.

[0237] According to an aspect of the present disclosure, non-adjacent spatial neighborhoods can be used for affine mode.

[0238] In an affine mode with non-adjacent spatial neighbors (NA-AFF), non-adjacent spatial neighbors can be obtained.

[0239] Figures 28A to 28B show patterns for obtaining non - adjacent spatial neighborhoods in some examples. Similar to existing non - adjacent regular merge candidates, the distance between the current CUT and the non - adjacent spatial neighborhood in NA - AFF is also defined based on the width and height of the current CU.

[0240] The motion information of the non - adjacent spatial neighborhood is used to generate additional inherited and constructed affine merge / AMVP candidates. Figure 28A represents generating additional inherited affine merge / AMVP candidates, and Figure 28B represents generating additional constructed affine merge / AMVP candidates.

[0241] Specifically, as shown by Figure 28A, for inherited candidates, the same derivation process of the inherited affine merge / AMVP candidates in VVC is not changed except that the CPMV is inherited from the non - adjacent spatial neighborhood. The non - adjacent spatial neighborhoods are checked based on their distances to the current block, that is, from near to far. At a specific distance, only the first available neighborhoods (coded in affine mode) from each side (e.g., left and top) of the current block are included for the derivation of the inherited candidates. As shown by the dashed arrows in Figure 28, the checking order of the neighborhoods on the left and top sides is bottom - to - top and right - to - left, respectively.

[0242] For the first type of constructed candidates, as shown by Figure 28B, the positions of the non - adjacent spatial neighborhood one to the left and the non - adjacent spatial neighborhood one above are first determined independently. Then, the position of the upper - left neighborhood can be determined accordingly, which can enclose a rectangular virtual block together with the left non - adjacent neighborhood and the upper non - adjacent neighborhood.

[0243] Next, as shown in Figure 29, the motion information of three neighboring spatial neighborhoods is used to form CPMV at the upper - left (A), upper - right (B), and lower - left (C) of the virtual block, and then the virtual block is projected onto the current CU to generate the corresponding constructed candidate.

[0244] In some examples, for the second type of configured candidates, the derivation process is similar to the configuration scheme in history-based affine model inheritance (HAMI). However, instead of using a history-based look-up table, the non-translational affine parameters are inherited from non-adjacent spatial neighborhoods. Specifically, the second type of affine configured candidates are generated from 1) the translational affine parameters of the 4×4 block of adjacent neighborhoods and 2) the non-translational affine parameters inherited from the non-adjacent spatial neighborhoods defined in FIG. 28A.

[0245] In some examples, the NA-AFF candidates are inserted into the existing affine merge candidate list and the affine AMVP candidate list in a specific order.

[0246] In one example, in the affine merge mode, the order is: 1. SbTMVP candidates, if available; 2. Inherited from adjacent neighborhoods; 3. Inherited from non-adjacent neighborhoods; 4. Configured from adjacent neighborhoods; 5. The second type of configured affine candidates from non-adjacent neighborhoods; 6. The first type of configured affine candidates from non-adjacent neighborhoods; 7. Including zero MV.

[0247] In other examples, in the affine AMVP mode, the order is: 1. Inherited from adjacent neighborhoods; 2. Configured from adjacent neighborhoods; 3. Translational MV from adjacent neighborhoods; 4. Translational MV from temporal neighborhoods; 5. Inherited from non-adjacent neighborhoods; 6. The first type of configured affine candidates from non-adjacent neighborhoods; 7. Including zero MV.

[0248] With the inclusion of the additional candidates generated by NA-AFF, the size of the affine merge candidate list increases from 5 to 15. The subgroup of ARMC for the affine merge mode increases from 3 to 15.

[0249] In some video coders (e.g., ECM-5.0 software), NA-AFF is implemented without adding constraints on memory usage.

[0250] In some examples (e.g., VCC), a geometric partitioning mode (GPM) is supported for inter prediction. The geometric partitioning mode is signaled as a kind of merge mode using a CU-level flag together with other merge modes such as regular merge mode, MMVD mode, CIIP mode, sub-block merge mode, etc. In some examples, a total of 64 partitions are supported by the geometric partitioning mode for each of the possible CU sizes w×h = 2m×2n (m,n ∈ {3···6}) except 8×64 and 64×.

[0251] In some examples, when the geometric partitioning mode is used, a CU is divided into two parts by a geometrically positioned straight line, also called a dividing line. The position of the dividing line can be mathematically derived based on the angular parameter and offset parameter of a specific partition. Each part of the two geometric partitions by the dividing line within the CU is inter predicted using its own motion, and only uni prediction is allowed for each partition. Thus, each part has one motion vector and one reference index. The uni prediction motion constraint is applied to ensure that a CU in GPM mode can be coded as a conventional bi prediction, e.g., two motion-compensated predictions are performed for each CU. In some examples, when the geometric partitioning mode is currently used for a CU, a geometric partition index indicating the partition mode of the geometric partition (e.g., indicating the angle and offset) and two merge indexes (one for each partition) are further signaled.

[0252] Intra Block Copy (IBC) is, in some examples, also referred to as current picture referencing and is applied in various video codecs such as High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), AOMedia Video 1 (AV1), etc. IBC coding tools are used in the IBC mode for image / video coding. Different video codecs may have specific features or IBC coding tools. IBC can include various modes in different standards such as Intra Block Copy (IBC) mode, Intra Block Copy (IntraBC) mode, Intra Template Matching Prediction (IntraTMP), etc.

[0253] Some IBC coding tools are used as current picture referencing (CPR) in the HEVC Screen Content Coding (SCC) extension. The IBC mode can use coding techniques that are used for inter prediction where the current picture is used as a reference picture in the IBC mode. The advantage of using the IBC mode is the reference structure of the IBC mode where 2D (two-dimensional) spatial vectors can be used as the representation of the addressing mechanism to reference samples. The architectural advantage of the IBC mode is that the changes to the specifications required for the integration of IBC are relatively small, and the implementation burden can be reduced when manufacturers have already implemented specific inter prediction techniques such as HEVC version 1. CPR in the HEVC SCC extension can be a special inter prediction mode, and as a result, the same syntax structure as that of the inter prediction mode and a decoding process similar to that of the inter prediction mode can be obtained.

[0254] The IBC mode can be integrated into the inter prediction process. In some examples, the IBC mode (or CPR) is an inter prediction mode, and an intra-only prediction slice will become a prediction slice that allows the use of the IBC mode. When the IBC mode is applicable, the coder can extend the reference picture list by only 1 entry such that the pointer points to the current picture. For example, the current picture is used as a 1-picture-size buffer of the shared decoded picture buffer (DBP). The signaling of the IBC mode can be implicit. For example, if the selected reference picture points to the current picture, the CU can use the IBC mode. In various embodiments, the reference samples used in the IBC process are not filtered, which is different from normal inter prediction. The corresponding reference picture used in the IBC process is a long-term reference. To minimize memory requirements, the coder can release the buffer after reconstructing the current picture. For example, the coder can release the buffer immediately after reconstructing the current picture. The filtered version of the reconstructed picture can be returned to the DPB by the coder as a short-term reference if the reconstructed picture is a reference picture.

[0255] In block vector (BV) coding, the reference to the reconstructed area can be performed by a 2D BV that is similar to that in inter prediction. The prediction and coding of the BV can reuse the prediction and coding of the MV in the inter prediction process. In some examples, the luma BV is at integer resolution instead of 1 / 4 precision of the MV used for a normally inter-coded CTU.

[0256] Figure 30 shows the BV currently associated with the CU (3001) according to an embodiment of the present disclosure. Each square (3000) can represent a CU. The gray, shaded area represents an already coded area (e.g., an encoded area), and the white, unshaded area represents an area to be coded (e.g., an area to be encoded). The current CTU (3000(4)) being reconstructed includes the current CU (3001), the coded area (3002), and the area to be coded (3003). In one example, the area (3003) is to be coded after coding the current CU (3001).

[0257] In the example, as seen in HEVC for example, the gray shaded area above the current CTU (3000(4)) excluding the two CTUs ((3000(1) to 3000(2))) can be used as a reference area in IBC mode to allow wavefront parallel processing (WPP). The BV allowed in HEVC can indicate a block within the reference area (the gray shaded area excluding the two CTUs (3000(1) to 3000(2))). For example, the BV (3005) allowed in HEVC indicates the reference block (3011).

[0258] In the example, as seen in VVC for example, in addition to the current CTU (3000(4)), the left adjacent CTU (3000(3)) to the left of the current CTU (3000(4)) is allowed as a reference area in IBC mode. In the example, the reference area used in IBC mode in VVC is within the dashed line area (3015) and includes coded samples. For example, the BV (3006) allowed in VVC indicates the reference block (3012).

[0259] In some examples, the decoded motion vector difference (MVD) of the BV (also referred to as BV difference (BVD)) can be left-shifted by 2 before adding to the corresponding BV predictor to reconstruct the final BV.

[0260] In some embodiments, special handling of the IBC mode may be required for implementation and execution, and the IBC mode and the inter prediction mode (e.g., the normal inter prediction mode) may be different, as described later. In an example, the reference samples used in the IBC mode are not filtered (e.g., the reconstructed samples before the in-loop filtering process such as DBF and sample adaptive offset (SAO) filtering are applied). Other inter prediction modes of HEVC (e.g., the normal inter prediction mode) can use filtered samples, e.g., reference samples filtered by the in-loop filtering process.

[0261] In some examples, luma sample interpolation is not performed in the IBC mode. Chroma sample interpolation can be performed in the IBC mode. In some examples, chroma sample interpolation is only necessary when the chroma BV is non-integer when the chroma BV is derived from the corresponding luma BV. In some examples, luma sample interpolation and chroma sample interpolation can be performed in the normal inter prediction mode.

[0262] In the IBC mode, in special cases, it may occur when the chroma BV is a non-integer BV and the reference block is close to the boundary of the available area (e.g., the reference area). For example, the surrounding reconstructed samples may be outside the boundary for performing chroma interpolation. In an example, the BV indicating a single line adjacent to the boundary may result in the surrounding reconstructed samples being outside the boundary.

[0263] Note that different video codecs may include additional features. For example, the IBC mode in ECM-5.0 has some improvements over the IBC method in VVC. In accordance with the aspects of the present disclosure, the IBC merge / AMVP list construction in ECM-5.0 is updated with some changes. In an example of the first change, the IBC merge / AMVP candidate can be inserted into the IBC merge / AMVP candidate list only when the IBC merge / AMVP candidate is valid. In an example of the second change, the top-right, bottom-left, and top-left spatial candidates and one pairwise average candidate can be added to the IBC merge / AMVP candidate list. In an example of the third change, template based adaptive reordering (ARMC-TM) is applied to the IBC merge list.

[0264] In ECM-5.0, history-based motion vector prediction (HMVP) can be used in the IBC mode, and the size of the IBC HMVP table increases to 25. In some examples, after up to 20 IBC merge candidates are derived by full pruning, they are sorted together. After sorting, the first 6 candidates with the lowest template matching cost are selected as the final candidates in the IBC merge list.

[0265] In ECM-5.0, the zero vector candidate for padding the IBC merge / AMVP candidate list is replaced with a set of BVP candidates located in the IBC reference region. The set of BVP candidates is called the padding candidate. The zero vector is not valid as a block vector in the IBC merge mode, and as a result, the zero vector is discarded as a BVP in the IBC candidate list.

[0266] FIG. 31 shows a diagram representing sets of padding candidates in some examples. In the example of FIG. 31, a buffer called the IBC buffer is allocated for reconstruction in IBC mode. During reconstruction, the IBC buffer is allocated to buffer a portion (3100) of a picture. The portion (3100) includes reconstructed samples within the reconstructed region (shown by being filled with diagonal lines), the current block (shown by being filled black), and the region to be reconstructed (shown by the white space). In FIG. 31, W represents the width of the current block and H represents the height of the current block. In the example of FIG. 31, a portion of the reconstructed region can be used for reference to reconstruct the current block, and that portion is called the reference region (3110). The reference region (3110) includes sub-regions (e.g., A, B, and C). In FIG. 31, three padding candidates (3101)-(3103) are located at the closest corners of the reference region (3110), and three additional padding candidates (3104)-(3106) are determined in the middle of the three sub-regions (A, B, and C). The coordinates of the padding candidates (3101)-(3106) are determined by the width and height of the current block and the parameters of ΔX and ΔY as shown in FIG. 31.

[0267] According to aspects of the present disclosure, template matching can be appropriately used in current picture reference, such as seen in, for example, IBC merge mode, IBC AMVP mode, etc. In some examples, a merge list called the IBC-TM merge list is used. The IBC-TM merge list is different from the normal merge list used in normal IBC merge mode. For example, candidates are selected according to a pruning method such as a pruning method based on the motion distance between candidates in normal TM merge mode. The trailing zero vector can be replaced by padding candidates such as left (-W,0) (shown as (3101) in FIG. 31), up (0,-H) (shown as (3102) in FIG. 31), and upper left (-W,-H) (shown as (3103) in FIG. 31), where W and H are the width and height of the current CU.

[0268] In the IBC-TM merge mode, as an example, the selected candidates are refined by template matching prior to rate distortion optimization (e.g., at the encoder side) or the decoding process (e.g., at the decoder side). In some examples, when both the IBC-TM merge mode and the normal IBC merge mode are used, the TM merge flag may be signaled to indicate whether the IBC-TM merge mode or the normal IBC merge mode is used.

[0269] In the IBC-TM AMVP mode, up to three candidates are selected from the IBC-TM merge list. Each of the three selected candidates is refined using template matching refinement and sorted according to the template matching cost. Then, only the first two candidates with lower template matching costs are considered in the motion estimation process as usual.

[0270] The template matching refinement for both the IBC-TM merge mode and the AMVP mode follows the IBC motion vector constraints: (i) being an integer and (ii) within the reference region.

[0271] FIG. 32 shows a diagram representing the IBC reference region depending on the position of the current CU. In the example of FIG. 32, the IBC buffer is allocated the CTU size. Each CTU contains four sub-regions, and the IBC buffer is configured to be updated according to the sub-regions. In the example of FIG. 32, the left CTU contains sub-regions 0 to 3, and the current CTU contains sub-regions 4 to 7. During reconstruction, as shown by state (1), when the current block is in sub-region 4, the IBC reference region includes sub-regions 1, 2, and 3; as shown by state (2), when the current block is in sub-region 5, the IBC reference region includes sub-regions 2, 3, and 4; as shown by state (3), when the current block is in sub-region 6, the IBC reference region includes sub-regions 3, 4, and 5; as shown by state (4), when the current block is in sub-region 7, the IBC reference region includes sub-regions 4, 5, and 6.

[0272] In some examples, in the IBC-TM merge mode, template matching refinement is performed with integer precision. In some examples, in the IBC-TM AMVP mode, template matching refinement is performed either with integer or 4-perl precision depending on the AMVR value. In some examples, template matching refinement in the IBC-TM merge mode and the IBC-TM AMVP mode is applied to samples without interpolation. In both the IBC-TM merge mode and the IBC-TM AMVP mode, the refined motion vectors (also called refined block vectors) and the templates used in each refinement step are required to satisfy the constraints of the reference region (e.g., required to be within the IBC reference region represented by FIG. 32).

[0273] In related examples, the diversity reordering of merge candidates in the merge candidate list employs template matching (TM) cost as an indicator for evaluating the diversity of merge candidates. In related examples, TM may not always be able to give an accurate estimate of the actual cost of the current block, for example, when the template area has different characteristics from the current block. Further, in some examples, when the TM tool is invalid or not available in the codec, the above diversity reordering method cannot be applied.

[0274] Note that in the following description, the template of a block can refer to any suitable portion of the neighboring samples of the block, such as the upper, left, right, and lower neighboring samples of the block.

[0275] FIG. 33 shows an example of the current block and the template of the current block. The template (shown by the gray area) includes the reconstructed samples adjacent above and to the left.

[0276] Figure 34 shows another example of the current block and the template of the current block. The template (indicated by the gray area) includes the reconstructed samples adjacent above and to the left.

[0277] In addition to the template patterns of FIGS. 33 and 34, other suitable patterns can also be used in the techniques disclosed in this disclosure.

[0278] Some aspects of the present disclosure provide techniques for applying diversity rearrangement based on candidate characteristics such as candidate motion information. The motion information can be translational motion information or affine motion information.

[0279] In some embodiments, the motion information of the translational MV is used to evaluate the diversity of candidates, and the evaluation is called diversity evaluation based on translational motion.

[0280] In some examples, the diversity evaluation based on translational motion is applied to merge candidates with translational MVs such as regular merge mode, merge by MMVD, etc.

[0281] In some examples, in the diversity evaluation based on translational motion, a candidate is considered redundant if the MV difference between the candidate and its preceding candidate (the candidate in the merge candidate list) is smaller than the MV threshold. For example, MV1 and MV2 represent the MV values of the candidate and its preceding candidate. In one example, the MV difference between the candidate and its preceding candidate, represented by deltaMV, is calculated by deltaMV = abs(MV1 - MV2). The MV difference has an x-direction component represented by deltaMV x and a y-direction component represented by deltaMV y In one example, the diversity evaluation based on translational motion is max(deltaMV x , deltaMV y)For example, the maximum component value of the MV difference, such as <T, is expressed as being smaller than the threshold value. Here, T is the threshold value. For example, the condition for diversity evaluation based on translational motion is that when the maximum component value of the MV difference between a candidate and its preceding candidate is smaller than the threshold value T, the candidate is regarded as redundant.

[0282] The threshold value T can be appropriately set. In one example, the threshold value T is set to 1 / 2 pixel. In other examples, the threshold value T is set to 1 pixel. In other examples, the threshold value T is adaptively set according to the current block size.

[0283] In some examples, when both candidates (the candidate and its preceding candidate) are bi-predictive, the diversity evaluation based on translational motion may be applied to the MV values in both reference picture lists (for example, each reference picture list provides a reference picture). For example, the condition for diversity evaluation based on translational motion is that when both the candidate and its preceding candidate are bi-predictive candidates, the candidate is considered redundant when the first maximum component value of the first MV difference of the reference picture list L0 (referred to as the first reference picture) between the candidate and its preceding candidate is smaller than the threshold value T, and the second maximum component value of the second MV difference of the reference picture list L1 (also referred to as the second reference picture) between the candidate and its preceding candidate is smaller than the threshold value T. For example, deltaMV1 represents the first MV difference related to the reference picture list L0, and deltaMV2 represents the second MV difference related to the reference picture list L1. The condition is that max(deltaMV1 x , deltaMV1 y ) < T and max(deltaMV2 x , deltaMV2 y ) < T is regarded as redundant.

[0284] In some examples, when both candidates (the candidate and its preceding candidate) are GPM candidates, the diversity evaluation based on translational motion may be applied to the MV values of each partition. For example, the condition for the diversity evaluation based on translational motion is that when both the candidate and its preceding candidate are GPM candidates, the candidate is considered redundant when the first maximum component value of the first MV difference of the first partition between the candidate and its preceding candidate is smaller than the threshold T, and the second maximum component value of the second MV difference of the second partition between the candidate and its preceding candidate is smaller than the threshold T. For example, deltaMV1P represents the first MV difference related to the first partition, and deltaMV2P represents the first MV difference related to the second partition. The condition is considered redundant when max(deltaMV1P x ,deltaMV1P y ) < T and max(deltaMV2P x ,deltaMV2P y ) < T.

[0285] In one example, when the candidate is a double prediction and its preceding candidate is a single prediction, the candidate is not considered redundant. In other examples, when the candidate is a single prediction and its preceding candidate is a double prediction, the candidate is not considered redundant.

[0286] In some embodiments, for a merge candidate with affine motion information, one or more affine parameters can be used to evaluate the diversity of the candidate, and the evaluation is called diversity evaluation based on affine motion. In some examples, one or more affine parameters are used to evaluate the diversity of candidates in the merge candidate list for affine motion.

[0287] In some examples, the diversity evaluation based on affine motion is applied to affine candidates in the affine merge mode. In some examples, the diversity evaluation based on affine motion is applied to affine candidates in the affine MMVD.

[0288] Various affine parameters, such as control points and affine parameters of an affine model, can be used in the diversity evaluation based on affine motion.

[0289] In some examples, in the diversity evaluation based on affine motion, an affine candidate is regarded as redundant when the difference of the translation component (the first control point motion vector CPMV0 which is an affine parameter) between the affine candidate and its preceding candidate (the affine candidate in the merge candidate list for affine motion) is smaller than a threshold. For example, MV1 and MV2 represent the MV values of VPMV0 of the affine candidate and its preceding candidate. In one example, the difference of CPMV0 between the affine candidate and its preceding candidate, represented by deltaMV, is calculated by deltaMV = abs(MV1 - MV2). The difference of CPMV0 has the x - direction component represented by deltaMV x and the y - direction component represented by deltaMV y . In one example, the diversity evaluation based on affine motion is expressed as max(deltaMV x , deltaMV y ) < T mv etc., that is, the maximum component value of the MV difference of CPMV0 is smaller than the MV threshold. Here, T mv is the MV threshold. For example, the condition of the diversity evaluation based on affine motion is that the affine candidate is regarded as redundant when the maximum component value of the difference (or delta) of CPMV0 between the affine candidate and its preceding candidate is smaller than the MV threshold T mv .

[0290] The MV threshold T mv can be set appropriately. In one example, the MV threshold T mv is set to 1 / 2 pixel. In other examples, the MV threshold T mv is set to 1 pixel. In other examples, the MV threshold T mv is set adaptively according to the evaluation of the absolute value of CPMV0 of one or all candidates.

[0291] In some examples, in the diversity evaluation based on affine motion, an affine candidate is considered redundant if the absolute difference of the affine parameters of the affine model between the affine candidate and its preceding candidate (the affine candidate within the merge candidate list for affine motion) is less than a threshold value T p In some examples, the affine parameters include parameters a, b, d, e of Equation (4) in the case of a 6-parameter affine model, or include parameters a, b of Equation (2) in the case of a 4-parameter affine model.

[0292] In one example, using a 4-parameter affine, the first affine model of the affine candidate includes parameters a1 and b1, and the second affine model of the preceding candidate of the affine candidate includes parameters a2 and b2. In one example, the condition for diversity evaluation based on affine motion is max(|a1 - a2|, |b1 - b2|) < T p such that the maximum delta value of the parameters between the affine candidate and its preceding candidate is less than the threshold value T p and the affine candidate is considered redundant.

[0293] In one example, using a 6-parameter affine, the first affine model of the affine candidate includes affine parameters a1, b1, d1, and e1, and the second affine model of the preceding candidate of the affine candidate includes affine parameters a2, b2, d2, and e2. In one example, the condition for diversity evaluation based on affine motion is max(|a1 - a2|, |b1 - b2|, |d1 - d2|, |e1 - e2|) < T p such that the maximum delta value of the affine parameters of the 6-parameter affine model between the affine candidate and its preceding candidate is less than the threshold value T p and the affine candidate is considered redundant.

[0294] In some examples, the delta value of one or a combination of a plurality of affine parameters can be used as a diversity evaluation criterion.

[0295] In some examples, when both candidates (the affine candidate and its preceding candidate) are double predictions, the diversity evaluation based on the affine motion may be applied to the affine parameters in both reference picture lists. For example, the condition for the diversity evaluation based on the affine motion is that when both the affine candidate and its preceding candidate are double prediction candidates, the affine candidate is considered redundant when the first maximum delta value of the first affine parameter of the reference picture list L0 between the affine candidate and its preceding candidate is smaller than the threshold, and the second maximum delta value of the second affine parameter of the reference picture list L1 between the affine candidate and its preceding candidate is smaller than the threshold.

[0296] In one example, deltaMV1 represents the first CPMV0 difference related to the reference picture list L0, and deltaMV2 represents the second CPMV0 difference related to the reference picture list L1. The affine candidate is x , deltaMV1 y ) < T mv and max(deltaMV2 x , deltaMV2 y ) < T mv is considered redundant.

[0297] In other examples, using 4-parameter affine, the first affine model of the affine candidate in the reference picture list L0 includes parameters a1 and b1, and the second affine model of the preceding candidate in the reference picture list L0 includes parameters a2 and b2. Further, the first affine model of the affine candidate in the reference picture list L1 includes parameters a1' and b1', and the second affine model of the preceding candidate in the reference picture list L1 includes parameters a2' and b2'. In one example, the condition for the diversity evaluation based on the affine motion is that the first maximum delta value of the affine parameters of the reference picture list L0 between the affine candidate and its preceding candidate, such as max(|a1 - a2|, |b1 - b2|) < T p is smaller than the threshold T p , and max(|a1' - a2'|, |b1' - b2'|) < T pwhen the second maximum delta value of the affine parameters of the reference picture list L1 between the affine candidate and its preceding candidate, such as, is less than the threshold value T p the affine candidate is regarded as redundant.

[0298] In other embodiments, using 6-parameter affine, the first affine model of the affine candidate in the reference picture list L0 includes parameters a1, b1, d1, and e1, and the second affine model of the preceding candidate in the reference picture list L0 includes parameters a2, b2, d2, and e2. Further, the first affine model of the affine candidate in the reference picture list L1 includes parameters a1', b1', d1', and e1', and the second affine model of the preceding candidate in the reference picture list L1 includes parameters a2', b2', d2', and e2'. In one example, the condition for diversity evaluation based on affine motion is max(|a1 - a2|, |b1 - b2|, |d1 - d2|, |e1 - e2|) < T p when the first maximum delta value of the affine parameters of the reference picture list L0 between the affine candidate and its preceding candidate, such as, is less than the threshold value T p and max(|a1' - a2'|, |b1' - b2'|, |d1' - d2'|, |e1' - e2'|) < T p when the second maximum delta value of the affine parameters of the reference picture list L1 between the affine candidate and its preceding candidate, such as, is less than the threshold value T p the affine candidate is regarded as redundant.

[0299] In one example, when the affine candidate is a double prediction and its preceding candidate is a single prediction, the affine candidate is not regarded as redundant. In other examples, when the affine candidate is a single prediction and its preceding candidate is a double prediction, the affine candidate is not regarded as redundant.

[0300] In some examples, the diversity evaluation based on translational motion and / or the diversity evaluation based on affine motion is applied when requirements such as the size of the current block being greater than a threshold, less than a threshold, or within a predefined threshold range are met. In some examples, the diversity evaluation based on translational motion and / or the diversity evaluation based on affine motion is applied when requirements such as the size of the source block associated with a candidate (e.g., a candidate for translational motion or a candidate for affine motion, also referred to as an affine candidate) being greater than a threshold, less than a threshold, or within a predefined threshold range are met.

[0301] In some examples, the diversity evaluation based on translational motion and / or the diversity evaluation based on affine motion is applied when requirements such as the block coordinates (e.g., the coordinates of the upper left corner of the source block) associated with a candidate (e.g., a candidate for translational motion or a candidate for affine motion, also referred to as an affine candidate) being greater than a threshold, less than a threshold, or within a predefined threshold range are met.

[0302] In some examples, when a candidate (e.g., a candidate for translational motion or a candidate for affine motion, also referred to as an affine candidate) is determined to be redundant after comparison with its preceding candidate, the candidate and the preceding candidate are compared by block size. The smaller block size is considered redundant and not placed in the merge candidate list. For example, if the candidate has a block size smaller than the preceding candidate, the candidate is considered redundant and not placed in the merge candidate list, or if the candidate has a block size larger than the preceding candidate, the candidate is considered redundant and not placed in the merge candidate list.

[0303] In some embodiments, the diversity rearrangement is based on both the motion information of the candidate and the TM cost criterion. The TM cost criterion is |D1 - D2| < T DIt is represented that when the template matching cost difference between a candidate and its preceding candidate, such as, is less than a threshold value, the candidate is regarded as redundant, where D1 and D2 represent the template matching costs obtained during ARMC - TM ordering, and T D represents the threshold value. In one example, T D is set to λ, which is the Lagrange parameter used in the RD criterion on the encoder side.

[0304] In some examples, a diversity evaluation based on the TM cost is applied to the candidates in the merge candidate list, and then a diversity evaluation based on the motion information is applied to the candidates in the merge candidate list. In some examples, a diversity evaluation based on the motion information is applied to the candidates in the merge candidate list, and then a diversity evaluation based on the TM cost is applied to the candidates in the merge candidate list.

[0305] In some examples, a candidate is regarded as redundant when both the diversity evaluation based on the motion information and the diversity evaluation based on the TM cost evaluate the candidate as redundant.

[0306] In some examples, when both the diversity evaluation based on the motion information and the diversity evaluation based on the TM cost evaluate a candidate as redundant, the candidate is placed at the end of the merge candidate list.

[0307] In some examples, when both the diversity evaluation based on the motion information and the diversity evaluation based on the TM cost evaluate a candidate as redundant, the candidate is not placed in the merge candidate list (pruning).

[0308] In some embodiments, the diversity rearrangement is based on both the TM cost diversity evaluation and the block size associated with the candidate. In some examples, when the TM cost difference between two candidates is within a given threshold, the candidate associated with the smaller block size is regarded as redundant. In one example, when a candidate is regarded as redundant, the candidate is placed at the end of the merge candidate list. In other examples, when a candidate is regarded as redundant, the candidate is not placed in the merge candidate list (pruning).

[0309] FIG. 35 shows a flowchart illustrating a process (3500) according to an embodiment of the present disclosure. The process (3500) may be used in a video encoder. In various embodiments, the process (3500) is performed by processing circuitry such as in terminal devices (310), (320), (330), and (340), processing circuitry that performs the functions of video encoder (403), processing circuitry that performs the functions of video encoder (603), processing circuitry that performs the functions of video encoder (703), and the like. In some embodiments, since the process (3500) is implemented by software instructions, when the processing circuitry executes the software instructions, the processing circuitry performs the process (3500). The process (3500) begins at (S3501) and proceeds to (S3510).

[0310] (S3510), a candidate list is determined that includes a plurality of candidates (also referred to as MV candidates) for coding of the current block.

[0311] (S3520), the candidate list is modified by a diversity evaluation based on motion information that prioritizes a subset of candidates with a larger difference in motion information.

[0312] (S3530), a particular candidate is selected from the candidate list for prediction of the current block. In one example, the particular candidate is selected based on rate distortion optimization.

[0313] (S3540), an index indicating the particular candidate from the modified candidate list is encoded in the bitstream carrying the video including the current block.

[0314] In some embodiments, the diversity evaluation based on motion information includes a diversity evaluation based on translational motion. In some embodiments, the diversity evaluation based on motion information includes a diversity evaluation based on affine motion.

[0315] In some embodiments, it is determined whether the first candidate in the candidate list satisfies the requirements. The diversity evaluation based on the motion information is applied to the first candidate and a second candidate preceding the first candidate in response to the first candidate satisfying the requirements. In one example, the requirements include that the block size associated with the first candidate satisfies the size requirement. In other examples, the requirements include that the block coordinates of the first candidate satisfy the coordinate requirement.

[0316] In some embodiments, the first candidate and the second candidate are determined to have sufficient diversity in response to one of the first candidate and the second candidate being a single prediction candidate and the other of the first candidate and the second candidate being a dual prediction candidate.

[0317] In some embodiments, the diversity evaluation based on the motion information is a diversity evaluation based on translational motion. In some examples, to change the candidate list based on the diversity evaluation based on translational motion, a first motion vector difference between the first candidate and the second candidate in the candidate list is calculated. Based on the first motion vector difference, it is determined whether one of the first candidate and the second candidate is redundant. For example, a first maximum component value of the first motion vector difference (e.g., an absolute x component value or an absolute y component value) is compared with a threshold value. In response to the first maximum component value being smaller than the threshold value, it is determined that one of the first candidate and the second candidate is redundant. The threshold value can be a fixed constant value such as 1 / 2 pixel, 1 pixel, etc., or can be a value that is a function of the size of the current block.

[0318] In some examples, when the first candidate and the second candidate are dual prediction candidates, the first motion vector difference is related to a first reference picture. Further, a second motion vector difference between the first candidate and the second candidate is calculated, and the second motion vector difference is related to a second reference picture. In response to both the first maximum component value of the first motion vector difference and the second maximum component value of the second motion vector difference being smaller than the threshold value, it is determined that one of the first candidate and the second candidate is redundant.

[0319] In some examples, the first candidate and the second candidate are geometric partitioning mode (GPM) candidates, and the first motion vector difference is associated with a first partition of each of the first candidate and the second candidate. Further, a second motion vector difference between the first candidate and the second candidate is calculated. The second motion vector difference is associated with a second partition of each of the first candidate and the second candidate. In response to both the first maximum component value of the first motion vector difference and the second maximum component value of the second motion vector difference being less than a threshold, one of the first candidate and the second candidate is determined to be redundant.

[0320] In some embodiments, the diversity evaluation based on motion information is a diversity evaluation based on affine motion. In some examples, to change a candidate list by the diversity evaluation based on affine motion, a first control point motion vector (CPMV) difference value between a first pair of first control point motion vectors (CPMVs) from each of the first candidate and the second candidate in the candidate list is calculated. Then, based on the first CPMV difference value, it is determined whether one of the first candidate and the second candidate is redundant. In some examples, the first maximum component value (e.g., the absolute x component value or the absolute y component value) of the first CPMV difference value is compared with a threshold. In response to the first maximum component value being less than the threshold, one of the first candidate and the second candidate is determined to be redundant. The threshold can be a fixed constant value such as 1 / 2 pixel, 1 pixel, etc., or can be a value that is a function of the size of the current block.

[0321] In some examples, the first candidate and the second candidate are dual prediction candidates, and the first CPMV difference value is between a first pair of first CPMVs associated with a first reference picture. Further, a second CPMV difference value between a second pair of first CPMVs from each of the first candidate and the second candidate is calculated, and the second pair of first CPMVs is associated with a second reference picture. In response to both the first maximum component value of the first CPMV difference value and the second maximum component value of the second CPMV difference value being less than a threshold, one of the first candidate and the second candidate is determined to be redundant.

[0322] In some embodiments, the diversity evaluation based on motion information is a diversity evaluation based on affine motion. In some examples, to change the candidate list by the diversity evaluation based on affine motion, a plurality of first affine parameter difference values between the first pair of affine models are calculated from each of the first candidate and the second candidate in the candidate list. Based on the plurality of first affine parameter difference values, it is determined whether one of the first candidate and the second candidate is redundant.

[0323] In some examples, the first maximum value among the plurality of first affine parameter difference values is compared with a threshold value. In response to the first maximum value being smaller than the threshold value, it is determined that one of the first candidate and the second candidate is redundant.

[0324] In some examples, the first candidate and the second candidate are dual prediction candidates, and the plurality of first affine parameter difference values are related to the first reference picture. A plurality of second affine parameter difference values between the second pair of affine models are calculated from each of the first candidate and the second candidate. The second pair of affine models is related to the second reference picture. In response to both the first maximum value among the plurality of first affine parameter difference values and the second maximum value among the plurality of second affine parameter difference values being smaller than the threshold value, it is determined that one of the first candidate and the second candidate is redundant.

[0325] In some examples, to change the candidate list by the diversity evaluation based on motion information, based on the diversity evaluation based on motion information and the diversity evaluation based on template matching (TM) cost, it is determined whether one of the first candidate and the second candidate in the candidate list is redundant. In one example, in response to the motion difference between the first candidate and the second candidate (e.g., the metric of the motion vector difference, the metric of the control point motion vector difference, e.g., the metrics of the affine parameter difference) being smaller than the first threshold value, and the template matching cost difference between the first candidate and the second candidate being smaller than the second threshold value, it is determined that one of the first candidate and the second candidate is redundant.

[0326] In one example, to change the candidate list, a candidate determined to be a redundant candidate is moved to the end of the candidate list. In other examples, an MV candidate determined to be a redundant candidate is removed from the candidate list (not placed in the candidate list).

[0327] In some examples, one of a first candidate and a second candidate in a candidate list is determined to be a redundant candidate, the first candidate is associated with a first block having a first block size, and the second candidate is associated with a second block having a second block size. In response to the first block size being smaller than the second block size, the first candidate is selected as the redundant candidate. In response to the second block size being smaller than the first block size, the second candidate is selected as the redundant candidate.

[0328] Then, the process proceeds to (S3599) and ends.

[0329] Process (3500) can be suitably adapted. The steps of process (3500) can be changed and / or omitted. Additional steps can be added. Any suitable order of implementation can be used.

[0330] FIG. 36 shows a flowchart illustrating a process (3600) according to an embodiment of the present disclosure. Process (3600) can be used in a video decoder. In various embodiments, process (3600) is executed by a processing circuit in terminal devices (310), (320), (330), and (340), a processing circuit that executes the functions of video decoder (410), a processing circuit that executes the functions of video decoder (510), a processing circuit that executes the functions of video decoder (633), and the like. In some embodiments, since process (3600) is implemented by software instructions, when the processing circuit executes the software instructions, the processing circuit executes process (3600). Process (3600) starts from (S3601) and proceeds to (S3610).

[0331] (S3610) A candidate list including a plurality of candidates for prediction of the current block is determined. In some examples, a video bitstream including the current block is received. Then, a candidate list including a plurality of candidates (also referred to as MV candidates) for prediction of the current block is determined.

[0332] (S3620) The candidate list is modified (e.g., sorted and / or pruned) by a diversity evaluation based on motion information, which prioritizes a subset of candidates with a larger difference in motion information.

[0333] (S3630) An index indicating a selected candidate from the modified candidate list is decoded from the video bitstream including the current block.

[0334] (S3640) Based on the selected candidate, the current block is reconstructed.

[0335] In some embodiments, the diversity evaluation based on motion information includes a diversity evaluation based on translational motion. In some embodiments, the diversity evaluation based on motion information includes a diversity evaluation based on affine motion.

[0336] In some embodiments, it is determined whether a first candidate in the candidate list satisfies a requirement. The diversity evaluation based on motion information is applied to the first candidate and a second candidate preceding the first candidate in response to the first candidate satisfying the requirement. In one example, the requirement includes that a block size associated with the first candidate satisfies a size requirement. In other examples, the requirement includes that block coordinates of the first candidate satisfy a coordinate requirement.

[0337] In some embodiments, the first candidate and the second candidate are determined to have sufficient diversity in response to one of the first candidate and the second candidate being a single prediction candidate and the other of the first candidate and the second candidate being a dual prediction candidate.

[0338] In some embodiments, the diversity evaluation based on motion information is a diversity evaluation based on translational motion. In some examples, to modify a candidate list by the diversity evaluation based on translational motion, a first motion vector difference between a first candidate and a second candidate in the candidate list is calculated. Based on the first motion vector difference, it is determined whether one of the first candidate and the second candidate is redundant. For example, a first maximum component value (e.g., an absolute x component value or an absolute y component value) of the first motion vector difference is compared with a threshold value. In response to the first maximum component value being smaller than the threshold value, it is determined that one of the first candidate and the second candidate is redundant. The threshold value can be a fixed constant value such as 1 / 2 pixel, 1 pixel, or can be a value that is a function of the size of the current block.

[0339] In some examples, when the first candidate and the second candidate are dual prediction candidates, the first motion vector difference is related to a first reference picture. Further, a second motion vector difference between the first candidate and the second candidate is calculated, and the second motion vector difference is related to a second reference picture. In response to both a first maximum component value of the first motion vector difference and a second maximum component value of the second motion vector difference being smaller than the threshold value, it is determined that one of the first candidate and the second candidate is redundant.

[0340] In some examples, the first candidate and the second candidate are geometric partitioning mode (GPM) candidates, and the first motion vector difference is related to a first partition of each of the first candidate and the second candidate. Further, a second motion vector difference between the first candidate and the second candidate is calculated. The second motion vector difference is related to a second partition of each of the first candidate and the second candidate. In response to both a first maximum component value of the first motion vector difference and a second maximum component value of the second motion vector difference being smaller than the threshold value, it is determined that one of the first candidate and the second candidate is redundant.

[0341] In some embodiments, the diversity evaluation based on motion information is a diversity evaluation based on affine motion. In some examples, to change the candidate list by the diversity evaluation based on affine motion, a first plurality of affine parameter difference values between a first pair of affine models from each of a first candidate and a second candidate in the candidate list are calculated. Based on the first plurality of affine parameter difference values, it is determined whether one of the first candidate and the second candidate is redundant.

[0342] In some examples, the first candidate and the second candidate are dual prediction candidates, and the first CPMV difference value is between a first pair of first CPMVs associated with a first reference picture. Further, a second CPMV difference value between a second pair of first CPMVs from each of the first candidate and the second candidate is calculated, and the second pair of first CPMVs is associated with a second reference picture. In response to both the first maximum component value of the first CPMV difference value and the second maximum component value of the second CPMV difference value being smaller than a threshold value, it is determined that one of the first candidate and the second candidate is redundant.

[0343] In some embodiments, the diversity evaluation based on motion information is a diversity evaluation based on affine motion. In some examples, to change the candidate list by the diversity evaluation based on affine motion, a first CPMV difference value between a first pair of first CPMV from each of a first candidate and a second candidate in the candidate list is calculated. Based on the first CPMV difference value, it is determined whether one of the first candidate and the second candidate is redundant.

[0344] In some examples, the first maximum value among the first plurality of affine parameter difference values is compared to a threshold value. In response to the first maximum value being less than the threshold value, one of the first candidate and the second candidate is determined to be redundant.

[0345] In some examples, the first candidate and the second candidate are dual prediction candidates, and the first plurality of affine parameter difference values are associated with a first reference picture. A second plurality of affine parameter difference values between a second pair of affine models are calculated from each of the first candidate and the second candidate. The second pair of affine models is associated with a second reference picture. In response to both the first maximum value among the first plurality of affine parameter difference values and the second maximum value among the second plurality of affine parameter difference values being less than the threshold value, one of the first candidate and the second candidate is determined to be redundant.

[0346] In some examples, to modify a candidate list based on a diversity evaluation based on motion information, based on a diversity evaluation based on motion information and a diversity evaluation based on a template matching (TM) cost, it is determined whether one of the first candidate and the second candidate in the candidate list is redundant. In one example, in response to a motion difference between the first candidate and the second candidate (e.g., a metric of a motion vector difference, a metric of a control point motion vector difference, e.g., metrics of an affine parameter difference) being less than a first threshold value and a template matching cost difference between the first candidate and the second candidate being less than a second threshold value, one of the first candidate and the second candidate is determined to be redundant.

[0347] In one example, to modify the candidate list, a candidate determined to be a redundant candidate is moved to the end of the candidate list. In other examples, an MV candidate determined to be a redundant candidate is removed from the candidate list (not placed in the candidate list).

[0348] In some examples, one of a first candidate and a second candidate in a candidate list is determined to be a redundant candidate, the first candidate is associated with a first block having a first block size, and the second candidate is associated with a second block having a second block size. In response to the first block size being smaller than the second block size, the first candidate is selected as the redundant candidate. In response to the second block size being smaller than the first block size, the second candidate is selected as the redundant candidate.

[0349] Then, the process proceeds to (S3699) and ends.

[0350] Process (3600) can be suitably adapted. The steps of process (3600) can be changed and / or omitted. Additional steps can be added. Any suitable order of implementation can be used.

[0351] FIG. 37 shows a flowchart illustrating a process (3700) according to an embodiment of the present disclosure. Process (3700) can be used in a video encoder. In various embodiments, process (3500) is performed by a processing circuit in terminal devices (310), (320), (330), and (340), a processing circuit that executes the functions of video encoder (403), a processing circuit that executes the functions of video encoder (603), a processing circuit that executes the functions of video encoder (703), and the like. In some embodiments, since process (3700) is implemented by software instructions, when the processing circuit executes the software instructions, the processing circuit executes process (3700). Process (3700) starts from (S3701) and proceeds to (S3710).

[0352] (In S3710), a candidate list including a plurality of candidates (also referred to as MV candidates) for prediction of the current block is determined.

[0353] In (S3720), the candidate list is changed by diversity evaluation based on template matching (TM) that gives higher priority to a subset of candidates with a larger difference in template matching cost and a larger block size.

[0354] In (S3730), a specific candidate is selected from the candidate list for prediction of the current block. In one example, a specific candidate from the candidate list for prediction of the current block is selected based on rate distortion optimization.

[0355] In (S3740), an index indicating a specific candidate from the changed candidate list is encoded in the bitstream carrying the video including the current block.

[0356] In some embodiments, to change the candidate list, one of a first candidate and a second candidate in the candidate list is determined to be a redundant candidate in response to the TM cost difference between the first candidate and the second candidate satisfying a requirement. The first candidate is associated with a first block having a first block size, and the second candidate is associated with a second block having a second block size. The first candidate is selected as a redundant candidate in response to the subject block size being smaller than the second block size. The second candidate is selected as a redundant candidate in response to the second block size being smaller than the first block size.

[0357] In one example, the redundant candidate is moved to the end of the candidate list. In other examples, the redundant candidate is removed from the candidate list.

[0358] Then, the process proceeds to (S3799) and ends.

[0359] Process (3700) can be appropriately adapted. The steps of process (3700) can be changed and / or omitted. Additional steps can be added. Any suitable order of implementation can be used.

[0360] FIG. 38 shows a flowchart illustrating a process (3800) according to an embodiment of the present disclosure. The process (3800) may be used in a video decoder. In various embodiments, the process (3800) is performed by a processing circuit in terminal devices (310), (320), (330), and (340), a processing circuit that executes the functions of video decoder (410), a processing circuit that executes the functions of video decoder (510), a processing circuit that executes the functions of video decoder (633), and the like. In some embodiments, since the process (3800) is implemented by software instructions, when the processing circuit executes the software instructions, the processing circuit executes the process (3800). The process (3800) starts at (S3801) and proceeds to (S3810).

[0361] (S3810), a candidate list including a plurality of candidates for prediction of the current block is determined.

[0362] (S3820), the candidate list is modified by a diversity evaluation based on template matching (TM) that prioritizes a subset of candidates with a larger difference in template matching cost and a larger block size.

[0363] (S3830), an index indicating a selected candidate from the modified candidate list is decoded from the video bitstream including the current block.

[0364] (S3840), based on the selected candidate, the current block is reconstructed.

[0365] In some embodiments, to modify the candidate list, in response to the TM cost difference between a first candidate and a second candidate in the candidate list satisfying a requirement, one of the first candidate and the second candidate is determined to be a redundant candidate. The first candidate is associated with a first block having a first block size, and the second candidate is associated with a second block having a second block size. The first candidate is selected as a redundant candidate in response to the subject block size being smaller than the second block size. The second candidate is selected as a redundant candidate in response to the second block size being smaller than the first block size.

[0366] In one example, the redundant candidate is moved to the end of the candidate list. In other examples, the redundant candidate is removed from the candidate list.

[0367] Then, the process proceeds to (S3899) and ends.

[0368] The process (3800) can be suitably adapted. The steps of the process (3800) can be changed and / or omitted. Additional steps can be added. Any suitable order of implementation can be used.

[0369] The above technology can be implemented as computer software that uses computer-readable instructions and is physically stored on one or more computer-readable media. For example, FIG. 39 shows a computer system (3900) suitable for implementing a particular embodiment of the disclosed subject matter.

[0370] The computer software can be coded in any suitable machine code or computer language according to mechanisms such as assembly, compilation, and linking to generate code including instructions that can be executed directly or through interpretation, microcode execution, etc. by one or more central processing units (CPUs), graphics processing units (GPUs), etc.

[0371] The command is executable on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, game consoles, devices for the Internet of Things, and the like.

[0372] The components shown in FIG. 39 with respect to the computer system (3900) are illustrative in nature and are not intended to suggest any limitation with respect to the use or functionality of the computer software implementing the embodiments of the present disclosure. The configuration of the components should not be construed as having any dependency or requirement with respect to any one or combination of the components described in the exemplary embodiments of the computer system (3900).

[0373] The computer system (3900) may include a specific human interface input device. Such a human interface input device may respond to input by one or more users through, for example, tactile input (e.g., keyboard, swipe, data glove movement), voice input (e.g., voice, clap), visual input (e.g., gesture), olfactory input (not shown). The human interface device can also be used to capture certain media that is not necessarily directly related to conscious human input, such as voice (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still camera), video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0374] The input human interface device may include one or more of a keyboard (3901), a mouse (3902), a trackpad (3903), a touch screen (3910), a data glove (not shown), a joystick (3905), a microphone (3906), a scanner (3907), a camera (3908) (each shown only once).

[0375] The computer system (3900) may also include certain human interface output devices. Such human interface output devices can stimulate the senses of one or more users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices include tactile output devices (e.g., tactile feedback by a touch screen (3910), a data glove (not shown), or a joystick (3905), but there may also be tactile feedback devices that do not function as input devices.), voice output devices (e.g., a speaker (3909), headphones (not shown)), visual output devices (e.g., a CRT screen, an LCD screen, a plasma screen, an OLED screen, regardless of whether they have a touch screen input function or not, and regardless of whether they have a tactile feedback function or not, and some of them can output two-dimensional visual output or output of more than three dimensions by means such as stereoscopic output, virtual reality glasses (not shown), holographic display, and a smoke tank (not shown) (3910)), and a printer (not shown).

[0376] The computer system (3900) can also include memory devices and their associated media accessible to humans, such as a CD / DVD ROM / RW (3920) including a CD / DVD or similar media (3921), a thumb drive (3922), a removable hard disk or solid state drive (3923), legacy magnetic media such as tapes and floppy (registered trademark) disks (not shown), dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown), etc.

[0377] One skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include a transmission medium, a carrier wave, or other transient signals.

[0378] The computer system (3900) can also include an interface (3954) to one or more communication networks (3955). The network can be, for example, wireless, wireline, optical. The network can further be local, wide area, metropolitan, vehicle and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet (registered trademark), wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wireline or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicle and factory networks including CAN bus, etc. Certain networks generally require an external network interface adapter attached to a specific general-purpose digital port or peripheral bus (3949) (such as a USB port of the computer system (3900)). Others are generally incorporated into the core of the computer system (3900) by attachment to a system bus as described later (such as an Ethernet network to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (3900) can communicate with other entities. Such communication can be unidirectional receive-only (such as broadcast TV) or unidirectional transmit-only (such as CAN bus to a specific CAN bus device), or can be bidirectional to other computer systems using, for example, local or wide area digital networks. Specific protocols or protocol stacks are available for each of the networks and network interfaces as described above.

[0379] The above human interface device, human-accessible storage device, and network interface can be attached to the core (3940) of the computer system (3900).

[0380] The core (3940) can include one or more central processing units (CPUs) (3941), a graphics processing unit (GPU) (3942), a dedicated programmable processing unit in the form of a field programmable gate array (FPGA) (3943), a hardware accelerator for specific tasks (3944), a graphics adapter (3950), etc. These devices may be connected through a system bus (3948) together with a read only memory (ROM) (3945), a random access memory (RAM) (3946), a built-in large-capacity storage device such as an internal hard drive inaccessible to users, an SSD, etc. (3947). In some computer systems, the system bus (3948) can be accessible in the form of one or more physical plugs to allow for expansion by additional CPUs, GPUs, etc. Peripheral devices can be attached directly to the system bus (3948) of the core or through a peripheral bus (3949). In the example, a display (3910) can be connected to the graphics adapter (3950). Architectures for peripheral buses include PCI, USB, etc.

[0381] The CPU (3941), GPU (3942), FPGA (3943), and accelerator (3944) are capable of executing specific instructions that can together constitute the above computer code. The computer code can be stored in the ROM (3945) or RAM (3946). Temporary data can also be stored in the RAM (3946), while persistent data can be stored, for example, in the built-in large-capacity storage device (3947). Fast storage and retrieval to / from any of the memory devices can be enabled by the use of cache memory. The cache memory can be closely related to one or more CPUs (3941), GPUs (3942), large-capacity storage devices (3947), ROM (3945), RAM (3946), etc.

[0382] A computer-readable medium can have computer code for performing various computer-implemented operations. The medium and the computer code can be specially designed and configured for the purposes of this disclosure, or they can be of the kind well known and available to those of ordinary skill in the computer software arts.

[0383] As an example, and not by way of limitation, a computer system having an architecture (3900), specifically a core (3940), can provide functionality as a result of software embodied in one or more tangible computer-readable media being executed by a processor (including a CPU, GPU, FPGA, accelerator, etc.). Such a computer-readable media can be a media associated with the user-accessible mass storage device introduced above, in addition to a specific storage device of the core (3940) having a non-transitory nature, such as an on-core mass storage device (3947) or ROM (3945). The software implementing various embodiments of the present disclosure is stored in such a device and is executable by the core (3940). The computer-readable media can include one or more memory devices or chips depending on specific needs. The software can cause the core (3940), and specifically the processors (including a CPU, GPU, FPGA, etc.) therein, to define data structures stored in the RAM (3946) and modify such data structures according to processes defined by the software, thereby executing specific processes or specific portions of specific processes described herein. Additionally, or alternatively, the computer system can provide functionality as a result of logic (e.g., an accelerator (3944)) that is hard-wired or otherwise embodied in circuitry and that operates instead of or in conjunction with software to execute specific processes or specific portions of specific processes described herein. References to software can, as necessary, include logic, and vice versa. References to computer-readable media can, as necessary, include circuitry (e.g., an integrated circuit (IC)) storing software for execution, circuitry embodying logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0384] Appendix A: Acronyms JEM: Joint Exploration Model VVC: Versatile Video Coding BMS: Benchmark Set MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Supplementary Enhancement Information VUI: Video Usability Information GOP: Group of Picture(s) TU: Transform Unit(s) PU: Prediction Unit(s) CTU: Coding Tree Unit(s) CTB: Coding Tree Block(s) PB: Prediction Block(s) HRD: Hypothetical Reference Decoder SNR: Signal Noise Ratio CPU: Central Processing Unit(s) GPU: Graphics Processing Unit(s) CRT: Cathode Ray Tube LCD: Liquid-Crystal Display OLED: Organic Light-Emitting Diode CD: Compact Disc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application-Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile communications LTE: Long-Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Area(s) SSD: Solid-State Drive IC: Integrated Circuit CU: Coding Unit

[0385] Although the present disclosure has described several exemplary embodiments, there are alternatives, exchanges, and various replacement equivalents within the scope of the present disclosure. Thus, as is apparent, those skilled in the art can implement the principles of the present disclosure, and thus, can conceive of numerous systems and methods within its spirit and scope, even if not explicitly illustrated or described herein.

[0386] [Incorporation by Reference] This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 390,128, filed on July 18, 2022, under the title "Diversified Merge Candidate Reordering" and U.S. Patent Application No. 17 / 984,889, filed on November 10, 2022, under the title "METHOD FOR DIVERSIFIED MERGE CANDIDATE REORDERING". The disclosures of these prior applications are hereby incorporated by reference in their entireties into this application.

Claims

1. A method for video processing executed by a decoder, comprising: receiving a video bitstream including a current block; determining a candidate list including a plurality of motion vector (MV) candidates for predicting the current block; modifying the plurality of MV candidates in the candidate list according to a diversity evaluation based on motion information of the plurality of MV candidates, wherein modifying the plurality of MV candidates prioritizes a subset of MV candidates with a larger difference in motion information; decoding, from the bitstream, an index indicating a selected MV candidate from the modified candidate list; reconstructing the current block based on the selected MV candidate; and a method having the above steps.

2. The diversity evaluation based on the motion information includes: a diversity evaluation based on translational motion, and a diversity evaluation based on affine motion including at least one of them, The method according to claim 1.

3. determining whether a first MV candidate in the candidate list satisfies a requirement; in response to the first MV candidate satisfying the requirement, applying a diversity evaluation based on the motion information to the first MV candidate and a second MV candidate preceding the first MV candidate; wherein the requirement includes: the block size related to the first MV candidate satisfies a size requirement, and the block coordinates of the first MV candidate satisfy a coordinate requirement including at least one of them, The method according to claim 1.

4. determining that the first MV candidate and the second MV candidate have sufficient diversity in response to one of the first MV candidate and the second MV candidate being a single prediction candidate and the other being a dual prediction candidate; The method according to claim 1, further comprising the above step.

5. The diversity evaluation based on the motion information is a diversity evaluation based on translational motion, and modifying the candidate list by the diversity evaluation based on translational motion includes: calculating a first motion vector difference between a first MV candidate and a second MV candidate in the candidate list; determining whether one of the first MV candidate and the second MV candidate is redundant based on the first motion vector difference; The method according to claim 1, further comprising the above steps.

6. A step of comparing a first maximum component value of the first motion vector difference with a threshold value; A step of determining that one of the first MV candidate and the second MV candidate is redundant in response to the first maximum component value being smaller than the threshold value The method according to claim 5, further comprising: **Claim 7** The threshold value is A fixed constant value, and A value that is a function of the size of the current block having The method according to claim 6. **Claim 8** The first MV candidate and the second MV candidate are dual prediction candidates, the first motion vector difference is related to a first reference picture, and the method A step of calculating a second motion vector difference between the first MV candidate and the second MV candidate, the second motion vector difference being related to a second reference picture; A step of determining that one of the first MV candidate and the second MV candidate is redundant in response to both a first maximum component value of the first motion vector difference and a second maximum component value of the second motion vector difference being smaller than the threshold value further comprising: The method according to claim 5. **Claim 9** The first MV candidate and the second MV candidate are geometric partitioning mode (GPM) candidates, the first motion vector difference is related to a first partition of each of the first MV candidate and the second MV candidate, and the method A step of calculating a second motion vector difference between the first MV candidate and the second MV candidate, the second motion vector difference being related to a second partition of each of the first MV candidate and the second MV candidate; A step of determining that one of the first MV candidate and the second MV candidate is redundant in response to both a first maximum component value of the first motion vector difference and a second maximum component value of the second motion vector difference being smaller than the threshold value further comprising: The method according to claim 5. **Claim 10** The diversity evaluation based on the motion information is a diversity evaluation based on an affine motion, and changing the candidate list by the diversity evaluation based on the affine motion A step of calculating a first control point motion vector (CPMV) difference value between a first pair of first control point motion vectors (CPMVs) from each of the first MV candidate and the second MV candidate in the candidate list; A step of determining whether one of the first MV candidate and the second MV candidate is redundant based on the first CPMV difference value, further comprising: The method according to claim 1.

11. Comparing the first maximum component value of the first CPMV difference value with a threshold value; Determining that one of the first MV candidate and the second MV candidate is redundant in response to the first maximum component value being smaller than the threshold value The method according to claim 10, further comprising:

12. The threshold value is a fixed constant value, and a value that is a function of the size of the current block having The method according to claim 11.

13. The first MV candidate and the second MV candidate are dual prediction candidates, and the first CPMV difference value is between a first pair of the first CPMVs associated with a first reference picture, and the method includes: Calculating a second CPMV difference value between a second pair of first CPMVs from each of the first MV candidate and the second MV candidate, the second pair of the first CPMVs being associated with a second reference picture; Determining that one of the first MV candidate and the second MV candidate is redundant in response to both the first maximum component value of the first CPMV difference value and the second maximum component value of the second CPMV difference value being smaller than the threshold value The method according to claim 10, further comprising: The method according to claim 10.

14. The diversity evaluation based on the motion information is a diversity evaluation based on an affine motion, and changing the candidate list by the diversity evaluation based on the affine motion includes: Calculating a first plurality of affine parameter difference values between a first pair of affine models from each of the first MV candidate and the second MV candidate in the candidate list; Determining whether one of the first MV candidate and the second MV candidate is redundant based on the first plurality of affine parameter difference values. The method according to claim 1 further comprises: The method according to claim 1.

15. Comparing a first maximum value among the first plurality of affine parameter difference values with a threshold value; Determining that one of the first MV candidate and the second MV candidate is redundant in response to the first maximum value being smaller than the threshold value The method according to claim 14, further comprising:

16. The first MV candidate and the second MV candidate are dual prediction candidates, and the first plurality of affine parameter difference values are related to a first reference picture, and the method Calculating a plurality of second affine parameter difference values between a second pair of affine models from each of the first MV candidate and the second MV candidate, wherein the second pair of affine models is related to a second reference picture; Determining that one of the first MV candidate and the second MV candidate is redundant in response to both a first maximum value among the plurality of first affine parameter difference values and a second maximum value among the plurality of second affine parameter difference values being smaller than a threshold; further comprising; The method according to claim 14.

17. Changing the candidate list by diversity evaluation based on the motion information comprises: further comprising determining whether one of a first MV candidate and a second MV candidate in the candidate list is redundant based on a diversity evaluation based on the motion information and a diversity evaluation based on a template matching (TM) cost; The method according to claim 1.

18. further comprising determining that one of the first MV candidate and the second MV candidate is redundant in response to a motion difference between the first MV candidate and the second MV candidate being smaller than a first threshold and a template matching cost difference between the first MV candidate and the second MV candidate being smaller than a second threshold; The method according to claim 17.

19. Changing the candidate list comprises: moving an MV candidate determined to be a redundant candidate to the end of the candidate list; and removing an MV candidate determined to be a redundant candidate from the candidate list further comprising at least one of; The method according to claim 1.

20. Changing the candidate list comprises: determining that one of a first MV candidate and a second MV candidate in the candidate list is a redundant candidate, wherein the first MV candidate is related to a first block having a first block size and the second MV candidate is related to a second block having a second block size; and selecting the first MV candidate as the redundant candidate in response to the first block size being smaller than the second block size; further comprising; The method according to claim 1.

21. A method of video processing executed by an encoder, comprising: determining a candidate list including a plurality of motion vector (MV) candidates for coding a current block; A step of changing the plurality of MV candidates in the candidate list according to a diversity evaluation based on motion information of the plurality of MV candidates, wherein changing the plurality of MV candidates prioritizes a subset of MV candidates having a larger difference in motion information more, and A step of selecting a specific candidate from the changed candidate list; A step of encoding an index indicating the specific candidate into a bitstream carrying a video including the current block; A method having the above steps.

Images