Inter-prediction Direction and BCW Index Coding in Merge Mode

Abstract

The processing circuit receives a video bitstream and obtains prediction information of a current block within a current picture from the video bitstream. The prediction information indicates whether the current block should be predicted in an inter prediction mode. In response to the current block being predicted in the inter prediction mode, the processing circuit determines a merge candidate from a merge candidate list and determines an inter prediction direction based on a syntax element signaled in the video bitstream. The inter prediction direction signaled separately from the merge candidate is one of a first mono-prediction from a first reference picture list, a second mono-prediction from a second reference picture list, and a bi-prediction from the first reference picture list and the second reference picture list. The processing circuit determines a motion vector and reconstructs the current block based on the inter prediction direction and the motion vector.

Description

Technical Field

[0001]

[0001] Incorporation by Reference This application claims the benefit of priority to U.S. Patent Application No. 17 / 984,907, filed Nov. 10, 2022, entitled “Inter-Prediction Direction and BCW Index Coding in Merge Mode,” which claims the benefit of priority to U.S. Provisional Application No. 63 / 390,565, filed Jul. 19, 2022, entitled “Inter-Prediction Direction and BCW Index Coding in Merge Mode.” The disclosure of the prior application is hereby incorporated by reference in its entirety.

[0002]

[0002] Technical Field This disclosure generally describes embodiments related to video coding.

Background Art

[0003]

[0003] The background description provided herein is for the purpose of generally presenting the context of the disclosure. The work done under the present inventors' names is not admitted as prior art to this disclosure, either expressly or impliedly, to the extent that the work is not described in a manner that would qualify as prior art at the time of filing this application and is not described in only this background section.

[0004] [

[0004] ] Uncompressed digital images and / or videos can contain a series of pictures, each picture having, for example, a spatial dimension of 1920×1080 luminance samples and associated chrominance samples. A series of pictures can have a fixed or variable picture rate (informally known as the frame rate), for example 60 pictures per second, or 60 Hz. Uncompressed images and / or videos have significant bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (luminance sample resolution of 1920×1080 at a frame rate of 60 Hz) requires a bandwidth close to 1.5 Gbit / s. One hour of such video requires a storage space of over 600 GB.

[0005] [

[0005] ] One of the purposes of image and / or video encoding and decoding can be said to be the reduction of redundancy in the input image and / or video signal by compression. Compression can, in some cases, help reduce the aforementioned bandwidth and / or storage space requirements by an order of magnitude or more. The description herein uses video encoding / decoding as an exemplary example, but the same techniques can be applied to image encoding / decoding in a similar manner without departing from the spirit of the present disclosure. Both lossless compression and lossy compression, as well as combinations thereof, can be used. Lossless compression (reversible compression) refers to a technique in which an exact copy of the original signal can be reconstructed from the compressed original signal. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original signal and the reconstructed signal is small enough such that the reconstructed signal is useful for the intended application. In the case of video, lossy compression is widely used. The amount of allowable distortion depends on the application; for example, users of certain consumer streaming applications may be able to tolerate higher distortion than users of television distribution applications. The achievable compression ratio can reflect the fact that higher allowable / tolerable distortion can result in a higher compression ratio.

[0006]

[0006] Video encoders and decoders can utilize techniques in several broad categories, including, for example, motion compensation, transform processing, quantization, and entropy coding.

[0007]

[0007] Video codec technology can include techniques known as intra coding. In intra coding, sample values are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially divided into blocks of samples. If all blocks of samples are coded in the intra mode, that picture can be an intra picture. Intra pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and thus can be used as the first picture in a coded video bitstream and video session or as a still image. Samples of an intra block can be subjected to a transform, and the transform coefficients can be quantized prior to entropy coding. Intra prediction can be a technique for minimizing sample values in the pre-transform domain. In some cases, the smaller the DC value after transformation and the smaller the AC coefficients, the fewer bits are required to represent the block after entropy coding for a given quantization step size.

[0008]

[0008] For example, conventional intra-coding used in MPEG-2 generation coding technology does not use intra prediction. However, some new video compression technologies include techniques that attempt to perform prediction based on surrounding sample data and / or metadata obtained during encoding and / or decoding of data blocks. Such techniques are hereinafter referred to as "intra prediction" techniques. It should be noted that in at least some cases, intra prediction uses only reference data from the current picture being reconstructed and does not use reference data from reference pictures.

[0009]

[0009] There can be many different forms of intra prediction. If more than one such technique may be used in a given video coding technology, the particular technique used may be coded as a particular intra prediction mode that uses that particular technique. In certain cases, the intra prediction mode may have submodes and / or parameters, in which case the submodes and / or parameters may be coded individually or included in a mode codeword, which defines the prediction mode being used. Which codeword to use for a given combination of mode, submode, and / or parameter can affect the coding efficiency gain due to intra prediction and can also affect the entropy coding technique used to convert the codeword into a bitstream.

[0010]

[0010] Specific modes of intra prediction were introduced in H.264, improved in H.265, and further refined in new coding technologies such as JEM (joint exploration model), VVC (versatile video coding), and MBS (benchmark set). A prediction block can be formed using sample values of samples in the vicinity of already available samples. The sample values of the neighboring samples are copied to a predictor block according to a certain direction. The reference for the direction in use can be coded in the bitstream or can be predicted itself.

[0011]

[0011] Referring to FIG. 1A, what is depicted at the lower right is a subset of 9 predictor directions out of the 33 possible predictor directions defined in H.265 (corresponding to 33 of the 35 intra - modes, the angular modes). The point (101) where the arrows converge represents the sample to be predicted. The arrows represent the directions from which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from the sample that is at an angle of 45 degrees from the horizontal and going up to the right. Similarly, arrow (103) indicates that sample (101) is predicted from the sample that is at an angle of 22.5 degrees from the horizontal and going towards the lower left of sample (101).

[0012]

[0012] Referring further to FIG. 1A, a square block (104) of 4×4 samples is depicted at the top left (shown by the thick dashed line). The square block (104) contains 16 samples, each labeled with an “S” and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample from the top in the Y dimension and the first sample from the left in the X dimension. Similarly, sample S44 is the fourth sample within block (104) in both the Y and X dimensions. Since the block size is 4×4 samples, S44 is at the bottom right. Further, reference samples following a similar numbering scheme are shown. The reference samples are labeled with the Y position (e.g., row index) and X position (column index) relative to block (104) and an R. In both H.264 and H.265, since the predicted samples are in the neighborhood of the block being reconstructed, there is no need to use negative values.

[0013]

[0013] Intra-picture prediction can be performed by copying the reference sample value from neighboring samples indicated by the signaled prediction direction. For example, assume that the coded video bitstream includes signaling indicating a prediction direction that matches arrow (102) for this block, i.e., assume that the samples are predicted from samples going diagonally up and to the right at an angle of 45 degrees from horizontal. In this case, samples S41, S32, S23, S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014]

[0014] In certain cases, the values of multiple reference samples can be combined, particularly when the direction is not evenly divisible by 45 degrees, for example, by interpolation, to calculate the reference sample.

[0015]

[0015] As video coding technology develops, the number of possible directions is increasing. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013). Currently, JEM / VVC / BMS can support up to 65 directions. Experiments are being conducted to identify the most likely directions, and specific techniques in entropy coding are used to represent these likely directions with fewer bits while accepting a specific penalty for less likely directions. Furthermore, the directions themselves can sometimes be predicted from neighboring directions already decoded in neighboring blocks.

[0016]

[0016] FIG. 1B shows a schematic diagram (110) depicting 65 intra prediction directions by JEM, indicating the increasing number of prediction directions over time.

[0017]

[0017] The mapping of intra prediction direction bits representing directions within a coded video bitstream can vary for each video coding technology. Such mapping can range from a simple direct mapping to complex adaptive schemes including codewords, the most likely modes, and similar techniques. However, in most cases, there may be specific directions (those that occur less statistically likely than other specific directions in the video content). Since the goal of video compression is redundancy reduction, these less likely directions are represented with more bits than the likely directions in well - operating video coding technologies.

[0018]

[0018] The encoding and decoding of images and / or videos may be performed using inter-picture prediction with motion compensation. Motion compensation may be a lossless compression technique, and a block of sample data from a previously reconstructed image or a part thereof (reference image) may be spatially shifted in the direction indicated by a motion vector (hereinafter referred to as MV) and then used for the prediction of a newly reconstructed picture or picture part. In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV can have two dimensions X and Y, or three dimensions, the third being an index of the reference picture used (the latter may indirectly be the time dimension).

[0019]

[0019] In some video compression techniques, the MV applicable to a given area of sample data can be predicted from other MVs, for example, from an MV related to another area of sample data that is spatially adjacent to the area being reconstructed and that precedes that MV in the decoding order. By doing so, the amount of data required to code the MV can be significantly reduced, thereby eliminating redundancy and increasing the compression ratio. For example, when coding an input video signal derived from a camera (known as natural video), there is a statistical likelihood that areas larger than the area to which a single MV is applicable move in a similar direction, and thus in some cases, it is possible to predict using a similar motion vector derived from the MVs of adjacent areas, so MV prediction can potentially function effectively. This results in an MV that is found to be similar to or the same as the MV predicted from surrounding MVs for a given area, which can be represented with fewer bits than would be used if the MV were directly coded after entropy coding. In some cases, MV prediction can be an example of lossless compression of the signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, MV prediction itself can be lossy due to rounding errors, for example, when calculating a predictor from several surrounding MVs.

[0020]

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, “High Efficiency Video Coding”, December 2016). Among the many MV prediction mechanisms provided by H.265, the one described with reference to FIG. 2 is a technique hereafter referred to as “spatial merge”.

[0021] Referring to FIG. 2, the current block (201) includes samples discovered by the encoder during the motion search process such that it can be predicted from previous blocks of the same size that are spatially shifted. Instead of directly coding its MV, the MV can be derived from the metadata associated with one or more reference pictures, using the MV associated with any of five surrounding samples (202 to 206 respectively), shown as A0, A1, and B0, B1, B2, from the latest reference picture (in the decoding order). In H.265, the MV prediction can use predictors from the same reference picture that neighboring blocks are using. SUMMARY OF THE INVENTION

[0022]

[0022] Aspects of the present disclosure provide methods and apparatuses for video encoding / decoding. In some examples, a video decoding device includes a processing circuit. In some embodiments, the processing circuit receives a video bitstream including a current block within a current picture. The processing circuit obtains prediction information from the video bitstream. The prediction information indicates whether the current block should be predicted in an inter prediction mode. In response to the current block being predicted in the inter prediction mode, the processing circuit determines a merge candidate from a merge candidate list and determines an inter prediction direction based on syntax elements signaled in the video bitstream. The inter prediction direction signaled separately from the merge candidate is one of a first uni-prediction from a first reference picture list, a second uni-prediction from a second reference picture list, and a bi-prediction from the first reference picture list and the second reference picture list. The processing circuit determines a motion vector for prediction of the current block based on the merge candidate and reconstructs the current block based on the inter prediction direction and the motion vector.

[0023]

[0023] In some examples, the processing circuit decodes one or more signals from a video bitstream, and the one or more signals indicate an inter prediction direction. In one example, the one or more signals are after a first syntax indicating whether the current block is in merge mode. In another example, the one or more signals are after a second syntax indicating the merge type of the current block.

[0024]

[0024] In some examples, the processing circuit infers that the inter prediction direction is bi-prediction in response to the merge type of the current block being based on bi-prediction.

[0025]

[0025] In some examples, the one or more signals include a syntax using a first value indicating uni-prediction and a second value indicating bi-prediction.

[0026]

[0026] In some examples, the one or more signals include a syntax using a first value indicating a first uni-prediction, a second value indicating a second uni-prediction, and a third value indicating bi-prediction.

[0027]

[0027] In some examples, the one or more signals include a first flag indicating whether the first reference picture list is in the inter prediction direction; and a second flag indicating whether the second reference picture list is in the inter prediction direction;

[0028]

[0028] In some examples, in response to the high-level syntax allowing bi-prediction, the processing circuit decodes one or more signals from the video bitstream.

[0029]

[0029] In some examples, the inter prediction direction is bi-prediction, and the processing circuit decodes from the video bitstream an index indicating a particular weighted candidate in the weighted candidate list. The weighted candidate list includes a plurality of weighted candidates each providing a weighted value for combining predictions from a first reference picture list and a second reference picture list. In some examples, the weighted candidate list includes at least: weighted candidates inherited from merge candidates and default equal-weight candidates.

[0030]

[0030] In some embodiments, the processing circuit decodes from the bitstream information indicating that the current block in the current picture is an inter prediction block in merge mode. The video bitstream carries a video including the current picture. The processing circuit constructs an extended merge candidate list based on at least a first merge candidate having motion parameters obtained from neighboring blocks. The extended merge candidate list includes at least the first merge candidate and non-redundant merge candidates extended from the first merge candidate. The non-redundant merge candidates have at least the motion vector of the first merge candidate. The processing circuit decodes an index indicating a particular merge candidate in the extended merge candidate list and reconstructs the current block based on the particular merge candidate.

[0031]

[0031] In some examples, the first merge candidate is a bi-prediction candidate using a first motion vector associated with a first reference picture list and a second motion vector associated with a second reference picture list. The processing circuit constructs an extended merge candidate list including the first merge candidate and at least one of a second merge candidate and a third merge candidate, where the second merge candidate is a uni-prediction candidate having the first motion vector associated with the first reference picture list; and the third merge candidate is a uni-prediction candidate having the second motion vector associated with the second reference picture list.

[0032]

[0032] In one example, the first merge candidate is a bi-prediction candidate that uses a first motion vector associated with a first reference picture list and a second motion vector associated with a second reference picture list, and the processing circuit forms an extended merge candidate list that includes the first merge candidate and at least one of a second merge candidate, a third merge candidate, a fourth merge candidate, a fifth merge candidate, and a sixth merge candidate. The second merge candidate is a bi-prediction candidate having a first motion vector associated with the second reference picture list and a second motion vector associated with the first reference picture list; The third merge candidate is a uni-prediction candidate having the first motion vector associated with the first reference picture list; The fourth merge candidate is a uni-prediction candidate having the second motion vector associated with the second reference picture list; The fifth merge candidate is a uni-prediction candidate having the second motion vector associated with the first reference picture list; and The sixth merge candidate is a uni-prediction candidate having the first motion vector associated with the second reference picture list.

[0033]

[0033] In some examples, the first merge candidate is a bi-prediction candidate coded using bi-prediction with CU level weight (BCW), and the first merge candidate includes a first weight value, a first motion vector associated with a first reference picture list, and a second motion vector associated with a second reference picture list and a second weight value. In response to the first weight value being greater than the second weight value, the processing circuit constructs an extended merge candidate list including the first merge candidate and a second merge candidate, and the second merge candidate is a uni-prediction candidate having a first motion vector associated with the first reference picture list. In response to the first weight value being less than the second weight value, the processing circuit constructs an extended merge candidate list including the first merge candidate and a third merge candidate, and the third merge candidate is a uni-prediction candidate having a second motion vector associated with the second reference picture list.

[0034]

[0034] In some examples, the first merge candidate is a uni-prediction candidate having a first motion vector associated with a first reference picture list. The processing circuit constructs an extended merge candidate list including the first merge candidate and a second merge candidate which is a uni-prediction candidate having a first motion vector associated with a second reference picture list.

[0035]

[0035] In some examples, the first merge candidate is a bi-prediction candidate coded using a first (bi-prediction with CU level weight (BCW)) index value, and the first merge candidate includes a first motion vector associated with a first reference picture list and a second motion vector associated with a second reference picture list. The processing circuit constructs an extended merge candidate list including the first merge candidate and one or more second merge candidates, and each of the one or more second merge candidates includes motion information in the first merge candidate and is coded using a BCW index value different from the first BCW index value.

[0036]

[0036] In one example, the first merge candidate is a bi-prediction candidate coded using a first (CU-level weighted bi-prediction (BCW)) index value corresponding to unequal weighting. The first merge candidate includes motion information having a first motion vector associated with a first reference picture list and a second motion vector associated with a second reference picture list. The processing circuit constructs an extended merge candidate list including the first merge candidate and a second merge candidate. The second merge candidate includes the motion information in the first merge candidate and is coded using a second BCW index value corresponding to equal weighting.

[0037]

[0037] In one example, the processing circuit orders the merge candidates in the extended merge candidate list according to a template matching cost.

[0038]

[0038] In some embodiments, the processing circuit decodes from the bitstream information indicating that the current block in the current picture is an inter-prediction block in merge mode. The video bitstream includes the current block. The processing circuit constructs a merge candidate list including a plurality of merge candidates obtained from neighboring blocks and decodes an index indicating a specific merge candidate in the merge candidate list. The processing circuit selects a specific inter-prediction direction associated with the specific merge candidate from among the plurality of direction candidates based on the template matching costs of the plurality of direction candidates. The processing circuit reconstructs the current block based on the specific merge candidate in the specific inter-prediction direction.

[0039]

[0039] In one example, the processing circuit determines a specific inter prediction direction for a specific merge candidate in response to the specific merge candidate being obtained from neighboring blocks. In another example, the processing circuit determines a specific inter prediction direction for a specific merge candidate in response to the completion of motion vector refinement for the specific merge candidate. In another example, the processing circuit determines a specific inter prediction direction for a specific merge candidate in response to the completion of template matching-based reordering of a plurality of merge candidates. In another example, the processing circuit determines a specific inter prediction direction for a specific merge candidate in response to the decoding of an index.

[0040]

[0040] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding, cause the computer to perform a video decoding method.

Brief Description of the Drawings

[0041]

[0041] Further features, characteristics, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

Figure 1A

[0042] FIG. 1A is a schematic diagram of an exemplary subset of intra prediction modes.

Figure 1B

[0043] FIG. 1B is a diagram of exemplary intra prediction directions.

Figure 2

[0044] FIG. 2 is a schematic diagram of a current block and its surrounding spatial merge candidates in one example.

Figure 3

[0045] FIG. 3 is a schematic diagram of a simplified block diagram of a communication system (300) according to an embodiment.

Figure 4

[0046] FIG. 4 is a schematic diagram of a simplified block diagram of a communication system (400) according to an embodiment.

Figure 5

[0047] FIG. 5 is a schematic diagram of a simplified block diagram of a decoder according to an embodiment.

Figure 6

[0048] FIG. 6 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment.

Figure 7

[0049] FIG. 7 shows a block diagram of an encoder according to another embodiment.

Figure 8

[0050] FIG. 8 shows a block diagram of a decoder according to another embodiment.

Figure 9

[0051] FIG. 9 shows the positions of spatial merge candidates according to an embodiment of the present disclosure.

Figure 10

[0052] FIG. 10 shows candidate pairs considered for redundancy checking of spatial merge candidates according to an embodiment of the present disclosure.

Figure 11

[0053] FIG. 11 shows exemplary motion vector scaling for temporal merge candidates.

Figure 12

[0054] FIG. 12 shows exemplary candidate positions for temporal merge candidates of a current coding unit.

Figure 13A

[0055] FIG. 13A shows an affine motion model in some examples.

Figure 13B

[0055] FIG. 13B shows an affine motion model in some examples.

Figure 14

[0056] FIG. 14 shows an affine model vector field in an example.

Figure 15

[0057] FIG. 15 shows the positions of spatial merge candidates in some examples.

Figure 16

[0058] FIG. 16 shows control point motion vector inheritance in some examples.

Figure 17

[0059] FIG. 17 shows candidate positions for an affine merge mode interpreted in some examples.

Figure 18

[0060] Figure 18 shows an exemplary diagram of the difference between the sample MV and the sub-block MV in some examples.

Figure 19

[0061] Figure 19 shows an exemplary SbTMVP process used in the SbTMVP mode.

Figure 20

[0061] Figure 20 shows an exemplary SbTMVP process used in the SbTMVP mode.

Figure 21

[0062] Figure 21 shows an example of the search process (2100) in the MMVD mode.

Figure 22

[0063] Figure 22 shows an example of search points in the MMVD mode.

Figure 23

[0064] Figure 23 shows a diagram showing the history parameter table in some examples.

Figure 24

[0065] Figure 24 shows a diagram showing the history parameter table stored in the line buffer in some examples.

Figure 25A

[0066] Figure 25A shows the pattern of non-adjacent spatial neighbors obtained in some examples.

Figure 25B

[0066] Figure 25A shows the pattern of non-adjacent spatial neighbors obtained in some examples.

Figure 26

[0067] Figure 26 shows a diagram showing an example for constructing merge candidates.

Figure 27

[0068] Figure 27 shows an example of template matching.

Figure 28

[0069] Figure 28 shows a diagram showing a reference sample of the template of the current block for the dual-prediction merge candidate.

Figure 29

[0070] Figure 29 shows a reference sample of a template for a current block using sub-block-based merge candidates and an example of template derivation.

Figure 30

[0071] Figure 30 shows a diagram illustrating the directions of refinement directions.

Figure 31

[0072] Figure 31 shows a flowchart showing an overview of a process according to some embodiments of the present disclosure.

Figure 32

[0073] Figure 32 shows a flowchart showing an overview of a process according to some embodiments of the present disclosure.

Figure 33

[0074] Figure 33 shows a flowchart showing an overview of a process according to some embodiments of the present disclosure.

Figure 34

[0075] Figure 34 shows a flowchart showing an overview of a process according to some embodiments of the present disclosure.

Figure 35

[0076] Figure 35 shows a flowchart showing an overview of a process according to some embodiments of the present disclosure.

Figure 36

[0077] Figure 36 shows a flowchart showing an overview of a process according to some embodiments of the present disclosure.

Figure 37

[0078] Figure 37 is a schematic diagram of a computer system according to an embodiment.

Embodiments for Carrying Out the Invention

[0042]

[0079] Figure 3 shows an exemplary block diagram of a communication system (300). The communication system (300) includes a plurality of terminal devices that can communicate with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via a network (350). In the example of Figure 3, the first pair of terminal devices (310) and (320) perform unidirectional data transmission. For example, the terminal device (310) can code video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to another terminal device (320) via the network (350). The coded video data can be transmitted in the form of one or more coded video bitstreams. The terminal device (320) can receive the coded video data from the network (350), decode the coded video data to restore the video pictures, and display the video pictures according to the restored video data. Unidirectional data transmission may be common in media serving applications and the like.

[0043]

[0080] In another example, the communication system (300) includes, for example, a second pair of terminal devices (330) and (340) that perform bidirectional transmission of coded video data, for example, during a video conference. With respect to the bidirectional transmission of data, for example, each of the terminal devices (330) and (340) can code video data (e.g., a stream of video pictures captured by a terminal device) for transmission to the other of the terminal devices (330) and (340) via the network (350). Each of the terminal devices (330) and (340) can also receive the coded video data transmitted by the other of the terminal devices (330) and (340), can decode the coded video data to restore the video pictures, and can display the video pictures on an accessible display device according to the restored video data.

[0044]

[0081] In the example of FIG. 3, the terminal devices (310), (320), (330), and (340) are shown as servers, personal computers, and smartphones, but the principles of the present disclosure need not be so limited. Embodiments of the present disclosure have applications using laptop computers, tablet computers, media players, and / or dedicated video conferencing devices. The network (350) represents any number of networks that carry coded video data between the terminal devices (310), (320), (330), and (340), including, for example, wired (wired) and / or wireless communication networks. The communication network (350) can exchange data over circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of the present disclosure, the architecture and topology of the network (350) may not be important for the operation of the present disclosure, unless otherwise described below.

[0045]

[0082] FIG. 4 shows a video encoder and a video decoder in a streaming environment as an application example of the disclosed subject matter. The disclosed subject matter is equally applicable to other video-enabled applications including, for example, video conferencing, digital TV, streaming services, storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.).

[0046]

[0083] The streaming system can include a video source (401), such as a digital camera, and may include a capture subsystem (413) capable of generating a stream of, for example, uncompressed video pictures (402). In one example, the stream of video pictures (402) includes samples taken by a digital camera. The stream of video pictures (402), drawn as a thick line to emphasize the large amount of data when compared to the encoded video data (404) (or coded video bitstream), can be processed by an electronic device (420) including a video encoder (403) coupled to the video source (401). The video encoder (403) includes hardware, software, or a combination thereof and can be operative to implement or realize aspects of the disclosed subject matter as detailed below. The encoded video data (404) (or encoded video bitstream), drawn as a thin line to emphasize the smaller amount of data when compared to the stream of video pictures (402), can be stored in a streaming server (405) for future use. One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include, for example, a video decoder (410) within an electronic device (430). The video decoder (410) decodes an incoming copy (407) of the encoded video data and generates an output stream of video pictures (411) that can be rendered on a display (412), such as a display screen, or other rendering device (not shown). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., video bitstream) can be encoded according to a particular video coding / compression standard.Examples of these standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0047]

[0084] Note that the electronic devices (420) and (430) can include other components (not shown). For example, the electronic device (420) can include a video decoder (not shown), and the electronic device (430) can also include a video encoder (not shown).

[0048]

[0085] FIG. 5 shows an exemplary block diagram of a video decoder (510). The video decoder (510) can be included in an electronic device (530). The electronic device (530) can include a receiver (531) (e.g., a receiving circuit). The video decoder (510) can be used in place of the video decoder (410) in the example of FIG. 4.

[0049]

[0086] Receiver (531) is capable of receiving one or more coded video sequences to be decoded by video decoder (510). In an embodiment, when decoding of each coded video sequence is independent from other coded video sequences, it is possible to receive one coded video sequence at a time. The coded video sequence can be received from channel (501), which may be a hardware / software link to a storage device storing the encoded video data. Receiver (531) can receive the encoded video data together with other data, such as coded audio data and / or auxiliary data streams, and these data can be transferred using respective entities (not shown). Receiver (531) can separate the coded video sequence from other data. To handle network jitter, buffer memory (515) may be coupled between receiver (531) and entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In a particular application, buffer memory (515) is part of video decoder (510). In other cases, it may be outside video decoder (510) (not shown). In yet another example, for example, to handle network jitter, there may be buffer memory (not shown) outside video decoder (510), and furthermore, for example, there may be another buffer memory (515) inside video decoder (510) to handle playback timing. If receiver (531) is receiving data from a store-and-forward device with sufficient bandwidth and controllability, or from a synchronous network, buffer memory (515) may not be required or can be made smaller.For use in a best-effort packet network such as the Internet, buffer memory (515) may be required, which may be relatively large and advantageously may be of an adaptable size and may be implemented at least in part in an operating system or similar element (not shown) outside of the video decoder (510).

[0050]

[0087] Video decoder (510) can include a parser (520) to reconstruct symbols (521) from the coded video sequence. The categories of these symbols include information used to manage the operation of the video decoder (510), and potentially information for controlling a rendering device (512) (e.g., a display screen) that is not an essential part of the electronic device (530) but can be coupled to the electronic device (530) as shown in FIG. 5. The control information for the rendering device may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (520) can parse / entropy decode the received coded video sequence. The coding of the coded video sequence can follow video coding techniques or standards and can follow various principles including variable length coding, Huffman coding, arithmetic coding with or without context influence, etc. The parser (520) can extract a set of subgroup parameters for at least one subgroup of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroup can include a Group of Pictures (GOP), picture, tile, slice, macroblock, Coding Unit (CU), block, Transform Unit (TU), Prediction Unit (PU), etc. The parser (520) can also extract from coded video sequence information such as transform coefficients, quantization parameter values, motion vectors, etc.

[0051]

[0088] The parser (520) can perform entropy decoding / analysis processing on the video sequence received from the buffer memory (515) to generate symbols (521).

[0052]

[0089] The reconstruction of the symbol (521) can include a plurality of different units according to the type of the coded video picture or a part thereof (e.g., inter and intra pictures, inter and intra blocks) and other factors. How each unit is included can be controlled by subgroup control information analyzed by the parser (520) from the coded video sequence. Such a flow of subgroup control information between the parser (520) and a plurality of subsequent units is not depicted for clarity.

[0053]

[0090] The video decoder (510) can be conceptually subdivided into a plurality of functional units as described below in addition to the functional blocks already described. In a practical implementation operating under commercial constraints, many of these units can interact closely with each other and can be at least partially integrated with each other. However, for the purpose of explaining the disclosed subject matter, the conceptual subdivision into the following functional units is appropriate.

[0054]

[0091] The first unit is the scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives not only the quantized transform coefficients but also control information (including the transform to be used, block size, quantization factor, quantization scaling matrix, etc.) as symbols (521) from the parser (520). The scaler / inverse transform unit (551) can output a block including sample values that can be input to the aggregator (555).

[0055]

[0092] In some cases, the output samples of the scaler / inverse transform unit (551) may be related to intra-coded blocks. An intra-coded block is a block that does not use prediction information from a previously reconstructed picture but can use prediction information from a previously reconstructed part of the current picture. Such prediction information can be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) uses the already reconstructed surrounding information taken from the buffer (558) of the current picture to generate a block of the same size and shape as the block being reconstructed. The current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (555) may, in some cases, add, for each sample, the prediction information generated by the intra prediction unit (552) to the output sample information as provided by the scaler / inverse transform unit (551).

[0056]

[0093] Otherwise, the output samples of the scaler / inverse transform unit (551) can be related to blocks that may be inter-coded and motion compensated. In such a case, the motion compensation prediction unit (553) can access the reference picture memory (557) to retrieve the samples to be used for prediction. According to the symbol (521) related to the block, after motion compensating the retrieved samples, these samples are added by the aggregator (555) to the output of the scaler / inverse transform unit (551) (in this case, called residual samples or residual signal), generating output sample information. The address in the reference picture memory (557) from which the motion compensation prediction unit (553) retrieves the prediction samples can be controlled by the motion vectors available to the motion compensation prediction unit (553) in the form of, for example, X, Y, and a symbol (521) that can have reference picture components. Also, motion compensation can include interpolation of sample values taken from the reference picture memory (557), a motion vector prediction mechanism, etc., when an exact motion vector of sub-samples is used.

[0057]

[0094] The output samples of the aggregator (555) can be affected by various loop filtering techniques within the loop filter unit (556). Video compression techniques can include in-loop filter techniques that are included in the coded video sequence (also called the coded video bitstream) and are controlled by parameters made available to the loop filter unit (556) as symbols (521) from the parser (520). Also, video compression can be responsive to meta information obtained during the decoding of a previously decoded portion of the coded picture or coded video sequence (in decoding order), and can also be responsive to previously reconstructed loop-filtered sample values.

[0058]

[0095] The output of the loop filter unit (556) can be not only output to the rendering device (512), but also a sample stream that can be stored in the reference picture memory (557) for future inter-picture prediction.

[0059]

[0096] Once a predetermined coded picture is completely reconstructed, it can be used as a reference picture for future prediction. For example, when the coded picture corresponding to the current picture is completely reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and the fresh current picture buffer can be reallocated before starting the reconstruction of subsequent coded pictures.

[0060]

[0097] The video decoder (510) is capable of performing a decoding operation according to a standard such as ITU-T Rec.H.265 or a predetermined video compression technique. The coded video sequence can comply with the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence complies with both the syntax of the video compression technique or standard and the profile as documented in the video compression technique or standard. Specifically, the profile can select specific tools as the only tools available under that profile from all the tools available in the video compression technique or standard. Also, for compliance, it is necessary that the complexity of the coded video sequence falls within the range defined by the level of the video compression technique or standard. In some cases, the level restricts the maximum picture size, maximum frame rate, maximum reconstruction sample rate (measured, for example, in megasamples per second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further restricted by the Hypothetical Reference Decoder (HRD) specifications and metadata for HRD buffer management signaled in the coded video sequence.

[0061]

[0098] In an embodiment, the receiver (531) may receive additional (redundant) data together with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (510) to decode the data properly and / or to reconstruct the original video data more accurately. The additional data can be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0062]

[0099] Figure 6 shows an exemplary block diagram of a video encoder (603). The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used in place of the video encoder (403) in the example of FIG. 4.

[0063]

[0100] The video encoder (603) can receive video samples from a video source (601) (not part of the electronic device (620) in the example of FIG. 6) capable of capturing the video image to be coded by the video encoder (603). In another example, the video source (601) is part of the electronic device (620).

[0064]

[0101] The video source (601) can provide a source video sequence to be coded by the video encoder (603) in the form of a digital video sample stream that can be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit,...), any color space (e.g., BT.601 YCrCB, RGB,...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (601) may be a storage device storing pre-prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that convey motion when viewed in sequence. The picture itself can be organized as a spatial array of pixels, and each pixel can include one or more samples depending on the sampling structure, color space, etc. in use. One of ordinary skill in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0065]

[0102] According to an embodiment, the video encoder (603) can code and compress pictures of a source video sequence into a coded video sequence (643) in real time or under any other required time constraints. Enforcing an appropriate coding speed is one function of the controller (650). In some embodiments, the controller (650) controls other functional units and is functionally coupled to other functional units as described below. The coupling is not depicted for clarity. The parameters set by the controller (650) can include rate control related parameters (picture skip, quantizer, lambda value of rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other appropriate functions related to the video encoder (603) optimized for a particular system design.

[0066]

[0103] In some embodiments, the video encoder (603) is configured to operate in a coding loop. As an extremely simplified explanation, in one example, the coding loop can include a source coder (630) (which is responsible for generating symbols such as a symbol stream based on the input picture and reference pictures to be coded) and a (local) decoder (633) incorporated in the video encoder (603). The decoder (633) reconstructs the symbols to generate sample data in the same way as a (remote) decoder does. The reconstructed sample stream (sample data) is input into the reference picture memory (634). Since the decoding of the symbol stream results in a bit-exact result independent of the decoder's location (local or remote), the content in the reference picture memory (634) is also bit-exact between the local encoder and the remote encoder. In other words, the prediction part of the encoder "sees" exactly the same sample values as the samples that the decoder would "see" when using prediction during decoding as the reference picture samples. This basic principle of reference picture synchronization (and the resulting drift if synchronization cannot be maintained, e.g., due to channel errors) is also used similarly in some related technologies.

[0067]

[0104] It is possible to assume that the operation of the "local" decoder (633) is the same as that of a "remote" decoder such as the video decoder (510) already described in detail above in relation to FIG. 5. However, referring briefly to FIG. 5, since it is possible to assume that the symbols are available and the encoding / decoding of the symbols into the coded video sequence by the entropy coder (646) and the parser (520) is lossless, the entropy decoding part of the video decoder (510) including the buffer memory (515) and the parser (520) may not be fully realized in the local decoder (633).

[0068]

[0105] In an embodiment, decoder technologies other than parsing / entropy decoding existing in the decoder are present in the corresponding encoder in an ideal or substantially identical functional form. Therefore, the disclosed subject matter focuses on the operation of the decoder. Since the description of encoder technology is the reverse of the decoder technology described comprehensively, it can be omitted. In a specific area, a more detailed description will be given below.

[0069]

[0106] During operation, in some examples, the source coder (630) can perform motion-compensated predictive coding that predictive-codes an input picture by referring to one or more previously-coded pictures from a video sequence designated as a "reference picture". In this way, the coding engine (632) codes the difference between a pixel block of the input picture and a pixel block of a reference picture that can be selected as a predictive reference for the input picture.

[0070]

[0107] The local video decoder (633) is capable of decoding the coded video data of a picture that can be designated as a reference picture based on the symbols generated by the source coder (630). The operation of the coding engine (632) can advantageously be a lossless process. If the coded video data can be decoded by a video decoder (not shown in FIG. 6), the reconstructed video sequence may typically be a replica of the source video sequence with some errors. The local video decoder (633) can repeat the decoding process that can be executed by the video decoder in the reference picture, causing the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference picture having common content as the reconstructed reference picture obtained by the video decoder at the remote end (assuming no transmission errors).

[0071]

[0108] The predictor (635) can perform a prediction search for the coding engine (632). That is, for a new picture to be coded, the predictor (635) can search the reference picture memory (634) for sample data (as a candidate reference pixel block) or predetermined metadata (reference picture motion vectors, block shapes, etc.), which may serve as an appropriate prediction reference for the new picture. The predictor (635) can operate on a sample block - pixel block basis to find an appropriate prediction reference. In some cases, the input picture may have a prediction reference derived from a plurality of reference pictures stored in the reference picture memory (634) as determined by the search result obtained by the predictor (635).

[0072]

[0109] The controller (650) can manage the coding operation of the source coder (630), including setting parameters and subgroup parameters used, for example, to encode video data.

[0073]

[0110] All outputs of the aforementioned functional units can be entropy-coded in the entropy coder (645). The entropy coder (645) converts the symbols generated by the various functional units into a coded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable-length coding, arithmetic coding, etc.

[0074]

[0111] The transmitter (640) can buffer the coded video sequence as created by the entropy coder (645) and prepare it for transmission via the communication channel (660), which may be a hardware / software link to a storage device storing the encoded video data. The transmitter (640) can merge the coded video data from the video coder (603) with other data to be transmitted, such as, for example, coded audio data and / or auxiliary data streams (source not shown).

[0075]

[0112] The controller (650) can manage the operation of the video encoder (603). During coding, the controller (650) can assign a specific coded picture type to each of the coded pictures, which may affect the coding technique applicable to each picture. For example, a picture may often be assigned as one of the following picture types:

[0113] An Intra Picture (I Picture) can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs, for example, allow different types of Intra Pictures, including Independent Decoder Refresh (IDR) Pictures. Those skilled in the art are aware of these variations of I Pictures, as well as their respective uses and characteristics.

[0076]

[0114] A Predictive Picture (P Picture) can be encoded and decoded using Intra prediction or Inter prediction that uses at most one motion vector and a reference index to predict the sample values of each block.

[0077]

[0115] A Bi - directional Predictive Picture (B Picture) can be encoded and decoded using Intra prediction or Inter prediction that uses at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple predictive pictures can use more than two reference pictures and related metadata for the reconstruction of one block.

[0078]

[0116] The source picture is typically spatially subdivided into a plurality of sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples each) and can be coded block by block. The blocks can be prediction coded by referring to other (already coded) blocks as determined by the coding assignment applied to each block of the picture. For example, blocks of an I picture can be non-prediction coded or they can be prediction coded by referring to already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of a P picture can be prediction coded by spatial or temporal prediction by referring to one previously coded reference picture. Blocks of a B picture can be prediction coded by spatial or temporal prediction by referring to one or two previously coded reference pictures.

[0079]

[0117] The video encoder (603) can perform coding operations according to a predetermined video coding technology or standard such as ITU-T Rec.H.266. In this operation, the video encoder (603) can execute various compression operations including prediction coding operations that utilize temporal and spatial redundancies in the input video sequence. The coded video data can thus conform to the syntax specified by the video coding technology or standard being used.

[0080]

[0118] In an embodiment, the transmitter (640) can transmit additional data together with the coded video. The source coder (630) can include such data as part of the coded video sequence. The additional data can include temporal / spatial / SNR enhancement layers, other forms of redundant data (redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.).

[0081]

[0119] Video can be captured as a plurality of source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra prediction) utilizes the spatial correlation in a given picture, and inter-picture prediction utilizes the (temporal or other) correlation between pictures. In one example, a specific picture under encoding / decoding called the current picture is partitioned into blocks. If a block in the current picture is similar to a reference block in a reference picture that has been previously coded and is still buffered in the video, the block in the current picture can be coded by a vector called a motion vector. The motion vector points to the reference block in the reference picture and can have a third dimension to identify the reference picture when multiple reference pictures are used.

[0082]

[0120] In some embodiments, it is possible to use dual-prediction techniques for inter-picture prediction. According to the dual-prediction technique, two reference pictures such as a first reference picture and a second reference picture that both precede the current picture in decoding order within the video (however, they may be in the past and future respectively in display order) are used. A block in the current picture can be coded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. The block can be predicted by a combination of the first reference block and the second reference block.

[0083]

[0121] Furthermore, in order to improve coding efficiency, it is possible to use merge mode techniques for inter-picture prediction.

[0084]

[0122] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are partitioned into coding tree units (CTUs) for compression, and the CTUs within a picture have the same size such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU includes three coding tree blocks (CTBs) which are one luma CTB and two chroma CTBs. Each CTU can be recursively quad-tree partitioned into one or more coding units (CUs). For example, a 64×64 pixel CTU can be partitioned into one 64×64 pixel CU, four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one example, each CU is analyzed to determine a prediction type of the CU, such as an inter prediction type or an intra prediction type. The CU is partitioned into one or more prediction units (PUs) depending on temporal and / or spatial predictability. Generally, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, the prediction operation in coding (encoding / decoding) is performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of values (e.g., luma values) for pixels such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0085]

[0123] FIG. 7 shows an exemplary diagram of a video encoder (703). The video encoder (703) is configured to receive a processing block (e.g., a prediction block) of sample values within a current video picture in a sequence of video pictures and encode the processing block into a coded picture that is part of a coded video sequence. In one example, the video encoder (703) is used in place of the video encoder (403) of the example of FIG. 4.

[0086]

[0124] In an example of HEVC, a video encoder (703) receives a matrix of sample values of a processing block, such as a prediction block of 8×8 samples. The video encoder (703) determines whether the processing block is best coded, for example using rate distortion optimization, using an intra-mode, an inter-mode, or a bi-prediction mode. If the processing block is to be coded in the intra-mode, the video encoder (703) can use intra prediction techniques to code the processing block into the coded picture; if the processing block is to be coded in the inter-mode or the bi-prediction mode, the video encoder (703) can use inter prediction techniques or bi-prediction techniques respectively to code the processing block into the coded picture. In certain video coding techniques, the merge mode may be an inter-picture prediction sub-mode, in which case the motion vector is derived from one or more motion vector predictors without the benefit of coded motion vector components outside the predictor. In certain other video coding techniques, there may be motion vector components applicable to the target block. In one example, the video encoder (703) includes other components, such as a mode decision module (not shown), to determine the mode of the processing block.

[0087]

[0125] In the example of FIG. 7, the video encoder (703) includes an inter encoder (730), an intra encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general-purpose controller (721), and an entropy encoder (725) all coupled together as shown in FIG. 7.

[0088]

[0126] The inter - encoder (730) receives samples of a current block (e.g., a processing block), compares the block with one or more reference blocks in a reference picture (e.g., blocks in a previous picture and blocks in a subsequent picture), generates inter - prediction information (e.g., a description of redundant information by an inter - coding technique, a motion vector, merge - mode information), and is configured to calculate an inter - prediction result (e.g., a predicted block) based on the inter - prediction information using some suitable technique. In some examples, the reference picture is a decoded reference picture decoded based on the encoded video information.

[0089]

[0127] The intra - encoder (722) receives samples of a current block (e.g., a processing block), optionally compares the block with blocks already coded within the same picture, generates quantized coefficients after transformation, and optionally also generates intra - prediction information (e.g., intra - prediction direction information according to one or more intra - coding techniques). In one example, the intra - encoder (722) also calculates an intra - prediction result (e.g., a predicted block) based on the intra - prediction information and reference blocks within the same picture.

[0090]

[0128] The general-purpose controller (721) is configured to determine general control data and control other components of the video encoder (703) based on the general control data. In one example, the general-purpose controller (721) determines the mode of a block and provides a control signal to the switch (726) based on that mode. For example, when the mode is the intra mode, the general-purpose controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select the intra prediction information and include the intra prediction information in the bitstream; also, when the mode is the inter mode, the general-purpose controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select the inter prediction information and include the inter prediction information in the bitstream.

[0091]

[0129] The residual calculator (723) is configured to calculate the difference (residual data) between the received block and the prediction result selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) is configured to operate based on the residual data to encode the residual data and generate transform coefficients. In one example, the residual encoder (724) is configured to convert the residual data from the spatial domain to the frequency domain and generate transform coefficients. The transform coefficients are then subjected to quantization processing to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be appropriately used by the intra-encoder (722) and the inter-encoder (730). For example, the inter-encoder (730) can generate a decoded block based on the decoded residual data and the inter-prediction information, and the intra-encoder (722) can generate a decoded block based on the decoded residual data and the intra-prediction information. The decoded block is appropriately processed to generate a decoded picture, and the decoded picture is buffered in a memory circuit (not shown) and can be used as a reference picture in some examples.

[0092]

[0130] The entropy encoder (725) is configured to format a bitstream to include an encoded block. The entropy encoder (725) is configured to include various information in the bitstream according to an appropriate standard such as the HEVC standard. In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra prediction information or inter prediction information), residual information, and other appropriate information in the bitstream. Note that there is no residual information when coding a block in either the inter-mode or the merge sub-mode of the bi-prediction mode according to the disclosed subject matter.

[0093]

[0131] FIG. 8 shows an exemplary diagram of a video decoder (810). The video decoder (810) is configured to receive a coded picture that is part of a coded video sequence and decode the coded picture to generate a reconstructed picture. In one example, the video decoder (810) is used instead of the video decoder (410) in the example of FIG. 4.

[0094]

[0132] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872) coupled together as shown in FIG. 8.

[0095]

[0133] Entropy decoder (871) can be configured to reconstruct from the coded picture certain symbols representing the syntax elements that make up the coded picture. Such symbols can include, for example, the mode in which a block is coded (e.g., intra mode, inter mode, bi-prediction mode, merge sub-mode or another sub-mode in the latter two), prediction information (e.g., intra prediction information or inter prediction information), which can identify certain samples or metadata used for prediction by intra decoder (872) or inter decoder (880) respectively. Also, the symbols can include, for example, residual information in the form of quantized transform coefficients. In one example, when the prediction mode is inter or bi-prediction mode, inter prediction information is provided to inter decoder (880); when the prediction type is intra prediction type, intra prediction information is provided to intra decoder (872). The residual information can be inverse quantized and provided to residual decoder (873).

[0096]

[0134] Inter decoder (880) is configured to receive inter prediction information and generate an inter prediction result based on the inter prediction information.

[0097]

[0135] Intra decoder (872) is configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0098]

[0136] Residual decoder (873) is configured to perform inverse quantization to extract non-quantized transform coefficients, process the non-quantized transform coefficients, and convert the residual information from the frequency domain to the spatial domain. Residual decoder (873) may also require certain control information (including quantization parameter (QP)), and that information may be provided by entropy decoder (871) (since this may be only a small amount of control information, the data path is not depicted).

[0099]

[0137] The reconstruction module (874) is configured to combine, in the spatial domain, residual information as an output by the residual decoder (873) and a prediction result (which may be output by an inter or intra prediction module in some cases) to form a reconstructed block, which is part of a reconstructed picture, and the picture may be part of a reconstructed video. It should be noted that other appropriate processes, such as deblocking processing, may be performed to improve visual quality.

[0100]

[0138] The video encoders (403), (603), and (703), and the video decoders (410), (510), and (810) can be implemented using any appropriate technology. In an embodiment, the video encoders (403), (603), and (703), and the video decoders (410), (510), and (810) can be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (703), and the video decoders (410), (510), and (810) can be implemented using one or more processors that execute software instructions.

[0101]

[0139] Aspects of the present disclosure provide techniques for coding inter prediction directions and BCW indices in merge mode.

[0102]

[0140] In VVC, it is possible to use various inter-prediction modes. Regarding the CU to be inter-predicted, the motion parameters can include the MV, one or more reference picture indices, the reference picture list usage index, additional information for a predetermined coding feature to be used for generating the inter-predicted samples. The motion parameters can be signaled in an explicit or implicit manner. When the CU is coded in skip mode, the CU can be associated with the PU and may not have significant residual coefficients, a coded motion vector delta, or an MV difference (e.g., MVD) or a reference picture index. It is possible to specify the merge mode, where the motion parameters for the current CU are obtained from neighboring CU(s) (spatial candidates and / or temporal candidates), and optionally, additional information such as that introduced in VVC. The merge mode can be applied not only for the skip mode but also for the inter-predicted CU. In one example, an alternative to the merge mode is the explicit transmission of the motion parameters, in which case the MV(s), the corresponding reference picture index for each reference picture list, the reference picture list usage flag, and other information are explicitly signaled for each CU.

[0103]

[0141] In embodiments such as VVC, the VVC Test model (VTM) reference software includes one or more refined inter-prediction coding tools, which are: extended merge prediction, merge motion vector difference (MMVD) mode, adaptive motion vector prediction (AMVP) mode using symmetric MVD signaling, affine motion compensation prediction, Subblock-based temporal motion vector prediction (SbTMVP), Adaptive motion vector resolution (AMVR), Motion field storage (1 / 16 luma sample MV storage, and 8×8 motion field compression), Bi-prediction with CU-level weights (BCW), Bi-directional optical flow (BDOF), Prediction refinement using optical flow (PROF), Decoder side motion vector refinement (DMVR), Combined inter and intra prediction (CIIP), Geometric partitioning mode (GPM) and the like are included. Inter prediction and related methods are described in detail below.

[0104]

[0142] In some examples, it is possible to use extended merge prediction. In an example such as in VTM4, the merge candidate list is constructed by sequentially including the following five types of candidates: Motion vector predictor(s) (MVP) from spatial neighboring CUs, Temporal MVP from collocated CU(s), History-based MVP(s) (HMVP) from a first-in-first-out (FIFO) table, Pairwise average MVP, and zero MV(s).

[0105]

[0143] The size of the merge candidate list can be signaled in the slice header. In one example, the maximum allowable size of the merge candidate list is 6 in VTM4. For each CU coded in merge mode, the index of the best merge candidate (e.g., the merge index) can be coded using truncated unary binarization (TU). The first bin of the merge index can be coded using context (e.g., context-adaptive binary arithmetic coding (CABAC)), and bypass coding can be used for the other bins.

[0106]

[0144] Some examples of the generation process for each category of merge candidates are given below. In an embodiment, the spatial candidates are derived as follows. It is possible to assume that the derivation of the spatial merge candidates in VVC is the same as that in HEVC. In one example, out of the candidates at the positions shown in FIG. 9, a maximum of 4 merge candidates are selected. FIG. 9 shows the positions of the spatial merge candidates according to an embodiment of the present disclosure. Referring to FIG. 9, the order of derivation is B1, A1, B0, A0, and B2. The position B2 is considered only if none of the CUs at the positions A0, B0, B1, and A1 are available (e.g., due to the CU belonging to another slice or another tile), or if it is intra-coded. After the candidate at position A1 is added, the addition of the remaining candidates is subject to a redundancy check, which ensures that candidates with the same motion information are excluded from the candidate list, thereby improving the coding effect.

[0107]

[0145] To reduce the computational complexity, not all possible candidate pairs are considered in the above redundancy check. Instead, only the pairs connected by arrows in FIG. 10 are considered, and a candidate is only added to the candidate list when the corresponding candidates used for the redundancy check do not have the same motion information. FIG. 10 shows candidate pairs considered for the redundancy check of spatial merge candidates according to an embodiment of the present disclosure. Referring to FIG. 10, the pairs connected by each arrow are A1 and B1, A1 and A0, A1 and B2, B1 and B0, B1 and B2. Thereby, candidates at positions B1, A0, and / or B2 can be compared with candidates at position A1, and candidates at positions B0 and / or B2 can be compared with candidates at position B1.

[0108]

[0146] In an embodiment, the time candidates are derived as follows. In one example, only one time merge candidate is added to the candidate list. FIG. 11 shows an exemplary motion vector scaling for time merge candidates. To derive the time merge candidate of the current CU (1111) in the current picture (1101), the scaled MV (1121) (e.g., as indicated by the dotted line in FIG. 11) can be derived based on the collocated CU (1112) belonging to the collocated reference picture (1104) at the same position. The reference picture list used to derive the collocated CU (1112) can be explicitly signaled in the slice header. The scaled MV (1121) for the time merge candidate can be obtained as shown by the dotted line in FIG. 11. The scaled MV (1121) can be scaled from the MV of the collocated CU (1112) using the picture order count (POC) distances tb and td. The POC distance tb can be defined as the POC difference between the current reference picture (1102) of the current picture (1101) and the current picture (1101). The POC distance td can be defined as the POC difference between the collocated reference picture (1104) of the collocated picture (1103) and the collocated picture (1103). The reference picture index of the time merge candidate can be set to zero.

[0109]

[0147] FIG. 12 shows exemplary candidate positions (e.g., C0 and C1) for the temporal merge candidate of current CU. The positions for the temporal merge candidate can be selected from candidate positions C0 and C1. Candidate position C0 is located at the lower right corner of CU (1210) at the equivalent position of the current CU. Candidate position C1 is located at the center of CU (1210) at the equivalent position of the current CU. If the CU at candidate position C0 is not available, is intra-coded, or is outside the current row of the CTU, candidate position C1 is used to derive the temporal merge candidate. Otherwise, for example, if the CU at candidate position C0 is available, is intra-coded, and in the current row of the CTU, candidate position C0 is used to derive the temporal merge candidate.

[0110]

[0148] In HEVC, a translational motion model is applied to motion compensation prediction (MCP). In the real world, there may be many types of motions such as zoom-in / out, rotation, diagonal motion, and other irregular motions. Block-based affine transform motion compensation prediction may be applied as in VTM. FIG. 13A shows an affine motion field of a block (1302) described by the motion information of two control points (4 parameters). FIG. 13B shows an affine motion field of a block (1304) described by three control point motion vectors (6 parameters).

[0111]

[0149] As shown in FIG. 13A, in the 4-parameter affine motion model, the motion vector at the sample position (x, y) within block (1302) can be derived by Equation Eq. (1) as follows:

[0112]

Equation

[0113]

Eq.

[0150] As shown in FIG. 13B, in the 6-parameter affine motion model, the motion vector at the sample position (x, y) within the block (1304) can be derived by Eq. (3) as follows:

[0114]

Eq.

[0115]

Eq.

[0116] (mv 1x , mv 1y ) can be considered as the motion vector of the top-right corner control point.

[0117] (mv 2x , mv 2y ) can be considered as the motion vector of the bottom-left corner control point.

[0118]

[0151] As shown in FIG. 14, in order to simplify motion compensation prediction, it is possible to apply block-based affine transform prediction. To derive the motion vector for each 4×4 luma sub-block, the motion vector (e.g., (1402)) of the central sample of the sub-block (e.g., (1404)) within the current block (1400) is calculated according to equations (1)-(4) and can be rounded to 1 / 16 fractional precision. Then, it is possible to apply a motion compensation interpolation filter to generate the prediction for each sub-block using the derived motion vector. Also, the sub-block size of the chroma component may be set to 4×4. The MV of a 4×4 chroma sub-block can be calculated as the average of the MVs of four corresponding 4×4 luma sub-blocks.

[0119]

[0152] In affine merge prediction, for a CU where both the width and height are 8 or more, it is possible to apply the affine merge (AF_MERGE) mode. The CPMVs of the current CU can be generated based on the motion information of spatially neighboring CUs. Up to five CPMVP candidates can be applied to affine merge prediction, and an index can be signaled to indicate which of the five CPMVP candidates can be used for the current CU. In affine merge prediction, it is possible to form an affine merge candidate list using the following three types of CPMV candidates: (1) Inherited affine merge candidates estimated from the CPMVs of neighboring CUs, (2) Affine merge candidates constructed using CPMVPs derived using the translational MVs of neighboring CUs, and (3) Zero MVs.

[0120]

[0153] In VTM3, up to two inherited affine candidates can be applied. The two inherited affine candidates can be derived from the affine motion models of neighboring blocks. For example, one inherited affine candidate can be derived from the left neighboring CU, and the other inherited affine candidate can be derived from the above neighboring CU. It is possible to assume that the exemplary candidate blocks are those shown in FIG. 15. As shown in FIG. 15, in the case of the left predictor (or left inherited affine candidate), it is possible to assume that the scan order is A0→A1, and in the case of the above predictor (or above inherited affine candidate), it is possible to assume that the scan order is B0→B1→B2. Therefore, only the first available inherited candidate can be selected from each side. Between the two inherited candidates, the pruning check may not be executed. When the neighboring affine CU is identified, the control point motion vectors of the neighboring affine CU can be used to derive the CPMV candidates in the affine merge list of the current CU. As shown in FIG. 16, when the lower left neighboring block A of the current block (1604) is coded in the affine mode, the motion vectors v2, v3, v4 of the top left corner, above right corner, and lower left corner of the CU (1602) including block A can be obtained. When block A is coded in the four-parameter affine model, the two CPMVs of the current CU (1604) can be calculated according to v2, v3 of CU (1602). When block A is coded in the six-parameter affine model, the three CPMVs of the current CU (1604) can be calculated according to v2, v3, v4 of CU (1602).

[0121]

[0154] The constructed affine candidates of the current block can be considered as candidates constructed by combining the translational motion information in the vicinity of each control point of the current block. The motion information of the control points can be derived from specific spatial and temporal vicinities that can be shown in Figure 17. As shown in Figure 17, CPMV k (k = 1, 2, 3, 4) represents the k-th control point of the current block (1302). In the case of CPMV1, it is possible to check the B2->B3->A2 blocks and use the MV of the first available block. In the case of CPMV2, it is possible to check the B1->B0 block. In the case of CPMV3, it is possible to check the A1->A0 block. If CPMV4 is not available, TMVP may be used as CPMV4.

[0122]

[0155] After the MVs of the four control points are obtained, based on the motion information of the four control points, an affine merge candidate for the current block (1702) can be constructed. For example, it is possible to construct an affine merge candidate based on the combination of the MVs of the four control points in the following order: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, and {CPMV1, CPMV3}.

[0123]

[0156] A combination of three CPMVs can construct a 6-parameter affine merge candidate, and a combination of two CPMVs can construct a 4-parameter affine merge candidate. To avoid the motion scaling process, if the reference indices of the control points are different, it is possible to discard the related combinations of the control point MVs.

[0124]

[0157] After the inherited affine merge candidates and the constructed affine merge candidates are checked, if the list is still not full, it is possible to insert zero MVs at the end of the list.

[0125]

[0158] In affine AMVP prediction, it is possible to apply the affine AMVP mode to a CU where both the width and height are 16 or more. The affine flag at the CU level can be signaled in the bitstream to indicate whether the affine AMVP mode is used, and then another flag can be signaled to indicate whether 4-parameter affine or 6-parameter affine is applied. In affine AMVP prediction, it is possible to signal the predictors of the CPMVPs of the current CU and the differences of the CPMV of the current CU in the bitstream. The size of the affine AMVP candidate list can be assumed to be 2, and the affine AMVP candidate list can be generated by using four types of CPMV candidates in the following order: (1) Inherited affine AMVP candidates estimated from the CPMVs of neighboring CUs, (2) Affine AMVP candidates constructed using the CPMVPs derived using the translational MVs of neighboring CUs, (3) Translational MVs from neighboring CUs, and (4) Zero MVs.

[0126]

[0159] The order of checking the inherited affine AMVP candidates can be the same as the order of checking the inherited affine merge candidates. To determine the AVMP candidates, it is possible to consider only the affine CUs having the same reference picture as the current block. When the inherited affine motion predictor is inserted into the candidate list, the pruning process may not be applied.

[0127]

[0160] The constructed AMVP candidates can be derived from the specified spatial neighborhood. As shown in FIG. 17, it is possible to apply the same checking order as that at the time of constructing the affine merge candidates. Also, it is possible to check the reference picture indices of neighboring blocks. The first block in the checking order may be inter-coded and have the same reference picture as the current CU (1702). When the current CU (1702) is coded in the 4-parameter affine mode, one constructed AMVP candidate can be determined and both mv0 and mv1 can be used. The constructed AMPV candidates can be further added to the affine AMVP list. When the current CU (1702) is coded in the 6-parameter affine mode and all three CPMVs are available, the constructed AMVP candidates can be added as one candidate in the affine AMVP list. Otherwise, the constructed AMVP candidates can be set as unavailable.

[0128]

[0161] After the inherited affine AMVP candidates and the constructed AMVP candidates are checked, if the number of candidates in the affine AMVP list is still less than 2, mv0, mv1, and mv2 may be added in order. mv0, mv1, and mv2, if available, may function as translational MVs for predicting all control point MVs of the current CU (e.g., (1702)). Finally, if the affine AMVP is still not full, zero MVs can be used to fill the affine AMVP list.

[0129]

[0162] Sub-block-based affine motion compensation can save memory access bandwidth and reduce computational complexity compared to pixel-based motion compensation at the expense of prediction accuracy penalty. To achieve finer granularity of motion compensation, prediction refinement with optical flow (PROF) can be used to refine sub-block-based affine motion compensation prediction without increasing the memory access bandwidth for motion compensation. In VVC, after sub-block-based affine motion compensation is performed, luma prediction samples can be refined by adding the differences derived from the optical flow equation. PROF can be described in the following four steps:

[0163] Step (1): Sub-block-based affine motion compensation can be performed to generate a sub-block prediction I(i,j).

[0130]

[0164] Step (2): The spatial gradients g x (i,j) and g y (i,j) can be calculated at each sample position using a 3-tap filter [-1,0,1]. It is possible to assume that the gradient calculation is the same as that of BDOF. For example, the spatial gradients g x (i,j) and g y (i,j) can be calculated based on Eqs. (5) and (6) respectively.

[0131]

Number

[0132]

[0165] Step (3): Luma prediction refinement can be calculated by the optical flow equation as shown in Eq. (7).

[0133]

Number

[0134]

[0166] Since the affine model parameters and the sample positions relative to the center of the sub - block may not change between sub - blocks, Δv(i,j) is calculated for the first sub - block (e.g., (1802)) and can be reused for other sub - blocks (e.g., (1810)) within the same CU (e.g., (1800)). Let the horizontal offset and vertical offset from the sample position (i,j) to the center of the sub - block (x SB ,y SB ) be dx(i,j) and dy(i,j), then Δv(x,y) can be derived as follows by Eqs. (8) and (9):

[0135]

Equation

[0167] To maintain accuracy, the center of the sub - block (x SB ,y SB ) can be calculated as ((W SB - 1) / 2,(H SB - 1) / 2) where W SB and H SB are the width and height of the sub - block respectively.

[0136]

[0168] Once Δv(x,y) is obtained, it is possible to obtain the parameters of the affine model. For example, in the case of a 4 - parameter affine model, the parameters of the affine model can be expressed as in Eq. (10):

[0137]

Equation

[0138]

Equation

[0139]

[0169] Step (4): Finally, the luma prediction refinement ΔI(i,j) can be added to the sub-block prediction. The final prediction I' can be generated as shown in Equation Eq. (12):

[0140]

Number

[0170] PROF may not be applicable to the affine-coded CU in the following two cases: (1) the case where all control point MVs are the same, indicating that the CU has only translational motion; and (2) the case where the affine motion parameters are larger than the specified limits, because the sub-block-based affine MC degrades with respect to the CU-based MC to avoid large memory access bandwidth requirements.

[0141]

[0171] To improve coding efficiency and reduce the transmission overhead of MVs, sub-block level MV refinement can be applied to extend the temporal motion vector prediction (TMVP) at the CU level. In one example, the subblock-based TMVP (SbTMVP) mode enables inheriting motion information at the sub-block level from a reference picture at an equivalent position. Each sub-block of a current CU (e.g., a current CU having a large size) within a current picture can have its respective motion information without explicitly transmitting the block partition structure or respective motion information. In the SbTMVP mode, the motion information of each sub-block can be obtained, for example, in three steps as follows.

[0142] In the first step, it is possible to derive the displacement vector (DV) of the current CU.

[0143] In the second step, it is possible to check the availability of SbTMVP candidates and derive the central motion (e.g., the central motion of the current CU). [[ID=ID=10]]

[0144] In the third step, the sub-block motion information can be derived from the corresponding sub-block in the equivalent position block using the DV. The three steps can be integrated into one or two steps, and / or the order of the three steps may be changed.

[0145]

[0172] Different from deriving temporal MVs from blocks at equivalent positions within a reference frame or reference picture for TMVP candidate derivation, in the SbTMVP mode, for each sub-block of a current CU within the current picture, a DV (e.g., a DV derived from the MV of a neighboring CU to the left of the current CU) may be applied to identify the corresponding sub-block within the equivalent position picture. If the corresponding sub-block is not inter-coded, the motion information of the current sub-block can be set to the motion at the center of the equivalent position block.

[0146]

[0173] The SbTMVP mode can be supported by various video coding standards including, for example, VVC. Similar to the TMVP mode, in HEVC, for example, in the SbTMVP mode, a motion field (also called a motion information field or an MV field) in the equivalent position picture can be used to improve the MV prediction and merge mode for CUs within the current picture. In one example, the same equivalent position picture used by the TMVP mode is used in the SbTMVP mode. In one example, the SbTMVP mode differs from the TMVP mode in the following aspects: (i) The TMVP mode predicts motion information at the CU level, while the SbTMVP mode predicts motion information at the sub-CU level; (ii) The TMVP mode fetches temporal MVs from equivalent position blocks in the equivalent position picture (e.g., the equivalent position block is the bottom-right or central block relative to the current CU), while the SbTMVP mode can apply a motion shift before fetching temporal motion information from the equivalent position picture. In one example, the motion shift used in the SbTMVP mode is obtained from the MV of one of the spatial neighboring blocks of the current CU.

[0147]

[0174] Figures 19-20 show an exemplary SbTMVP process used in SbTMVP mode. The SbTMVP process can predict the motion vectors (MVs) of sub-CUs (e.g., sub-blocks) within a current CU (e.g., current block) (1901) in a current picture (2011) in, for example, two steps. In the first step, the spatial neighborhood (e.g., A1) of the current block (1901) in FIGS. 19-20 is examined. If the spatial neighborhood (e.g., A1) has a MV (2021) that uses an equivalent position picture (20 12) as a reference picture for the spatial neighborhood (e.g., A1), the MV (2021) can be selected to be the motion shift (or DV) to be applied to the current block (1901). If no such MV (e.g., an MV that uses an equivalent position picture (2012) as a reference picture) is identified, the motion shift or DV can be set to a zero MV (e.g., (0,0)). In some examples, the MV(s) of additional spatial neighborhoods such as A0, B0, B1 and the like are checked if such an MV is not identified for the spatial neighborhood A1.

[0148]

[0175] In a second step, the motion shift or DV(2021) identified in the first step is applied to the current block (1901) (e.g., DV(2021) is added to the coordinates of the current block), and sub-CU level motion information (e.g., including an MV and a reference index) is obtained from the co-located picture (2012). In the example shown in FIG. 20, the motion shift or DV(2021) is set to be the MV of the spatial neighborhood A1 (e.g., block A1) of the current block (1901). For each sub-CU or sub-block (2031) within the current block (1901), it is possible to derive the motion information of the sub-CU or sub-block (2031) using the motion information of the corresponding co-located block (2001) within the co-located picture (2012) (e.g., the motion information of the smallest motion grid covering the central sample of the co-located block (2001)). After the motion information of the co-located sub-CU (2032) within the co-located block (2001) is identified, the motion information of the co-located sub-CU (2032) can be converted into the motion information (e.g., an MV and one or more reference indices) of the current sub-CU (2031) using a scaling method such as a method similar to the TMVP process used in HEVC, where temporal motion scaling is applied to align the reference picture of the temporal MV with the reference picture of the current CU.

[0149]

[0176] The motion field of the current block (1901) derived based on DV(2021) can include the motion information of each sub-block (2031) within the current block (1901), such as an MV and one or more related reference indices. The motion field of the current block (1901) can also be referred to as an SbTMVP candidate and corresponds to DV(2021).

[0150]

[0177] FIG. 20 shows an example of a motion field of a current block (1901) or an SbTMVP candidate. The motion information of the bi-predicted sub-block (2031(1)) includes a first MV, a first index indicating a first reference picture in reference picture list 0 (L0), a second MV, and a second index indicating a second reference picture in reference picture list 1 (L1). In one example, the motion information of the uni-predicted sub-block (2031(2)) includes an MV and an index indicating a reference picture in L0 or L1.

[0151]

[0178] In one example, DV(2021) is applied to the center position of the current block (1901) to determine the displaced center position in the co-located picture (2012). If the block containing the displaced center position is not inter-coded, the SbTMVP candidate is considered unavailable. Otherwise, when the block containing the displaced center position (e.g., the co-located block (2001)) is inter-coded, the motion information of the center position of the current block (1901), called the center motion of the current block (1901), can be derived from the motion information of the block containing the displaced center position within the co-located picture (2012). In one example, it is possible to derive the center motion of the current block (1901) from the motion information of the block containing the displaced center position within the co-located picture (2012) using a scaling process. When the SbTMVP candidate is available, DV(2021) can be applied to find the corresponding sub-block (2032) within the co-located picture (2012) for each sub-block (2031) of the current block (1901). The motion information of the corresponding sub-block (2032) can be used to derive the motion information of the sub-block (2031) within the current block (1901) in the same way as used to derive the center motion of the current block (1901). In one example, when the corresponding sub-block (2032) is not inter-coded, the motion information of the current sub-block (2031) is set to be the center motion of the current block (1901).

[0152]

[0179] In some examples, bi-prediction with CU-level weight (BCW) can be used to weight predictions from different reference pictures in various ways. In one example (e.g., HEVC), the bi-prediction signal is generated by averaging two prediction signals obtained from two different reference pictures and / or by using two different motion vectors. In some examples (e.g., VVC), the bi-prediction mode is extended to allow weighted averaging (or weighted mean) of two prediction signals beyond simple arithmetic mean as shown by Equation Eq. (13):

[0153]

Number

[0154]

[0180] In some examples, five values are allowed for BCW weighting in weighted averaging of bi-prediction, e.g., w ∈ {-2, 3, 4, 5, 10}. In some examples, BCW weighting can be represented by a BCW weighting index. For each bi-predicted CU, the value of the weighting parameter w is determined in one of two ways: 1) For non-merge CUs, the weighting index is signaled after the motion vector difference, and the weighting index indicates a value selected from a list; 2) For merge CUs, the weighting index is inferred from neighboring blocks based on the merge candidate index. In some examples, BCW is applied only to CUs having 256 or more luma samples (i.e., those with CU width times CU height being 256 or more). In one example, for low-delay pictures, all five weighting values are used. For non-low-delay pictures, only three weights (e.g., w ∈ {3, 4, 5}) are used.

[0155]

[0181] In some examples, on the encoder side, a search algorithm, such as a fast search algorithm, is applied to find the weighted index without significantly increasing the complexity of the encoder. When BCW is combined with AMVR, if the current picture is a low-delay picture, unequal weighting is only conditionally checked for 1-pel and 4-pel motion vector accuracy.

[0156]

[0182] In some examples, when BCW is combined with affine, affine ME is performed on unequal weighting only if the affine mode is selected as the current best mode.

[0157]

[0183] In some examples, unequal weighting is conditionally checked when the two reference pictures in bi-prediction are the same.

[0158]

[0184] In some examples, unequal weighting is not searched if certain conditions, such as the POC distance between the current picture and its reference picture, the coding QP, and the temporal level, are met.

[0159]

[0185] In some examples, the weighted index for BCW is coded using the first context-coded bin followed by a bypass-coded bin. The first context-coded bin indicates whether equal weighting is used; if unequal weighting is used, additional bins are signaled using bypass coding to indicate which unequal weighting is used from a list of weighting parameter values.

[0160]

[0186] Some video codecs, such as the H.264 / AVC and HEVC standards, support a coding tool called weighted prediction (WP) to efficiently code video content with fading. Support for WP is also added to the VVC standard. WP enables the signaling of weighting parameters (weights and offsets) for each reference picture in each of the reference picture lists L0 and L1. Then, during motion compensation, the weights and offsets of the corresponding reference pictures are applied. WP and BCW are designed for different types of video content. To avoid the interaction between WP and BCW that would complicate the VVC decoder design, when a CU uses WP, the BCW weight index is not signaled and w is inferred to be 4 (i.e., equal weighting is applied). For a merge CU, the weight index is inferred from neighboring blocks based on the merge candidate index. This can be applied to both the normal merge mode and the inherited affine merge mode. For the constructed affine merge mode, the affine motion information is composed based on the motion information of at most 3 blocks. The BCW index of a CU using the constructed affine merge mode is simply set to be equal to the BCW index of the first control point MV.

[0161]

[0187] In VVC, CIIP and BCW cannot be applied together to a CU. When a CU is coded in the CIIP mode, the BCW index of the current CU is set to 2, for example, equal weighting.

[0162]

[0188] In an embodiment, like in VVC, a merge with motion vector difference (MMVD) mode is used, in which case samples of a CU (e.g., the current CU) can be predicted using implicitly derived motion information. The MMVD mode is used for either the skip mode or the merge mode using a motion vector representation method. The MMVD merge flag can be signaled to specify whether the MMVD mode is used for a CU, for example, after signaling a skip flag or a merge flag.

[0163]

[0189] In some examples, MMVD reuses merge candidates. The candidates can be selected from among the merge candidates and are further extended by the motion vector representation method. MMVD provides a motion vector representation with simplified signaling. In some examples, the motion vector representation method includes a starting point, a magnitude of motion, and a direction of motion.

[0164]

[0190] In some examples (e.g., VVC), the MMVD technique can use a merge candidate list to select a candidate for the starting point. However, in one example, the only candidate (MRG_TYPE_DEFAULT_N), which is the default merge type, is considered for the extension of MMVD.

[0165]

[0191] In some examples, a base candidate index (IDX) is used to define the starting point. The base candidate index indicates the best candidate among the candidates in the list shown in Table 1. For example, the list is a merge candidate list having motion vector predictors (MVPs). The base candidate index can indicate the best candidate within the merge candidate list.

[0166] Table 1 - Examples of Base Candidate Index (IDX)

[0167]

Table 1

[0192] Note that in one example, the number of basic candidates is equal to 1, and in that case, the basic candidate IDX is not signaled.

[0168]

[0193] In the MMVD mode, after a merge candidate (also called an MV base or an MV start point) is selected, it is possible to refine the merge candidate with additional information such as the signaled MVD information. The additional information is an index used to specify the magnitude of the motion (e.g., a distance index, e.g., mmvd_distance_idx[x0][y0]) and an index used to indicate the direction of the motion (e.g., a direction index, e.g., mmvd_direction_idx[x0][y0]) can be included. In the MMVD mode, it is possible to select one of the first two candidates in the merge list as the MV base. For example, a merge candidate flag (e.g., mmvd_cand_flag[x0][y0]) indicates one of the first two candidates in the merge list. The merge candidate flag can be signaled to indicate (e.g., specify) which of the first two candidates is selected. The additional information can indicate the MVD (or motion offset) with respect to the MV base. For example, the magnitude of the motion indicates the magnitude of the MVD, and the direction of the motion indicates the direction of the MVD.

[0169]

[0194] In one example, a merge candidate selected from the merge candidate list is used to provide a starting point or an MV starting point in the reference picture. The motion vector of the current block can be represented by a starting point and a motion offset (or MVD) including the magnitude and direction of the motion with respect to the starting point. On the encoder side, the selection of the merge candidate and the determination of the motion offset can be performed based on a search process (evaluation process) as shown in FIG. 21. On the decoder side, the selected merge candidate and the motion offset can be determined based on the signaling from the encoder side.

[0170]

[0195] FIG. 21 shows an example of a search process (2100) in the MMVD mode. FIG. 22 shows an example of search points in the MMVD mode. In some examples, a subset or the entire set of the search points in FIG. 22 is used in the search process (2100) of FIG. 21. For example, by executing the search process (2100) on the encoder side, additional information including a merge candidate flag (e.g., mmvd_cand_flag[x0][y0]), a distance index (e.g., mmvd_distance_idx[x0][y0]), a direction index (e.g., mmvd_direction_idx[x0][y0]) can be determined for the current block (2101) in the current picture (or current frame).

[0171]

[0196] The first motion vector (2111) and the second motion vector (2121) belonging to the first merge candidate are shown. The first motion vector (2111) and the second motion vector (2121) are the starting points of the MVs used in the search process (2100). The first merge candidate may be a merge candidate in the merge candidate list constructed for the current block (2101). The first and second motion vectors (2111) and (2121) can be associated with two reference pictures (2102) and (2103) in the reference picture lists L0 and L1, respectively. Referring to FIGS. 21-22, the first and second motion vectors (2111) and (2121) can indicate two starting points (2011) and (2021) in the reference pictures (2102) and (2103), respectively, as shown in FIG. 22.

[0172]

[0197] Referring to FIG. 22, the two starting points (2211) and (2221) in FIG. 22 can be determined in the reference pictures (2102) and (2103). In one example, based on the starting points (2211) and (2221), a plurality of predetermined points extending in the vertical direction (+Y or -Y) or the horizontal direction (+X, -X) in the reference pictures (2202) and (2203) can be evaluated starting from the starting points (2211) and (2221). In one example, a pair of points (2214) and (2224) (e.g., as indicated by a shift of 1S in FIG. 21), or a pair of points (2215) and (2225) (e.g., as indicated by a shift of 2S in FIG. 21), which are pairs of points at mirror image positions with respect to each other for each starting point (2211) or (2221), can be used to determine a pair of motion vectors (e.g., MVs (2113) and (2123) in FIG. 21) that can form a motion vector predictor candidate for the current block (2101). Also, motion vector predictor candidates (e.g., MVs (2113) and (2123) in FIG. 21) determined based on predetermined points around the starting point (2211) or (2221) can be evaluated.

[0173]

[0198] The distance index (e.g., mmvd_distance_idx[x0][y0]) specifies the magnitude of motion and can also indicate a predetermined offset (e.g., 1S or 2S in FIG. 21) from the starting point indicated by the merge candidate flag. Note that the predetermined offset is also referred to as an MMVD step in one example.

[0174]

[0199] Referring to FIG. 21, the offset (e.g., MVD(2112) or MVD(2122)) can be applied (e.g., added) to the horizontal or vertical component of the starting MV (e.g., MV(2111) or (2121)). An exemplary relationship between the distance index (IDX) and the predefined offset is specified in Table 2.

[0175] When full-pel MMVD is off, for example, if the full-pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is equal to 0, it is possible to assume that the range of the predefined offset of MMVD is from 1 / 4 luma sample to 32 luma samples. When full-pel MMVD is off, the predefined offset can have a non-integer value such as a fraction of a luma sample (e.g., 1 / 4 pixel or 1 / 2 pixel).

[0176] When full-pel MMVD is on, for example, if the full-pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is equal to 1, it is possible to assume that the range of the predefined offset of MMVD is from 1 luma sample to 128 luma samples. In one example, when full-pel MMVD is on, the predefined offset only has an integer value such as one or more luma samples.

[0177] Table 2 - Exemplary relationship between distance index and offset (e.g., predetermined offset)

[0178]

Table 2

[0200] The direction index can represent the direction (or movement direction) of the MVD with respect to the starting point. In one example, the direction index represents one of the four directions shown in Table 3. The meaning of the MVD sign in Table 3 may vary depending on the information of the starting MV. In one example, when the starting MV is a single-prediction MV, or when the starting MV is a dual-prediction MV and both reference lists point to the same side of the current picture (for example, when the POCs of both reference pictures are greater than the POC of the current picture, or when the POCs of both reference pictures are less than the POC of the current picture), the MVD sign in Table 3 indicates the sign (or symbol) of the MV offset (or MVD) added to the starting MV.

[0179]

[0201] The starting MV is a bi-predicted MV. When the two MVs point to different sides of the current picture (for example, when the POC of one reference picture is larger than the POC of the current picture and the POC of the other reference picture is smaller than the POC of the current picture), the MVD sign in Table 3 indicates the sign of the MV offset (or MVD) added to the list0 MV component of the starting MV, and the MVD sign of the list1 MV has the opposite value. Referring to Figure 21, the starting MVs (2111) and (2121) are bi-predicted MVs, and the two MVs (2111) and (2121) point to different sides of the current picture. The POC of the L1 reference picture (2103) is larger than the POC of the current picture, and the POC of the L0 reference picture (2102) is smaller than the POC of the current picture. The MVD sign (for example, the sign "+" with respect to the x-axis) indicated by the direction index (e.g., 00) in Table 3 specifies the sign (for example, the sign "+" with respect to the x-axis) of the MVD (e.g., MVD(2112)) added to the list0 MV component of the starting MV (e.g., (2111)), and the MVD sign of MVD(2122) with respect to the list1 MV component of the starting MV (e.g., (2121)) has the opposite value, for example, the sign "-" opposite to the sign "+" of MVD(2112).

[0180]

[0202] Referring to Table 3, the direction index 00 indicates the positive direction on the x-axis, the direction index 01 indicates the negative direction on the x-axis, the direction index 10 indicates the positive direction on the y-axis, and the direction index 11 indicates the negative direction on the y-axis.

[0181] Exemplary relationship between the sign of the MV offset in Table 3 and the direction index

[0182]

Table 3

[0203] The syntax element mmvd_merge_flag[x0][y0] can be used to represent the MMVD merge flag of the current CU. In one example, An MMVD merge flag (e.g., mmvd_merge_flag[x0][y0]) equal to 1 indicates that the MMVD mode is used to generate the inter-prediction parameters of the current CU.

[0183] An MMVD merge flag (e.g., mmvd_merge_flag[x0][y0]) equal to 0 indicates that the MMVD mode is not used to generate the inter-prediction parameters. The array indices x0 and y0 can specify the position (x0, y0) of the top-left luma sample of the coding block (e.g., current CB) being considered with respect to the top-left luma sample of the picture (e.g., current picture).

[0184]

[0204] If the MMVD merge flag (e.g., mmvd_merge_flag[x0][y0]) does not exist for the current CU, the MMVD merge flag (e.g., mmvd_merge_flag[x0][y0]) may be inferred to be equal to 0 for the current CU.

[0185]

[0205] In some examples such as the VVC specification, a single context is used to signal the MMVD merge flag (e.g., mmvd_merge_flag). For example, a single context is used to code (e.g., encode and / or decode) the MMVD merge flag in context-adaptive binary arithmetic coding (CABAC).

[0186]

[0206] The syntax element mmvd_cand_flag[x0][y0] can represent a merge candidate flag. In one example, the merge candidate flag (e.g., mmvd_cand_flag[x0][y0]) specifies whether the first (0) or second (1) candidate in the merge candidate list is used together with the MVD derived from the distance index (e.g., mmvd_distance_idx[x0][y0]) and the direction index (e.g., mmvd_direction_idx[x0][y0]). The array indices x0 and y0 can specify the position (x0, y0) of the top-left luma sample of the coding block under consideration (e.g., the current CB) relative to the top-left luma sample of the picture (e.g., the current picture).

[0187]

[0207] If the merge candidate flag (e.g., mmvd_cand_flag[x0][y0]) does not exist, the merge candidate flag (e.g., mmvd_cand_flag[x0][y0]) may be inferred to be equal to 0.

[0188]

[0208] The syntax element mmvd_distance_idx[x0][y0] can represent a distance index. In one example, the distance index (e.g., mmvd_distance_idx[x0][y0]) specifies the index used to derive MmvdDistance[x0][y0] as specified in Table 4. The array indices x0 and y0 can specify the position (x0, y0) of the top-left luma sample of the coding block under consideration (e.g., the current CB) relative to the top-left luma sample of the picture (e.g., the current picture).

[0189] Table 4 - Exemplary relationship between MmvdDistance[x0][y0] and mmvd_distance_idx[x0][y0]

[0190]

Table 4

[0209] The first column of Table 4 indicates the distance index (e.g., mmvd_distance_idx[x0][y0]).

[0191] The second column of Table 4 indicates the magnitude of motion (e.g., MmvdDistance[x0][y0]) when full-pel MMVD is off, e.g., when the full-pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is equal to 0.

[0192] The third column of Table 4 indicates the magnitude of motion (e.g., MmvdDistance[x0][y0]) when full-pel MMVD is on, e.g., when the full-pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is equal to 1.

[0193]

[0210] In one example, the units of the second and third columns in Table 4 are 1 / 4 luma samples. Referring to the first column of Table 4, when the distance index (e.g., mmvd_distance_idx[x0][y0]) is 0, the magnitude of motion (e.g., MmvdDistance[x0][y0]) is 1 if full-pel MMVD is off (e.g., slice_fpel_mmvd_enabled_flag is 0). The magnitude of motion (e.g., MmvdDistance[x0][y0]) is 1 × 1 / 4 luma sample or 1 / 4 luma sample.

[0194] when the distance index (e.g., mmvd_distance_idx[x0][y0]) is 0, the magnitude of motion (e.g., MmvdDistance[x0][y0]) is 4 if full-pel MMVD is on (e.g., slice_fpel_mmvd_enabled_flag is 1). The magnitude of motion (e.g., MmvdDistance[x0][y0]) is 4 × 1 / 4 luma sample or 1 luma sample.

[0195]

[0211] In one example, the second column (1 / 4 luma sample unit) in Table 4 corresponds to the second row (luma sample unit) in Table 1, and the third column (1 / 4 luma sample unit) in Table 4 corresponds to the third row (luma sample unit) in Table 2.

[0196]

[0212] The syntax element mmvd_direction_idx[x0][y0] can represent a direction index. In one example, the direction index (e.g., mmvd_direction_idx[x0][y0]) indicates an index used to derive a motion direction (e.g., MmvdSign[x0][y0]), as shown in Table 5. The array indices x0 and y0 indicate the position (x0, y0) of the top-left luma sample of the coding block under consideration (e.g., the current CB) relative to the top-left luma sample of a picture (e.g., the current picture).

[0197] The first column in Table 5 indicates a direction index (e.g., mmvd_distance_idx[x0][y0]).

[0198] The second column in Table 5 indicates the first sign (e.g., MmvdSign[x0][y0][0]) of the first component of the MVD (e.g., MVD x or MmvdOffset[x0][y0][0]).

[0199] The third column in Table 5 indicates the second sign (e.g., MmvdSign[x0][y0][1]) of the second component of the MVD (e.g., MVD y or MmvdOffset[x0][y0][1]).

[0200] Exemplary relationship between Table 5 - MmvdSign[x0][y0] and mmvd_direction_idx[x0][y0]

[0201]

Table 5

[0213] The first component of the MVD (e.g., MmvdOffset[x0][y0][0]) and the second component (e.g., MmvdOffset[x0][y0][1]) or the offset MmvdOffset[x0][y0] can be derived as follows:

[0202]

Number

[0214] In one example, the distance index (e.g., mmvd_distance_idx[x0][y0]) is 3 and the direction index (e.g., mmvd_distance_idx[x0][y0]) is 2. Based on Table 5 and the direction index (e.g., mmvd_direction_idx[x0][y0]) being 2, the first sign of the first component of the MVD (e.g., MVD x or MmvdOffset[x0][y0][0]) is "0", and the second sign of the second component of the MVD (e.g., MVD y or MmvdOffset[x0][y0][1]) is "+1". In this example, the MVD is along the positive vertical direction (+y) and has no horizontal component.

[0203]

[0215] When the full - pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is equal to 0 and the full - pel MMVD is off, based on Table 4 and the distance index (e.g., mmvd_distance_idx[x0][y0]) being 3, the magnitude of the motion indicated by MmvdDistance[x0][y0] is 8. Based on Eqs. 14 - 15, the first component of the MVD (e.g., MmvdOffset[x0][y0][0]) is (8 << 2)×0 = 0 and the second component of the MVD (e.g., MmvdOffset[x0][y0][1]) is (8 << 2) × (+1) = 2 (luma samples) results in

[0204]

[0216] When the full - pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is equal to 1 and the full - pel MMVD is on, based on the table 4 and the distance index (e.g., mmvd_distance_idx[x0][y0]) being 3, the magnitude of the motion indicated by MmvdDistance[x0][y0] is 32. Based on Equation Eq.35 and Equation Eq.36, the first component of the MVD (for example, MmvdOffset[x0][y0][0]) is (32 << 2) × 0 = 0 and the second component of the MVD (for example, MmvdOffset[x0][y0][1]) is (32 << 2) × (+1) = 8 (luma samples) results in

[0205]

[0217] According to one aspect of the present disclosure, it is possible to use affine merge with motion vector difference (affine MMVD) in video coding. Affine MMVD selects available affine merge candidates as base predictors from a sub - block - based merge list. Affine MMVD applies a motion vector offset to the motion vector value of each control point from the base predictor. In one example, if no affine merge candidate is available, affine MMVD will not be used. In some examples, the distance index and the offset direction index can be signaled thereafter.

[0206]

[0218] In some examples, a distance index is signaled to indicate which distance offset to use from an offset table as shown in table 6: Table 6: Example of offset table

[0207]

Table 6

[0219] In some examples, the direction index can represent four directions as shown in Table 7, where only the x or y direction may have an MV difference, not both directions.

[0208] Table 7: Example of a direction table

[0209]

Table 7

[0220] In some examples, the inter prediction is single - prediction, and to generate a result including the MV value of each control point, the signaled distance offset is applied in the offset direction of each control point predictor.

[0210]

[0221] In some examples, the inter prediction is dual - prediction, and the signaled distance offset can be applied in the signaled offset direction with respect to the L0 motion vector of the control point predictor, and the offset applied to the L1 MV can be applied to a mirrored or scaled base as in the following specific example.

[0222] In a specific example, the inter prediction is dual - prediction, and the signaled distance offset is applied in the signaled distance offset direction with respect to the L0 motion vector of the control point predictor. In the case of L1 CPMV, the offset is applied to a mirrored base, which means that the same amount of distance offset is applied in the opposite direction.

[0211]

[0223] In another specific example, a POC distance-based offset mirroring method is used for dual-prediction. When the base candidate is dual-predicted, the offset applied to L0 is as signaled, and the offset for L1 depends on the temporal positions of the reference pictures in lists L0 and L1. For example, if both reference pictures are on the same side of the temporal current picture, the same distance offset and the same offset direction are applied to the CPMVs of both L0 and L1. In another example, if the two reference pictures are on different sides of the current picture, the CPMVs of L1 can have a distance offset applied in the opposite offset direction.

[0212]

[0224] In another specific example, a POC distance-based offset scaling method is used for dual-prediction. When the base candidate is dual-predicted, the offset applied to L0 is as signaled, and the offset for L1 can be scaled based on the temporal distances of the reference pictures in lists 0 and 1.

[0213]

[0225] In some examples, the range of distance offset values is extended. For example, it is possible to provide three sets of distance offset values, and one set of distance offset values can be adaptively selected based on the picture resolution. In one example, an offset table is selected based on the picture resolution. Table 8 shows an example of an extended distance offset table that includes three sets of distance offset values associated with different picture resolutions. The set of distance offset values can be selected based on the picture resolution.

[0214] Table 8: Example of an extended distance-offset table

[0215]

Table 8

[0226] In some examples, diversity reordering is applied based on the template matching cost. Diversity reordering can increase diversity and improve index coding efficiency. In some examples, candidates that are overly redundant in the sense of rate-distortion (RD) are detected to introduce diversity into the merge candidate list. In one example, if the template matching cost difference between a candidate and its preceding candidate is below the lambda value as expressed by |D1 - D2| < λ, where D1 and D2 are the template matching costs obtained during the first ARMC ordering and λ is the Lagrangian parameter used in the RD criterion on the encoder side, the candidate is considered redundant.

[0216]

[0227] In some examples, it is possible to execute an algorithm for diversity reordering. The algorithm can determine the minimum template matching cost difference between each candidate and its preceding candidate among all candidates in the merge candidate list. If the minimum template matching cost difference is equal to or above λ, the merge candidate list is considered to be diverse enough and the sorting or reordering stops. If the minimum template matching cost difference is below λ, the candidate with that minimum template matching cost difference is considered redundant and is moved to a more distant position within the merge candidate list. The more distant position is the first position where the candidate is sufficiently different compared to its preceding candidate.

[0217]

[0228] The algorithm can stop after a finite number of iterations (if the minimum cost difference is not inferior to λ).

[0218]

[0229] In some examples, the algorithm is applied to various merge modes such as the regular merge mode in ECM-5.0, the merge mode using template matching, the merge mode using bilateral matching, and the affine merge mode. In some examples, a similar algorithm is applied to merge MMVD and the sign MVD prediction method that uses ARMC for rearrangement.

[0219]

[0230] In some examples, the value of λ is the λ of the rate distortion criterion used to select the best merge candidate on the encoder side for a low-latency configuration, and is set equal to the value λ corresponding to the QP for the random access setting. In some examples, a set of λ values corresponding to each signaled QP offset is provided in the SPS, or for QP offsets not present in the SPS, is provided in the slice header.

[0220]

[0231] In some examples, MVD sign prediction technology is used. In one example, possible combinations of MVD signs (e.g., various combinations of signs in the x and y directions) are sorted according to the template matching cost of the possible combinations of MVD signs, an index corresponding to the true combination of MVD signs is derived, and context coding is performed. According to the MVD sign prediction technology, the true combination of MVD signs has a high probability at the beginning of the sorted order. Therefore, an appropriate signaling technology can be used to signal the index at a low signaling cost.

[0221]

[0232] In one example, on the decoder side, the true MVD sign can be derived. For example, it is possible to analyze the magnitude of the MVD components, and the context-coded MVD sign prediction index is analyzed from the bitstream that carries the video. Further, the MV candidates can be formed by creating combinations from among the possible combinations of MVD signs, and the magnitude of the MVD components and the MV candidates can be added to the MV predictor list. It is possible to calculate the template matching cost for the MV candidates in the MV predictor list. The MV candidates in the MV predictor list can be sorted according to the template matching cost. Then, the context-coded MVD sign prediction index is used to select the combination of true MVD signs from the MV predictor list. The MVD sign prediction technique can be applied to various modes including MVD, such as the inter-AMVP, affine AMVP, MMVD, and affine MMVD modes.

[0222]

[0233] In some examples, it is possible to use a technique called history-parameter-based affine model inheritance. Specifically, in some examples, a first history-parameter table (HPT) and a second HPT are set.

[0223]

[0234] FIG. 23 shows a diagram (2300) showing the first HPT and the second HPT in some examples.

[0224]

[0235] As shown in FIG. 23, the entry of the first HPT stores a set of affine parameters of an affine model such as a, b, c, d, and each affine parameter is represented by a 16-bit signed integer. The entries in the first HPT are classified by a reference list (e.g., reference picture list L0 or reference picture list L1) and a reference index. Five reference indexes are supported for each reference list in the first HPT. As an example, expressed by a mathematical formula, the category of the first HPT (denoted as HPTCat) is calculated as shown in Equation Eq. (16):

[0225]

Number

[0226]

[0236] For each category, it is possible to store up to 7 entries. As a result, there are a total of 70 entries in the first HPT. At the beginning of each CTU row, the number of entries in each category is initialized to 0. After decoding the CU affine-coded using the reference list RefList cur and RefIdx cur , the affine parameters are used to update the entry of the category HPTCat (RefList cur , RefIdx cur ) in the same way as updating the HMVP table.

[0227]

[0237] In one example, a history-affine-parameter-based candidate (HAPC) is derived from one of seven neighboring 4×4 blocks shown as A0, A1, A2, B0, B1, B2, or B3 in FIG. 23 and a set of affine parameters stored in the corresponding entry in the first HPT. The MV of the neighboring 4×4 block is provided as the base MV. In the formulation, the MV of the current block at position (x, y) is calculated as in Eq. (17):

[0228]

Number

[0229]

[0238] A second history parameter table (HPT) with base MV information is also added. The second HPT can include nine entries, and the entries can include the base MV, the reference index and four affine parameters for each reference list, and the base position. In one example, an additional merge HAPC can be generated from the base MV information and the corresponding affine model (e.g., affine parameters) stored in the entries of the second HPT.

[0230]

[0239] Further, in some examples, pairwise affinity merge candidates are generated by two affinity merge candidates, one of which is history-derived and the other is not history-derived. In one example, pairwise affinity merge candidates are generated by averaging the CPMVs of existing affinity merge candidates in a candidate list.

[0231]

[0240] In some examples, in response to the introduction of a new HAPC, the size of the sub-block-based merge candidate list is increased from 5 to 15, all of which may be involved in the ARMC process.

[0232]

[0241] In the above description, the HPTs (e.g., the first HPT and the second HPT) are updated online. In addition to the HPT updated by one line, the HPTs stored in the upper / upper-right CTUs of the current CTU can, in some examples, be used by the blocks within the current CTU. After coding / decoding the CTU, the HPT may be stored in a line buffer for use in the next CTU line.

[0233]

[0242] FIG. 24 shows, in some examples, a history parameter table stored in a line buffer. In FIG. 24, a picture (2400) is partitioned into CTUs. FIG. 24 shows CTU row k and CTU row k + 1. The current CTU (2410) is within CTU row k + 1. For coding the current block (2411), the HPT (2401) stored in the CTU above the current CTU (2410) and the HPT (2402) stored in the upper-right CTU of the current CTU (2410) can, in some examples, be used by the blocks within the current CTU. The HPTs (e.g., the first HPT and the second HPT) are updated online and stored in the line buffer of the current CTU (2410) after decoding the last coding block within the current CTU.

[0234]

[0243] According to one aspect of the present disclosure, non-adjacent spatial neighbors can be used in the affine mode.

[0235]

[0244] In the affine mode with non-adjacent spatial neighbors (NA-AFF), it is possible to obtain non-adjacent spatial neighbors.

[0236]

[0245] Figures 25A-25B show the patterns of the obtained non-adjacent spatial neighbors in some examples. Similar to the existing non-adjacent regular merge candidates, the distance between the current CU and the non-adjacent spatial neighbors in NA-AFF is also defined based on the width and height of the current CU.

[0237]

[0246] Using the motion information of the non-adjacent spatial neighbors, additional inherited and constructed affine merge / AMVP candidates are generated. Figure 25A shows generating additional inherited affine merge / AMVP candidates, and Figure 25B shows generating additional constructed affine merge / AMVP candidates.

[0238]

[0247] Specifically, as shown in Figure 25A, for the inherited candidates, the same derivation process of the inherited affine merge / AMVP candidates in VVC is maintained without change except that the CPMVs are inherited from the non-adjacent spatial neighbors. The non-adjacent spatial neighbors are checked based on their distances to the current block, i.e., the distances from the near ones to the far ones. At a specific distance, only the first available neighbors (coded in the affine mode) from each side (e.g., the left side and the upper side) of the current block are included for the derivation of the inherited candidates. As indicated by the dashed arrows in Figure 25A, the checking order of the neighbors on the left side and the upper side is from bottom to top and from right to left, respectively.

[0239]

[0248] For the first type of constructed candidate, as shown in Figure 25B, the positions of one left and upper non-adjacent spatial neighborhood are first determined independently. Subsequently, the position of the top-left neighborhood can then be determined accordingly, which can enclose a rectangular virtual block using the left and upper non-adjacent neighborhoods.

[0240]

[0249] Next, as shown in Figure 26, the motion information of three non-adjacent neighborhoods is used to form CPMVs at the top-left (A), top-right (B), and bottom-left (C) of the virtual block, and then the motion information is projected onto the current CU to generate the corresponding constructed candidate.

[0241]

[0250] In some examples, for the second type of constructed candidate, the derivation process is similar to the construction scheme in history-based affine model inheritance (HAMI). However, instead of using a history-based lookup table, non-translational affine parameters are inherited from non-adjacent spatial neighborhoods. Specifically, the affine construction candidate of the second type is 1) the translational affine parameters of adjacent neighborhood 4x4 blocks; and 2) non-translational affine parameters inherited from non-adjacent spatial neighborhoods as defined in Figure 25A; generated from a combination of these.

[0242]

[0251] In some examples, NA-AFF candidates are inserted into the existing affine merge candidate list and affine AMVP candidate list according to a specific order.

[0243]

[0252] In one example, in the affine merge mode, the order includes the following: 1. SbTMVP candidates (if available); 2. Inherited from adjacent neighborhoods; 3. Inherited from non - adjacent neighborhoods; 4. Constructed from adjacent neighborhoods; 5. Second - type constructed affine candidates from non - adjacent neighborhoods; 6. First - type constructed affine candidates from non - adjacent neighborhoods; 7. Zero MV.

[0244]

[0253] In another example, in the affine AMVP mode, the order includes the following: 1. Inherited from adjacent neighborhoods; 2. Constructed from adjacent neighborhoods; 3. Translational MV from adjacent neighborhoods; 4. Translational MV from temporal neighborhoods; 5. Inherited from non - adjacent neighborhoods; 6. First - type constructed affine candidates from non - adjacent neighborhoods; 7. Zero MV.

[0245]

[0254] Due to including additional candidates generated by NA - AFF, the size of the affine merge candidate list is increased from 5 to 15. The subgroup size of ARMC for the affine merge mode is increased from 3 to 15.

[0246]

[0255] In some video codecs (e.g., ECM - 5.0 software), NA - AFF is implemented without adding constraints on memory usage.

[0247]

[0256] In some examples (e.g., VVC), a Geometric Partitioning Mode (GPM) is supported for inter prediction. The geometric partitioning mode is signaled using a CU-level flag as a type of merge mode with other merge modes such as regular merge mode, MMVD mode, CIIP mode, sub-block merge mode, etc. In some examples, a total of 64 partitions are supported by the geometric partitioning mode for each of m ×2 n w×h = 2, excluding 8x64 and 64x8, where m, n ∈ {3...6}.

[0248]

[0257] In some examples, when the geometric partitioning mode is used, the CU is divided into two parts by a geometrically positioned line (also called a dividing line). The position of the dividing line can be mathematically derived based on the angle and offset parameters of a specific partition. Each part of the two geometric partitions by the dividing line within the CU is inter predicted using its own motion; only uni-prediction is allowed for each partition. Thus, each part has one motion vector and one reference index. The uni-prediction motion constraint is applied to ensure that a CU in GPM mode can be coded as a conventional bi-prediction, e.g., two motion compensated predictions are performed for each CU. In some examples, when the geometric partitioning mode is used for the current CU, a geometric partitioning index indicating the partitioning mode of the geometric partition (e.g., indicating the angle and offset) and two merge indices (one for each partition segment) are further signaled.

[0249]

[0258] The template matching (TM) technology is a decoder-side MV derivation technology for refining the motion information of the current CU by finding the closest match between a template in the current picture (i.e., the top and / or left neighboring blocks of the current CU) and a block in the reference picture (i.e., of the same size as the template).

[0250]

[0259] Figure 27 shows an example of template matching (2700). By determining the closest match between a template (e.g., current template) (2721) of the current CU (2701) in the current picture (2710) and a template (e.g., reference template) (e.g., one of the plurality of possible templates is template (2725)) among the plurality of possible templates in the reference picture (2711), it is possible to derive the motion information of the current CU (e.g., current block) (2701) (e.g., derive the final motion information from the initial motion information such as the initial MV 2702). The template (2721) of the current CU (2701) can have any appropriate shape and any appropriate size.

[0251]

[0260] In an embodiment, the template (2721) of the current CU (2701) includes a top template (2722) and a left template (2723). Each of the top template (2722) and the left template (2723) can have any appropriate shape and any appropriate size.

[0252]

[0261] The top template (2722) can include samples within one or more top neighboring blocks of the current CU (2701). In one example, the top template (2722) includes four sample rows within one or more top neighboring blocks of the current CU (2701).

[0253] The left template (2723) can include samples within one or more left neighboring blocks of the current CU (2701). In one example, the left template (2723) includes four sample columns within one or more left neighboring blocks of the current CU (2701).

[0254]

[0262] Each of the plurality of possible templates within the reference picture (2711) (e.g., template (2725)) corresponds to a template (2721) within the current picture (2710). In an embodiment, the initial MV (2702) points to a reference block (2703) within the reference picture (2711) from the current CU (2701). Each of the plurality of template candidates within the reference picture (2711) (e.g., template (2725)) and the template (2121) within the current picture (2710) can have the same shape and the same size. For example, the template (2725) of the reference block (2703) includes a top template (2726) of the reference picture (2711) and a left template (2727) of the reference picture (2711). The top template (2726) can include samples within one or more top neighboring blocks of the reference block (2703). The left template (2727) can include samples within one or more left neighboring blocks of the reference block (2703).

[0255]

[0263] The TM cost can be determined based on a pair of templates such as a template (e.g., the current template) (2721) and a template (e.g., the reference template) (2725). The TM cost can indicate the matching between the template (2721) and the template (2725). An optimized MV (or final MV) can be determined based on the search around the initial MV (2702) of the current CU (2701) within the search range (2715). The search range (2715) can have any suitable shape and any suitable number of reference samples. In one example, the search range (2715) within the reference picture (2711) includes a range of [-L,L]-pel, where L is a positive integer such as 8 (e.g., 8 samples). For example, based on the search range (2715), a difference (e.g., [0,1]) is determined, and an intermediate MV is determined by the sum of the initial MV (2702) and the difference (e.g., [0,1]). Based on the intermediate MV, an intermediate reference block and a corresponding template within the reference picture (2711) can be determined. The TM cost can be determined based on the template (2721) and the intermediate template of the reference picture (2711). The TM cost can correspond to the difference (e.g., [0,0], [0,1], etc. corresponding to the initial MV (2702)) determined based on the search range (2715). In one example, the difference corresponding to the minimum TM cost is selected, and the optimized MV is the sum of the difference corresponding to the minimum TM cost and the initial MV (2702). As described above, TM can derive the final motion information (e.g., the optimized MV) from the initial motion information (e.g., the initial MV 2102).

[0256]

[0264] In the example of FIG. 27, within a search range such as [-8pel, +8pel], a better MV can be searched around the initial motion vector of the current CU.

[0257]

[0265] In some examples, the search step size in template matching is determined based on the AMVR mode, and the TM can be cascaded with the bilateral matching process in the merge mode.

[0258]

[0266] In some examples, in the AMVP mode, to select the one that reaches the minimum difference between the current block template and the reference block template, the MVP candidate is determined based on the template matching error, and then the TM is executed only for this specific MVP candidate for MV refinement. For example, the TM starts from the full-pel MVD accuracy (or 4-pel in the case of the 4-pel AMVR mode) within the [-8, +8]-pel search range by using iterative diamond search to refine this MVP candidate. The AMVP candidate can be further refined by sequentially using half-pel and quarter-pel ones according to the AMVR mode as specified in Table 9 after a cross-search using the full-pel MVD accuracy (or 4-pel in the case of the 4-pel AMVR mode). This search process ensures that the MVP candidate maintains the same MV accuracy as specified by the AMVR mode after the TM process. In the search process, when the difference between the previous minimum cost and the current minimum cost in an iteration is smaller than the threshold equal to the area of the block, the search process ends.

[0259] Table 9. Search Patterns for Merge Mode Using AMVR and AMVR

[0260]

Table 9

[0267] In one example, in the merge mode, a similar search process is applied to the merge candidates indicated by the merge index. As shown in Table 9, TM can execute all the way to 1 / 8 pel MVD accuracy or skip them beyond half pel MVD accuracy, depending on whether an alternative interpolation filter (used when AMVR is in half pel mode) is used according to the merged motion information. Further, when the TM mode is enabled, template matching can operate as an independent process or an additional MV refinement process between block-based and sub-block-based bilateral matching (BM) methods, depending on whether BM can be enabled according to that enable condition check.

[0261]

[0268] According to one aspect of the disclosure, it is possible to reduce signaling overhead by using candidate reordering techniques based on template matching. For example, a technique referred to as adaptive reordering of merge candidates with template matching (ARMC-TM) can be used.

[0262]

[0269] In some examples, using ARMC-TM, the merge candidates are adaptively reordered using template matching (TM). ARMC-TM can be applied to the regular merge mode, the template matching (TM) merge mode, and the affine merge mode (excluding SbTMVP candidates). In the case of the TM merge mode, the merge candidates are reordered before the refinement process.

[0263]

[0270] In one example, after using ARMC-TM to construct the merge candidate list, the merge candidates are divided into several subgroups. In one example, the subgroup size is set to 5 for both the regular merge mode and the TM merge mode. In another example, the subgroup size is set to 3 for the affine merge mode. The merge candidates within each subgroup are sorted in ascending order according to the cost value based on template matching. For simplicity, in some examples, the merge candidates of the latest subgroup, rather than the first subgroup, are not sorted.

[0264]

[0271] The template matching cost of the merge candidate is measured by the sum of absolute differences (SAD) between the samples of the template of the current block and the reference sample corresponding to the template (which is also called the reference template in one example). The template includes a set of reconstructed samples in the vicinity of the current block. The reference sample of the template is arranged according to the motion information of the merge candidate.

[0265]

[0272] When the merge candidate utilizes bidirectional prediction, the reference sample of the template of the merge candidate is also generated by bidirectional prediction.

[0266]

[0273] Figure 28 illustrates a reference sample of the template of the current block for a dual-prediction merge candidate. In Figure 28, the current picture (2810) includes the current block for coding. When the merge candidate is a dual-prediction merge candidate, the MV of the merge candidate may point to a first reference block in the first reference picture (2820) and a second reference block in the second reference picture (2830). The template of the current block is indicated by (T), and the template includes a set of reconstructed samples adjacent to the current block. The first set of reference samples of the template is in the first reference picture (2820) near the first reference block, and the second set of reference samples of the template is in the second reference picture (2830) near the second reference block. In one example, the template matching cost of the dual-prediction merge candidate is calculated by adding a first sum of absolute differences (SAD) between the samples of the template of the current block and the first set of reference samples of the template, and a second sum of absolute differences (SAD) between the samples of the template of the current block and the second set of reference samples of the template.

[0267]

[0274] In some examples, the merge candidate may be a sub-block-based merge candidate. In one example, for a sub-block-based merge candidate having a sub-block size equal to Wsub×Hsub, the top template may include several sub-templates having a size of Wsub×1, and the left template may include several sub-templates having a size of 1×Hsub. Wsub is the width of the sub-block, and Hsub is the height of the sub-block.

[0268]

[0275] It is possible to assume that an exemplary derivation of a template and a reference sample of the template for a current block having sub-block-based merge candidates is as shown in FIG. 29. As shown in FIG. 29, the current block (2902) can be included in the current picture (2904). The current block (2902) may include sub-blocks A-G in the first row and the first column. The current block (2902) can include a template (2906) adjacent to the top side and the left side of the current block (2902). The collocated block (2908) for the current block (2902) is within the reference picture (2910). The collocated block (2908) may include sub-blocks A-G in the first row and the first column, which correspond to the sub-blocks A-G within the current block (2902). The sub-block motion information of sub-blocks A-G in the first row and the first column of the current block (2902) (for example, corresponding to the affine motion vectors) can be used to derive the reference sample (or sub-reference template) of the sub-template of the collocated block (2908).

[0269] For example, the motion information of sub-blocks A, E, F, and G of the current block (2902) can be applied to derive the reference sample of the sub-template arranged adjacent to the left side of sub-blocks A, E, F, and G of the collocated block (2908). The sub-template adjacent to the left side of sub-blocks A, E, F, G of the collocated block (2908) can form the left reference template of the collocated block (2908).

[0270] The motion information of sub-blocks A, B, C, and D of the current block (2902) can be applied to derive reference samples of a sub-template arranged adjacent to the upper sides of sub-blocks A, B, C, and D of the equivalent position block (2908). The sub-template adjacent to the top sides of sub-blocks A, B, C, and D of the equivalent position block (2908) can further form the upper reference template of the equivalent position block (2908).

[0271]

[0276] In some examples, it is possible to use ARMC based on the MV candidate type. For example, a merge candidate of a certain single candidate type, such as TMVP or non-adjacent MVP (NA-MVP), is sorted based on the ARMC TM cost value. Then, the sorted candidates are added to the merge candidate list. For example, the TMVP candidate type ARMC can add more TMVP candidates with more temporal positions and different inter-prediction directions and perform sorting and selection. Further, the NA-MVP candidate type ARMC expands non-adjacent MVPs (non-adjacent MVPs) that are more spatially non-adjacent. The target reference picture of the TMVP candidate can be selected from any one of the reference pictures in the list according to the scaling factor. For example, the selected reference picture is the one whose scaling factor is closest to 1.

[0272]

[0277] According to one aspect of the disclosure, candidate sorting based on template matching can be performed for MMVD and affine MMVD.

[0273]

[0278] In some examples, the MMVD offset is extended to more positions for the MMVD mode and the affine MMVD mode.

[0274]

[0279] Figure 30 is a diagram showing directions in which a refinement direction can be added with respect to the MMVD. In Figure 30, additional refinement positions along the diagonal (diagonal direction angle) of k×π / 8 are added, where k is an integer. Position (3001) corresponds to the base candidate and can be the starting point, and positions (3011)-(3014) are in the directions of 0, π / 2, π, and 3π / 2 respectively. More directions may be added. For example, positions (3021)-(3024) are in the directions of π / 4, 3π / 4, 5π / 4, 7π / 4 respectively; positions (2531)-(2538) are in the directions of π / 8, 3π / 8, 5π / 8, 7π / 8, 9π / 8, 11π / 8, 13π / 8, 15π / 8 respectively. Therefore, the number of directions increases from 4 to 16. Furthermore, in one example, each direction may have 6 MMVD refinement positions. The total number of possible MMVD refinement positions is 16×6.

[0275]

[0280] According to one aspect of the present disclosure, the SAD cost between the current template (e.g., the row above and the column to the left of the current block) and the reference template can be calculated for each refinement position. Based on the SAD cost of the refinement positions, all possible MMVD refinement positions (16×6) for each base candidate are sorted. Then, the upper part of the refinement positions, for example, the upper 1 / 8 of the refinement positions (e.g., 12), for example, those with the minimum template SAD cost, are ultimately retained as the positions available for MMVD index coding. The MMVD index is binarized by a rice code having a parameter equal to 2.

[0276]

[0281] In some examples, it is possible to increase the refinement positions for the affine MMVD and apply the candidate reordering based on template matching to the reordering of the affine MMVD. For example, the affine MMVD refinement positions are in directions along the diagonal angular directions of k×π / 4, such as the 8 directions of 0, π / 4, π / 2, 3π / 4, π, 5π / 4, 3π / 2, and 7π / 4 respectively. Each direction may have 6 affine MMVD refinement positions. The total number of possible affine MMVD refinement positions is 8×6. In one example, the SAD cost between the current template (e.g., the top 1 row and left 1 column for the current block) and the reference template can be calculated for each refinement position. Based on the SAD cost of the refinement positions, all possible affine MMVD refinement positions (8×6) for each base candidate are reordered. Then, the upper part of the refinement positions, e.g., the upper 1 / 2 of the refinement positions (e.g., 24), e.g., those with the minimum template SAD cost, are retained as the positions available for affine MMVD index coding.

[0277]

[0282] According to one aspect of the present disclosure, in some related examples (e.g., current VVC and ECM), the inter-prediction direction information (e.g., single-prediction or bi-prediction, or single-prediction L0, or single-prediction L1) in the merge mode is determined based on the motion information from the merge candidates.

[0278]

[0283] Some aspects of the present disclosure provide techniques for coding the inter-prediction direction and the BCW index in the merge mode to improve coding efficiency. For example, the inter-prediction direction for the merge mode can be determined (e.g., signaled), and the motion vector information is signaled in addition to the inter-prediction direction.

[0279]

[0284] According to one aspect of the present disclosure, inter-prediction direction signaling such as dual-prediction or single-prediction is added at the coding block level for merge mode, for example, for regular merge mode, merge MMVD mode, affine merge mode, affine MMVD mode, and merge mode using TM refinement.

[0280]

[0285] In an embodiment, the syntax of the inter-prediction direction is signaled after a general merge flag. For example, after a flag indicating the merge mode is signaled, the syntax indicating the inter-prediction direction is signaled.

[0281]

[0286] In another embodiment, after a merge type flag is signaled, the syntax of the inter-prediction direction is signaled. For example, after a flag indicating a merge type such as sub-block based merge, regular merge, etc., the syntax indicating the inter-prediction direction is signaled. In some examples, some merge types are associated with dual-prediction, and thus the inter-prediction direction can be inferred. For example, the GPM mode is a dual-prediction based merge type. Thus, when the merge type is determined to be GPM, the syntax of the inter-prediction direction can be inferred to be dual-prediction.

[0282]

[0287] In an embodiment, the inter-prediction direction information is signaled by a single syntax. In one example, the single syntax has two possible values, each indicating single-prediction and dual-prediction respectively. In another example, the single syntax has three possible values, such as a first possible value indicating single-prediction of reference list L0, a second possible value indicating single-prediction of reference list L1, and a third possible value indicating dual-prediction.

[0283]

[0288] In another embodiment, the inter prediction direction information is signaled by two flags, one for each reference list, indicating which reference list or both reference lists are used. In one example, The first flag inter_pred_idc[0] is used to indicate whether the reference picture list L0 is involved in inter prediction, and the second flag inter_pred_idc[1] is used to indicate whether the reference picture list L1 is involved in inter prediction.

[0284]

[0289] In an embodiment, the signaling of the inter prediction direction is used only when bi-prediction is allowed at a higher level such as slice level, picture level, etc. In one example, bi-prediction is allowed for the current slice when the slice type of the current slice is a B slice. For blocks within the current slice, the signaling of the inter prediction direction is performed at the coding block level.

[0285]

[0290] In some embodiments, when a merge candidate is signaled with bi-prediction which is the inter prediction direction, the BCW index is explicitly signaled. In one example, the BCW index is always explicitly signaled for bi-prediction merge candidates. In another example, for bi-prediction merge candidates, a flag (e.g., signal_bcw_idx) is signaled first to indicate whether the BCW is to be signaled. If the flag is signaled to be true, a BCW index selected from a set of BCW weighting candidates excluding those inherited from the merge candidate is then signaled. In one example, if the flag is signaled to be false, the BCW index is inherited from the merge candidate.

[0286]

[0291] In some examples, the BCW index value is mapped to a reordered list. In one example, the default (equal weighting, e.g., the BCW weighting value is 4 and it is in the third position in the original BCW value list, e.g., {-2, 3, 4, 5, 10}) has the lowest index value, e.g., 0; the BCW index inherited from the merge candidate has the second lowest value, e.g., 1; and then the remaining BCW index values in the reordered list follow. Then, it is possible to signal the index for the reordered list. In some examples, the default and inherited BCW indexes are more likely to be used, and thus coding the BCW using the reordered list has higher coding efficiency than using the original BCW index.

[0287]

[0292] In another example, to map the BCW index value to a reordered list, the inherited BCW index has the lowest value, e.g., 0; the default (equal weighting) has the second lowest index value, e.g., 1; and the remaining BCW index values can be placed at later positions in the reordered list.

[0288]

[0293] In some examples, only the inherited BCW index of the merge candidate and the default BCW index for equal weighting can be used in relation to the merge candidate. Thus, one flag (e.g., 1 bit) can be used to signal which BCW index to use. In some examples, when the inherited BCW index indicates equal weighting, a predefined unequal weighting (e.g., a second default BCW index) can be an alternative weighting option. Then, in one example, a flag is signaled to indicate whether the inherited equal weighting is applied or the predefined unequal weighting is used.

[0289]

[0294] According to one aspect of the present disclosure, by extending existing merge candidates in more possible inter-prediction directions, additional merge candidates may be formed, and the additional merge candidates can be added to the extended list of candidates together with the existing merge candidates. In one example, the extended list of candidates can be directly used in signaling. In another example, the extended list of candidates can be sorted using TM cost-based candidate sorting before signaling.

[0290]

[0295] In an embodiment, for bi-prediction merge candidates, an extension process is performed to generate more merge candidates in bi-prediction and / or uni-prediction. After the extension process, both the original merge candidates and the newly extended merge candidates are placed in the extended list of candidates.

[0291]

[0296] In one example, the original bi-prediction merge candidates are extended to at most three merge candidates, including the original merge candidates and two uni-prediction merge candidates, using the motion information from each reference picture list of the original merge candidates.

[0292]

[0297] In another example, the original bi-prediction merge candidates are: The original bi-prediction merge candidates; Bi-prediction candidates using the MV when the two reference picture lists are exchanged; Two uni-prediction candidates generated by the motion information of each reference picture list of the original bi-prediction candidates; Two uni-prediction candidates generated by the motion information of the MV swapped bi-prediction candidate; And are extended to at most six candidates.

[0293]

[0298] In one example, when a bi-prediction candidate is coded by BCW, the bi-prediction candidate can be converted to a uni-prediction candidate using a reference list and a motion vector associated with a weight value greater than a pre-defined threshold.

[0294]

[0299] In some embodiments, for a single-prediction candidate, the single-prediction candidate can be extended to two single-prediction candidates that have the same MV but are on different reference picture lists.

[0295]

[0300] In some embodiments, the bi-prediction merge candidate can be extended to multiple candidates based on the BCW index value. In one example, the bi-prediction candidate is extended to multiple candidates, each having the same MV value but different applicable BCW index values.

[0296]

[0301] In another example, if the bi-prediction candidate has a BCW index that does not correspond to the default equal weighting, the bi-prediction candidate is extended to two candidates including the original bi-prediction candidate with the original BCW index and an additional bi-prediction candidate with a BCW index for equal weighting.

[0297]

[0302] In some embodiments, when template matching is allowed, the extended candidate list can be sorted based on the template matching cost to form a sorted list. Then, the index of the sorted list is signaled in the bitstream to specify which candidate to use. In some embodiments, when template matching is not allowed, the index of the extended candidate list is directly signaled to determine which candidate to use.

[0298]

[0303] According to one aspect of the present disclosure, when a merge candidate is a bi-directional predictor, the inter-prediction direction of the merge candidate can be determined using template matching during merge candidate list construction. For example, when a merge candidate is a bi-directional predictor, a template matching process is applied to the merge candidate to calculate the template-matching (TM) cost for each of only reference picture list L0, only reference picture list L1, and both reference picture lists L0 and L1. The inter-prediction direction having the minimum cost is determined for the merge candidate. Note that the total number of merge candidates in the candidate list does not change during merge candidate list construction.

[0299]

[0304] In some examples, the determination of the inter-prediction direction for a merge candidate by using the template matching process is applicable when the merge candidate is determined from spatially or temporally neighboring blocks.

[0300]

[0305] In some examples, the determination of the inter-prediction direction for a merge candidate by using the template matching process is applicable when the motion vector refinement of the merge candidate is completed.

[0301]

[0306] In some examples, the determination of the inter-prediction direction for a merge candidate by using the template matching process is applied to the merge candidate when the template matching-based rearrangement of the merge candidate list construction is completed.

[0302]

[0307] Figure 31 shows a flowchart illustrating an overview of a process (3100) according to an embodiment of the present disclosure. The process (3100) can be used in a video encoder. In various embodiments, the process (3100) is executed by a processing circuit such as a processing circuit in terminal devices (310), (320), (330), (340), a processing circuit that executes the functions of video encoder (403), a processing circuit that executes the functions of video encoder (603), a processing circuit that executes the functions of video encoder (703), and the like. In some embodiments, the process (3100) is executed by software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit executes the process (3100). The process starts at (S3101) and proceeds to (S3110).

[0303]

[0308] At (S3110), a merge candidate list including a plurality of merge candidates is constructed for inter prediction of a current block within a current picture.

[0304]

[0309] At (S3120), a specific merge candidate from the merge candidate list and an inter prediction direction for inter prediction of the current block are determined. The specific merge candidate provides at least a motion vector for inter prediction, and the inter prediction direction is selected as one of a first uni-prediction from a first reference picture list, a second uni-prediction from a second reference picture list, and a bi-prediction from the first reference picture list and the second reference picture list. 【030——5】

[0310] At (S3130), a first index indicating the specific merge candidate is encoded into a bitstream (also referred to as a video bitstream), and the bitstream conveys a video including the current picture.

[0306]

[0311] In (S3140), one or more signals indicating an inter prediction direction in the bitstream are encoded in the bitstream. Note that in some examples, (S3140) is executed before (S3130).

[0307]

[0312] In some examples, the one or more signals are after a first syntax indicating whether the current block is in merge mode. In another example, the one or more signals are after a second syntax indicating the merge type of the current block.

[0308]

[0313] In some examples, the one or more signals include a syntax using a first value indicating single-prediction and a second value indicating dual-prediction.

[0309]

[0314] In some examples, the one or more signals include a syntax using a first value indicating a first single-prediction, a second value indicating a second single-prediction, and a third value indicating dual-prediction.

[0310]

[0315] In some examples, the one or more signals include a first flag indicating whether the first reference picture list is in the inter prediction direction; and a second flag indicating whether the second reference picture list is in the inter prediction direction;

[0311]

[0316] In some examples, the inter prediction direction is dual-prediction, a specific weighting candidate in the weighting candidate list is determined, and the weighting candidate list includes a plurality of weighting candidates (e.g., a set of BCW weights, a set of BCW indices) each providing a weighting value for combining predictions from the first reference picture list and the second reference picture list. A second index is encoded to specify a specific weighting candidate from the weighting candidate list.

[0312]

[0317] In one example, the weighted candidates include at least the default equal-weight candidates (e.g., a BCW weight of 4, or a BCW index of 2), and the inherited weighted candidates associated with a particular merge candidate.

[0313] In one example, the weighted candidate list includes, in order, the default equal-weight candidates, the inherited weighted candidates from the merge candidates, and other weighted candidates. Thus, the default equal-weight candidates correspond to a second index of 0, and the inherited weighted candidates correspond to a second index of 1.

[0314] In another example, the weighted candidate list includes, in order, the inherited weighted candidates from the merge candidates, the default equal-weight candidates, and other weighted candidates. Thus, the default equal-weight candidates correspond to a second index of 1, and the inherited weighted candidates correspond to a second index of 0.

[0315]

[0318] In another example, the weighted candidate list only includes the default equal-weight candidates and the inherited weighted candidates from the merge candidates. Then, a flag is encoded, and the flag specifies which one to use. If the inherited weighted candidates are equal weights, a secondary default BCW index may be used as the other weighted candidate.

[0316]

[0319] The process then proceeds to (S3199) and ends.

[0317]

[0320] The process (3100) can be appropriately adapted. The steps in the process (3100) can be modified and / or omitted. Additional steps can be added. Any appropriate execution order can be used.

[0318]

[0321] Figure 32 shows a flowchart illustrating an overview of a process (3200) according to an embodiment of the present disclosure. The process (3200) can be used in a video decoder. In various embodiments, the process (3200) is executed by a processing circuit such as a processing circuit in a terminal device (310), (320), (330), (340), a processing circuit that executes the functions of a video decoder (410), a processing circuit that executes the functions of a video decoder (510), and the like. In some embodiments, the process (3200) is executed by software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit executes the process (3200). The process begins at (S3201) and proceeds to (S3210).

[0319]

[0322] At (S3210), information indicating an inter prediction for predicting a current block in a current picture of a video is decoded from a bitstream (also referred to as a video bitstream) that carries the video. In some examples, the processing circuit receives the video bitstream and obtains prediction information for a current block in the current picture from the video bitstream. The prediction information indicates whether the current block should be predicted in an inter prediction mode.

[0320]

[0323] At (S3220), a merge candidate list is constructed and a merge candidate is selected from the merge candidate list. For example, in response to the current block being predicted in an inter prediction mode, the processing circuit determines a merge candidate from the merge candidate list.

[0321]

[0324] In (S3230), the inter prediction direction is determined separately from the merge candidate, and the inter prediction direction is one of a first mono - prediction from the first reference picture list, a second mono - prediction from the second reference picture list, and a bi - prediction from the first reference picture list and the second reference picture list. For example, the processing circuit determines the inter prediction direction based on a syntax element signaled in the video bitstream. The inter prediction direction signaled separately from the merge candidate is one of a first mono - prediction from the first reference picture list, a second mono - prediction from the second reference picture list, and a bi - prediction from the first reference picture list and the second reference picture list.

[0322]

[0325] In (S3240), the motion vector for the prediction of the current block is determined based on the merge candidate.

[0323]

[0326] In (S3250), the current block is reconstructed based on the inter prediction direction and the motion vector.

[0324]

[0327] In some embodiments, to determine the inter prediction direction, one or more signals are decoded from the bitstream, and the one or more signals indicate the inter prediction direction. In one example, the one or more signals are after a first syntax indicating whether the current block is in the merge mode. In another example, the one or more signals are after a second syntax indicating the merge type of the current block.

[0325]

[0328] In some examples, in response to the merge type of the current block being based on bi - prediction, the inter prediction direction is inferred to be bi - prediction.

[0326]

[0329] In one example, the one or more signals include a syntax using a first value indicating mono - prediction and a second value indicating bi - prediction.

[0327]

[0330] In another example, one or more signals include a syntax that uses a first value indicating a first side-prediction, a second value indicating a second side-prediction, and a third value indicating a bi-prediction.

[0328]

[0331] In another example, one or more signals include a first flag indicating whether the first reference picture list is in the inter prediction direction; and a second flag indicating whether the second reference picture list is in the inter prediction direction.

[0329]

[0332] In another example, one or more signals are decoded from a video bitstream in response to the high-level syntax allowing bi-prediction.

[0330]

[0333] In some examples, the inter prediction direction is bi-prediction, and an index indicating a particular weighting candidate in a weighting candidate list is decoded. The weighting candidate list includes a plurality of weighting candidates (e.g., a set of BCW weights, a set of BCW indexes) each providing a weighting value for combining predictions from a first reference picture list and a second reference picture list.

[0331]

[0334] In one example, a flag is decoded, and the flag indicates whether BCW is signaled. If the flag is true, the index can be decoded. If the flag is false, it is possible to use an inherited BCW weight or BCW index from a merge candidate.

[0332]

[0335] In some examples, the weighting candidates include at least a default equal-weighting candidate (e.g., a BCW weight of 4, or a BCW index of 2) and an inherited weighting candidate associated with a particular merge candidate.

[0333] In one example, the weighted candidate list includes, in order, the default equal-weight candidates, the inherited weighted candidates from the merge candidates, and other weighted candidates. Thus, the default equal-weight candidates correspond to a second index of 0, and the inherited weighted candidates correspond to a second index of 1.

[0334] In another example, the weighted candidate list includes, in order, the inherited weighted candidates from the merge candidates, the default equal-weight candidates, and other weighted candidates. Thus, the default equal-weight candidates correspond to a second index of 1, and the inherited weighted candidates correspond to a second index of 0.

[0335]

[0336] In another example, the weighted candidate list only includes the default equal-weight candidates and the inherited weighted candidates from the merge candidates. Then, a flag is encoded, and the flag specifies which one to use. If the inherited weighted candidates are equal weights, a secondary default BCW index may be used as the other weighted candidate.

[0336]

[0337] The process then proceeds to (S3299) and ends.

[0337]

[0338] The process (3200) can be suitably adapted. The steps in the process (3200) can be modified and / or omitted. Additional steps can be added. Any suitable execution order can be used.

[0338]

[0339] FIG. 33 shows a flowchart illustrating an overview of a process (3300) according to an embodiment of the present disclosure. The process (3300) can be used in a video encoder. In various embodiments, the process (3300) is executed by a processing circuit such as a processing circuit in terminal devices (310), (320), (330), (340), a processing circuit that executes the functions of the video encoder (403), a processing circuit that executes the functions of the video encoder (603), a processing circuit that executes the functions of the video encoder (703), and the like. In some embodiments, the process (3300) is executed by software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit executes the process (3300). The process starts at (S3301) and proceeds to (S3310).

[0339]

[0340] At (S3310), an extended merge candidate list is constructed based on at least a first merge candidate having motion parameters obtained from blocks in the vicinity of the current block for inter prediction of the current block in the current picture. The neighboring blocks may be spatial neighboring blocks or temporal neighboring blocks of the current block. The extended merge candidate list includes at least the first merge candidate and non-redundant merge candidates extended from the first merge candidate. The non-redundant merge candidates have at least the motion vectors of the first merge candidate.

[0340]

[0341] At (S3320), a specific merge candidate is determined from the extended merge candidate list for inter prediction.

[0341]

[0342] At (S3330), an index indicating a specific merge candidate in the extended merge candidate list is encoded in the bitstream, and the bitstream carries a video including the current picture.

[0342]

[0343] In some examples, the first merge candidate is a bi-prediction candidate that uses a first motion vector associated with a first reference picture list and a second motion vector associated with a second reference picture list. The extended merge candidate list is constructed to include the first merge candidate and at least one of a second merge candidate (a uni-prediction candidate having a first motion vector associated with the first reference picture list) and a third merge candidate (a uni-prediction candidate having a second motion vector associated with the second reference picture list).

[0343]

[0344] In some examples, the first merge candidate is a bi-prediction candidate that uses a first motion vector associated with a first reference picture list and a second motion vector associated with a second reference picture list. The extended merge candidate list includes the first merge candidate and at least one of a second merge candidate, a third merge candidate, a fourth merge candidate, a fifth merge candidate, and a sixth merge candidate, the second merge candidate is a bi-prediction candidate having a first motion vector associated with the second reference picture list and a second motion vector associated with the first reference picture list; the third merge candidate is a uni-prediction candidate having a first motion vector associated with the first reference picture list; the fourth merge candidate is a uni-prediction candidate having a second motion vector associated with the second reference picture list; the fifth merge candidate is a uni-prediction candidate having a second motion vector associated with the first reference picture list; and the sixth merge candidate is a uni-prediction candidate having a first motion vector associated with the second reference picture list.

[0344]

[0345] In one example, the first merge candidate is a bi-prediction candidate coded using CU-level weighted bi-prediction (BCW), and the first merge candidate includes a first weight value, a first motion vector associated with a first reference picture list, and a second motion vector associated with a second reference picture list and a second weight value.

[0345] In one example, in response to the first weight value being greater than the second weight value, the extended merge candidate list includes a first merge candidate and a second merge candidate, and the second merge candidate is a uni-prediction candidate having a first motion vector associated with the first reference picture list.

[0346] In another example, in response to the first weight value being less than the second weight value, the extended merge candidate list includes the first merge candidate and a third merge candidate, and the third merge candidate is a uni-prediction candidate having a second motion vector associated with the second reference picture list.

[0347]

[0346] In one example, the first merge candidate is a uni-prediction candidate having a first motion vector associated with the first reference picture list. The extended merge candidate list includes the first merge candidate and a second merge candidate, which is a uni-prediction candidate having the first motion vector associated with the second reference picture list.

[0348]

[0347] In one example, the first merge candidate is a bi-prediction candidate coded using a first (CU-level weighted bi-prediction (BCW)) index value, and the first merge candidate includes a first motion vector associated with the first reference picture list and a second motion vector associated with the second reference picture list. The extended merge candidate list includes the first merge candidate and one or more second merge candidates, and each of the one or more second merge candidates includes motion information in the first merge candidate and is coded using a BCW index value different from the first BCW index value.

[0349]

[0348] In some examples, the first merge candidate is a bi-prediction candidate that is coded using a first (CU-level weighted bi-prediction (BCW)) index value corresponding to unequal weighting, and the first merge candidate includes motion information having a first motion vector associated with a first reference picture list and a second motion vector associated with a second reference picture list. The extended merge candidate list includes the first merge candidate and a second merge candidate, and the second merge candidate includes the motion information in the first merge candidate and is coded using a second BCW index value corresponding to equal weighting.

[0350]

[0349] In some examples, the merge candidates in the extended merge candidate list are sorted according to a template matching cost.

[0351]

[0350] The process then proceeds to (S3399) and ends.

[0352]

[0351] Process (3300) can be suitably adapted. The steps in process (3300) can be modified and / or omitted. Additional steps can be added. Any suitable execution order can be used.

[0353]

[0352] FIG. 34 shows a flowchart illustrating an overview of a process (3400) according to an embodiment of the present disclosure. Process (3400) can be used in a video decoder. In various embodiments, process (3400) is executed by a processing circuit such as a processing circuit in terminal devices (310), (320), (330), (340), a processing circuit that executes the functions of video decoder (410), a processing circuit that executes the functions of video decoder (510), and the like. In some embodiments, process (3400) is executed by software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit executes process (3400). The process starts at (S3401) and proceeds to (S3410).

[0354]

[0353] In (S3410), information indicating that the current block in the current picture is an inter-prediction block in merge mode is decoded from the bitstream. The bitstream carries a video including the current picture.

[0355]

[0354] In (S3420), an extended merge candidate list is constructed based on at least a first merge candidate having motion parameters obtained from neighboring blocks. The neighboring blocks may be spatial neighboring blocks or temporal neighboring blocks of the current block. The extended merge candidate list includes at least the first merge candidate and non-redundant merge candidates extended from the first merge candidate. The non-redundant merge candidates have at least the motion vector of the first merge candidate.

[0356]

[0355] In (S3430), an index indicating a specific merge candidate in the extended merge candidate list is decoded.

[0357]

[0356] In (S3440), the current block is reconstructed based on the specific merge candidate.

[0358]

[0357] In some examples, the first merge candidate is a bi-prediction candidate using a first motion vector associated with a first reference picture list and a second motion vector associated with a second reference picture list. The extended merge candidate list includes the first merge candidate and at least one of a second merge candidate and a third merge candidate, The second merge candidate is a uni-prediction candidate having a first motion vector associated with the first reference picture list; The third merge candidate is a uni-prediction candidate having a second motion vector associated with the second reference picture list.

[0359]

[0358] In some examples, the first merge candidate is a bi-prediction candidate that uses a first motion vector associated with a first reference picture list and a second motion vector associated with a second reference picture list. The extended merge candidate list constitutes an extended merge candidate list that includes the first merge candidate and at least one of a second merge candidate, a third merge candidate, a fourth merge candidate, a fifth merge candidate, and a sixth merge candidate. The second merge candidate is a bi-prediction candidate having a first motion vector associated with the second reference picture list and a second motion vector associated with the first reference picture list; The third merge candidate is a uni-prediction candidate having a first motion vector associated with the first reference picture list; The fourth merge candidate is a uni-prediction candidate having a second motion vector associated with the second reference picture list; The fifth merge candidate is a uni-prediction candidate having a second motion vector associated with the first reference picture list; and The sixth merge candidate is a uni-prediction candidate having a first motion vector associated with the second reference picture list.

[0360]

[0359] In some examples, the first merge candidate is a bi-prediction candidate coded using CU-level weighted bi-prediction (BCW), and the first merge candidate includes a first weight value and a first motion vector associated with a first reference picture list, and a second weight value and a second motion vector associated with a second reference picture list. In response to the first weight value being greater than the second weight value, the extended merge candidate list includes the first merge candidate and a second merge candidate, and the second merge candidate is a uni-prediction candidate having a first motion vector associated with the first reference picture list. In response to the first weight value being less than the second weight value, the extended merge candidate list includes the first merge candidate and a third merge candidate, and the third merge candidate is a uni-prediction candidate having a second motion vector associated with the second reference picture list.

[0361]

[0360] In some examples, the first merge candidate is a uni-prediction candidate having a first motion vector associated with a first reference picture list, and the extended merge candidate list includes the first merge candidate and a second merge candidate that is a uni-prediction candidate having a first motion vector associated with a second reference picture list.

[0362]

[0361] In some examples, the first merge candidate is a bi-prediction candidate coded using a first (CU-level weighted bi-prediction (BCW)) index value. The first merge candidate includes a first motion vector associated with a first reference picture list and a second motion vector associated with a second reference picture list. The extended merge candidate list includes the first merge candidate and one or more second merge candidates, each of the one or more second merge candidates includes the motion information in the first merge candidate and is coded using a BCW index value different from the first BCW index value.

[0363]

[0362] In some examples, the first merge candidate is a bi-prediction candidate coded using a first (CU-level weighted bi-prediction (BCW)) index value corresponding to unequal weighting, and the first merge candidate includes motion information having a first motion vector associated with a first reference picture list and a second motion vector associated with a second reference picture list. The extended merge candidate list includes the first merge candidate and a second merge candidate, the second merge candidate includes the motion information in the first merge candidate and is coded using a second BCW index value corresponding to equal weighting.

[0364]

[0363] In some examples, the merge candidates in the extended merge candidate list are sorted according to a template matching cost.

[0365]

[0364] The process then proceeds to (S3499) and ends.

[0366]

[0365] The process (3400) can be suitably adapted. The steps in the process (3400) can be modified and / or omitted. Additional steps can be added. Any suitable execution order can be used.

[0367]

[0366] FIG. 35 shows a flowchart illustrating an overview of a process (3500) according to an embodiment of the present disclosure. The process (3500) can be used in a video encoder. In various embodiments, the process (3500) is executed by a processing circuit such as a processing circuit in the terminal devices (310), (320), (330), (340), a processing circuit that executes the function of the video encoder (403), a processing circuit that executes the function of the video encoder (603), a processing circuit that executes the function of the video encoder (703), and the like. In some embodiments, the process (3500) is executed by software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit executes the process (3500). The process starts at (S3501) and proceeds to (S3510).

[0368]

[0367] In (S3510), for inter prediction of the current block in the current picture, a merge candidate list including a plurality of merge candidates obtained from neighboring blocks is constructed. The neighboring blocks may be spatial neighboring blocks or temporal neighboring blocks of the current block.

[0369]

[0368] In (S3520), for each merge candidate in the merge candidate list, an inter prediction direction is determined from among a plurality of direction candidates based on the template matching cost of the plurality of direction candidates.

[0370]

[0369] In (S3530), a specific merge candidate is determined from the merge candidate list.

[0371]

[0370] In (S3540), an index indicating a specific merge candidate from the merge candidate list is encoded in a bitstream that conveys a video including the current picture.

[0372]

[0371] In one example, an inter prediction direction is determined for a merge candidate in response to the merge candidate being obtained from neighboring blocks. In another example, an inter prediction direction is determined for a merge candidate in response to the completion of motion vector refinement for the merge candidate. In another example, an inter prediction direction is determined for a merge candidate in response to the completion of template matching-based rearrangement of a plurality of merge candidates.

[0373]

[0372] Next, the process proceeds to (S3599) and ends.

[0374]

[0373] Process (3500) can be suitably adapted. Steps in process (3500) can be modified and / or omitted. Additional steps can be added. Any suitable execution order can be used.

[0375]

[0374] FIG. 36 shows a flowchart outlining a process (3600) according to an embodiment of the present disclosure. Process (3600) can be used in a video decoder. In various embodiments, process (3600) is executed by a processing circuit such as a processing circuit in terminal devices (310), (320), (330), (340), a processing circuit that executes the functions of video decoder (410), a processing circuit that executes the functions of video decoder (510), and the like. In some embodiments, process (3600) is executed by software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit executes process (3600). The process starts at (S3601) and proceeds to (S3610).

[0376]

[0375] In (S3610), information indicating that the current block in the current picture is an inter-prediction block in merge mode is decoded from the bitstream. The bitstream carries video including the current picture.

[0377]

[0376] In (S3620), a merge candidate list including a plurality of merge candidates obtained from neighboring blocks is configured. The neighboring blocks may be spatial neighboring blocks or temporal neighboring blocks of the current block.

[0378]

[0377] In (S3630), an index indicating a specific merge candidate in the merge candidate list is decoded from the bitstream.

[0379]

[0378] In (S3640), based on the template matching costs of a plurality of direction candidates, a specific inter-prediction direction related to a specific merge candidate is selected from among the plurality of direction candidates.

[0380]

[0379] In (S3650), the current block is reconstructed based on a specific merge candidate in a specific inter-prediction direction.

[0381]

[0380] In one example, in response to a specific merge candidate being obtained from a neighboring block, a specific inter-prediction direction related to the specific merge candidate is determined (selected).

[0382]

[0381] In another example, in response to the completion of motion vector refinement for a specific merge candidate, a specific inter-prediction direction related to the specific merge candidate is determined (selected).

[0383]

[0382] In another example, in response to the completion of template matching-based rearrangement of a plurality of merge candidates, a specific inter-prediction direction related to a specific merge candidate is determined.

[0384]

[0383] In another example, a specific inter-prediction direction for a particular merge candidate is determined in response to the index being decoded.

[0385]

[0384] The process then proceeds to (S3699) and ends.

[0386]

[0385] The process (3600) can be suitably adapted. The steps in the process (3600) can be modified and / or omitted. Additional steps can be added. Any suitable order of execution can be used.

[0387]

[0386] The techniques described above can be implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, FIG. 37 shows a computer system (3700) suitable for implementing a particular embodiment of the disclosed subject matter.

[0388]

[0387] The computer software can be coded using any suitable machine code or computer language that can be the subject of an assembly, compilation, linking, or similar mechanism to create code that includes instructions that can be directly executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or instructions that are executed via interpretation or microcode execution.

[0389]

[0388] The instructions can be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet-of-things devices, etc.

[0390]

[0389] The components shown in FIG. 37 with respect to the computer system (3700) are essentially exemplary and are not intended to suggest any limitation as to the scope or functionality of the computer software implementing the embodiments of the present disclosure. Also, the component configuration should not be construed as having any dependency or requirement regarding any one or combination of the components shown in the exemplary embodiments of the computer system (3700).

[0391]

[0390] The computer system (3700) can include a specific human interface input device. Such a human interface input device can respond to input by one or more human users via, for example, tactile input (e.g., keystrokes, swipes, movement of a data glove), auditory input (e.g., voice, clapping), visual input (e.g., gesture), and olfactory input (not shown). Also, the human interface device can be used to capture specific media that is not necessarily directly related to conscious human input, such as audio (e.g., conversation, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., 2D video, 3D video including stereoscopic pictures).

[0392]

[0391] The input human interface device can include one or more of a keyboard (3701), a mouse (3702), a trackpad (3703), a touch screen (3710), a data glove (not shown), a joystick (3705), a microphone (3706), a scanner (3707), and a camera (3708) (although only one of each is depicted).

[0393]

[0392] The computer system (3700) can also include a specific human interface output device. Such a human interface output device can stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such a human interface output device can be a tactile output device (e.g., a touch screen (3710), a data glove (not shown), tactile feedback by a joystick (3705), although there may be a tactile feedback device that does not serve as an input device), an auditory output device (e.g., a speaker (3709), headphones (not shown)), a visual output device (e.g., a screen (3710) including a CRT screen, an LCD screen, a plasma screen, an OLED screen, each of which may or may not have a touch screen input function, each of which may or may not have a tactile feedback function, and some of them may be capable of outputting three-dimensional or higher-dimensional output by means such as two-dimensional visual output and stereoscopic output; virtual reality glasses (not shown), holographic display, and smoke tank (not shown)), and a printer (not shown).

[0394]

[0393] The computer system (3700) can also include an optical medium such as a CD / DVD ROM / RW (3720) using a medium (3721) such as a CD / DVD, a thumb drive (3722), a removable hard drive or solid state drive (3723), a legacy magnetic medium (not shown) such as a tape and a floppy disk (not shown), and a human-accessible storage device and related media such as a specialized ROM / ASIC / PLD-based device such as a security dongle (not shown).

[0395] One of ordinary skill in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include a transmission medium, a carrier wave, or other transient signals.

[0396]

[0395] A computer system (3700) may also include an interface to one or more communication networks (3755). The network can be, for example, wireless, wired, or optical. The network can further be related to local, wide area, metropolitan, vehicle industry, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), wired or wireless wide area digital networks for TV (including cable TV, satellite TV, and over-the-air broadcast TV), vehicle industries including CANBus, etc. Certain networks generally require an external network interface adapter attached to a specific general-purpose data port or peripheral bus (3749) (e.g., the USB port of a computer system (3700)); others are generally integrated commonly into the core of a computer system (3700) by attaching to a system bus as described below (e.g., an Ethernet interface is integrated within a PC computer system, and a cellular network interface is integrated within a smartphone computer system). Using any of these networks, the computer system (3700) can communicate with other entities. Such communication can be one-way receive-only (e.g., broadcast TV), one-way transmit-only (e.g., CANbus for certain CANbus devices), or two-way, e.g., for other computer systems using local or wide area digital networks. Specific protocols and protocol stacks can be used at each of those networks and network interfaces as described above.

[0397]

[0396] The aforementioned human interface device, human accessible storage device, and network interface can be attached to the core (3740) of the computer system (3700).

[0398]

[0397] The core (3740) can include one or more central processing units (CPUs) (3741), a graphics processing unit (GPU) (3742), a special programmable processing unit in the form of a field programmable gate array (FPGA) (3743), a hardware accelerator for specific tasks (3744), a graphics adapter (3750), etc. These devices can be connected via a system bus (3748) together with a read only memory (ROM) (3745), a random access memory (3746), and an internal mass storage device (e.g., an internal non-user accessible hard drive, SSD, etc.) (3747). In some computer systems, the system bus (3748) may be accessible in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices can be attached directly to the core's system bus (3748) or via a peripheral bus (3749). In one example, the screen (3710) can be connected to the graphics adapter (3750). The architecture of the peripheral bus includes PCI, USB, etc.

[0399]

[0398] The CPU (3741), GPU (3742), FPGA (3743), and accelerator (3744) can be combined to execute specific instructions capable of constituting the aforementioned computer code. The computer code can be stored in the ROM (3745) or RAM (3746). Temporary data can be stored in the RAM (3746), while persistent data can be stored, for example, in the internal mass storage (3747). Fast storage and retrieval for any memory device may be made possible by utilizing cache memory, which can be closely associated with one or more CPUs (3741), GPUs (3742), mass storage (3747), ROM (3745), RAM (3746), etc.

[0400]

[0399] A computer-readable medium can have computer code therein for performing various computer-implemented operations. The medium and the computer code can be considered to be specially designed and constructed for the purposes of this disclosure, or they can be considered to be of the kind well-known and available to those of ordinary skill in the field of computer software.

[0401] As an example, and not by way of limitation, a computer system having an architecture (3700), specifically a core (3740), can provide functionality as an operation result of software embodied in one or more tangible computer-readable media by a processor (including a CPU, GPU, FPGA, accelerator, etc.). Such a computer-readable media can be media related to user-accessible mass storage as described above, as well as specific storage of the core (3740) of a non-transitory nature such as a mass storage (3747) inside the core or a ROM (3745). The software implementing various embodiments of the present disclosure can be stored in such a device and executed by the core (3740). The computer-readable media can include one or more memory devices or chips according to specific needs. The software includes defining a data structure stored in a RAM (3746) and modifying such a data structure according to a process defined by the software, and causing a specific process or a specific part of a specific process described herein to be executed by the core (3740) and in particular a processor (including a CPU, GPU, FPGA, etc.) therein. Additionally or alternatively, the computer system can provide functionality as a result of logic wired or otherwise incorporated within a circuit (e.g., an accelerator (3744)), which circuit can execute a specific process or a specific part of a specific process described herein instead of or together with the software. References to software include logic and, if necessary, vice versa. References to computer-readable media can include a circuit (such as an integrated circuit (IC)) storing software for execution, a circuit embodying logic for execution, or both, as appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0402] Appendix A: Acronyms JEM: joint exploration model VVC: versatile video coding BMS: benchmark set MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Supplementary Enhancement Information VUI: Video Usability Information GOPs: Groups of Pictures TUs: Transform Units, PUs: Prediction Units CTUs: Coding Tree Units CTBs: Coding Tree Blocks PBs: Prediction Blocks HRD: Hypothetical Reference Decoder SNR: Signal Noise Ratio CPUs: Central Processing Units GPUs: Graphics Processing Units CRT: Cathode Ray Tube LCD: Liquid-Crystal Display OLED: Organic Light-Emitting Diode CD: Compact Disc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application-Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile communications LTE: Long-Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Areas SSD: solid-state drive IC: Integrated Circuit CU: Coding Unit

[0401] While this disclosure describes several exemplary embodiments, there are changes, substitutions, and various alternative equivalents that fall within the scope of this disclosure. Accordingly, although not explicitly illustrated or described herein, it will be understood that those skilled in the art will be able to devise many systems and methods that embody the principles of this disclosure and thus fall within its spirit and scope.

Claims

1. A video processing method in a decoder, comprising: Receiving a video bitstream including a current block within a current picture; Obtaining prediction information from the video bitstream, the prediction information indicating whether the current block is to be predicted in an inter prediction mode; Determining a merge candidate from a merge candidate list in response to the current block being predicted in the inter prediction mode; Determining an inter prediction direction based on a syntax element signaled in the video bitstream, the inter prediction direction being signaled separately from the merge candidate and being one of a first uni - prediction from a first reference picture list, a second uni - prediction from a second reference picture list, and a bi - prediction from the first reference picture list and the second reference picture list; Determining a motion vector for prediction of the current block based on the merge candidate; and Reconstructing the current block based on the inter prediction direction and the motion vector. A method comprising the above.

2. The method according to Claim 1, wherein the step of determining the inter prediction direction further comprises: Decoding one or more signals from the video bitstream, the one or more signals indicating the inter prediction direction. A method comprising the above.

3. The method according to Claim 1, wherein the video bitstream is compiled in an enhanced compression mode (ECM).

4. The method according to Claim 1, wherein the step of determining the inter prediction direction further comprises: Inferring that the inter prediction direction is a bi - prediction in response to the merge type of the current block being based on bi - prediction. A method comprising the above.

5. The method according to Claim 2, wherein the one or more signals comprise: A syntax using a first value indicating uni - prediction and a second value indicating bi - prediction.

6. The method according to Claim 2, wherein the one or more signals comprise: A syntax using a first value indicating the first uni - prediction, a second value indicating the second uni - prediction, and a third value indicating the bi - prediction.

7. The method according to claim 2, wherein the one or more signals are: a first flag indicating whether the first reference picture list is in the inter prediction direction; and a second flag indicating whether the second reference picture list is in the inter prediction direction; A method comprising the above.

8. The method according to claim 2, wherein the step of decoding one or more signals in the video bitstream further comprises: decoding one or more signals from the video bitstream in response to the high level syntax allowing bi-prediction; A method comprising the above.

9. The method according to claim 1, wherein the inter prediction direction is bi-prediction, and the method further comprises: decoding, from the video bitstream, an index indicating a specific weighted candidate in the weighted candidate list, the weighted candidate list including a plurality of weighted candidates each providing a weighted value for combining predictions from the first reference picture list and the second reference picture list; A method comprising the above.

10. The method according to claim 9, wherein the weighted candidate list includes at least: weighted candidates inherited from the merge candidates and default equal-weighted candidates.

11. A video processing method in a decoder, comprising: receiving a video bitstream including a current block in a current picture; decoding, from the bitstream, information indicating that the current block in the current picture is an inter prediction block in merge mode; constructing an extended merge candidate list based on at least a first merge candidate having motion parameters obtained from neighboring blocks, the extended merge candidate list including at least the first merge candidate and non-redundant merge candidates extended from the first merge candidate, the non-redundant merge candidates having at least the motion vector of the first merge candidate; decoding an index indicating a specific merge candidate in the extended merge candidate list; and reconstructing the current block based on the specific merge candidate; A method comprising the above.

12. In the method according to claim 11, the first merge candidate is a dual-prediction candidate that uses a first motion vector associated with a first reference picture list and a second motion vector associated with a second reference picture list, and the step of constructing the extended merge candidate list further comprises: including the step of constructing the extended merge candidate list including the first merge candidate and at least one of a second merge candidate and a third merge candidate; the second merge candidate is a single-prediction candidate having a first motion vector associated with the first reference picture list; and the third merge candidate is a single-prediction candidate having a second motion vector associated with the second reference picture list.

13. In the method according to claim 11, the first merge candidate is a dual-prediction candidate that uses a first motion vector associated with a first reference picture list and a second motion vector associated with a second reference picture list, and the step of constructing the extended merge candidate list further comprises: including the step of constructing the extended merge candidate list including the first merge candidate and at least one of a second merge candidate, a third merge candidate, a fourth merge candidate, a fifth merge candidate, and a sixth merge candidate; the second merge candidate is a dual-prediction candidate having a first motion vector associated with the second reference picture list and a second motion vector associated with the first reference picture list; the third merge candidate is a single-prediction candidate having a first motion vector associated with the first reference picture list; the fourth merge candidate is a single-prediction candidate having a second motion vector associated with the second reference picture list; the fifth merge candidate is a single-prediction candidate having a second motion vector associated with the first reference picture list; and the sixth merge candidate is a single-prediction candidate having a first motion vector associated with the second reference picture list.

14. In the method according to claim 11, the first merge candidate is a bi-prediction candidate coded using CU-level weighted bi-prediction (BCW), and the first merge candidate includes a first weight value and a first motion vector associated with a first reference picture list, and a second weight value and a second motion vector associated with a second reference picture list, and the step of constructing the extended merge candidate list further comprises: including the step of constructing the extended merge candidate list including the first merge candidate and a second merge candidate or a third merge candidate; the second merge candidate is a uni-prediction candidate having a first motion vector associated with the first reference picture list in response to the first weight value being greater than the second weight value; the third merge candidate is a uni-prediction candidate having a second motion vector associated with the second reference picture list in response to the first weight value being less than the second weight value.

15. In the method according to claim 11, the first merge candidate is a uni-prediction candidate having a first motion vector associated with a first reference picture list, and the step of constructing the extended merge candidate list further comprises: including the step of constructing an extended merge candidate list including the first merge candidate and a second merge candidate which is a uni-prediction candidate having a first motion vector associated with a second reference picture list.

16. In the method according to claim 11, the first merge candidate is a bi-prediction candidate coded using a first CU-level weighted bi-prediction (BCW) index value, and the first merge candidate includes a first motion vector associated with a first reference picture list and a second motion vector associated with a second reference picture list, and the step of constructing the extended merge candidate list further comprises: including the step of constructing the extended merge candidate list including the first merge candidate and one or more second merge candidates; each of the one or more second merge candidates includes motion information in the first merge candidate and is coded using a BCW index value different from the first BCW index value.

17. In the method according to claim 11, the first merge candidate is a bi-prediction candidate coded using a first CU-level weighted bi-prediction (BCW) index value corresponding to unequal weighting, the first merge candidate includes motion information having a first motion vector associated with a first reference picture list and a second motion vector associated with a second reference picture list, and the step of constructing the extended merge candidate list further comprises: comprising the step of constructing the extended merge candidate list including the first merge candidate and a second merge candidate; wherein the second merge candidate includes the motion information in the first merge candidate and is coded using a second BCW index value corresponding to equal weighting, the method.

18. In the method according to claim 11, the step of constructing the extended merge candidate list further comprises: including the step of ordering the merge candidates in the extended merge candidate list according to the template matching cost, the method.

19. A video processing method in a decoder, comprising: receiving a video bitstream including a current block in a current picture; decoding, from the bitstream, information indicating that the current block in the current picture is an inter-prediction block in a merge mode; constructing a merge candidate list including a plurality of merge candidates obtained from neighboring blocks; decoding an index indicating a specific merge candidate in the merge candidate list; selecting, from among a plurality of direction candidates, a specific inter-prediction direction related to the specific merge candidate based on the template matching cost of the plurality of direction candidates; and reconstructing the current based on the specific merge candidate in the specific inter-prediction direction; the method comprising.

20. In the method according to claim 19, the step of selecting the specific inter-prediction direction comprises: determining a specific inter-prediction direction related to the specific merge candidate in response to the specific merge candidate being obtained from neighboring blocks; determining a specific inter-prediction direction related to the specific merge candidate in response to the completion of motion vector refinement for the specific merge candidate; Determining a specific inter-prediction direction for the specific merge candidate in response to the completion of template matching-based reordering of the plurality of merge candidates; and Determining a specific inter-prediction direction for the specific merge candidate in response to the decoding of the index; A method comprising at least one of the above. **Claim 21** A video processing method in an encoder, comprising: Transmitting a video bitstream including a current block in a current picture to a decoder; wherein the video bitstream includes prediction information indicating whether the current block should be predicted in an inter-prediction mode; When the current block is predicted in the inter-prediction mode, a merge candidate is determined from a merge candidate list; The inter-prediction direction is specified by a syntax element signaled in the video bitstream, the inter-prediction direction being signaled separately from the merge candidate and being one of a first uni-prediction from a first reference picture list, a second uni-prediction from a second reference picture list, and a bi-prediction from the first reference picture list and the second reference picture list; A motion vector for prediction of the current block is determined based on the merge candidate; and The current block is reconstructed based on the inter-prediction direction and the motion vector.

Images