Method and device for video processing and medium
By determining whether the sample set is outside the boundary during video encoding and decoding and applying a weighted process to generate predictions, the problem of insufficient encoding and decoding efficiency in existing technologies is solved, and the performance of video encoding and decoding is improved.
Patent Information
- Application Number
- CN202380017858.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-01-19
- Filing Date
- 2023-01-16
- Publication Date
- 2024-11-26
AI Technical Summary
Existing video encoding and decoding technologies still have room for improvement in terms of encoding and decoding efficiency, especially in out-of-bounds prediction and local illumination compensation.
By determining whether the sample set is outside the boundary of the video unit during the video encoding and decoding process, and applying a weighted process to generate predictions, the encoding and decoding efficiency is improved.
It improves the efficiency of video encoding and decoding, especially in performance for out-of-bounds prediction and local illumination compensation.
Smart Images

Figure CN119032571A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present disclosure generally relate to video coding technology, and more specifically, to an out-of-bounds prediction, local illumination compensation (LIC) advanced motion vector prediction-merging (AMVP-MERGE) mode in image / video coding. Background Art
[0002] Nowadays, digital video capabilities are being applied to all aspects of people's lives. Various types of video compression technologies, such as MPEG-2, MPEG-4, ITU-TH.263, ITU-TH.264 / MPEG-4 Part 10 Advanced Video Codec (AVC), ITU-TH.265 High Efficiency Video Codec (HEVC) standard, and Versatile Video Codec (VVC) standard have been proposed for video encoding / decoding. However, the encoding and decoding efficiency of video encoding and decoding technologies is generally expected to be further improved. Summary of the invention
[0003] Embodiments of the present disclosure provide a solution for video processing.
[0004] In a first aspect, a method for video processing is presented. The method includes: during conversion between a video unit of a video and a bitstream of the video unit, determining whether at least one of a first sample set or a second sample set is outside a boundary associated with the video unit; applying a weighting process to the first sample set and the second sample set based on the determination; generating a prediction based on the weighted first sample set and the weighted second sample set; and performing conversion based on the prediction. Compared to conventional techniques, out-of-bounds prediction samples have been handled. In addition, encoding and decoding efficiency can be improved.
[0005] In a second aspect, an apparatus for processing video data is presented. The apparatus for processing video data comprises a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to the first aspect.
[0006] In a third aspect, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores instructions for causing a processor to execute the method according to the first aspect.
[0007] In a fourth aspect, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by a video processing device. The method includes: determining whether at least one of a first sample set or a second sample set is outside a boundary associated with a video unit of the video; applying a weighting process to the first sample set and the second sample set based on the determination; generating a prediction based on the weighted first sample set and the weighted second sample set; and generating a bitstream of the video unit based on the prediction.
[0008] In a fifth aspect, a method for storing a bitstream of a video includes: determining whether at least one of a first sample set or a second sample set is outside a boundary associated with a video unit of the video; applying a weighting process to the first sample set and the second sample set based on the determination; generating a prediction based on the weighted first sample set and the weighted second sample set; generating a bitstream of the video unit based on the prediction; and storing the bitstream in a non-transitory computer-readable recording medium.
[0009] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become more apparent through the following detailed description with reference to the accompanying drawings. In the exemplary embodiments of the present disclosure, the same reference numerals generally refer to the same components.
[0011] Figure 1 illustrates a block diagram illustrating an example video encoding and decoding system according to some embodiments of the present disclosure;
[0012] Figure 2 illustrates a block diagram illustrating a first example video encoder according to some embodiments of the present disclosure;
[0013] Figure 3 illustrates a block diagram illustrating an example video decoder according to some embodiments of the present disclosure;
[0014] Figure 4 The positions of spatial merging candidates are illustrated;
[0015] Figure 5 illustrates candidate pairs considered for redundancy check of spatial merging candidates;
[0016] Figure 6 is an illustration of motion vector scaling for temporal merging candidates;
[0017] Figure 7 The candidate positions of the temporal merge candidates C0 and C1 are shown;
[0018] Figure 8 The MMVD search point is shown;
[0019] Fig. 9 shows the extended CU area used in BDOF;
[0020] Fig.10 It is a diagram for the symmetric MVD mode;
[0021] Fig.11 An affine motion model based on control points is shown;
[0022] Fig.12 The affine MVF of each sub-block is shown;
[0023] Fig.13 The position of the inherited affine motion predictor is shown;
[0024] Fig.14 Control point motion vector inheritance is shown;
[0025] Fig.15 The positions of candidate positions for the constructed affine merge pattern are shown;
[0026] Fig.16 is a diagram of the use of motion vectors for the proposed combination method;
[0027] Fig.17 Sub-block MV VSB and pixel Δv(i, j) are shown;
[0028] Fig.18a and Fig.18b The SbTMVP process in VVC is shown in the figure, where Fig.18a illustrates the spatially adjacent blocks used by ATVMP, and Fig.18b illustrates the derivation of a sub-CU motion field by applying motion shifts from spatial neighbors and scaling motion information from corresponding co-located sub-CUs;
[0029] Fig.19 The extended CU area used in BDOF is shown.
[0030] Fig. 20 Decoding side motion vector refinement is shown;
[0031] Fig.21 The upper and left neighboring blocks used in the derivation of CIIP weights are shown;
[0032] Fig. 22 An example of a GPM split grouped by the same angle is shown;
[0033] Fig.23 shows the unidirectional prediction MV selection for geometric partitioning mode;
[0034] Fig.24 illustrates an example generation of warp weights w0 using a geometric partitioning pattern;
[0035] Fig.25 shows spatial neighboring blocks used to derive spatial merging candidates;
[0036] Fig.26 shows the template matching performed on the search area around the initial MV;
[0037] Fig. 27 A diamond-shaped area in the search area is shown;
[0038] Fig.28 The frequency response of the interpolation filter and the VVC interpolation filter at a half-pixel phase is shown;
[0039] Fig.29 A template and a reference sample of the template in a reference picture are shown;
[0040] Fig.30 A template and a reference sample of the template for a block with sub-block motion using motion information of a sub-block of a current block are shown;
[0041] Fig.31 The diagram shows the bidirectional MC block in the current ECM;
[0042] Fig.32 A flowchart illustrating a method according to an embodiment of the present disclosure; and
[0043] Fig.33 Illustrated is a block diagram of a computing device in which various embodiments of the present disclosure may be implemented.
[0044] Throughout the drawings, same or similar reference numbers generally refer to same or similar elements. DETAILED DESCRIPTION
[0045] The principle of the present disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are described for illustrative purposes only, and help those skilled in the art to understand and implement the present disclosure, without implying any limitation on the scope of the present disclosure. The disclosure described herein can be implemented in various ways except as follows.
[0046] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.
[0047] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but it is not required that every embodiment include the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Further, when a particular feature, structure, or characteristic is described with respect to an example embodiment, it is considered within the knowledge of those skilled in the art to affect such feature, structure, or characteristic with respect to other embodiments, whether or not explicitly described.
[0048] It should be understood that although the terms "first" and "second" etc. can be used in this article to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first element can be called the second element, and similarly, the second element can be called the first element without departing from the scope of the exemplary embodiment. As used herein, the term "and / or" includes any and all combinations of one or more of the listed items.
[0049] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are also intended to include the plural forms. It should also be understood that the terms "include", "comprises", "has", "has", "includes", and / or "comprising" when used herein specify the presence of stated features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof. Example Environment
[0050] Figure 1 1 is a block diagram illustrating an example video codec system 100 that may utilize the techniques of the present disclosure. As shown, the video codec system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a video encoding device, and the destination device 120 may also be referred to as a video decoding device. In operation, the source device 110 may be configured to generate encoded video data, and the destination device 120 may be configured to decode the encoded video data generated by the source device 110. The source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.
[0051] The video source 112 may include a source such as a video capture device. Examples of a video capture device include, but are not limited to, an interface for receiving video data from a video content provider, a computer graphics system for generating video data, and / or a combination thereof.
[0052] The video data may include one or more pictures. The video encoder 114 encodes the video data from the video source 112 to generate a bitstream. The bitstream may include a sequence of bits that form a coded representation of the video data. The bitstream may include coded pictures and related data. The coded pictures are coded representations of pictures. Related data may include sequence parameter sets, picture parameter sets and other grammatical structures. The I / O interface 116 may include a modulator / demodulator and / or a transmitter. The coded video data may be directly sent to the destination device 120 via the network 130A via the I / O interface 116. The coded video data may also be stored on a storage medium / server 130B for access by the destination device 120.
[0053] The destination device 120 may include an I / O interface 126, a video decoder 124, and a display device 122. The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain encoded video data from the source device 110 or the storage medium / server 130B. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to the user. The display device 122 may be integrated with the destination device 120, or may be outside the destination device 120, which is configured to interface with an external display device.
[0054] The video encoder 114 and the video decoder 124 may operate according to a video compression standard such as the High Efficiency Video Codec (HEVC) standard, the Versatile Video Codec (VVC) standard, and other current and / or future standards.
[0055] Figure 2 is a block diagram illustrating an example of a video encoder 200 according to some embodiments of the present disclosure, which may be Figure 1 An example of a video encoder 114 in the system 100 is illustrated.
[0056] Video encoder 200 may be configured to implement any or all of the techniques of this disclosure. Figure 2 In the example of , video encoder 200 includes multiple functional components. The techniques described in this disclosure can be shared between various components of video encoder 200. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.
[0057] In some embodiments, the video encoder 200 may include a partition unit 201, a prediction unit 202, a residual generation unit 207, a transform unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transform unit 211, a reconstruction unit 212, a buffer 213 and an entropy coding unit 214, and the prediction unit 202 may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205 and an intra-frame prediction unit 206.
[0058] In other examples, the video encoder 200 may include more, fewer, or different functional components. In an example, the prediction unit 202 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode in which at least one reference picture is a picture in which the current video block is located.
[0059] Furthermore, although some components, such as the motion estimation unit 204 and the motion compensation unit 205, may be integrated, for purposes of illustration, these components are described in detail in the following sections. Figure 2 are represented separately in the example.
[0060] The partition unit 201 may partition a picture into one or more video blocks. The video encoder 200 and the video decoder 300 may support various video block sizes.
[0061] The mode selection unit 203 may select one of a plurality of codec modes (intra-frame codec or inter-frame codec), for example, based on the error result, and provide the resulting intra-frame codec block or inter-frame codec block to the residual generation unit 207 to generate residual block data, and to the reconstruction unit 212 to reconstruct the codec block for use as a reference picture. In some examples, the mode selection unit 203 may select a combination of intra-frame and inter-frame prediction (CIIP) modes, where the prediction is based on an inter-frame prediction signal and an intra-frame prediction signal. In the case of inter-frame prediction, the mode selection unit 203 may also select a resolution of motion vectors for the block (e.g., sub-pixel precision or integer pixel precision).
[0062] To perform inter-frame prediction on the current video block, the motion estimation unit 204 may generate motion information of the current video block by comparing the current video block with one or more reference frames from the buffer 213. The motion compensation unit 205 may determine a predicted video block of the current video block based on the motion information and decoded samples of pictures from the buffer 213 other than the picture associated with the current video block.
[0063] For example, depending on whether the current video block is in an I-slice, a P-slice, or a B-slice, the motion estimation unit 204 and the motion compensation unit 205 may perform different operations for the current video block. As used herein, an "I-slice" may refer to a portion of a picture consisting of macroblocks, all of which are based on macroblocks within the same picture. Furthermore, as used herein, in some aspects, a "P-slice" and a "B-slice" may refer to a portion of a picture consisting of macroblocks that are not dependent on macroblocks in the same picture.
[0064] In some examples, the motion estimation unit 204 may perform unidirectional prediction for the current video block, and the motion estimation unit 204 may search the reference picture of list 0 or list 1 to obtain the reference video block of the current video block. The motion estimation unit 204 may then generate a reference index and a motion vector, the reference index indicating the reference picture in list 0 or list 1 containing the reference video block, and the motion vector indicating the spatial displacement between the current video block and the reference video block. The motion estimation unit 204 may output the reference index, the prediction direction indicator, and the motion vector as the motion information of the current video block. The motion compensation unit 205 may generate a predicted video block of the current video block based on the reference video block indicated by the motion information of the current video block.
[0065] Alternatively, in other examples, the motion estimation unit 204 may perform bidirectional prediction for the current video block. The motion estimation unit 204 may search the reference pictures in list 0 to obtain a reference video block of the current video block, and may also search the reference pictures in list 1 to obtain another reference video block of the current video block. The motion estimation unit 204 may then generate reference indexes and motion vectors, which indicate the reference pictures in list 0 and list 1 that contain the reference video blocks, and which indicate the spatial displacement between the reference video blocks and the current video block. The motion estimation unit 204 may output the reference index and motion vector of the current video block as the motion information of the current video block. The motion compensation unit 205 may generate a predicted video block of the current video block based on the reference video block indicated by the motion information of the current video block.
[0066] In some examples, motion estimation unit 204 may output a complete set of motion information for use in a decoding process by a decoder. Alternatively, in some embodiments, motion estimation unit 204 may signal motion information for a current video block with reference to motion information of another video block. For example, motion estimation unit 204 may determine that the motion information of the current video block is sufficiently similar to the motion information of a neighboring video block.
[0067] In one example, motion estimation unit 204 may indicate a value in a syntax structure associated with the current video block that indicates to video decoder 300 that the current video block has the same motion information as another video block.
[0068] In another example, the motion estimation unit 204 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 300 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.
[0069] As discussed above, the video encoder 200 may predictively signal motion vectors.Two examples of predictive signaling techniques that may be implemented by the video encoder 200 include Advanced Motion Vector Prediction (AMVP) and merge mode signaling.
[0070] The intra prediction unit 206 may perform intra prediction on the current video block. When the intra prediction unit 206 performs intra prediction on the current video block, the intra prediction unit 206 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a prediction video block and various syntax elements.
[0071] The residual generation unit 207 may generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) the prediction video block(s) of the current video block from the current video block. The residual data for the current video block may include residual video blocks corresponding to different sample components of the samples in the current video block.
[0072] In other examples, such as in skip mode, there may be no residual data for the current video block, and the residual generation unit 207 may not perform a subtraction operation.
[0073] Transform processing unit 208 may generate one or more transform coefficient video blocks for a current video block by applying one or more transforms to the residual video block associated with the current video block.
[0074] After transform processing unit 208 generates a transform coefficient video block associated with the current video block, quantization unit 209 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values associated with the current video block.
[0075] Inverse quantization unit 210 and inverse transform unit 211 may apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct a residual video block from the transform coefficient video block. Reconstruction unit 212 may add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by prediction unit 202 to generate a reconstructed video block associated with the current video block for storage in buffer 213.
[0076] After the reconstruction unit 212 reconstructs the video block, a loop filtering operation may be performed to reduce video block artifacts in the video block.
[0077] The entropy coding unit 214 may receive data from other functional components of the video encoder 200. When the entropy coding unit 214 receives the data, the entropy coding unit 214 may perform one or more entropy coding operations to generate entropy-coded data and output a bitstream including the entropy-coded data.
[0078] Figure 3 is a block diagram illustrating an example of a video decoder 300 according to some embodiments of the present disclosure, which may be Figure 1 An example of a video decoder 124 in the system 100 is illustrated.
[0079] Video decoder 300 may be configured to perform any or all of the techniques of this disclosure. Figure 3 In the example of , video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared between various components of video decoder 300. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.
[0080] exist Figure 3 In the example of , the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, a reconstruction unit 306 and a buffer 307. In some examples, the video decoder 300 can perform a decoding stage that is generally reciprocal to the encoding stage described with respect to the video encoder 200.
[0081] The entropy decoding unit 301 may retrieve an encoded bitstream. The encoded bitstream may include entropy-encoded video data (e.g., a coded block of video data). The entropy decoding unit 301 may decode the entropy-encoded video data, and from the entropy-decoded video data, the motion compensation unit 302 may determine motion information, including motion vectors, motion vector precision, reference picture list indexes, and other motion information. The motion compensation unit 302 may determine such information, for example, by performing AMVP and merge mode. AMVP is used, including deriving several most likely candidates based on data and reference pictures from adjacent PBs. The motion information typically includes horizontal and vertical motion vector displacement values, one or two reference picture indexes, and in the case of a prediction area in a B stripe, an identification of which reference picture list is associated with each index. As used herein, in some aspects, "merge mode" may refer to deriving motion information from blocks that are adjacent in space or time.
[0082] The motion compensation unit 302 may generate a motion compensated block, possibly performing interpolation based on an interpolation filter. An identifier of an interpolation filter to be used with sub-pixel precision may be included in a syntax element.
[0083] Motion compensation unit 302 may calculate interpolated values of sub-integer pixels of a reference block using interpolation filters as used during encoding of the video block by video encoder 200. Motion compensation unit 302 may determine the interpolation filters used by video encoder 200 from received syntax information and use these interpolation filters to produce a prediction block.
[0084] The motion compensation unit 302 may use at least a portion of the syntax information to determine the size of blocks used to encode (one or more) frames and / or (one or more) slices of the coded video sequence, partition information describing how each macroblock of a picture of the coded video sequence is partitioned, a mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-frame coding block, and other information used to decode the coded video sequence. As used herein, in some aspects, a "slice" may refer to a data structure that can be decoded independently of other slices of the same picture in terms of entropy coding, signal prediction, and residual signal reconstruction. A slice may be an entire picture or a region of a picture.
[0085] The intra prediction unit 303 may form a prediction block from spatially neighboring blocks using, for example, an intra prediction mode received in the bitstream. The inverse quantization unit 304 inverse quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 305 applies an inverse transform.
[0086] The reconstruction unit 306 may obtain a decoded block, for example, by adding the residual block to the corresponding prediction block generated by the motion compensation unit 302 or the intra prediction unit 303. Optionally, a deblocking filter may also be applied to filter the decoded block to remove blocking artifacts. The decoded video block is then stored in a buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction and also generates a decoded video for presentation on a display device.
[0087] Some exemplary embodiments of the present disclosure will be described in detail below. It should be noted that the section titles are used in this document to facilitate ease of understanding, and the embodiments disclosed in the section are not limited to that section. In addition, although certain embodiments are described with reference to general video codecs or other specific video codecs (codecs), the disclosed technology is also applicable to other video codec technologies. In addition, although some embodiments describe the video coding and decoding steps in detail, it should be understood that the corresponding decoding steps of the cancellation codec will be implemented by the decoder. In addition, the term video processing includes video coding or compression, video decoding or decompression, and video transcoding, in which video pixels are represented from one compression format to another compression format or at different compression bit rates. 1. Overview The present disclosure relates to video coding technology. Specifically, the present disclosure is about enhanced functions based on DMVR / BDOF in image / video coding. The present disclosure can be applied to existing video coding standards such as HEVC, VVC, etc. The present disclosure can also be applied to future video coding standards or video codecs. 2. Background technology Video codec standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T developed H.261 and H.263, ISO / IEC developed MPEG-1 and MPEG-4 ViSual, and the two organizations jointly developed H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Codec (AVC) and H.265 / HEVC standards. Since H.262, video codec standards have been based on a hybrid video codec structure in which temporal prediction plus transform codec is utilized. In order to explore future video codec technologies beyond HEVC, VCEG and MPEG jointly founded the Joint Video Exploration Team (JVET) in 2015. JVET meetings are held quarterly at the same time, and the new video codec standard was officially named the Versatile Video Codec (VVC) at the JVET meeting in April 2018, and the first version of the VVC Test Model (VTM) was released at that time. The VVC working draft and the test model VTM are updated after each meeting. The VVC project achieved technical completion (FDIS) at the July 2020 meeting. 2.1. Existing inter-frame prediction codec tools For each inter-predicted CU, motion parameters including motion vector, reference picture index and reference picture list usage index and additional information required by the new coding features of VVC will be used for inter-predicted sample generation. Motion parameters can be signaled explicitly or implicitly. When a CU is encoded and decoded in skip mode, the CU is associated with one PU and has no significant residual coefficients, no coded motion vector increments or reference picture indexes. A merge mode is specified, whereby the motion parameters of the current CU are obtained from neighboring CUs (including spatial candidates and temporal candidates) and the additional timetable introduced in VVC. Merge mode can be applied to any inter-predicted CU, not just for skip mode. An alternative to merge mode is explicit transmission of motion parameters, where motion vectors, corresponding reference picture indexes for each reference picture list and reference picture list usage flags, as well as other required information, are explicitly signaled per CU. In addition to the inter-frame coding features in HEVC, VVC also includes many new and refined inter-frame prediction coding tools listed below: - Extended combined forecast; - Merge mode with MVD (MMVD); - Symmetric MVD (SMVD) signaling; -Affine motion compensated prediction; - Sub-block based temporal motion vector prediction (SbTMVP); - Adaptive Motion Vector Resolution (AMVR); - Motion field storage: 1 / 16th brightness sample MV storage and 8x8 motion field compression; - Bidirectional prediction with CU level weights (BCW); - Bidirectional Optical Flow (BDOF); - Decoder side motion vector refinement (DMVR); - Geometric Partitioning Mode (GPM); - Combined intra and inter prediction (CIIP). Details on those inter prediction methods specified in VVC are provided below. 2.1.1. Extended Merge Prediction In VVC, the merge candidate list is constructed by including the following five types of candidates in order: 1) Spatial MVP from spatially neighboring CUs; 2) Time MVP from the same CU; 3) History-based MVP from FIFO table; 4) Paired average MVP; 5) Zero MV. The size of the merge list is signaled in the sequence parameter set header, and the maximum allowed size of the merge list is 6. For each CU code in merge mode, the index of the best merge candidate is encoded using truncated unary binarization (TU). The first bin of the merge index is coded with context, while bypass coding is used for other bins. The derivation process of merge candidates for each category is provided in this section. As done in HEVC, VVC also supports parallel derivation of merge candidate lists for all CUs in a region of a certain size. 2.1.1.1. Spatial candidate export The derivation of spatial merge candidates in VVC is the same as that in HEVC, except that the positions of the first two merge candidates are swapped. Figure 4 is a schematic diagram 400 illustrating the locations of spatial merging candidates. Figure 4 A maximum of four merge candidates are selected from the candidates at the depicted positions. The derivation order is B0, A0, B1, A1, and B2. Position B2 is considered only when one or more CUs at positions B0, A0, B1, and A1 are not available (for example, because it belongs to another slice or tile) or is intra-coded. After the candidate at position A1 is added, the addition of the remaining candidates undergoes a redundancy check that ensures that candidates with the same motion information are excluded from the list, so that the coding efficiency is improved. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Figure 5is a diagram 500 illustrating candidate pairs considered for redundancy checking of spatial merging candidates. Figure 5 The pairs linked with arrows in , and the candidate is added to the list only if the corresponding candidate for redundancy check does not have the same motion information. 2.1.1.2. Time candidate export In this step, only one candidate is added to the list. In particular, in the derivation of this temporal merge candidate, the scaled motion vector is derived based on the co-located CU belonging to the co-located reference picture. The reference picture list to be used for deriving the co-located CU is explicitly signaled in the slice header. The scaled motion vector of the temporal merge candidate is as follows Figure 6 The scaled motion vector is obtained as illustrated by the dashed line in the diagram 600 of , which is scaled from the motion vector of the co-located CU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal merge candidate is set equal to zero. Figure 7 700 is a schematic diagram illustrating candidate positions of temporal merging candidates C0 and C1. Figure 7 As depicted, the position of the temporal candidate is selected between candidates C0 and C1. If the CU at position C0 is not available, intra-coded, or outside the current row of CTUs, then position C1 is used. Otherwise, position C0 is used in the derivation of the temporal merge candidate. 2.1.1.3. History-based Merge Candidate Derivation History-based MVP (HMVP) merge candidates are added to the merge list after spatial MVP and TMVP. In this method, the motion information of previously coded blocks is stored in a table and used as the MVP of the current CU. A table with multiple HMVP candidates is maintained during the encoding / decoding process. The table is reset (cleared) when a new CTU row is encountered. Whenever there is a non-sub-block inter-coded CU, the relevant motion information is added to the last entry of the table as a new HMVP candidate. The HMVP table size S is set to 6, which indicates that up to 6 history-based MVP (HMVP) candidates can be added to the table. When a new motion candidate is inserted into the table, a restricted first-in-first-out (FIFO) rule is used, where a redundancy check is first applied to find if there is an identical HMVP in the table. If found, the identical HMVP is removed from the table and all subsequent HMVP candidates are moved forward. HMVP candidates can be used in the merge candidate list construction process. The latest few HMVP candidates in the table are checked in order and inserted into the candidate list after the TMVP candidate. Redundancy check is applied to HMVP candidates to spatial or temporal merge candidates. To reduce the number of redundant checking operations, the following simplifications are introduced: 1. The number of HMPV candidates for merge list generation is set to (N<=4)?M:(8-N), where N indicates the number of existing candidates in the merge list and M indicates the number of available HMVP candidates in the table. 2. Once the total number of available merge candidates reaches the maximum allowed merge candidate minus 1, the merge candidate list construction process from HMVP is terminated. 2.1.1.4. Pairwise average merge candidate derivation The pairwise average candidate is generated by averaging predefined pairs of candidates in the existing merge candidate list, and the predefined pairs are defined as {(0, 1), (0, 2), (1, 2), (0, 3), (1, 3), (2, 3)}, where the number represents the merge index of the merge candidate list. The average motion vector is calculated separately for each reference list. If two motion vectors are available in one list, they are averaged even when they point to different reference pictures; if only one motion vector is available, the one motion vector is used directly; if no motion vector is available, this list is kept invalid. When the merge list is not full after adding pairwise average merge candidates, zero MVPs are inserted at the end until the maximum number of merge candidates is encountered. 2.1.1.5. Merge estimated regions The merge estimation region (MER) allows the merge candidate list to be derived independently for a CU in the same merge estimation region (MER). For the generation of the merge candidate list for the current CU, candidate blocks in the same MER as the current CU are not included. In addition, the update process for the history-based motion vector predictor candidate list is only updated when (xCb+cbWidth)>>Log2ParMrgLevel is greater than xCb>>Log2ParMrgLevel and (yCb+cbHeight)>>Log2parMrglevel is greater than (yCb>>Log2ParMrgLevel) and when (xCb, yCb) is the top left luma sample position of the current CU in the picture and (cbWidth, cbHeight) is the CU size. The MER size is selected at the encoder side and signaled in the sequence parameter set as log2_paral-lel_merge_level_minus2. 2.1.2. Merge Mode with MVD (MMVD) In addition to the merge mode where the implicitly derived motion information is directly used for prediction sample generation of the current CU, a merge mode with motion vector difference (MMVD) is introduced in VVC. The MMVD flag is signaled just after the skip flag and merge flag are sent to specify whether the MMVD mode is used for the CU. In MMVD, after a merge candidate is selected, it is further refined by the MVD information that is signaled. Further information includes a merge candidate flag, an index to specify the magnitude of motion, and an index to indicate the direction of motion. In MMVD mode, one of the first two candidates in the merge list is selected to be used as the MV basis. The MMVD candidate flag is signaled to specify which candidate is used. The distance index specifies the motion magnitude information and indicates a predefined offset from the starting point. Figure 8 As shown, the offset is added to the horizontal component or the vertical component of the starting MV. The relationship between the distance index and the predefined offset is specified in Table 1. Table 1 - Relationship between distance index and predefined offset The direction index indicates the direction of the MVD relative to the starting point. The direction index can represent four directions as shown in Table 2. Note that the meaning of the MVD symbol can change according to the information of the starting MV. When the starting MV is a non-predicted MV or a bidirectionally predicted MV in which the two lists point to the same side of the current picture (that is, the POCs of the two references are both greater than the POC of the current picture, or both are less than the POC of the current picture), the symbol in Table 1 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectionally predicted MV in which the two MVs point to different sides of the current picture (that is, the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture), the symbol in Table 2 specifies the sign of the MV offset added to the list0 MV component of the starting MV and the sign of the list1 MV has an opposite value. Table 2 - Sign of MV offsets specified by direction index Direction Index 00 01 10 11 x-axis + - N / A N / A y-axis N / A N / A + - 2.1.2.1. Bidirectional Prediction with CU-Level Weights (BCW) In HEVC, the bidirectional prediction signal is generated by averaging two prediction signals obtained from two different reference pictures and / or using two different motion vectors. In VVC, the bidirectional prediction mode is extended beyond simple averaging to allow weighted averaging of the two prediction signals: P bi-pred=((8-w)*P0+w*P1+4)>>3 (2-1). Five weights are allowed in weighted average bidirectional prediction, w∈{-2, 3, 4, 5, 10}. For each bidirectionally predicted CU, the weight w is determined in one of two ways: 1) For non-merged CUs, the weight index is signaled after the motion vector difference; 2) For merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is only applicable to CUs with 256 or more luma samples (i.e., CU width times CU height is greater than or equal to 256). For low-latency pictures, all 5 weights are used. For non-low-latency pictures, only 3 weights (w∈{3, 4, 5}) are used. - At the encoder, a fast search algorithm is applied to find the weight index without significantly increasing the encoder complexity. These algorithms are summarized as follows. When combined with AMVR, unequal weights are only conditionally checked for 1-pixel and 4-pixel motion vector accuracy if the current picture is a low-latency picture. - When combined with affine, affine ME will be performed for unequal weights if and only if the affine mode is selected as the current best mode. - When the two reference pictures in bi-directional prediction are the same, unequal weights are only checked conditionally. - Unequal weights are not searched when certain conditions are met, which depend on the POC distance between the current picture and its reference pictures, the codec QP and the temporal level. The BCW weight index is encoded using one context encoded bit followed by a bypass encoded bit. The first context encoded bit indicates whether equal weights are used; and if unequal weights are used, an additional bit is signaled using bypass codec to indicate which unequal weights are used. Weighted prediction (WP) is a codec tool supported by the H.264 / AVC and HEVC standards to efficiently encode and decode video content in fading situations. Support for WP has also been added to the VVC standard. WP allows weighting parameters (weights and offsets) to be signaled for each reference picture in each of the reference picture lists L0 and L1. Then, during motion compensation, the corresponding (one or more) weights and (one or more) offsets of the reference picture(s) are applied. WP and BCW are designed for different types of video content. In order to avoid the interaction between WP and BCW (which will complicate the VVC decoder design), if the CU uses WP, the BCW weight index is not signaled, and w is inferred to be 4 (i.e., equal weights are applied). For the merged CU, the weight index is inferred from the neighboring blocks based on the merge candidate index. This can be applied to both the normal merge mode and the inherited affine merge mode. For the constructed affine merge mode, affine motion information is constructed based on the motion information of up to 3 blocks. The BCW index of the CU using the constructed affine merge mode is simply set equal to the BCW index of the first control point MV. In VVC, CIIP and BCW cannot be applied jointly to a CU. When encoding and decoding a CU in CIIP mode, the BCW index of the current CU is set to 2, for example, equal weight. 2.1.2.2. Bidirectional Optical Flow (BDOF) The Bidirectional Optical Flow (BDOF) tool is included in VVC. BDOF (formerly known as BIO) is included in JEM. Compared to the JEM version, BDOF in VVC is a simpler version that requires much less computation, especially in terms of the number of multiplications and multiplier size. BDOF is used to refine the bidirectional prediction signal of a CU at the 4×4 sub-block level. If BDOF meets all the following conditions, it is applicable to a CU: - The CU is encoded and decoded using the "true" bidirectional prediction mode, that is, one of the two reference pictures is before the current picture in the display order, and the other reference picture is after the current picture in the display order; - The distances from the two reference pictures to the current picture (i.e., POC differences) are the same; - Both reference images are short-term reference images; -CU does not use affine mode or SbTMVP merge mode for encoding and decoding; -CU has more than 64 luminance samples; -CU height and CU width are both greater than or equal to 8 luma samples; - BCW weight index indicates equal weight; -WP is not enabled for the current CU; -CIIP mode is not used for the current CU. BDOF is only applicable to the luma component. As its name indicates, BDOF mode is based on the concept of optical flow, which assumes that the motion of objects is smooth. For each 4×4 sub-block, motion refinement (v x , v y ) is calculated by minimizing the difference between the L0 prediction samples and the L1 prediction samples. Motion refinement is then used to adjust the bi-prediction sample values in the 4x4 sub-block. The following steps are applied in the BDOF process. First, the horizontal and vertical gradients of the two prediction signals are calculated by directly computing the difference between two neighboring samples: and Right now Among them I (k) (i, j) is the sample value at coordinate (i, j) of the prediction signal in list k (k=0, 1), and shift1 is calculated based on the luma bit depth bitDepth as shift1=max(6, bitDepth-6). Then, the autocorrelations and cross-correlations of the gradients S1, S2, S3, S5, and S6 are calculated as in where Ω is a 6×6 window around the 4×4 sub-block, and n a and n b The values of are set equal to min(1, bitDepth-11) and min(4, bitDepth-8) respectively. Then, the motion refinement (v) is derived using the cross-correlation and autocorrelation terms as follows: x , v y ): in th′ BIO =2 max(5,BD-7) . is the round-down (floor) function, and Based on the motion refinement and gradients, the following adjustments are computed for each sample in the 4×4 sub-block: Finally, the BDOF samples of the CU are calculated by adjusting the bidirectional prediction samples as follows: pred BDOF (x, y) = (I (0)(x,y)+I (1) (x, y) + b(x, y) + O offset )>>shift (2-7). These values are chosen so that the multipliers in the BDOF process do not exceed 15 bits and the maximum bit width of the intermediate parameters in the BDOF process is kept within 32 bits. In order to derive the gradient value, some prediction samples I in the list k (k = 0, 1) outside the current CU boundary (k) (i, j) needs to be generated. Fig. 9 FIG. 2 is a schematic diagram showing an extended CU region used in BDOF. Fig. 9 As depicted in the schematic diagram 900 of FIG. 1 , BDOF in VVC uses an extended row / column around the boundary of the CU. In order to control the computational complexity of generating prediction samples outside the boundary, the extended area (in Fig. 9 The prediction samples in 910 are generated by directly taking the reference samples at nearby integer positions without interpolation (using the floor() operation on the coordinates), and the normal 8-head motion compensation interpolation filter is used to generate the CU (in Fig. 9 The predicted samples within the CU boundary are represented as 920 in FIG. These extended sample values are only used in the gradient calculation. For the remaining steps in the BDOF process, if any samples and gradient values outside the CU boundary are needed, these sample values and gradient values are filled (i.e., repeated) from their nearest neighbors. When the width and / or height of a CU is greater than 16 luma samples, it will be split into sub-blocks with a width and / or height equal to 16 luma samples, and the sub-block boundaries are regarded as CU boundaries in the BDOF process. The maximum unit size for the BDOF process is limited to 16×16. For each sub-block, the BDOF process can be skipped. When the SAD between the initial L0 prediction sample and the L1 prediction sample is less than the threshold, the BDOF process is not applied to the sub-block. The threshold is set equal to (8*W*(H>>1)), where W indicates the sub-block width and H indicates the sub-block height. In order to avoid the additional complexity of the SAD calculation, the SAD between the initial L0 prediction sample and the L1 prediction sample calculated in the DVMR process is reused here. If BCW is enabled for the current block, that is, the BCW weight index indicates unequal weights, bidirectional optical flow is disabled. Similarly, if WP is enabled for the current block, that is, luma_weight_1x_flag is 1 for either of the two reference pictures, BDOF is also disabled. BDOF is also disabled when the CU is encoded or decoded in symmetric MVD mode or CIIP mode. 2.1.2.3. Symmetric MVD Codec (SMVD) In VVC, in addition to normal unidirectional prediction and bidirectional prediction mode MVD signaling, symmetric MVD mode for bidirectional prediction MVD signaling is also applied. In symmetric MVD mode, the motion information including reference picture indexes of both list-0 and list-1 and MVD of list-1 is not signaled but derived. The decoding process of the symmetric MVD mode is as follows: 1) At the stripe level, the variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1 are derived as follows: - If mvd_11_zero_flag is 1, BiDirPredFlag is set equal to 0. Otherwise, if the nearest reference picture in list-0 and the nearest reference picture in list-1 form a preceding pair of reference pictures or a following pair of reference pictures, BiDirPredFlag is set to 1, and both the list-0 reference picture and the list-1 reference picture are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0. 2) At the CU level, if the CU is bidirectionally predicted and BiDirPredFlag is equal to 1, the symmetric mode flag indicating whether the symmetric mode is used is explicitly signaled. When the symmetric mode flag is true, only mvp_10_flag, mvp_11_flag and MVD0 are explicitly signaled. The reference index of list-0 and list-1 are set equal to the reference picture pair, respectively. MVD1 is set equal to (-MVD0). The final motion vector is shown in the formula below. Fig.10 is an illustration for the symmetric MVD mode. In the encoder, symmetric MVD motion estimation starts with an initial MV evaluation. A set of initial MV candidates includes MVs obtained from unidirectional prediction search, MVs obtained from bidirectional prediction search, and MVs from the AMVP list. The one with the lowest rate-distortion cost is selected as the initial MV for the symmetric MVD motion search. 2.1.3. Affine Motion Compensated Prediction In HEVC, only the translational motion model is applied to motion compensated prediction (MCP). However, in the real world, there are many kinds of motion, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, block-based affine transformation motion compensated prediction is applied. Fig.11 As shown, the affine motion field of a block is described by motion information of two control point motion vectors (4 parameters) or three control point motion vectors (6 parameters). for Fig.11 Using the 4-parameter affine motion model 1110 in , the motion vector at the sample position (x, y) in the block is derived as: for Fig.11 Using the 6-parameter affine motion model 1120 in , the motion vector at the sample position (x, y) in the block is derived as: Where (mv 0x , mv 0y ) is the motion vector of the upper left control point, (mv 1x , mv 1y ) is the motion vector of the upper right control point, and (mv 2x , mv 2y ) is the motion vector of the lower left control point. To simplify motion compensated prediction, block-based affine transform prediction is applied. Fig.12 A schematic diagram 1200 of the affine MVF for each sub-block is shown. To derive the motion vector for each 4×4 luminance sub-block, as Fig.12 As shown, the motion vector of the center sample of each subblock is calculated according to the above equation and rounded to 1 / 16 fractional precision. Then, the motion compensated interpolation filter is applied to generate a prediction for each subblock with the derived motion vector. The subblock size of the chroma component is also set to 4×4. The MV of a 4×4 chroma subblock is calculated as the average of the MVs of the four corresponding 4×4 luminance subblocks. As done for translational motion inter prediction, there are also two affine motion inter prediction modes: affine merge mode and affine AMVP mode. 2.1.3.1. Affine Merge Prediction AF_MERGE mode can be applied to CUs with width and height greater than or equal to 8. In this mode, the CPMV of the current CU is generated based on the motion information of spatially neighboring CUs. There can be up to five CPMVP candidates, and the index is signaled to indicate the CPMVP to be used for the current CU. The following three types of CPVM candidates are used to form the affine merge candidate list: - Inherited affine merge candidates extrapolated from the CPMV of neighboring CUs; - Affine merged candidate CPMVP constructed using the translation MV of the neighboring CU; -Zero MV. In VVC, there are at most two inherited affine candidates derived from the affine motion models of neighboring blocks, one from the left neighboring CU and one from the above neighboring CU. Fig.13 Schematic diagram 1300 illustrating the positions of inherited affine motion predictors. Candidate blocks such as Fig.13 As shown. For the left predictor, the scan order is A0->A1, while for the upper predictor, the scan order is B0->B1->B2. Only the first inheritance candidate from each side is selected. No pruning check is performed between two inheritance candidates. When a neighboring affine CU is identified, its control point motion vector is used to derive the CPMVP candidate in the affine merge list of the current CU. Fig.14 A schematic diagram 1400 of control point motion vector inheritance is shown. Fig.14 As shown, if the adjacent lower left block A 1410 is encoded and decoded in the affine mode, the motion vectors v2, v3 and v4 of the upper left corner, upper right corner and lower left corner of the CU 1420 containing the block A 1410 are obtained. When the block A 1410 is encoded and decoded using the 4-parameter affine model, two CPMVs of the current CU are calculated based on v2 and v3. In the case of encoding and decoding the block A using the 6-parameter affine model, three CPMVs of the current CU are calculated based on v2, v3 and v4. The constructed affine candidate means that the candidate is constructed by combining the neighboring translational motion information of each control point. The motion information of the control point is obtained from Fig.15 The specified spatial neighbors and specified temporal neighbors shown are derived. Fig.15 A schematic diagram 1500 is shown illustrating the positions of candidate positions for a constructed affine merge pattern. k (k=1, 2, 3, 4) represents the kth control point. For CPMV1, B2->B3->A2 blocks are checked and the MV of the first available block is used. For CPMV2, B1->B0 blocks are checked, and for CPMV3, A1->A0 blocks are checked. For TMVP, TMVP is used as CPMV4 if available. After the MVs of the four control points are obtained, an affine merge candidate is constructed based on the motion information. The following combinations of control point MVs are used to construct in order: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}. A combination of 3 CPMVs constructs a 6-parameter affine merge candidate, and a combination of 2 CPMVs constructs a 4-parameter affine merge candidate. To avoid the motion scaling process, the relevant combination of control point MVs is discarded if the reference index of the control points is different. After the inherited affine merge candidates and the constructed affine merge candidates are checked, if the list is still not full, a zero MV is inserted at the end of the list. 2.1.3.2. Affine AMVP prediction Affine AMVP mode can be applied to CUs with width and height greater than or equal to 16. An affine flag at the CU level is signaled in the bitstream to indicate whether the affine AMVP mode is used, and then another flag is signaled to indicate whether it is 4-parameter affine or 6-parameter affine. In this mode, the difference between the CPMVP of the current CU and its predicted sub-CPMVP is signaled in the bitstream. The affine AVMP candidate list size is 2, and it is generated by using the following four types of CPVM candidates in order: - Inherited affine AMVP candidates extrapolated from the CPMV of neighboring CUs; - A constructed affine AMVP candidate CPMVP derived using the translation MV of the neighboring CU; -Translated MV from neighboring CU; -Zero MV. The order of checking inherited affine AMVP candidates is the same as that of checking inherited affine merge candidates. The only difference is that for AVMP candidates, only affine CUs with the same reference picture as in the current block are considered. The pruning process is not applied when inserting inherited affine motion predictors into the candidate list. The constructed AMVP candidates are Fig.15 Derived from the specified spatial neighbors shown. The same check order as done in the affine merge candidate construction is used. In addition, the reference picture index of the neighboring blocks is also checked. The first block in the check order that is inter-coded and has the same reference picture as in the current CU is used. There is only one. When the current CU is encoded and decoded with a 4-parameter affine mode and both mv0 and mv1 are available, they are added as a candidate in the affine AMVP list. When the current CU is encoded and decoded with a 6-parameter affine mode and all three CPMVs are available, they are added as a candidate in the affine AMVP list. Otherwise, the constructed AMVP candidate is set to unavailable. If the affine AMVP list candidate is still less than 2 after the valid inherited affine AMVP candidates and constructed AMVP candidates are checked, mv0, mv1, and mv2 are added in order as translation MVs to predict all control point MVs of the current CU when available. Finally, if the affine AMVP list is still not full, zero MVs are used to fill the affine AMVP list. 2.1.3.3. Affine motion information storage In VVC, the CPMV of an affine CU is stored in a separate buffer. The stored CPMV is only used to generate inherited CPMVs for the most recently encoded CU in affine merge mode and affine AMVP mode. The sub-block MVs derived from the CPMVs are used for motion compensation, merging of translation MVs / MV derivation of AMVP lists, and deblocking. To avoid picture line buffering for additional CPMV, affine motion data inheritance from CUs from upper CTUs is handled differently than inheritance from normal neighboring CUs. If the candidate CU for affine motion data inheritance is in the upper CTU row, the lower left and lower right sub-block MVs in the line buffer are used for affine MVP derivation instead of the CPMVs. In this way, the CPMVs are stored only in the local buffer. If the candidate CU is decoded by 6-parameter affine codec, the affine model is downgraded to a 4-parameter model. Fig.16 As shown, along the top CTU boundary, the bottom left and bottom right sub-block motion vectors of the CU are used for affine inheritance of the CU in the bottom CTU. 2.1.3.4. Prediction refinement using optical flow for affine mode (PROF) Compared with pixel-based motion compensation, sub-block-based affine motion compensation can save memory access bandwidth and reduce computational complexity, but at the cost of prediction accuracy penalty. In order to achieve finer motion compensation granularity, prediction refinement with optical flow (PROF) is used to refine sub-block-based affine motion compensation prediction without increasing the memory access bandwidth of motion compensation. In VVC, after sub-block-based affine motion compensation is performed, the brightness prediction samples are refined by adding the differences derived from the optical flow equation. PROF is described as the following four steps: Step 1) Sub-block based affine motion compensation is performed to generate a sub-block prediction I(i, j). Step 2) Use a 3-tap filter [-1, 0, 1] to calculate the spatial gradient g of the sub-block prediction at each sample position x (i, j) and g y (i, j). The gradient calculation is exactly the same as that in BDOF. g x (i,j)=(I(i+1,j)>>shift1)-(I(i-1,j))shift1) (2-11) g y (i,j)=(I(i,j+1))shift1)-(I(i,j-1))shift1) (2-12) Shift1 is used to control the accuracy of the gradient. Sub-block (i.e. 4×4) prediction is extended by one sample on each side of the gradient calculation. To avoid additional memory bandwidth and additional interpolation calculations, these extended samples on the extended boundary are copied from the nearest integer pixel position in the reference picture. Step 3) The brightness prediction refinement is calculated by the following optical flow equation. ΔI(i, j) = g x (i, j)*Δv x (i, j)+g y (i, j)*Δv y (i, j) (2-13) where Δv(i, j) is the difference between the sample MV calculated for the sample position (i, j) (denoted as v(i, j)) and the sub-block MV of the sub-block to which the sample (i, j) belongs, as Fig.17 Δv(i, j) (shown as arrow 1710) is quantized in units of 1 / 32 luma sample precision. Since the affine model parameters and the sample positions relative to the sub-block center do not change from sub-block to sub-block, Δv(i, j) can be calculated for the first sub-block and reused for other sub-blocks in the same CU. Let dx(i, j) and dy(i, j) be the distance from the sample position (i, j) to the sub-block center (x SB ,y SB ), Δv(x, y) can be derived by the following equations: To maintain accuracy, the sub-block center (x SB ,y SB ) is calculated as ((WSB-1) / 2, (HSB-1) / 2), where W SB and H SB are the sub-block width and height respectively. For the 4-parameter affine model, For the 6-parameter affine model, Where (v 0x , v 0y )、(v 1x , v 1y )、(v 2x , v 2y ) are the upper left, upper right and lower left control point motion vectors, w and h are the width and height of the CU. Step 4) Finally, the luma prediction refinement ΔI(i, j) is added to the sub-block prediction I(i, j) and the final prediction I' is generated as the following equation. I′(i,j)=I(i,j)+ΔI(i,j) PROF is not applicable to affine-coded CUs in two cases: 1) all control point MVs are the same, indicating that the CU has only translational motion; 2) the affine motion parameters are larger than the specified limit because sub-block-based affine MCs are downgraded to CU-based MCs to avoid large memory access bandwidth requirements. A fast codec method is applied to reduce the coding complexity of affine motion estimation using PROF. PROF is not applied in the affine motion estimation stage in the following two cases: a) If this CU is not a root block and its parent block does not select the affine mode as its best mode, PROF is not applied because the possibility that the current CU selects the affine mode as the best mode is low; b) If the magnitudes of the four affine parameters (C, D, E, F) are all less than a predefined threshold and the current picture is not a low-latency picture, PROF is not applied. In this way, affine motion estimation using PROF can be accelerated. 2.1.4. Sub-block based temporal motion vector prediction (SbTMVP) VVC supports a sub-block based temporal motion vector prediction (SbTMVP) method. Similar to the temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses the motion field in the co-located picture to improve the motion vector prediction and merge mode for the CU in the current picture. The same co-located picture used by TMVP is used for SbTMVP. SbTMVP differs from TMVP in the following two main aspects: -TMVP predicts motion at the CU level, but SbTMVP predicts motion at the sub-CU level; -Whereas TMVP takes the temporal motion vector from the co-located block in the co-located picture (the co-located block is the bottom-right or center block relative to the current CU), SbTMVP applies a motion shift before taking the temporal motion information from the co-located picture, where the motion shift is obtained from the motion vector from one of the spatial neighboring blocks of the current CU. The SbTMVP process is shown in Fig.18a and Fig.18b middle. Fig.18a Schematic diagram 1110 of spatial neighboring blocks used by SbTMVP is illustrated. SbTMVP predicts the motion vector of a sub-CU within the current CU in two steps. In the first step, Fig.18a The spatial neighbor A1 in is checked. If A1 has a motion vector that uses the co-located picture as its reference picture, this motion vector is selected as the motion shift to be applied. If no such motion is identified, the motion offset is set to (0, 0). Fig.18bA schematic diagram of deriving a sub-CU motion field by applying motion shifts from spatial neighbors and scaling the motion information from the corresponding co-located sub-CU is illustrated. In a second step, the motion shift identified in step 1 is applied (i.e., added to the coordinates of the current block) to obtain sub-CU level motion information (motion vectors and reference indices) from the co-located picture as shown in FIG18 . Fig.18b The example in assumes that the motion shift is set to the motion of block A1. Then, for each sub-CU, the motion information of its corresponding block in the co-located picture (the minimum motion grid covering the center sample) is used to derive the motion information of the sub-CU. After the motion information of the co-located sub-CU is identified, it is converted into a motion vector and reference index of the current sub-CU in a manner similar to the TMVP process of HEVC, where temporal motion scaling is applied to align the reference picture of the temporal motion vector with the reference picture of the current CU. In VVC, a sub-block based merge list containing a combination of both SbTMVP candidates and affine merge candidates is used for signaling of sub-block based merge mode. The SbTMVP mode is enabled / disabled by a sequence parameter set (SPS) flag. If the SbTMVP mode is enabled, the SbTMVP predictor is added as the first entry in the list of sub-block based merge candidates, followed by the affine merge candidates. The size of the sub-block based merge list is signaled in the SPS, and the maximum allowed size of the sub-block based merge list is 5 in VVC. The sub-CU size used in SbTMVP is fixed to 8×8, and as done for affine merge mode, SbTMVP mode is only applicable to CUs whose width and height are both greater than or equal to 8. The encoding logic of the additional SbTMVP merge candidates is the same as that for other merge candidates, that is, for each CU in a P slice or a B slice, an additional RD check is performed to decide whether to use the SbTMVP candidate. 2.1.5. Adaptive Motion Vector Resolution (AMVR) In HEVC, when use_integer_mv_flag is equal to 0 in the slice header, the motion vector difference (MVD) (between the CU's motion vector and the predicted motion vector) is signaled in units of quarter luma samples. In VVC, the CU-level adaptive motion vector resolution (AMVR) scheme is introduced. AMVR allows the MVD of a CU to be encoded and decoded with different precisions. Depending on the mode of the current CU (normal AMVP mode or affine AVMP mode), the MVD of the current CU can be adaptively selected as follows: - Normal AMVP mode: quarter brightness samples, half brightness samples, integer brightness samples, or quad brightness samples. - Affine AMVP mode: quarter luma samples, integer luma samples, or 1 / 16 luma samples. If the current CU has at least one non-zero MVD component, the CU-level MVD resolution indication is conditionally signaled.If all MVD components (i.e., both horizontal and vertical MVD for reference list L0 and reference list L1) are zero, then quarter luma sample MVD resolution is inferred. For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter luma sample MVD precision is used for the CU. If the first flag is 0, no further signaling is required and quarter luma sample MVD precision is used for the current CU. Otherwise, a second flag is signaled to indicate that half luma samples or other MVD precision (integer or four luma samples) are used for a normal AMVP CU. In the case of half luma samples, a 6-tap interpolation filter is used for half luma sample positions instead of the default 8-tap interpolation filter. Otherwise, a third flag is signaled to indicate whether integer luma sample MVD precision or four luma sample MVD precision is used for a normal AMVP CU. In the case of an affine AMVP CU, a second flag is used to indicate whether integer luma sample MVD precision or 1 / 16 luma sample MVD precision is used. In order to ensure that the reconstructed MV has a predetermined precision (quarter luma sample, half luma sample, integer luma sample, or four luma samples), the motion vector predictor for the CU will be rounded to the same precision as the MVD before being added to the MVD. The motion vector predictor is rounded towards zero (that is, a negative motion vector predictor is rounded towards positive infinity, and a positive motion vector predictor is rounded towards negative infinity). The encoder uses the RD check to determine the motion vector resolution of the current CU. In order to avoid always performing four CU-level RD checks for each MVD resolution, in VTM11, the RD check of MVD precision other than quarter luma samples is only conditionally called. For normal AVMP mode, the RD cost of quarter luma sample MVD precision and integer luma sample MV precision is first calculated. Then, the RD cost of integer luma sample MVD precision is compared with the RD cost of quarter luma sample MVD precision to decide whether it is necessary to further check the RD cost of four luma sample MVD precision. When the RD cost of quarter luma sample MVD precision is much smaller than the RD cost of integer luma sample MVD precision, the RD check of four luma sample MVD precision is skipped. Then, if the RD cost of integer luma sample MVD precision is significantly greater than the best RD cost of the previously tested MVD precision, the check of half luma sample MVD precision is skipped. For the affine AMVP mode, if the affine inter mode is not selected after checking the rate-distortion cost of the affine merge / skip mode, merge / skip mode, quarter luma sample MVD accuracy normal AMVP mode, and quarter luma sample MVD accuracy affine AMVP mode, the 1 / 16 luma sample MV accuracy and 1-pixel MV accuracy affine inter modes are not checked. In addition, the affine parameters obtained in the quarter luma sample MV accuracy affine inter mode are used as the starting search points in the 1 / 16 luma sample and quarter luma sample MV accuracy affine inter modes. 2.1.6. Bidirectional Prediction with CU-Level Weights (BCW) In HEVC, a bidirectional prediction signal is generated by averaging two prediction signals obtained from two different reference pictures and / or using two different motion vectors. In VVC, the bidirectional prediction mode is extended beyond simple averaging to allow weighted averaging of the two prediction signals. P bi-pred =((8-w)*P0+w*P1+4)>>3 (2-18). Five weights are allowed in weighted average bidirectional prediction, w∈{-2, 3, 4, 5, 10}. For each bidirectionally predicted CU, the weight w is determined in one of two ways: 1) For non-merged CUs, the weight index is signaled after the motion vector difference; 2) For merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is only applicable to CUs with 256 or more luma samples (i.e., CU width times CU height is greater than or equal to 256). For low-latency pictures, all 5 weights are used. For non-low-latency pictures, only 3 weights (w∈{3, 4, 5}) are used. - At the encoder, a fast search algorithm is applied to find the weight index without significantly increasing the encoder complexity. These algorithms are summarized below. For further details, the reader is referred to the VTM software and the document JVET-L0646. When combined with AMVR, if the current picture is a low-delay picture, the unequal weights are only conditionally checked for 1-pixel and 4-pixel motion vector accuracy. - When combined with affine, affine ME will be performed for unequal weights if and only if the affine mode is selected as the current best mode. - Unequal weights are only checked conditionally when the two reference pictures in bidirectional prediction are the same. - Do not search for unequal weights when certain conditions are met, which depend on the POC distance between the current picture and its reference pictures, the codec QP, and the temporal level. The BCW weight index is encoded using one context encoded bit followed by a bypass encoded bit. The first context encoded bit indicates whether equal weights are used; and if unequal weights are used, an additional bit is signaled using bypass codec to indicate which unequal weights are used. Weighted prediction (WP) is a codec tool supported by the H.264 / AVC and HEVC standards to efficiently encode and decode video content in fading situations. Support for WP has also been added to the VVC standard. WP allows weighting parameters (weights and offsets) to be signaled for each reference picture in each of the reference picture lists L0 and L1. Then, during motion compensation, the corresponding (one or more) weights and (one or more) offsets of the reference picture(s) are applied. WP and BCW are designed for different types of video content. In order to avoid the interaction between WP and BCW (which will complicate the VVC decoder design), if the CU uses WP, the BCW weight index is not signaled, and w is inferred to be 4 (i.e., equal weights are applied). For the merged CU, the weight index is inferred from the neighboring blocks based on the merge candidate index. This can be applied to both the normal merge mode and the inherited affine merge mode. For the constructed affine merge mode, affine motion information is constructed based on the motion information of up to 3 blocks. The BCW index of the CU using the constructed affine merge mode is simply set equal to the BCW index of the first control point MV. In VVC, CIIP and BCW cannot be applied jointly to a CU. When encoding and decoding a CU in CIIP mode, the BCW index of the current CU is set to 2, for example, equal weight. 2.1.7. Bidirectional Optical Flow (BDOF) The Bidirectional Optical Flow (BDOF) tool is included in VVC. BDOF (formerly known as BIO) is included in JEM. Compared to the JEM version, BDOF in VVC is a simpler version that requires much less computation, especially in terms of the number of multiplications and multiplier size. BDOF is used to refine the bidirectional prediction signal of a CU at the 4×4 sub-block level. If BDOF meets all the following conditions, it is applicable to a CU: - The CU is encoded and decoded using the "true" bidirectional prediction mode, that is, one of the two reference pictures is before the current picture in the display order, and the other reference picture is after the current picture in the display order; - The distances from the two reference pictures to the current picture (i.e., POC differences) are the same; - Both reference images are short-term reference images; -CU does not use affine mode or SbTMVP merge mode for encoding and decoding; -CU has more than 64 luminance samples; -CU height and CU width are both greater than or equal to 8 luma samples; -BCW weight index indicates equal weight; -WP is not enabled for the current CU; -CIIP mode is not used for the current CU. BDOF is only applicable to the luma component. As its name indicates, the BDOF mode is based on the concept of optical flow, which assumes that the motion of objects is smooth. For each 4×4 sub-block, the motion refinement (v x , v y ) is calculated by minimizing the difference between the L0 prediction sample and the L1 prediction sample. Motion refinement is then used to adjust the bi-prediction sample values in the 4x4 sub-block. The following steps are applied in the BDOF process. First, the horizontal and vertical gradients of the two prediction signals are calculated by directly calculating the difference between two neighboring samples: and Right now Among them I (k) (i, j) is the sample value at coordinate (i, j) of the prediction signal in list k (k=0, 1), and shift1 is calculated based on the luma bit depth bitDepth as shift1=max(6, bitDepth-6). Then, the autocorrelations and cross-correlations of the gradients S1, S2, S3, S5, and S6 are calculated as in where Ω is a 6×6 window around the 4×4 sub-block, and n a and n b The values of are set equal to min(1, bitDepth-11) and min(4, bitDepth-8) respectively. Then, the motion refinement (v) is derived using the cross-correlation and autocorrelation terms as follows: x , v y ): in th′ BIO =2 max(5,BD-7) . is the round-down (floor) function, and Based on the motion refinement and gradients, the following adjustments are computed for each sample in the 4×4 sub-block: Finally, the BDOF samples of the CU are calculated by adjusting the bidirectional prediction samples as follows: pred BDOF (x, y) = (I (0) (x,y)+I (1) (x, y) + b(x, y) + o offset )>>shift (2-24). These values are chosen so that the multipliers in the BDOF process do not exceed 15 bits and the maximum bit width of the intermediate parameters in the BDOF process is kept within 32 bits. In order to derive the gradient value, some prediction samples I in the list k (k = 0, 1) outside the current CU boundary (k) (i, j) needs to be generated. Fig.19 FIG. 2 is a schematic diagram showing an extended CU region used in BDOF. Fig.19 As depicted in the schematic diagram 1900 of FIG. 1 , the BDOF in VVC uses an extended row / column around the boundary of the CU. In order to control the computational complexity of generating prediction samples outside the boundary, the extended area (in Fig.19 The prediction samples in 1910 are generated by directly taking the reference samples at nearby integer positions without interpolation (using the floor() operation on the coordinates), and the normal 8-head motion compensation interpolation filter is used to generate the CU (in Fig.19The predicted samples within the CU boundary are represented as 1920 in FIG. These extended sample values are only used in the gradient calculation. For the remaining steps in the BDOF process, if any samples and gradient values outside the CU boundary are needed, these sample values and gradient values are filled (i.e., repeated) from their nearest neighbors. When the width and / or height of a CU is greater than 16 luma samples, it will be split into sub-blocks with a width and / or height equal to 16 luma samples, and the sub-block boundaries are regarded as CU boundaries in the BDOF process. The maximum unit size for the BDOF process is limited to 16×16. For each sub-block, the BDOF process can be skipped. When the SAD between the initial L0 prediction sample and the L1 prediction sample is less than the threshold, the BDOF process is not applied to the sub-block. The threshold is set equal to (8*W*(H>>1)), where W indicates the sub-block width and H indicates the sub-block height. In order to avoid the additional complexity of the SAD calculation, the SAD between the initial L0 prediction sample and the L1 prediction sample calculated in the DVMR process is reused here. If BCW is enabled for the current block, that is, the BCW weight index indicates unequal weights, bidirectional optical flow is disabled. Similarly, if WP is enabled for the current block, that is, luma_weight_lx_flag is 1 for either of the two reference pictures, BDOF is also disabled. BDOF is also disabled when the CU is encoded or decoded in symmetric MVD mode or CIIP mode. 2.1.8. Decoder-side Motion Vector Refinement (DMVR) To improve the accuracy of the MV in merge mode, decoder-side motion vector refinement based on bilateral matching (BM) is applied in VVC. In bidirectional prediction operation, the refined MV is searched around the initial MV in reference picture list L0 and reference picture list L1. The BM method calculates the distortion between two candidate blocks in reference picture list L0 and list L1. Fig. 20 FIG. 4 is a schematic diagram showing the refinement of the motion vector at the decoding side. Fig. 20 As shown, based on each MV candidate around the initial MV, the SAD between block 2010 and block 2012 is calculated, where for the current picture 2002, block 2010 is in the reference picture 2001 in list L0 and block 2012 is in the reference picture 2003 in list L1. The MV candidate with the lowest SAD becomes the refined MV and is used to generate a bidirectional prediction signal. In VVC, the application of DMVR is restricted and is only applied to CUs of the following modes and feature codecs: - CU level merge mode with bi-predictive MV; - relative to the current picture, one reference picture is in the past and the other reference picture is in the future; - The distances from the two reference pictures to the current picture (i.e., POC differences) are the same; - Both reference images are short-term reference images; -CU has more than 64 luminance samples; -CU height and CU width are both greater than or equal to 8 luma samples; -BCW weight index indicates equal weight; - WP is not enabled for the current block; -CIIP mode is not used for the current block. The refined MV derived by the DMVR process is used to generate inter prediction samples and also used in temporal motion vector prediction for future picture coding, whereas the original MV is used in the deblocking process and also used in spatial motion vector prediction for future CU coding. Additional features of DMVR are mentioned in the following sub-clauses. 2.1.8.1. Search scheme In DVMR, the search point is around the initial MV, and the MV offset obeys the MV difference mirror rule. In other words, any point that passes the DMVR check represented by the candidate MV pair (MV0, MV1) obeys the following two equations: MV0′=MV0+MV_offset (2-25) MV1′=MV1-MV_offset (2-26) Where MV_offset represents the refinement offset between the initial MV and the refined MV in one of the reference pictures. The refinement search range is two integer luma samples from the initial MV. The search includes an integer sample offset search phase and a fractional sample refinement phase. A 25-point full search is applied to the integer sample offset search. The SAD of the initial MV pair is first calculated. If the SAD of the initial MV pair is less than the threshold, the integer sampling stage of DMVR is terminated. Otherwise, the SAD of the remaining 24 points is calculated and checked in raster scan order. The point with the smallest SAD is selected as the output of the integer sample offset search stage. In order to reduce the penalty of uncertainty in DMVR refinement, it is proposed to support the original MV during the DMVR process. The SAD between the reference blocks referenced by the initial MV candidates is reduced by 1 / 4 of the SAD value. The integer sample search is followed by fractional sample refinement. To save computational complexity, fractional sample refinement is derived by using the parameter error surface equation instead of an additional search with SAD comparison. Fractional sample refinement is conditionally called based on the output of the integer sample search stage. When the integer sample search stage terminates with a center with minimum SAD in the first iteration search or the second iteration search, fractional sample refinement is further applied. In the sub-pixel offset estimation based on the parametric error surface, the cost at the center position and the costs at the four neighboring positions from the center are used to fit a 2D parabolic error surface equation of the following form E(x, y) = A(x min ) 2 +B(yy min ) 2 +C (2-27) Where (x min ,y min ) corresponds to the fractional position with the minimum cost, and C corresponds to the minimum cost value. By solving the above equation using the cost values of the five search points, (x min ,y min ) is calculated as: x min =(E(-1,0)-E(1,0)) / (2(E(-1,0)+E(1,0)-2E(0,0))) (2-28) y min =(E(0,-1)-E(0,1)) / (2((E(0,-1)+E(0,1)-2E(0,0))) (2-29). x min and min The value of is automatically limited to between -8 and 8, since all cost values are positive and the minimum is E(0,0). This corresponds to a half-pixel offset with 1 / 16th pixel MV precision in VVC. The calculated fraction (x min ,y min ) is added to the integer distance refinement MV to get the sub-pixel accurate refinement deltaMV. 2.1.8.2. Bilinear interpolation and sample padding In VVC, the resolution of MV is 1 / 16 luminance sample. An 8-tap interpolation filter is used to interpolate samples at fractional positions. In DMVR, the search point is around the initial fractional pixel MV with an integer sample offset, so the samples at these fractional positions need to be interpolated for the DMVR search process. In order to reduce the computational complexity, a bilinear interpolation filter is used to generate fractional samples for the search process in DMVR. Another important effect is that by using a bilinear filter, DVMR does not access more reference samples than the normal motion compensation process in the case of a 2-sample search range. After obtaining the refined MV using the DMVR search process, a normal 8-tap interpolation filter is applied to generate the final prediction. In order not to access more reference samples of the normal MC process, samples will be filled from those available samples, which are not required for the interpolation process based on the original MV, but are required for the interpolation process based on the refined MV. 2.1.8.3. Maximum DMVR processing unit When the width and / or height of a CU is greater than 16 luma samples, it will be further split into sub-blocks with width and / or height equal to 16 luma samples. The maximum unit size for the DMVR search process is limited to 16×16. 2.1.9. Combined Inter and Intra Prediction (CIIP) In VVC, when a CU is encoded and decoded in merge mode, if the CU contains at least 64 luma samples (that is, the CU width multiplied by the CU height is equal to or greater than 64), and if both the CU width and the CU height are less than 128 luma samples, an additional flag is signaled to indicate whether the combined inter- and intra-prediction (CIIP) mode is applicable to the current CU. As its name indicates, CIIP prediction combines the inter-prediction signal with the intra-prediction signal. The inter-prediction signal P in CIIP mode inter is derived using the same inter-prediction process as applied to the normal merge mode; and the intra-prediction signal P intra is derived following the conventional intra prediction process with planar mode. Fig.21 The top and left neighboring blocks used in the CIIP weight derivation are shown. The intra prediction signal and the inter prediction signal are then combined using a weighted average, where the weight values depend on the codec mode of the top and left neighboring blocks (in Fig.21 is calculated as follows: - if the top neighbor is available and intra-coded, set isIntraTop to 1, otherwise set isIntraTop to 0; - If the left neighbor is available and intra-coded, set isIntraLeft to 1, otherwise set isIntraLeft to 0; - If (isIntraLeft + isIntraTop) is equal to 2, then wt is set to 3; - Otherwise, if (isIntraLeft+isIntraTop) is equal to 1, wt is set to 2; - Otherwise, set wt to 1. The CIIP forecast is formed as follows: P CIIP =((4-wt)*P inter +wt*P intra +2)>>2 (2-30). 2.1.10. Geometric Partitioning Mode (GPM) In VVC, geometric partition mode is supported for inter prediction. The geometric partition mode is signaled using a CU level flag as a merge mode, where other merge modes include normal merge mode, MMVD mode, CIIP mode, and sub-block merge mode. For each possible CU size w×h=2 excluding 8×64 and 64×8 m ×2 n , where m,n∈{3…6}, a total of 64 partitions are supported through the geometric partitioning mode. Fig. 22 An example of a GPM split with equal angle grouping is shown. When this mode is used, the CU is split into two parts by geometrically positioned straight lines ( Fig. 22 ). The position of the split line is mathematically derived from the angle and offset parameters of the specific partition. Each part of the geometric partition in a CU uses its own motion for inter-frame prediction; each partition only allows unidirectional prediction, that is, each part has one motion vector and one reference index. Unidirectional prediction motion constraints are applied to ensure that only two motion compensated predictions are required per CU, the same as regular bidirectional prediction. If geometry partition mode is used for the current CU, a geometry partition index indicating the partition mode (angle and offset) of the geometry partition and two merge indices (one for each partition) are further signaled. The number of maximum GPM candidate sizes is explicitly signaled in the SPS and the syntax binarization for the GPM merge indices is specified. After predicting each part of the geometry partition, a hybrid process with adaptive weights is used to adjust the sample values along the edges of the geometry partitions. This is the prediction signal for the entire CU and the transform and quantization processes will be applied to the entire CU as in other prediction modes. Finally, the motion field of the CU predicted using the geometry partition mode is stored. 2.1.10.1. One-way prediction candidate list construction The unidirectional prediction candidate list is derived directly from the merge candidate list constructed according to the extended merge prediction process. Denote n as the index of the unidirectional prediction motion in the geometric unidirectional prediction candidate list. The LX motion vector of the nth extended merge candidate is used as the nth unidirectional prediction motion vector of the geometric partition mode, where X is equal to the parity of n. Fig.23 The unidirectional prediction MV selection for the geometric partition mode is shown. These motion vectors are Fig.23 In the case where the corresponding LX motion vector of the n-th extended merge candidate does not exist, the L(1-X) motion vector of the same candidate is used as the unidirectional prediction motion vector of the geometric partition mode instead. 2.1.10.2. Blending Along Geometric Partition Edges After predicting each part of the geometric partition using its own motion, blending is applied to the two prediction signals to derive samples around the geometric partition edges. The blending weights for each position of the CU are derived based on the distance between the individual position and the partition edge. The distance from a position (x, y) to the edge of the partition is derived as: where i, j are the angle and offset indices of the geometry partition, which depend on the geometry partition index sent with the signal. x,j and ρ y,j The sign of depends on the angle index i. The weights for each part of the geometric partition are derived as follows: wIdxL(x,y)=partIdx? 32+d(x,y):32-d(x,y) w1(x, y) = 1 - w0(x, y) partIdx depends on the angle index i. Fig.24 An example of the generation of a warp weight w0 using a geometric partitioning pattern is shown. An example of the weight w0 is illustrated in Fig.24 middle. 2.1.10.3. Motion Field Storage for Geometry Partitioning Mode Mv1 from the first part of the geometry partition, Mv2 from the second part of the geometry partition, and the combination Mv of Mv1 and Mv2 are stored in the motion field of the CU encoded in the geometry partition mode. The type of motion vector for each individual position stored in the motion field is determined as: sType=abs(motionIdx)<32?2: (motionIdx≤0?(1-partIdx):partIdx) where motionIdx is equal to d(4x+2, 4y+2). partIdx depends on the angle index i. If sType is equal to 0 or 1, then Mv0 or Mv1 is stored in the corresponding motion field, otherwise if sTType is equal to 2, the combined Mv from Mv0 and Mv2 is stored. The combined Mv is generated using the following process: 1) If Mv1 and Mv2 are from different reference picture lists (one from L0 and the other from L1), then Mv1 and Mv2 are simply combined to form a bi-directional prediction motion vector. 2) Otherwise, if Mv1 and Mv2 are from the same list, only the unidirectional predicted motion Mv2 is stored. 2.1.11. Local Illumination Compensation (LIC) LIC is an inter-prediction technique used to model the local illumination variation between the current block and its prediction block as a function of the local illumination variation between the current block template and the reference block template. The parameters of the function can be represented by a scale α and an offset β, which form a linear equation, i.e., α*p[x]+β to compensate for illumination variation, where p[x] is the reference sample pointed to by the MV at position x on the reference picture. Since α and β can be derived based on the current block template and the reference block template, they do not require signaling overhead except for signaling a LIC flag for AMVP mode to indicate the use of LIC. The local illumination compensation proposed in JVET-O0066 is used for unidirectionally predicted inter CUs with the following modifications. Intra-frame neighbor samples can be used in LIC parameter derivation; Disable LIC for blocks with less than 32 luma samples; For both non-sub-block mode and affine mode, LIC parameter derivation is performed based on the template block samples corresponding to the current CU instead of the partial template block samples corresponding to the top-left first 16×16 unit; • Samples of the reference block template are generated by using MC with the block MV without rounding it to integer pixel precision. 2.1.12. Non-adjacent spatial candidates Non-adjacent spatial merging candidates as in JVET-L0399 are inserted after the TMVP in the regular merging candidate list. Fig.25 The spatial neighboring blocks used to derive the spatial merge candidates are shown in FIG. Fig.25 The distance between non-adjacent spatial candidates and the current codec block is based on the width and height of the current codec block. No line buffer limit is applied. 2.1.13. Template Matching (TM) Template Matching (TM) is a decoder-side MV derivation method used to refine the motion information of the current CU by finding the closest match between a template in the current picture (i.e., the top and / or left neighboring blocks of the current CU) and a block in a reference picture (i.e., the same size as the template). Fig.26 2600 is a diagram illustrating template matching performed on a search area around an initial MV. Fig.26 As shown, a better MV is searched around the initial motion of the current CU in the [-8, +8] pixel search range. Template matching previously proposed in JVET-J0021 is adopted in this contribution with two modifications: the search step size is determined based on the adaptive motion vector resolution (AMVR) mode and the TM can be cascaded with the bilateral matching process in the merge mode. In AMVP mode, MVP candidates are determined based on template matching error to select the one that achieves the minimum difference between the current block template and the reference block template, and then TM is performed only for this specific MVP candidate for MV refinement. TM refines this MVP candidate starting with full-pixel MVD accuracy (or 4-pixel MVD accuracy for 4-pixel AMVR mode) within a [-8, +8] pixel search range using an iterative diamond search. The AMVP candidate can be further refined by using a cross search with full-pixel MVD accuracy (or 4-pixel MVD accuracy for 4-pixel AMVR mode), and then further refined by using half-pixel and quarter-pixel MVD accuracy depending on the AMVR mode as specified in Table 3. This search process ensures that the MVP candidate still maintains the same MV accuracy as indicated by the AMVR mode after the TM process. Table 3 Search model of AMVR and merge mode with AMVR. In merge mode, a similar search method is applied to the merge candidates indicated by the merge index. As shown in Table 3, TM can be performed until 1 / 8 pixel MVD accuracy or skip those beyond half-pixel MVD accuracy, depending on whether an alternative interpolation filter is used according to the merge motion information (used when AMVR is half-pixel mode). In addition, when TM mode is enabled, template matching can work as an independent process or an additional MV refinement process between a block-based bilateral matching (BM) method and a sub-block-based BM method, depending on whether BM can be enabled according to its enabling condition check. 2.1.14. Multi-pass decoder-side motion vector refinement (mpDMVR) Multiple passes of decoder-side motion vector refinement are applied. In the first pass, bilateral matching (BM) is applied to the codec block. In the second pass, BM is applied to each 16×16 sub-block within the codec block. In the third pass, the MVs in each 8×8 sub-block are refined by applying bidirectional optical flow (BDOF). The refined MVs are stored for spatial and temporal motion vector prediction. 2.1.14.1. First pass - Block-based bilateral matching MV refinement In the first pass, refined MVs are derived by applying BM to the codec blocks. Similar to decoder side motion vector refinement (DMVR), in bidirectional prediction operation, refined MVs are searched around two initial MVs (MV0 and MV1) in reference picture lists L0 and L1. Refined MVs (MV0_pass1 and MV1_pass1) are derived around the initial MVs based on the minimum bilateral matching cost between the two reference blocks in L0 and L1. BM performs a local search to derive integer sample precision intDeltaMV. The local search applies a 3×3 square search pattern to loop through the search range [-sHor, sHor] in the horizontal direction and the search range [-sVer, sVer] in the vertical direction, where the values of sHor and sVer are determined by the block dimension and the maximum value of sHor and sVer is 8. The bilateral matching cost is calculated as: bilCost = mvDistanceCost + SadCost. When the block size cbW*cbH is greater than 64, the MRSAD cost function is applied to remove the DC effect of the distortion between the reference blocks. When the bilCost at the center point of the 3×3 search pattern has the minimum cost, the intDeltaMV local search is terminated. Otherwise, the current minimum cost search point becomes the new center point of the 3×3 search pattern and continues to search for the minimum cost until it reaches the end of the search range. The existing fractional sample refinement is further applied to derive the final deltaMV. The refined MV after the first pass is then exported as: MV0_pass1 = MV0 + deltaMV; MV1_pass1=MV1-deltaMV. 2.1.14.2. Second pass - sub-block based bilateral matching MV refinement In the second pass, refined MVs are derived by applying BM to 16×16 grid sub-blocks. For each sub-block, the refined MVs are searched around the two MVs (MV0_pass1 and MV1_pass1) obtained at the first pass in the reference picture lists L0 and L1. The refined MVs (MV0_pass2(sbIdx2) and MV1_pass2(sbIdx2)) are derived based on the minimum bilateral matching cost between the two reference sub-blocks in L0 and L1. For each subblock, BM performs a full search to derive integer sample precision intDeltaMV. The full search has a search range of [-sHor, sHor] in the horizontal direction and a search range of [-sVer, sVer] in the vertical direction, where the values of sHor and sVer are determined by the block dimension and the maximum value of sHor and sVer is 8. The bilateral matching cost is calculated by applying the cost factor to the SATD cost between the two reference sub-blocks as: bilCost = satdCost * costFactor. The search area (2*sHor+1)*(2*sVer+1) is divided into Fig. 27Up to 5 diamond-shaped search regions are shown in FIG. 2700. Each search region is assigned a costFactor, which is determined by the distance (intDeltaMV) between each search point and the starting MV, and each diamond region is processed sequentially starting from the center of the search region. In each region, the search points are processed in a raster scan order starting from the upper left corner to the lower right corner of the region. When the minimum bilCost in the current search region is less than or equal to the threshold of sbW*sbH, the integer pixel full search is terminated, otherwise, the integer pixel full search continues to the next search region until all search points are checked. The existing VVC DMVR fractional sample refinement is further applied to derive the final deltaMV (sbIdx2). The refined MV in the second pass is then derived as: ·MV0_pass2(sbIdx2)=MV0_pass1+deltaMV(sbIdx2), ·MV1_pass2(sbIdx2)=MV1_pass1-deltaMV(sbIdx2). 2.1.14.3. Third pass - sub-block based bidirectional optical flow MV refinement In the third pass, refined MVs are derived by applying BDOF to 8×8 grid sub-blocks. For each 8×8 sub-block, BDOF refinement is applied to derive scaled Vx and Vy without clipping from the refined MV of the parent sub-block of the second pass. The derived bioMv(Vx, Vy) is rounded to 1 / 16 sample precision and clipped between -32 and 32. The refined MVs in the third pass (MV0_pass3(sbIdx3) and MV1_pass3(sbIdx3)) are derived as: ·MV0_pass3(sbIdx3)=MV0_pass2(sbIdx2)+bioMv, ·MV1_pass3(sbIdx3)=MV0_pass2(sbIdx2)-BioMv. 2.1.15.OBMC When OBMC is applied, the top and left boundary pixels of the CU are refined using motion information of neighboring blocks in a weighted prediction as described in JVET-L0101. The conditions under which OBMC should not be applied are as follows: When OBMC is disabled at SPS level; When the current block has intra mode or IBC mode; When the current block applies LIC; When the current luminance block area is less than or equal to 32. Sub-block boundary OBMC is performed by applying the same blending to the top, left, bottom and right sub-block boundary pixels using the motion information of the neighboring sub-blocks. It is enabled for sub-block based codecs: Affine AMVP mode; Affine merge mode and sub-block based temporal motion vector prediction (SbTMVP); Sub-block based bilateral matching. 2.1.16. Sample-based BDOF In sample-based BDOF, instead of deriving the motion refinement (Vx, Vy) on a block basis, it is performed per sample. The codec block is divided into 8×8 sub-blocks. For each sub-block, whether to apply BDOF is determined by checking the SAD between two reference sub-blocks against a threshold. If it is decided to apply BDOF to a sub-block, for each sample in the sub-block, a sliding 5×5 window is used and the existing BDOF process is applied for each sliding window to derive Vx and Vy. The derived motion refinement (Vx, Vy) is applied to adjust the bidirectional prediction sample value of the center sample of the window. 2.1.17. Interpolation The 8-tap interpolation filter used in VVC is replaced with a 12-tap filter. The interpolation filter is derived from a sinc function whose frequency response is cut off at the Nyquist frequency and is clipped by a cosine window function. Table 4 gives the filter coefficients for all 16 phases. Fig.28 The frequency responses of the interpolation filter and the VVC interpolation filter at half-pixel phase are shown. It compares the frequency responses of the interpolation filter and the VVC interpolation filter all at half-pixel phase. Table 4. Filter coefficients for 12-tap interpolation filter 2.1.18. Multiple Hypothesis Prediction (MHP) In the multi-hypothesis inter prediction mode (JVET-M0425), one or more additional motion compensated prediction signals are signaled in addition to the conventional bidirectional prediction signal. The resulting overall prediction signal is obtained by sample-by-sample weighted superposition. bi and the first additional inter-frame prediction signal / hypothesis h3, the resulting prediction signal p3 is obtained as follows: p3=(1-α)p bi +αh3. The weight factor α is specified by a new syntax element add_hyp_weight_idx according to the following mapping. add_hyp_weight_idx α 0 1 / 4 1 -1 / 8 Similar to above, more than one additional prediction signal may be used.The resulting overall prediction signal is iteratively accumulated with each additional prediction signal. p n+1 =(1-α n+1 ) n +α n+1 h n+1 Get the overall prediction signal as the last p n (i.e., the p with the largest index n n n). Within this EE, at most two additional prediction signals may be used (i.e., n is limited to 2). The motion parameters for each additional prediction hypothesis can be signaled explicitly by specifying the reference index, motion vector predictor index and motion vector difference, or implicitly by specifying the merge index. A separate multiple hypothesis merge flag distinguishes these two signaling modes. For inter-AMVP mode, MHP is applied only when unequal weights in BCW are selected in bi-prediction mode. A combination of MHP and BDOF is possible, however BDOF is only applicable to the bidirectional prediction signal part of the prediction signal (ie the original first two hypotheses). 2.1.19. Adaptive Re-ranking of Merging Candidates by Template Matching (ARMC-TM) The merge candidates are adaptively reordered by Template Matching (TM). The reordering method is applicable to the regular merge mode, Template Matching (TM) merge mode and Affine merge mode (excluding SbTMVP candidates). For TM merge mode, the merge candidates are reordered before the refinement process. After constructing the merge candidate list, the merge candidates are divided into several subgroups. For the normal merge mode and the TM merge mode, the subgroup size is set to 5. For the affine merge mode, the subgroup size is set to 3. The merge candidates in each subgroup are reordered in ascending order according to the cost value based on template matching. For simplicity, the merge candidates in the last subgroup instead of the first subgroup are not reordered. The template matching cost of the merge candidate is measured by the sum of absolute differences (SAD) between the samples of the template of the current block and its corresponding reference samples. The template includes a set of reconstructed samples adjacent to the current block. The reference samples of the template are located by the motion information of the merge candidate. Fig.29 A schematic diagram 2900 is shown of a template and reference samples of the template in reference list 0 and reference list 1. When the merge candidate uses bidirectional prediction, the reference samples of the template of the merge candidate are also Fig.29As shown, the template of the merge candidate is generated by bidirectional prediction. When the merge candidate utilizes bidirectional prediction, the reference sample of the template of the merge candidate is represented by RT, and RT may be generated by RT0 derived from the reference picture 2920 in the reference picture list 0 and RT1 derived from the reference picture 2930 in the reference picture list 1. In one example, RT0 includes a set of reference samples indicated by the reference index of the merge candidate of the reference picture in the reference list 0 on the reference picture 2920 of the current block in the current picture 2910 with the MV of the merge candidate of the reference list 0. In one example, RT1 includes a set of reference samples indicated by the reference index of the merge candidate of the reference picture in the reference list 1 on the reference picture 2930 of the current block with the MV of the merge candidate of the reference list 1. For a sub-block based merge candidate with a sub-block size equal to Wsub×Hsub, the above template includes several sub-templates with a size of Wsub×1, and the left template includes several sub-templates with a size of 1×Hsub. Fig.30 A template and a reference sample of the template for a block with sub-block motion using motion information of a sub-block of a current block are shown. Fig.30 As shown, the motion information of the sub-blocks in the first row and first column of the current block is used to derive the reference samples of each sub-template. 2.1.20. Geometric Partitioning Mode (GPM) with Merged Motion Vector Difference (MMVD) GPM in VVC is extended by applying motion vector refinement on top of the existing GPM unidirectional MV. A flag is first signaled for the GPMCU to specify whether to use this mode. If this mode is used, each geometric partition of the GPM CU can further decide whether to signal MVD. If MVD is signaled for a geometric partition, the motion of the partition is further refined by the signaled MVD information after selecting the GPM merge candidate. All other processes remain the same as in GPM. The MVD is signaled as a distance and direction pair, similar to that in MMVD. There are nine candidate distances (1 / 4 pixel, 1 / 2 pixel, 1 pixel, 2 pixels, 3 pixels, 4 pixels, 6 pixels, 8 pixels, 16 pixels) and eight candidate directions (four horizontal / vertical directions and four diagonal directions) involved in GPM with MMVD (GPM-MMVD). In addition, when pic_fpel_mmvd_enabled_flag is equal to 1, the MVD is left shifted by 2 as in MMVD. 2.1.21. Geometric Partitioning Pattern (GPM) with Template Matching (TM) Template matching is applied to GPM. When GPM mode is enabled for a CU, a CU level flag is signaled to indicate whether TM is applied to both geometry partitions. The motion information of each geometry partition is refined using TM. When TM is chosen, the template is constructed using the left, top, or left and top neighboring samples depending on the partition angle, as shown in Table 5. The motion is then refined by minimizing the difference between the current template and the template in the reference picture using the same search pattern of merge mode with half-pixel interpolation filter disabled. Table 5 Templates for the 1st and 2nd geometric partitions, where A means using the top sample, L means using the left sample, and L+A means using both the left and top samples. Partition Angle 0 2 3 4 5 8 11 12 13 14 First Division A A A A L+A L+A L+A L+A A A Second Division L+A L+A L+A L L L L L+A L+A L+A Partition Angle 16 18 19 20 21 24 27 28 29 30 First Division A A A A L+A L+A L+A L+A A A Second Division L+A L+A L+A L L L L L+A L+A L+A The GPM candidate list is constructed as follows: 1. Interleaved List-0 MV candidates and List-1 MV candidates are directly derived from the regular merge candidate list, where List-0 MV candidates have higher priority than List-1 MV candidates. A pruning method with adaptive threshold based on the current CU size is applied to remove redundant MV candidates. 2. Interleaved List-1MV candidates and List-0 MV candidates are further derived directly from the regular merge candidate list, where List-1MV candidates have higher priority than List-0 MV4 candidates. The same pruning method with adaptive threshold is also applied to remove redundant MV candidates. 3. Zero MV candidates are filled until the GPM candidate list is full. GPM-MMVD and GPM-TM are exclusively enabled for one GPM CU. This is done by first signaling the GPM-MMVD syntax. When both GPM-MMVD control flags are equal to false (i.e., GPM-MMVD is disabled for both GPM partitions), the GPM-TM flag is signaled to indicate whether template matching is applied to both GPM partitions. Otherwise (at least one GPM-MMVD flag is equal to true), the value of the GPM-TM flag is inferred to be false. 2.1.22. GPM with inter-frame and intra-frame prediction (GPM inter-intra) With GPM inter-intra, in addition to merging candidates for each non-rectangular split area in the CU to which GPM is applied, predefined intra prediction modes along the geometric partition lines can also be selected. In the proposed method, a flag from the encoder is used to determine whether the intra prediction mode or the inter prediction mode is for each GPM separation area. When it is inter prediction mode, a unidirectional prediction signal is generated from the MV from the merged candidate list. On the other hand, when it is intra prediction mode, the unidirectional prediction signal is generated from the neighboring pixels for the intra prediction mode specified by the index from the encoder. The variation of possible intra prediction modes is subject to geometric constraints. Finally, the two unidirectional prediction signals are mixed in the same way as ordinary GPM. 2.1.23. Adaptive decoder-side motion vector refinement (Adaptive DMVR) The adaptive decoder-side motion vector refinement method includes two new merge modes introduced to refine the MV in one direction (L0 or L1) of bidirectional prediction for merge candidates that meet the DMVR conditions. A multi-pass DMVR process is applied to the selected merge candidate to refine the motion vector, however MVD0 or MVD1 is set to zero in the first pass (i.e. PU level) DMVR. Like the regular merge mode, the merge candidates for the proposed merge mode are derived from spatially neighboring coded blocks, TMVP, non-neighboring blocks, HMVP, and paired candidates. The difference is that only those merge candidates that satisfy the DMVR condition are added to the candidate list. Both proposed merge modes use the same merge candidate list, and the merge index is encoded and decoded as in the regular merge mode. 2.1.24. Bilateral Matching AMVP-MERGE Mode (AMVP-MERGE) In AMVP merge mode, a bidirectional predictor consists of an AMVP predictor in one direction and a merged predictor in the other direction. The AMVP part of the proposed mode is signaled as regular unidirectional AMVP, i.e., the reference index and MVD are signaled, and it has a derived MVP index if template matching is used (TM_AMVP), or the MVP index is signaled when template matching is disabled. The merge index is not signaled, and the merged predictor is selected from the candidate list with the minimum template or bilateral matching cost. When the selected merged predictor and AMVP predictor meet the DMVR condition, that is, there is at least one reference picture from the past and one reference picture from the future relative to the current picture, and the distances from the two reference pictures to the current picture are the same, bilateral matching MV refinement is applied with the merged MV candidate and AMVP MVP as the starting point. Otherwise, if the template matching function is enabled, template matching MV refinement is applied to the merged predictor or AMVP predictor with a higher template matching cost. The third pass of 8×8 sub-PU BDOF refinement as multi-pass DMVR is enabled for the AMVP merge mode codec block. 2.2. Out-of-bounds problem in bidirectional motion compensation In ECM-3.0, due to the reference sample filling of the reference picture, for inter-frame CU Fig.31 The illustrated reference blocks are valid if they are partially or fully located outside the reference picture. Fig.31 The bidirectional MC block in the current ECM is shown. Fig.31 In ECM, bidirectional motion compensation is performed to generate an inter-frame prediction block for the current block. In this example, the list 0 reference block is partially outside the boundary (OOB), while the list 1 reference block is completely inside the reference picture. However, the OOB portion of the motion compensated block generally provides lower prediction efficiency because the OOB portion is just repeated samples derived from boundary samples within the reference picture. However, in ECM, the lower efficiency of the OOB portion is not considered for inter-frame prediction. 3. Combined inter-frame intra-frame prediction and transform Interaction between Adaptive DMVR and other tools 1. In one example, Adaptive DMVR can be applied to other codecs besides Adaptive DMVR itself. a. For example, adaptive DMVR may refer to a DMVR method that fixes a motion vector in one prediction direction (such as LX) and then refines the motion vector in another direction (such as L(1-X)), where the motion vector is predicted bidirectionally. b. For example, the motion vectors in the merge candidate list can be further refined by adaptive DMVR. c. For example, the motion vectors of CIIP can be further refined by adaptive DMVR. d. For example, the motion vectors of GPM can be further refined by adaptive DMVR. e. For example, the motion vectors of MMVD can be further refined by adaptive DMVR. f. For example, the motion vectors of sub-block merging (eg, affine merging, sub-block TMVP, etc.) can be further refined by adaptive DMVR. g. For example, the motion vectors between AMVP frames can be further refined by adaptive DMVR. h. For example, the motion vectors of SMVD can be further refined by adaptive DMVR. i. For example, the motion vector of a sub-block AMVP (eg, affine AMVP, etc.) may be further refined by adaptive DMVR. j. For example, when adaptive DMVR is applied to other codecs, signaling of the use of adaptive DMVR for a block may be unnecessary. i. For example, if certain conditions are met, adaptive DMVR is applied to a block without signaling. 2. For example, whether adaptive DMVR or conventional DMVR (and / or BDMVR) is used at the video unit level may be signaled in the bitstream. a. For example, the video unit level can be the level of sequence / picture / slice / tile group / tile / sub-picture / PB / TB / CB / PU / TU / CU / VPDU / CTU / CTU row / other kinds of regions containing more than one sample or pixel. Interaction between AMVP-MERGE and other tools 3. In one example, AMVP-MERGE can be applied to other codecs besides AMVP-MERGE itself. a. For example, AMVP-MERGE may refer to an inter-frame coding method that generates a motion vector based on the LX motion of an AMVP candidate and the L(1-X) motion of a MERGE candidate. b. For example, the motion vector generated by AMVP-MERGE can be used in CIIP. c. For example, the motion vectors generated by AMVP-MERGE can be used in GPM. d. For example, the motion vectors generated by AMVP-MERGE can be used in MMVD. e. For example, the motion vector generated by AMVP-MERGE can be used in sub-block merging (eg, affine merging, sub-block TMVP, etc.). f. For example, the motion vectors generated by AMVP-MERGE can be used in GPM. g. For example, the motion vectors generated by AMVP-MERGE can be used in AMVP inter prediction (eg, as AMVP candidates for conventional AMVP inter coding). h. For example, the motion vector generated by AMVP-MERGE can be used in SMVD. i. For example, when the motion vector generated by AMVP-MERGE is used in other codecs, it can be perceived as a motion candidate. 4. Decoder side MHP data export V0 1. In one example, a hypothetical prediction may be generated based on virtually constructed motion data that is not identical to any motion data in the original MHP candidate list. a. For example, predictions for additional hypotheses may be generated based on virtually constructed motion data. b. For example, the movement of virtual constructs can be based on the AMVP-MERGE candidate list. i. For example, an AMVP-MERGE candidate list may be generated based on at least one type of AMVP motion candidate information and at least one type of MERGE motion candidate information. ii. For example, the AMVP-MERGE candidate list may be reordered. iii. For example, the motion candidates in the AMVP-MERGE motion candidate list may be refined. c. For example, the motion of the virtual construct can be based on the bidirectional virtual motion candidate. i. For example, bidirectional virtual motion candidates may be generated based on a certain AMVP candidate list. ii. For example, the bidirectional virtual motion candidates may be generated based on one of the following MERGE candidate lists: a regular merge candidate list, an MMVD candidate list, a TM merge list, a GEO merge list, a CIIP merge list, a sub-block merge list, etc. iii. For example, bidirectional virtual motion candidates may be generated based on the reordered motion candidate list. iv. For example, bidirectional virtual motion candidates may be further refined. d. For example, the motion of the virtual construct can be based on a unidirectional virtual motion candidate. i. For example, one-way virtual motion candidates may be generated based on a certain AMVP candidate list. ii. For example, the unidirectional virtual motion candidate may be generated based on one of the following MERGE candidate lists: a regular merge candidate list, an MMVD candidate list, a TM merge list, a GEO merge list, a CIIP merge list, a sub-block merge list, etc. iii. For example, a unidirectional virtual motion candidate may be generated based on the reordered motion candidate list. iv. For example, one-way virtual motion candidates can be further refined. e. For example, predictions of basic assumptions can be generated based on virtually constructed motion data. i. For example, an AMVP-MERGE codec block may be perceived as a base hypothesis, and at least one additional hypothesis may be applied thereto. Finally, a final prediction of the AMVP-MERGE codec block is generated by mixing the base hypothesis and the additional hypothesis together. 5. Question There are several problems in the existing video coding and decoding technology, which will be further improved to obtain higher coding and decoding gain. 1. In ECM-3.0, the existing AMVP-MERGE mode only supports 1 / 4 pixel precision MVD signaling, where AMVR is not supported. This can be further improved. 2. In ECM-3.0, AMVP-MERGE prediction cannot be used as one of the MHP assumptions, which may be suboptimal. 3. In ECM-3.0, the existing AMVP-MERGE mode generates a new prediction candidate list, which contains several bidirectional prediction motion candidates. Such motion candidates can be used in other coding modes to obtain higher coding efficiency. 4. In ECM-3.0, the existing adaptive DMVR mode generates a new prediction candidate list, which only contains bidirectional prediction motion candidates. Such motion candidates can be used in other codec modes to obtain higher codec efficiency. 5. In ECM-3.0, AMVP-MERGE mode uses DMVR and TM to determine (L0, L1) prediction candidate pairs and refine the prediction candidates. However, this codec tool is not controlled by the DMVR / TM / DMVR control flag, which can be further designed. 6. In ECM-3.0, the LIC mode is designed based on the assumption that there is a strong linear correlation between the neighboring samples of the current block and the temporally co-located block. More specifically, the LIC mode uses a least squares model with two parameters (such as a and b) to map the neighboring samples of the current block and the neighboring samples of the temporally co-located block. However, this assumption may not be obvious, especially when the neighbors and the reference are noisy. In such cases, the actual prediction may fail and the process results in suboptimal coding efficiency. 7. In ECM-3.0, the LIC for AMVP is signaled in association with a video unit, and the LIC for MERGE is inherited from an adjacent video unit. The signaling and determination of the LIC may be changed using a decoder-derived method. 8. In ECM-3.0, for bidirectional prediction, MHP prediction, GPM prediction, etc., a hypothetical prediction block is generated by weighting multiple hypothetical predictions. Due to repeated picture boundary filling, the weighting process is performed anyway, regardless of whether any prediction sample is out of bounds in any hypothetical prediction. However, further consideration can be given to how to deal with out-of-bounds prediction samples. 6. Embodiments of the present disclosure The following detailed implementation examples should be considered as examples to illustrate the general concept. These embodiments should not be interpreted narrowly. In addition, these embodiments can be combined in any way. The term “video unit” or “codec unit” or “block” may refer to a codec tree block (CTB), a codec tree unit (CTU), a codec block (CB), CU, PU, TU, PB, TB. In the present disclosure, regarding “encoding and decoding blocks in mode N”, “mode N” here can be a prediction mode (for example, MODE_INTRA, MODE_INTER, MODE_PLT, MODE_IBC, etc.) or a coding and decoding technology (for example, AMVP, Merge, SMVD, BDOF, PROF, DMVR, AMVR, TM, Affine, CIIP, GPM, GEO, TPM, MMVD, BCW, HMVP, SbTMVP, etc.). In the present disclosure, “bidirectional DMVR” may refer to conventional DMVR that refines both L0 motion vectors and L1 motion vectors, as described in detail in Section 2.1.14. In addition, “unidirectional DMVR” may refer to a DMVR process that refines only L0 motion vectors or L1 motion vectors, such as the adaptive DMVR described in detail in Section 2.1.23. In the following discussion, LIC parameters may refer to two parameters (such as slope parameter "a" and bias parameter "b") derived based on a linear model that is used to map neighboring samples of a current block and neighboring samples of a temporally co-located block (e.g., the temporally co-located block may be pointed to by a motion vector or a rounded motion vector of the current block). In addition, LIC parameters may be used to estimate prediction values of samples within the current video unit. In the following discussion, the AMVP mode may be a conventional AMVP mode, an affine AMVP mode, and / or an SMVD mode, and / or an AMVP-MERGE mode. Note that the terms mentioned below are not limited to the specific terms defined in existing standards. Any variants of the codec tool are also applicable. 6.1. To deal with the first problem, the following method is proposed: a. Propose to support multiple MVD / MV precisions for AMVP-MERGE mode. i. In one example, the supported precision candidates may be the same as those used for normal AMVP mode, e.g., for the non-affine case, half pixel, 1 / 4 pixel, 1 pixel, 4 pixels are applied; for the affine case, 1 / 16 pixel, 1 / 8 pixel, 1 / 4 pixel are applied. ii. In another example, at least one of the supported precision candidates may be different from the precision candidate used for normal AMVP mode. b. For example, the motion vector difference (eg, MVD) on the AMVP side of the AMVP-MERGE mode may be encoded and decoded with other precisions other than 1 / 4 pixel resolution. i. For example, the MVD value can be 4-pixel accurate. ii. For example, the MVD value can be 1 pixel accurate. iii. For example, the MVD value can be half-pixel accurate. iv. For example, the MVD value can be 1 / 8 pixel accurate. v. For example, the MVD value can be 1 / 16 pixel accurate. c. For example, a second interpolation filter (eg, a Y-tap filter such as Y=6) in addition to the first interpolation filter (eg, an X-tap filter such as X=8 or 12) may also be used for motion compensation. i. For example, when to use the second interpolation filter for an AMVP-MERGE codec block may depend on the MVD prediction and / or the final MV accuracy. ii. For example, when the signaled MVD is 1 / 2 pixel accurate and the final MV is also 1 / 2 pixel accurate, the second interpolation filter may be used. Otherwise, the first interpolation filter may be used. iii. In one example, the second interpolation filter used in the AMVP-MERGE mode may be the same as the second interpolation filter used for the normal AMVP mode (eg, the second interpolation filter for 1 / 2 pixel precision). iv. Alternatively, the second interpolation filter used in the AMVP-MERGE mode may be different from the second interpolation filter used for the normal AMVP mode (eg, an interpolation filter for 1 / 2 pixel precision). d. Alternatively, half-pixel MVD accuracy may not be allowed for AMVP-MERGE mode. e. For example, at least one syntax element (e.g., a flag and / or parameter index) may be signaled at a block level to indicate which motion vector precision is used to encode / signal the MVD value and / or MV of a video unit encoded in AMVP-MERGE mode. i. In addition, the MVD signaled at any resolution can be converted to internal precision (e.g., 1 / 16 pixel resolution) for subsequent processes such as motion compensation, etc. 6.2. To deal with the second problem, the following method is proposed: a. In one example, the prediction units generated based on the AMVP-MERGE mode can be used as the MHP hypothesis. i. For example, the AMVP-MERGE mode prediction block can be used as the basic assumption of the MHP block. ii. For example, syntax elements / structures related to MHP assumption data (e.g., whether there are additional assumptions associated with the video unit encoded in AMVP-MERGE mode; if so, whether the MHP additional assumptions based on AMVP or MERGE are used, etc.) can be signaled in the bitstream just after the video unit is identified as a video unit encoded in AMVP-MERGE mode. iii. For example, the AMVP-MERGE prediction block can be used as an additional hypothesis for the MHP block. iv. For example, additional hypotheses for MHP blocks may be generated based on AMVP-MERGE motion candidates. 1) For example, in this case, syntax elements related to this AMVP-MERGE motion candidate (e.g., which side is AMVP / MERGE encoded or decoded, the reference index of the AMVP side, the MVD value of the AMVP side, and / or the MVP index side of the AMVP) may be signaled in a multi-hypothesis data structure. v. For example, additional assumptions on whether LIC is used for AMVP-MERGE codec may be inherited from the base assumption on the use of LIC. 1) For example, if the base hypothesis is LIC codec, then the additional hypothesis of AMVP-MERGE codec is LIC codec, without signaling the use of LIC for this additional hypothesis. 2) Alternatively, if the base hypothesis is not LIC coded, then the additional hypothesis of AMVP-MERGE codec is also not LIC coded, without signaling the use of LIC for such additional hypothesis. vi. For example, the assumptions (base assumptions and / or additional assumptions) of whether LIC is used for AMVP-MERGE encoding and decoding may depend on the use of LIC on the merging side of the AMVP-MERGE candidate. 1) For example, suppose a certain AMVP-MERGE candidate consists of a uni-merge candidate in one side (L0 or L1) and a uni-AMVP candidate in the other side (L1 or L0), a. For example, the usage of LIC for such an AMVP-MERGE candidate may be inherited from its merge candidate. b. For example, if the merge candidate uses LIC, then LIC can be used for the prediction block of AMVP-MERGE codec. vii. Alternatively, the use of an assumed LIC for AMVP-MERGE codec may be signaled in the bitstream. 6.3. To deal with the third problem, the following method is proposed: a. For example, the AMVP-MERGE candidate can be used in one or more of the following encoding and decoding modes. i. CIIP mode (and / or its variants, such as conventional CIIP, CIIP-PDPC, CIIP-TM, etc.). ii. MMVD mode (and / or its variants, such as conventional MMVD, affine MMVD, etc.). iii. MHP model (and / or its variants, such as MHP basic assumptions and / or MHP additional assumptions, etc.). iv. GPM mode (and / or its variants, such as conventional GPM, GPM-TM, GPM-MMVD, GPM-Inter-Intra, etc.). b. For example, the AMVP-MERGE candidate may be first refined by a decoder-side motion vector refinement process (eg, TM- or DMVR-based motion vector refinement) and then used for a second encoding / decoding mode (eg, such as listed in the above sub-item). c. For example, an AMVP-MERGE candidate may be inserted into another candidate list. i. For example, AMVP-MERGE candidates can be inserted into the regular merge candidate list. 1) For example, AMVP-MERGE candidates can be used in regular merge mode and / or its variants. 2) For example, AMVP-MERGE candidates may be used in MMVD mode and / or its variants. 3) For example, AMVP-MERGE candidates may be used in CIIP mode and / or its variants. 4) For example, the AMVP-MERGE candidate may be used in the MHP mode and / or its variants. 5) For example, the AMVP-MERGE candidate may be used in the GPM mode and / or its variants. ii. For example, AMVP-MERGE candidates may be inserted into the regular TM merge candidate list. 1) For example, AMVP-MERGE candidates can be used in conventional TM merge mode and / or its variants. iii. Additionally, furthermore, AMVP-MERGE candidates may be inserted into another prediction list after the original candidates of that prediction list. d. For example, the AMVP-MERGE candidates may be reordered based on a decoder-derived method (by TM- or DMVR-based cost evaluation), and then M of them will be selected to be added to a second candidate list (such as a regular merge candidate list, a regular TM merge candidate list). i. Additionally, more than one AMVP-MERGE candidate may be re-ordered together. ii. Alternatively, the first candidate from the first AMVP-MERGE prediction list and the second candidate from the second prediction list may be reordered together. e. For example, once an AMVP-MERGE candidate is used for a codec block, additional syntax elements may be signaled to specify the prediction direction (L0 or L1) of the AMVP portion and / or the reference picture index of the selected AMVP candidate, and / or the motion vector prediction index of the selected AMVP candidate, and / or the motion vector difference associated with the AMVP motion vector predictor. i. Alternatively, the motion vector predictor index on the AMVP side of the AMVP-MERGE candidate may not be signaled (eg, it may be selected by a decoder-side method via a TM or DMVR based cost evaluation). f. Alternatively, once AMVP-MERGE candidates are used, additional syntax element(s) may be signaled specifying the predictor index of the merge candidate. i. Alternatively, the motion vector predictor index of the merge side of the AMVP-MERGE candidate may not be signaled (eg, it may be selected by a decoder-side method via a TM or DMVR based cost evaluation). g. In one example, before generating AMVP-MERGE candidates, the merged portion of the AMVP-MERGE mode may first be refined by a decoder-side motion refinement process (such as TM or DMVR). h. In one example, the AMVP portion of the AMVP-MERGE mode may first be refined by a decoder-side motion vector refinement process (such as TM or DMVR) before generating AMVP-MERGE candidates. 6.4. To deal with the fourth problem, the following method is proposed: a. For example, adaptive DMVR motion candidates can be used in one or more of the following codec modes. i. CIIP mode (and / or its variants, such as conventional CIIP, CIIP-PDPC, CIIP-TM, etc.). ii. MMVD mode (and / or its variants, such as conventional MMVD, affine MMVD, etc.). iii. MHP model (and / or its variants, such as MHP basic assumptions and / or MHP additional assumptions, etc.). iv. GPM mode (and / or its variants, such as conventional GPM, GPM-TM, GPM-MMVD, GPM-Inter-Intra, etc.). v. AMVP mode (and / or its variants, such as regular AMVP, SMVD, AMVP-MERGE, affine AMVP, etc.). b. For example, the adaptive DMVR motion candidates may be first refined by a decoder-side motion vector refinement process (eg, TM- or DMVR-based motion vector refinement) and then used in a second encoding / decoding mode (eg, such as listed in the above sub-item). c. For example, when adaptive DMVR motion candidates are used in AMVP mode, i. A DMVR motion candidate may refer to a motion vector pair containing both an L0 motion vector and an L1 motion vector, and / or both an L0 reference picture index and an L1 reference picture index. ii. It can be used as an MVP candidate. iii. The candidate indices of DMVR motion candidates (instead of L0 and L1 motion vector predictor indices and L0 and L1 reference picture indices) may be signaled in the bitstream for AMVP mode. iv. The reference picture index for one prediction direction (L0 or L1) may be signaled for AMVP mode, and the reference picture index for the other prediction direction (L1 or L0) may be inferred (eg, based on DMVR conditions). d. For example, the adaptive DMVR motion candidate can be inserted into another candidate list. i. For example, adaptive DMVR motion candidates can be inserted into the regular merge candidate list. 1) For example, adaptive DMVR motion candidates can be used in the regular merge mode and / or its variants. 2) For example, adaptive DMVR motion candidates can be used in MMVD mode and / or its variants. 3) For example, adaptive DMVR motion candidates can be used in CIIP mode and / or its variants. 4) For example, adaptive DMVR motion candidates can be used in MHP mode and / or its variants. 5) For example, adaptive DMVR motion candidates can be used in GPM mode and / or its variants. ii. For example, adaptive DMVR motion candidates can be inserted into the regular TM merge candidate list. 1) For example, adaptive DMVR motion candidates can be used in the conventional TM merge mode and / or its variants. iii. Additionally, furthermore, the adaptive DMVR motion candidates may be inserted into another prediction list after the original candidates of that prediction list. e. For example, the adaptive DMVR motion candidates may be reordered based on a decoder-derived method (via TM or DMVR-based cost evaluation), and then M of them will be selected to be added to a second candidate list (such as a regular merge candidate list, a regular TM merge candidate list). i. Additionally, more than one adaptive DMVR motion candidates may be reordered together. ii. Alternatively, the first candidate from the first adaptive DMVR merge list and the second candidate from the second prediction list may be reordered together. 6.5. To deal with the fifth problem, the following method is proposed: a. For example, the enabling / disabling of the first codec tool may be controlled by a second syntax element signaled at a syntax level higher than the codec block level. i. For example, a syntax level higher than the codec block level may indicate sequence level / picture group level / picture level / slice level / tile group level such as in a sequence header / picture header / SPS / VPS / DPS / DCI / PPS / APS / slice header / tile group header. ii. For example, a single codec may be controlled by the second syntax element. iii. For example, more than one codec may be controlled by the second syntax element. iv. For example, the first codec tool may be a prediction mode (such as AMVP-MERGE mode, etc.) utilizing a decoder-side motion derivation method. v. For example, the second syntax element may be a SPS / PPS / PH / SH flag that specifies whether decoder-side motion derivation methods (such as DMVR and TM, etc.) are allowed. vi. For example, the second syntax element may be a SPS / PPS / PH / SH flag that specifies whether decoder-side motion vector refinement (eg, DMVR, etc.) is allowed. vii. For example, the second syntax element may be a SPS / PPS / PH / SH flag that specifies whether decoder-side template matching (such as TM and / or inter-TM, etc.) is allowed. 6.6. Regarding LIC parameter derivation (e.g., as illustrated in the sixth question), assuming that the linear model for the LIC codec block is based on at least two parameters: a slope parameter "a" and a bias parameter "b", and the relationship between the neighboring samples of the current block and the neighboring samples of the temporally co-located block can be represented by "reconTempNeigh=a*reconCurNeigh+b", where "reconTempNeigh" represents the reconstructed / predicted value of the neighboring samples of the temporally co-located block, and "reconCurNeigh" represents the reconstructed / predicted value of the neighboring samples of the current block, the following method is proposed: a. For example, at least one adjustment factor may be applied to adjust at least one LIC parameter used for LIC model derivation. a. For example, the adjustment factor may be signaled / present in the bitstream. b. For example, the adjustment factors may be derived at both the encoder and the decoder. b. For example, at least one syntax element (eg, syntax parameter or index, or variable or offset value or integer) may be signaled at video unit level for calculating at least one LIC parameter of at least one LIC model. a. For example, the video unit level may be PU / CU / block level. i. For example, in addition, the video unit level can be a sequence / picture group / picture / slice / tile group / PB / TB / CB / PU / TU / CU / VPDU / CTU / CTU row / slice / tile / sub-picture level. b. For example, the syntax element(s) may be used to adjust the value of at least one LIC parameter of at least one LIC model. i. For example, syntax element(s) may be used as indicator(s) of adjustment factor(s). ii. Alternatively, syntax element(s) may be used to represent / indicate the value of at least one LIC parameter. iii. For example, an indicator may be signaled to adjust a parameter of a LIC model. c. For example, the derivation of LIC parameters may be based on both decoder-derived methods and signaled syntax elements. d. For example, a syntax element may be an indicator of an integer. i. For example, the value of the syntax element may be in the range of [-N, +N], for example, N=4. ii. For example, LIC parameters can be derived directly based on integers. e. For example, a syntax element may be an indicator of an index. i. For example, based on the first value of the index, the second value may be derived from a (predefined) lookup table for LIC parameter derivation. f. For example, how many syntax elements are signaled may depend on how many linear / LIC models are used for a video unit (eg, codec block). i. For example, if M LIC models are used for a video unit, M syntax elements may be signaled in association with the video unit. c. For example, the predicted sample value of the video unit for LIC coding and decoding may be derived based on the updated model: ValueAfter=a'*ValueBefore+b', where a'=a+Delta, b'=b-Delta*funcD. a. For example, the updated model can be used to estimate / derive the prediction samples inside the current block. i. Alternatively, the updated model can be used to modulate the relationship between neighboring samples of the current block and neighboring samples of the temporally co-located block. b. For example, "Delta" can be the slope adjustment / offset value for the LIC model. i. For example, at least one indicator of "Delta" may be signaled in the bitstream. ii. Alternatively, the value of "Delta" may be derived based on decoded information (eg, decoded sample values, decoded prediction modes of neighboring / reference blocks). iii. For example, "Delta" can be an integer. iv. For example, “Delta” may be an integer in the range of [-N, +N], for example, N=4. v. For example, "Delta" can be a number / value / integer / constant / variable derived from an index on a lookup table. c. For example, “funcD” may be calculated by averaging the reconstructed / predicted values of all available / appropriate / possible neighboring samples of the current block (or all available / appropriate / possible neighboring samples of the temporally co-located block). i. For example, "funcD" may be calculated by averaging neighboring / reference samples from both intra-coded blocks and inter-coded blocks. 1. Alternatively, "funcD" can be calculated by simply averaging neighboring / reference samples from inter-coded blocks. ii. For example, “funcD” may be calculated by averaging all available neighboring / reference samples located to the left and / or above the current block and / or temporally co-located block. 1. Alternatively, only a portion of such samples may be considered (eg, adjacent to the first top-left M×M unit, such as M=16 or 8 or 4 or 32). iii. For example, (neighboring samples of) the temporally co-located block may be retrieved / pointed to by a block motion vector (or its variants). 1. Alternatively, (neighboring samples of) the temporally co-located block may be retrieved / pointed to by a rounded block motion vector (eg, rounded to integer pixel precision). iv. For example, the averaging process can be handled with (or without) rounding factors. v. Alternatively, the averaging process can be replaced by other functions such as summation. d. For example, the updated model (eg, with adjustments) may be used / enabled for all LIC codec blocks. i. Alternatively, an updated model may be used / enabled for certain LIC codec blocks. 1. For example, “something” may be determined based on available / appropriate / possible neighboring samples (such as both left and top neighboring samples are available, or only left neighboring samples are available, or only top neighboring samples are available, etc.). 2. For example, "some kind" may be determined based on a prediction mode (such as AMVP codec or MERGE codec, unidirectional prediction or bidirectional prediction, etc.). ii. Alternatively, the updated model may be used / allowed only when both the left reference sample and the upper reference sample are available / appropriate / possible for the video unit. iii. Alternatively, the updated model can be used / allowed only when the video unit is predicted unidirectionally. d. The information for adjustment of LIC or CCLM or MM-CCLM can be encoded and decoded in a predictive manner. e. Information for adjustment of LIC or CCLM or MM-CCLM may be encoded or decoded using at least one context model. a. The context model may depend on codec information. b. Alternatively, it can be encoded and decoded in bypass mode. f. For example, the neighboring / reference samples used to derive the LIC model parameters may not be from all available / appropriate / possible neighboring / reference samples on the left and above sides adjacent to the coding block and the temporally co-located block. a. For example, it may refer to neighboring / reference samples from both intra-coded blocks and inter-coded blocks. i. Alternatively, it may refer only to neighboring / reference samples from inter-coded blocks. b. For example, it may refer to a neighboring / reference sample located to the left (or above) of the current block and / or the temporally co-located block. i. Alternatively, only a portion of such samples may be considered (eg, adjacent to the first top left M×M unit, such as M=16 or 8 or 4 or 32). c. For example, (neighboring samples of) the temporally co-located block may be retrieved / pointed to by a block motion vector (or its variants). i. Alternatively, (neighboring samples of) the temporally co-located block may be retrieved / pointed to by a rounded block motion vector (eg, rounded to integer pixel precision). g. For example, whether to apply / allow adjustments (eg, updated models) for a video unit may depend on the encoded information. a. For example, the original model (without adjustments) and the updated model (with adjustments) may be used / allowed. i. Alternatively, only updated models will be used / allowed. b. For example, whether adjustment-based LIC model update is allowed (or applied) can be signaled in the bitstream. i. For example, it may be signaled at (at least) one video unit level such as sequence / group of picture / picture / slice / tile group / PB / TB / CB / PU / TU / CU / VPDU / CTU / CTU row / slice / tile / sub-picture level. c. For example, whether to allow (or apply) the adjustment-based LIC model update can be derived at both the encoder side and the decoder side. 6.7. Whether and / or how to apply adjustments for CCLM, MM-CCLM or LIC may depend on codec information such as block dimension, codec mode, (transformed) residual, transform, etc. a. For example, if the width and / or height and / or size of the block is less than a threshold, no adjustment may be applied. 6.8. Regarding the signaling and determination of LIC (e.g., as illustrated in the seventh question), the following method is proposed: a. For example, the LIC flag at the video unit level (eg, CU / PU level LIC flag) may not be signaled but derived at both the encoder side and the decoder side. a. For example, the CU / PU level LIC flag for the AMVP codec block may not be signaled. b. For example, the CU / PU level LIC flag for the affine AMVP codec block may not be signaled. c. For example, the CU / PU level LIC flag for the AMVP-MERGE codec block may not be signaled. b. For example, whether to use LIC for a video unit may depend on coded information (eg, a decoder-derived method). a. For example, the LIC flag at the video unit level can be derived implicitly at both the encoder side and the decoder side. i. For example, the CU / PU level LIC flag for non-merging (such as AMVP and / or Affine AMVP) codec blocks may be derived implicitly. ii. For example, the CU / PU level LIC flag for the merge (and / or its variants such as TM, BM, MHP, ADMVR, CIIP, GPM, sbTMVP, affine merge, etc.) codec block may be derived implicitly. iii. For example, implicitly deriving methods that can be derived based on decoders. iv. For example, implicit derivation can be based on template matching. v. For example, implicit derivation can be based on bilateral matching. b. For example, the decoder-derived cost / error / distortion may be calculated for both the non-LIC case and the LIC case, and the one with the lower cost / error / distortion is determined as the codec method for the video unit. i. For example, the template (and / or bilateral) matching cost may be calculated for LIC coded video units and non-LIC coded video units, respectively. ii. For example, the cost / error / distortion is derived from temporally (co-located) neighboring samples and / or reference samples in a reference picture. iii. For example, the cost / error / distortion is not derived from the current block sample in the current picture. c. For example, the coded information for LIC mode may be neighboring / reference samples from both intra-coded blocks and inter-coded blocks. i. Alternatively, the coded information may be neighboring / reference samples from inter-coded blocks only. d. For example, the coded information for LIC mode may be all available neighboring / reference samples located to the left and / or above the current block and / or the temporally co-located block. i. Alternatively, only a portion of such samples may be considered (eg, adjacent to the first top left M×M unit, such as M=16 or 8 or 4 or 32). e. For example, a temporally co-located block may be retrieved / pointed to by a block motion vector (or its variants). i. Alternatively, the temporally co-located block may be retrieved / pointed to by a rounded block motion vector (eg, rounded to integer pixel precision). c. For example, the merge index of the merge block for LIC coding and decoding may not be signaled (eg, derived at both the encoder and the decoder). a. For example, the motion (eg, motion vector, reference index, prediction direction, etc.) of the LIC-encoded merged block can be derived at both the encoder side and the decoder side. b. For example, multiple template (and / or bilateral) matching costs / errors / distortions may be calculated for all (or multiple or predefined portions) available / possible / appropriate merge candidates, respectively. The one with the minimum cost / error / distortion is determined as the motion for the video unit. i. For example, the template is constructed from temporally (co-located) neighboring samples and / or reference samples in a reference picture. ii. For example, the template is not constructed from the current block samples in the current picture. iii. For example, the template is constructed with a sample without LIC. iv. For example, the template is constructed with a sample having LIC. d. For example, the best LIC parameter set (such as a and b calculated by a least squares fitting method) can be determined from more than one LIC parameter set. a. For example, more than one LIC parameter set may be applicable to a video unit of the LIC codec. b. For example, which LIC parameter set is used for a video unit can be derived at both the encoder side and the decoder side. i. Alternatively, a syntax element (eg, an index) may be signaled to specify the LIC parameter set for a video unit. c. For example, multiple template (and / or bilateral) matching costs / errors / distortions may be calculated for all appropriate LIC parameter sets, respectively. The one with the minimum cost / error / distortion is determined as the LIC parameter set for the video unit. i. For example, the template is constructed from temporally (co-located) neighboring samples and / or reference samples in a reference picture. ii. For example, the template is not constructed from the current block samples in the current picture. 6.9. Regarding out-of-bounds (OOB) prediction treatment (e.g., as illustrated in the eighth problem), the following method is proposed: a. For example, an OOB handling method may be used to blend the first prediction and the second prediction. a. For example, at least one prediction block / sub-block is a motion compensated prediction block / sub-block. i. For example, the first prediction may be non-inter (eg, intra) prediction, and the second prediction may be motion compensated prediction (such as CIIP mode, etc.). ii. For example, the prediction blocks / sub-blocks can all be motion compensated predicted. b. For example, two predictions may be generated from the same inter prediction direction (eg, MHP mode, GPM mode, etc.). c. For example, two predictions may be generated from different inter prediction directions (eg, bi-directional prediction weighting). d. For example, it can be used in MERGE mode and / or its variant modes (such as BM / ADMVR / CIIP / TM / MMVD / AffinMerge / sbTMVP / GPM / GPM-MMVD / GPM-TM mode, etc.). e. For example, it can be used in AMVP mode and / or its variant modes (such as SMVD / AMVVmerge / AffineAMVP, etc.). f. For example, it may be allowed / used for prediction based on decoder-side motion vector refinement (such as template matching and / or bilateral matching based motion vector refinement, etc.). i. For example, it may be allowed / used for predictions generated by decoder-side motion vector refinement related codec modes. ii. Alternatively, it may not be allowed / used for prediction based on decoder-side motion vector refinement. iii. Alternatively, it may not be allowed / used for prediction generated by decoder-side motion vector refinement related codec modes. g. For example, it may be allowed / used for predictions generated based on some kind of sub-block based prediction method (e.g., DMVR, TM, affine merge, affine AMVP, sbTMVP, OBMC, BDOF, AMVP-merge, GPM, etc.). i. For example, it may be allowed / used for certain kinds of CU / PU based prediction methods (eg, regular merge, CIIP, MMVD, MHP and / or regular AMVP, SMVD, etc.). ii. Alternatively, it may not be allowed / used for predictions generated based on some kind of sub-block based prediction method. 1. For example, it may not coexist with certain kinds of sub-block based prediction methods. iii. Alternatively, it may not be allowed / used for certain kinds of CU / PU based prediction methods. 1. For example, it may not coexist with certain CU / PU based prediction methods. b. For example, OOB may refer to outside reference picture boundaries. a. Alternatively, it may refer to outside the reference sub-picture boundaries. i. For example, the reference sub-picture ID is the same as the current sub-picture. b. Alternatively, it may refer to outside the reference strip boundaries. c. Alternatively, it may refer to outside the reference tile boundaries. i. For example, in the case of MCTS (ie, Motion Constrained Tile Set), motion vectors are constrained to be within tiles with the same coordinates / position in the reference picture. c. For example, the blending weights for OOB samples may be different from those for non-OOB samples inside the boundary. a. For example, the weighting factor for OOB samples of a motion compensated block / sub-block may be set according to a rule, for example, to a certain value (such as zero). b. For example, the weighting factors for motion compensated samples around the butting boundary of OOB samples and non-OOB samples may be set according to a rule, for example, gradually increasing / decreasing the weighting value as it moves away from the butting boundary. c. For example, the blending weights may be calculated based on boundary-based distances. d. For example, the blending weight may be calculated based on the distance based on the boundary between the OOB sample and the non-OOB sample. d. For example, when mixing the first prediction sample with the second prediction sample, if both motion compensated samples are OOB, the final prediction value can be generated without mixing. a. For example, non-OOB samples closer to the boundary can be obtained for final prediction sample generation. b. For example, the final prediction value may be generated based on non-OOB prediction samples within the current mixed block according to a rule (eg, the most recently available non-OOB prediction sample within the current mixed block). c. Alternatively, in this case, the final prediction value for such a sample may be generated by the same averaging / weighted blending as usual (eg, the same behavior as for samples inside the blending boundaries). e. For example, the values of the OOB samples of a motion compensated block / sub-block may be set according to a rule. a. For example, OOB samples may refer to OOB samples just after the (unidirectional) motion compensation process (but before the BDOF and blending / weighting processes). b. For example, the rules may be based on non-OOB motion compensated sample values inside the boundary. i. For example, the non-OOB motion compensated sample values at the first row located inside the boundary may be copied above for the above OOB samples. ii. For example, the non-OOB motion compensated sample values at the first column located inside the boundary may be duplicated on the left for the left OOB samples. iii. For example, the non-OOB motion compensated sample value at the top left corner located inside the boundary may be copied for the top left OOB sample. c. For example, OOB samples may refer to OOB samples after BDOF (if any) and before the mixing / weighting process. d. For example, the rule can be based on the sample value of the non-OOB BDOF refinement inside the boundary. i. For example, the non-OOB BDOF refined sample values at the first row located inside the boundary may be copied above for the above OOB samples. ii. For example, the non-OOB BDOF refined sample values at the first column located inside the boundary may be duplicated on the left for the left OOB samples. iii. For example, the non-OOB BDOF refined sample value at the top left corner located inside the boundary may be copied for the top left OOB sample. f. For example, if the prediction block / sub-block pointed to by the motion vector is OOB, a new prediction block / sub-block may be generated according to a rule. a. For example, a new prediction block / sub-block may be generated based on a zero motion vector (eg, (0, 0)). b. For example, the new prediction block / sub-block can be replaced by the co-located block. c. For example, the new prediction block / sub-block may be replaced with the non-OOB prediction block / sub-block closest to the OOO block / sub-block. g. For example, the OOB check may be based on motion vectors prior to decoder-side motion refinement (such as template matching based motion refinement and / or bilateral matching based motion refinement). a. Alternatively, the OOB check can be based on motion vectors after a certain stage of DMVR-based motion refinement (e.g., PU level or 16×16 sub-block level or 8×8 sub-block level). b. Alternatively, the OOB check can be based on motion vectors after all stages of DMVR-based motion refinement (e.g., PU level, 16x16 sub-block level, and 8x8 sub-block level). c. Alternatively, the OOB check can be based on the motion vectors after TM-based motion refinement. d. For example, the blending weights for OOB samples may be determined based on an OOB check. h. For example, if the reference block is partially (or entirely) OOB, the OOB portion of the motion compensated block / sub-block may not be used to generate the final prediction block. i. For example, if the reference block is partially (or entirely) OOB, the original motion compensation block / sub-block may be modified / shifted to another motion compensation block / sub-block. j. For example, if the reference block is partially (or fully) OOB, the original motion vector used to generate the motion compensated block / sub-block may be modified to another motion vector. k. For example, whether the OOB handling method is used for a video unit may be signaled at the video unit level. a. Alternatively, the OOB handling method can be forced for certain kinds of blocks (such as multiple hypothesis, bi-directionally predicted blocks) without video unit level signaling. b. Alternatively, whether to use OOB handling can be derived implicitly based on the encoded information. Miscellaneous aspects 6.10. For example, is it possible to control deblocking at the CTU (or CU) level? 6.11. For example, the intra-MTS type may be determined based on syntax elements signaled at the PPS level (eg, parameters of a threshold). 6.12. For example, adaptive mixing weights may be used for OBMC codec blocks, where the adaptive mixing weights may be based on coded information other than neighboring reconstructed / predicted samples. a. For example, the adaptive mixing weights can be determined by the temporal coding information in the reference picture. b. For example, adaptive blending weights may be selected based on more than one lookup table. a. For example, weight indices can be derived. b. For example, the weight index may be signaled. General 6.13. Whether and / or how to apply the above disclosed method may be signaled at the sequence level / picture group level / picture level / slice level / tile group level, such as in the sequence header / picture header / SPS / VPS / DPS / DCI / PPS / APS / slice header / tile group header. 6.14. Whether and / or how to apply the above disclosed methods can be signaled under PB / TB / CB / PU / TU / CU / VPDU / CTU / CTU row / slice / tile / sub-picture / other kind of region containing more than one sample or pixel. 6.15. Whether and / or how to apply the above disclosed method may depend on the coded information, such as block size, color format, single / dual tree partition, color component, slice / picture type.
[0088] Embodiments of the present disclosure relate to handling out-of-bounds samples.
[0089] As used herein, the terms "video unit" or "codec unit" or "block" used herein may refer to one or more of the following: a color component, a sub-picture, a slice, a tile, a codec tree unit (CTU), a CTU row, a CTU group, a codec unit (CU), a prediction unit (PU), a transform unit (TU), a codec tree block (CTB), a codec block (CB), a prediction block (PB), a transform block (TB), a block, a sub-block of a block, a sub-region within a block, or a region including more than one sample or pixel.
[0090] In the present disclosure, with respect to “encoding a block in mode N”, the term “mode N” may be a prediction mode (e.g., MODE_INTRA, MODE_INTER, MODE_PLT, MODE_IBC, etc.) or a coding technology (e.g., AMVP, Merge, SMVD, BDOF, PROF, DMVR, AMVR, TM, Affine, CIIP, GPM, MMVD, BCW, HMVP, SbTMVP, etc.).
[0091] In the following discussion, LIC parameters may refer to two parameters (such as slope parameter "a" and bias parameter "b") derived based on a linear model that is used to map neighboring samples of a current block and neighboring samples of a temporally co-located block (e.g., the temporally co-located block may be pointed to by a motion vector or a rounded motion vector of the current block). In addition, LIC parameters may be used to estimate prediction values of samples within the current video unit.
[0092] Fig.32 A flow chart of a method 3200 for video processing according to some embodiments of the present disclosure is illustrated. The method 3200 may be implemented during conversion between a video unit and a bitstream of the video unit.
[0093] like Fig.32 As shown, at block 3210, during conversion between a video unit of a video and a bitstream of the video unit, it is determined whether at least one of a first sample set or a second sample set is outside a boundary associated with the video unit. In other words, it can be determined whether the first sample set or the second sample set is an out-of-boundary (OOB) sample.
[0094] At block 3220, a weighted process is applied to the first sample set and the second sample set based on the determination. For example, OOB may refer to outside of a reference picture boundary. Alternatively, OOB may refer to outside of a reference sub-picture boundary. For example, the reference sub-picture ID is the same as the current sub-picture. Alternatively, OOB may refer to outside of a reference stripe boundary. Alternatively, OOB may refer to outside of a reference tile boundary. For example, in the case of a motion constrained tile set (MCTS), motion vectors are constrained to be within tiles with the same coordinates / position in a reference picture. In some embodiments, the boundary is one of: a reference picture boundary, a reference sub-picture boundary, a reference stripe boundary, or a reference tile boundary.
[0095] At block 3230, a prediction is generated based on the weighted first sample set and the weighted second sample set. For example, the prediction may be generated by weighting the first sample set and the second sample set.
[0096] At block 3240, a conversion is performed based on the prediction. In some embodiments, the conversion may include encoding the video unit into a bitstream. Alternatively, the conversion may include decoding the video unit from the bitstream. Compared to conventional solutions, some embodiments of the present disclosure may advantageously improve codec efficiency, codec gain, codec performance, and flexibility. In addition, out-of-bounds samples have been processed.
[0097] In some embodiments, if the first sample set is outside the boundary and the second sample set is inside the boundary, the first sample set may be mixed based on a first weighting factor and the second sample set may be mixed based on a second weighting factor. In this case, the first weighting factor may be different from the second weighting factor. For example, the mixing weights for OOB samples may be different from the mixing weights for those non-OOB samples inside the boundary.
[0098] In some embodiments, the first weighting factor for the first sample set of the motion compensated block or sub-block may be set according to a predetermined rule. For example, the first weighting factor is set to a specific value, such as zero.
[0099] In some embodiments, the third weighting factor for the motion compensated samples around the boundary of the first sample set and the second sample set can be set according to a predetermined rule. For example, the predetermined rule includes increasing the weighting value as the distance from the boundary is increased. Alternatively, the predetermined rule includes decreasing the weighting value as the distance from the boundary is decreased.
[0100] In some embodiments, the first weighting factor and the second weighting factor may be determined based on the distance to the boundary. Alternatively, the first weighting factor and the second weighting factor are determined based on the distance to the boundary of the first sample set and the second sample set.
[0101] In some embodiments, if the reference block is partially or completely outside the boundary, the outer boundary portion of the motion compensation block or sub-block may not be used to generate the prediction. For example, if the reference block is partially or completely OOB, the OOB portion of the motion compensation block / sub-block may not be used to generate the final prediction block.
[0102] In some embodiments, whether to generate a prediction by applying a weighted process based on a determination may be implicitly derived based on the coded information of the video unit. Alternatively, whether to generate a prediction by applying a weighted process based on a determination may be indicated at the video unit level.
[0103] In some embodiments, the prediction generated by applying a weighted process based on the determination can be forced for a block without indication. For example, a block can include a multi-hypothesis block. Or, a block can include a bidirectional prediction block.
[0104] In some embodiments, if the reference block is partially or completely outside the boundary, the original motion compensation block or sub-block can be modified / shifted to another motion compensation block or sub-block. Alternatively, if the reference block is partially or completely outside the boundary, the original motion vector used to generate the motion compensation block or sub-block can be modified to another motion vector.
[0105] In some embodiments, the first sample set is in the first prediction and the second sample set is in the second prediction. In this case, applying the weighting process may include blending the first prediction and the second prediction. For example, an OOB treatment method may be used to blend the first prediction and the second prediction.
[0106] In some embodiments, at least one prediction block or sub-block may be a motion compensated prediction block or sub-block. For example, the first prediction is non-inter prediction and the second prediction is motion compensated prediction. For example, the first prediction may be an intra prediction mode and the second prediction may be a CIIP mode. In some embodiments, both the first prediction and the second prediction are motion compensated predictions.
[0107] In some embodiments, the first prediction and the second prediction may be generated from the same inter-prediction direction. For example, the two predictions may be generated from the same inter-prediction direction (such as MHP mode, GPM mode).
[0108] In some embodiments, the first prediction and the second prediction are generated from different inter-prediction directions. For example, the two predictions may be generated from bi-directional prediction weights.
[0109] In some embodiments, the hybrid first prediction and the second prediction may be applied to at least one of the following: a merge (MERGE) mode or a variant mode of the MERGE mode. For example, the variant mode of the MERGE mode may include at least one of the following: a bilateral matching (BM) mode, an adaptive decoder-side motion vector refinement (ADMVR) mode, a combined inter-frame and intra-frame prediction (CIIP) mode, a template matching (TM) mode, a merge pattern vector difference (MMVD) mode, an affine merge mode, a sub-block-based temporal motion vector prediction (Sb TMVP) mode, a geometric partitioning mode (GPM) mode, a GPM-MMVD mode, or a GPM-TM mode.
[0110] In some embodiments, the hybrid first prediction and the second prediction may be applied to at least one of the following: an advanced motion vector prediction (AMVP) mode or a variant of the AMVP mode. For example, a variant of the AMVP mode may include at least one of the following: a symmetric motion vector difference (SMVD), an AMVP merge mode, or an affine AMVP mode.
[0111] In some embodiments, hybrid first prediction and second prediction may be allowed / used for prediction based on decoder side motion vector refinement.For example, decoder side motion vector refinement may include at least one of: template matching based motion vector refinement or bilateral matching based motion vector refinement.
[0112] In some embodiments, mixing the first prediction and the second prediction may be allowed for predictions generated by decoder-side motion vector refinement-related codec modes. Alternatively, mixing the first prediction and the second prediction may not be allowed for predictions based on decoder-side motion vector refinement. In some other embodiments, mixing the first prediction and the second prediction may not be allowed for predictions generated by decoder-side motion vector refinement-related codec modes.
[0113] In some embodiments, hybrid first prediction and second prediction may be allowed for predictions generated based on some kind of sub-block based prediction method. For example, the sub-block based prediction method may include one or more of the following: DMVR, TM, affine merge, affine AMVP, sbTMVP, OBMC, BDOF, AMVP merge, or GPM.
[0114] In some embodiments, a hybrid first prediction and a second prediction may be allowed for a certain type of prediction method based on a coding unit / prediction unit (CU / PU). For example, the prediction method based on a CU / PU may include at least one of the following: conventional merge, CIIP, MVD, MHP, conventional AMVP, or SMVD.
[0115] In some embodiments, hybrid first prediction and second prediction may not be allowed for predictions generated based on certain kinds of sub-block based prediction methods. For example, hybrid first prediction and second prediction do not coexist with certain kinds of sub-block based prediction methods.
[0116] In some embodiments, hybrid first prediction and second prediction may not be allowed for certain kinds of CU / PU based prediction methods. For example, hybrid first prediction and second prediction do not coexist with certain kinds of CU / PU based prediction methods.
[0117] In some embodiments, if both motion compensated samples are outside the boundary, the final prediction value can be generated without mixing the motion compensated samples. For example, when mixing the first prediction sample with the second prediction sample, if both motion compensated samples are OOB, the final prediction value can be generated without mixing.
[0118] In some embodiments, if the non-outer boundary samples are closer to the boundary, the non-outer boundary samples may be used to generate the final prediction value. For example, the non-OOB samples closer to the boundary may be obtained for the final prediction sample generation.
[0119] In some embodiments, the final prediction value may be generated based on non-external boundary prediction samples within the current hybrid block according to a rule. For example, the rule may include the most recently available non-OOB prediction sample within the current hybrid block.
[0120] In some embodiments, the final prediction value may be generated by weighting the mixture in the same manner as samples inside the mixture boundary. For example, it may refer to the same behavior as samples inside the mixture boundary.
[0121] In some embodiments, the values of the outer boundary samples of the motion compensated block / sub-block may be set according to a predetermined rule. For example, the outer boundary samples refer to the outer boundary samples after the motion compensation process. For example, OOB samples may refer to OOB samples just after the (unidirectional) motion compensation process (but before the BDOF and blending / weighting process).
[0122] In some embodiments, the predetermined rule may be based on non-external boundary motion compensated sample values inside the boundary. For example, the non-external boundary motion compensated sample values at the first row inside the boundary may be copied above for the upper external boundary sample. In some embodiments, the non-external boundary motion compensated sample values at the first column inside the boundary may be copied on the left for the left external boundary sample. Alternatively, the non-external boundary motion compensated sample values at the upper left corner inside the boundary may be copied for the upper left external boundary sample. In some embodiments, the external boundary sample may refer to the external boundary sample after bidirectional optical flow (BDOF) and before the weighting process.
[0123] In some embodiments, the rule may be based on the sample values of the non-external boundary BDOF refinements inside the boundary. For example, the sample values of the non-external boundary BDOF refinements at the first row inside the boundary may be copied above for the upper external boundary samples. In some embodiments, the sample values of the non-external boundary BDOF refinements at the first column inside the boundary may be copied on the left for the left external boundary samples. Alternatively, the sample values of the non-external boundary BDOF refinements at the upper left corner inside the boundary may be copied for the upper left external boundary samples.
[0124] In some embodiments, if the prediction block or sub-block pointed to by the motion vector is outside the boundary, a new prediction block or sub-block may be generated according to a predetermined rule. For example, a new prediction block or sub-block may be generated based on a zero motion vector (e.g., (0, 0)). In some embodiments, the new prediction block or sub-block may be replaced with a co-located block. Alternatively, the new prediction block or sub-block may be replaced with a non-external boundary prediction block or sub-block that is closest to the external boundary block or sub-block.
[0125] In some embodiments, the outer boundary check is based on motion vectors prior to decoder-side motion refinement.For example, the decoder-side motion refinement may include at least one of: template matching based motion refinement or bilateral matching based motion refinement.
[0126] In some embodiments, the outer boundary check may be based on motion vectors after a certain stage of DMVR-based motion refinement. For example, the certain stage may include one or more of the following: PU level, 16×16 sub-block level, or 8×8 sub-block level.
[0127] In some embodiments, the outer boundary check may be based on motion vectors after all stages of DMVR-based motion refinement. For example, these stages may include PU level, 16×16 sub-block level, and 8×8 sub-block level.
[0128] Alternatively, the outer boundary check may be based on motion vectors after TM-based motion refinement.In some other embodiments, blending weights for outer boundary samples may be determined based on the outer boundary check.
[0129] In some embodiments, deblocking may be controlled at a codec tree unit (CTU) or codec unit (CU) level. In some embodiments, the intra MTS type may be determined based on a syntax element indicated at a picture parameter set (PPS) level.
[0130] In some embodiments, whether adaptive mixing weights can be used for OBMC codec blocks. In this case, the adaptive mixing weights can be based on coded information other than neighboring reconstruction / prediction samples. For example, the adaptive mixing weights can be temporally coded information in a reference picture. Alternatively, the adaptive mixing weights are selected based on more than one lookup table. In some embodiments, a weight index can be derived. Alternatively, a weight index can be indicated.
[0131] In some embodiments, an indication of whether and / or how to apply the weighting process to the first sample set and the second sample set based on the determination may be indicated at one of the following: sequence level, picture group level, picture level, slice level, or tile group level. In some other embodiments, an indication of whether and / or how to apply the weighting process to the first sample set and the second sample set based on the determination may be indicated at one of the following: sequence header, picture header, sequence parameter set (SPS), video parameter set (VPS), dependency parameter set (DPS), decoding capability information (DCI), picture parameter set (PPS), adaptation parameter set (APS), slice header, or tile group header.
[0132] In some embodiments, an indication of whether and / or how to apply the weighting process to the first sample set and the second sample set based on the determination may be included in one of the following: a prediction block (PB), a transform block (TB), a codec block (CB), a prediction unit (PU), a transform unit (TU), a codec unit (CU), a virtual pipeline data unit (VPDU), a codec tree unit (CTU), a CTU row, a slice, a tile, a sub-picture, or a region containing more than one sample or pixel. Alternatively, whether and / or how to apply the weighting process to the first sample set and the second sample set may be determined based on the determination. In this case, the coded information may include at least one of the following: a block size, a color format, a single tree partition and / or a dual tree partition, a color component, a slice type, or a picture type.
[0133] In some embodiments, a non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by a video processing device. The method may include: determining whether at least one of a first sample set or a second sample set is outside a boundary associated with a video unit of the video; applying a weighting process to the first sample set and the second sample set based on the determination; generating a prediction based on the weighted first sample set and the second sample set; and generating a bitstream of the video unit based on the prediction.
[0134] In some embodiments, a method for storing a bitstream of a video may include determining whether at least one of a first sample set or a second sample set is outside a boundary associated with a video unit of the video. The method may also include applying a weighting process to the first sample set and the second sample set based on the determination. The method may include generating a prediction based on the weighted first sample set and the second sample set. The method may include generating a bitstream of the video unit based on the prediction. The method may also include storing the bitstream in a non-transitory computer-readable recording medium.
[0135] The embodiments of the present disclosure may be described in view of the following clauses, and features of these clauses may be combined in any reasonable way.
[0136] Clause 1. A method of video processing, comprising: during a conversion between a video unit of a video and a bitstream of the video unit, determining whether at least one of a first sample set or a second sample set is outside a boundary associated with the video unit; applying a weighting process to the first sample set and the second sample set based on the determination; generating a prediction based on the weighted first sample set and the weighted second sample set; and performing the conversion based on the prediction.
[0137] Clause 2. A method according to clause 1, wherein applying the weighting process to the first sample set and the second sample set includes: mixing the first sample set based on a first weighting factor based on a determination that the first sample set is outside the boundary and the second sample set is inside the boundary; and mixing the second sample set based on a second weighting factor, wherein the first weighting factor is different from the second weighting factor.
[0138] Clause 3. The method of clause 2, wherein the first weighting factor for the first set of samples of a motion compensated block or sub-block is set according to a predetermined rule.
[0139] Clause 4. The method of clause 3, wherein the first weighting factor is set to a specific value.
[0140] Clause 5. The method of clause 2, wherein a third weighting factor for motion compensated samples around a bordering boundary of the first sample set and the second sample set is set according to a predetermined rule.
[0141] Clause 6. The method according to clause 5, wherein the predetermined rule comprises one of: increasing the weighting value as the distance from the abutment limit increases; or decreasing the weighting value as the distance from the abutment limit increases.
[0142] Clause 7. The method of clause 2, wherein the first weighting factor and the second weighting factor are determined based on a distance to the boundary.
[0143] Clause 8. The method of clause 2, wherein the first weighting factor and the second weighting factor are determined based on distances to a bounding limit of the first sample set and the second sample set.
[0144] Clause 9. The method of clause 1, wherein an outer boundary portion of a motion compensated block or sub-block is not used to generate the prediction if a reference block is partially or fully outside the boundary.
[0145] Clause 10. The method of clause 1, wherein whether to generate the prediction by applying the weighting process based on the determination is implicitly derived based on information of a codec of the video unit.
[0146] Clause 11. The method of clause 1, wherein whether to generate the prediction by applying the weighting process based on the determination is indicated at a video unit level.
[0147] Clause 12. The method of clause 1, wherein generating the prediction by applying the weighting process based on the determination is forced for a certain block without indication.
[0148] Clause 13. The method of clause 12, wherein the certain block comprises at least one of: a multi-hypothesis block, or a bi-directional prediction block.
[0149] Clause 14. The method of clause 1, wherein if the reference block is partially or fully outside the boundary, the original motion compensated block or sub-block is modified to another motion compensated block or sub-block.
[0150] Clause 15. The method of clause 1, wherein if the reference block is partially or fully outside the boundary, an original motion vector used to generate the motion compensated block or sub-block is modified to another motion vector.
[0151] Clause 16. The method of clause 1, wherein the first set of samples is in a first prediction and the second set of samples is in a second prediction, and wherein applying the weighting process comprises blending the first prediction and the second prediction.
[0152] Clause 17. The method of clause 16, wherein at least one prediction block or sub-block is a motion compensated prediction block or sub-block.
[0153] Clause 18. The method of clause 16, wherein the first prediction is non-inter prediction and the second prediction is motion compensated prediction.
[0154] Clause 19. The method of clause 16, wherein the first prediction and the second prediction are both motion compensated predictions.
[0155] Clause 20. The method of clause 16, wherein the first prediction and the second prediction are generated from the same inter prediction direction.
[0156] Clause 21. The method of clause 16, wherein the first prediction and the second prediction are generated from different inter prediction directions.
[0157] Clause 22. The method of clause 16, wherein blending the first prediction and the second prediction is applied in at least one of: a merge mode or a variant mode of the merge mode.
[0158] Clause 23. A method according to clause 22, wherein the variant mode of the merge mode includes at least one of the following: bilateral matching (BM) mode, adaptive decoder side motion vector refinement (ADMVR) mode, combined inter-frame and intra-frame prediction (CIIP) mode, template matching (TM) mode, merge pattern vector difference (MMVD) mode, affine merge mode, sub-block based temporal motion vector prediction (Sb TMVP) mode, geometric partitioning mode (GPM) mode, GPM-MMVD mode, or GPM-TM mode.
[0159] Clause 24. The method of clause 16, wherein blending the first prediction and the second prediction is applied to at least one of: an advanced motion vector prediction (AMVP) mode or a variant of the AMVP mode.
[0160] Clause 25. The method of clause 24, wherein the variant of the AMVP mode comprises at least one of: symmetric motion vector difference (SMVD), AMVP merge mode, or affine AMVP mode.
[0161] Clause 26. The method of clause 16, wherein mixing the first prediction and the second prediction is allowed for prediction based on decoder-side motion vector refinement.
[0162] Clause 27. The method of clause 26, wherein mixing the first prediction and the second prediction is allowed for predictions generated by a decoder-side motion vector refinement related codec mode.
[0163] Clause 28. The method of clause 16, wherein mixing the first prediction and the second prediction is not allowed for prediction based on decoder-side motion vector refinement.
[0164] Clause 29. The method of clause 16, wherein mixing the first prediction and the second prediction is not allowed for predictions generated by a decoder-side motion vector refinement related codec mode.
[0165] Clause 30. The method of clause 16, wherein blending the first prediction and the second prediction is allowed for predictions generated based on some kind of sub-block based prediction method.
[0166] Clause 31. The method of clause 16, wherein mixing the first prediction and the second prediction is allowed for some kind of coding unit / prediction unit (CU / PU) based prediction method.
[0167] Clause 32. The method of clause 16, wherein mixing the first prediction and the second prediction is not allowed for predictions generated based on some kind of sub-block based prediction method.
[0168] Clause 33. The method of clause 16, wherein mixing the first prediction and the second prediction does not coexist with some kind of sub-block based prediction method.
[0169] Clause 34. The method of clause 16, wherein mixing the first prediction and the second prediction is not allowed for certain kinds of CU / PU based prediction methods.
[0170] Clause 35. The method of clause 34, wherein hybridizing the first prediction and the second prediction does not coexist with some kind of CU / PU based prediction method.
[0171] Clause 36. The method of any of clauses 1-34, wherein the boundary is one of: a reference picture boundary, a reference sub-picture boundary, a reference slice boundary, or a reference tile boundary.
[0172] Clause 37. The method of claim 36, wherein the identification of the reference sub-picture is the same as the identification of the current sub-picture.
[0173] Clause 38. The method of clause 36, wherein in the case of a motion constrained tile set (MCTS), motion vectors are constrained to be within tiles having the same coordinates or position in the reference picture.
[0174] Clause 39. The method of clause 1, wherein if two motion compensated samples are outside the boundary, a final prediction value is generated without blending the motion compensated samples.
[0175] 40. The method of clause 39, wherein a non-outer boundary sample is used to generate the final prediction value if the non-outer boundary sample is closer to the boundary.
[0176] Clause 41. The method of clause 39, wherein the final prediction value is generated according to a rule based on non-external boundary prediction samples inside the current mixed block.
[0177] Clause 42. The method of clause 39, wherein the final prediction value is generated by weighted blending in the same manner as samples inside the boundary are blended.
[0178] Clause 43. The method of clause 1, wherein the values of the outer boundary samples of the motion compensated block / sub-block are set according to a predetermined rule.
[0179] Clause 44. The method of clause 43, wherein the outer boundary samples refer to the outer boundary samples after a motion compensation process.
[0180] Clause 45. The method of clause 43, wherein the predetermined rule is based on non-external boundary motion compensated sample values that are interior to the boundary.
[0181] Clause 46. The method of clause 45, wherein the non-exterior boundary motion compensated sample values at a first row located inside the boundary are copied above for an above exterior boundary sample.
[0182] Clause 47. The method of clause 45, wherein the non-exterior boundary motion compensated sample values at a first column located inside the boundary are copied on the left side for the left exterior boundary samples.
[0183] Clause 48. The method of clause 45, wherein the non-exterior boundary motion compensated sample value at the upper left corner inside the boundary is copied for the upper left exterior boundary sample.
[0184] Clause 49. The method of clause 43, wherein the outer boundary samples refer to the outer boundary samples after bidirectional optical flow (BDOF) and before the weighting process.
[0185] Clause 50. The method of clause 43, wherein the rule is based on sample values of non-exterior boundary BDOF refinements that are interior to the boundary.
[0186] Clause 51. The method of clause 50, wherein sample values of the non-exterior boundary BDOF refinement at a first row located inside the boundary are copied above for upper exterior boundary samples.
[0187] Clause 52. The method of clause 50, wherein the sample values of the non-exterior boundary BDOF refinement at the first column located inside the boundary are replicated on the left side for the left exterior boundary samples.
[0188] Clause 53. The method of clause 50, wherein the sample value of the non-exterior boundary BDOF refinement located at the upper left corner of the interior of the boundary is replicated for the upper left exterior boundary sample.
[0189] Clause 54. The method according to clause 1, wherein if the prediction block or subblock pointed to by the motion vector is outside the boundary, a new prediction block or subblock is generated according to a predetermined rule.
[0190] Clause 55. The method of clause 54, wherein the new prediction block or sub-block is generated based on a zero motion vector.
[0191] Clause 56. The method of clause 54, wherein the new prediction block or sub-block is replaced with a co-located block.
[0192] Clause 57. The method of clause 54, wherein the new prediction block or sub-block is replaced with a non-external boundary prediction block or sub-block that is closest to the external boundary block or sub-block.
[0193] Clause 58. The method of clause 1, wherein the outer boundary check is based on motion vectors prior to decoder-side motion refinement.
[0194] Clause 59. The method of clause 58, wherein the decoder-side motion refinement comprises at least one of: template matching based motion refinement, or bilateral matching based motion refinement.
[0195] Clause 60. The method of clause 1, wherein the outer boundary check is based on motion vectors after some stage of DMVR-based motion refinement.
[0196] Clause 61. The method of clause 1, wherein the outer boundary check is based on motion vectors after all stages of DMVR-based motion refinement.
[0197] Clause 62. The method of clause 1, wherein the outer boundary check is based on motion vectors after TM-based motion refinement.
[0198] Clause 63. The method of clause 1, wherein blending weights for outer boundary samples are determined based on the outer boundary check.
[0199] Clause 64. The method of clause 1, wherein whether to deblock is controlled at a codec tree unit (CTU) or codec unit (CU) level.
[0200] Clause 65. The method of clause 1, wherein the intra MTS type is determined based on a syntax element indicated at a picture parameter set (PPS) level.
[0201] Clause 66. The method of clause 1, wherein adaptive mixing weights are used for OBMC codec blocks, wherein the adaptive mixing weights are based on codec information other than neighboring reconstructed / predicted samples.
[0202] Clause 67. The method of clause 66, wherein the adaptive blending weights are determined by temporally encoded information in a reference picture.
[0203] Clause 68. The method of clause 66, wherein the adaptive blending weights are selected based on more than one lookup table.
[0204] Clause 69. A method according to clause 68, wherein a weight index is derived, or wherein the weight index is indicated.
[0205] Clause 70. A method according to any of clauses 1-69, wherein an indication of whether and / or how to apply the weighting process to the first sample set and the second sample set based on the determination is indicated at one of: a sequence level, a picture group level, a picture level, a slice level, or a tile group level.
[0206] Clause 71. A method according to any one of clauses 1-69, wherein an indication of whether and / or how to apply the weighting process to the first sample set and the second sample set based on the determination is indicated in one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header.
[0207] Clause 72. A method according to any one of clauses 1-69, wherein an indication of whether and / or how to apply the weighting process to the first sample set and the second sample set based on the determination is included in one of the following: a prediction block (PB), a transform block (TB), a codec block (CB), a prediction unit (PU), a transform unit (TU), a codec unit (CU), a virtual pipeline data unit (VPDU), a codec tree unit (CTU), a CTU row, a slice, a tile, a sub-picture, or a region containing more than one sample or pixel.
[0208] Clause 73. The method according to any one of clauses 1-69 further includes: determining whether and / or how to apply the weighted process to the first sample set and the second sample set based on the encoded and decoded information of the video unit, the encoded and decoded information including at least one of the following: block size, color format, single-tree partitioning and / or double-tree partitioning, color component, slice type, or picture type.
[0209] Clause 74. The method of any of clauses 1-73, wherein the converting comprises encoding the video unit into the bitstream.
[0210] Clause 75. The method of any of clauses 1-73, wherein the converting comprises decoding the video unit from the bitstream.
[0211] Clause 76. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform a method according to any one of claims 1-105.
[0212] Clause 77. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform a method according to any of Claims 1-105.
[0213] Clause 78. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing device, wherein the method comprises: determining whether at least one of a first sample set or a second sample set is outside a boundary associated with a video unit of the video; applying a weighting process to the first sample set and the second sample set based on the determination; generating a prediction based on the weighted first sample set and the weighted second sample set; and generating a bitstream of the video unit based on the prediction.
[0214] Clause 79. A method for storing a bitstream of a video, comprising: determining whether at least one of a first sample set or a second sample set is outside a boundary associated with a video unit of the video; applying a weighting process to the first sample set and the second sample set based on the determination; generating a prediction based on the weighted first sample set and the weighted second sample set; generating a bitstream for the video unit based on the prediction; and storing the bitstream in a non-transitory computer-readable recording medium. Example Device
[0215] Fig.33 A block diagram of a computing device 3300 in which various embodiments of the present disclosure may be implemented is illustrated. The computing device 3300 may be implemented as or included in the source device 110 (or video encoder 114 or 200) or the destination device 120 (or video decoder 124 or 300).
[0216] It should be understood that Fig.33 The computing device 3300 shown is for illustrative purposes only and is not intended to in any way imply any limitation on the functionality and scope of the embodiments of the present disclosure.
[0217] like Fig.33 As shown, computing device 3300 includes a general computing device 3300. Computing device 3300 may include at least one or more processors or processing units 3310, memory 3320, storage unit 3330, one or more communication units 3340, one or more input devices 3350, and one or more output devices 3360.
[0218] In some embodiments, computing device 3300 can be implemented as any user terminal or server terminal with computing power. Server terminal can be a server, large computing device, etc. provided by a service provider. User terminal can be, for example, any type of mobile terminal, fixed terminal or portable terminal, including mobile phone, station, unit, device, multimedia computer, multimedia tablet, Internet node, communicator, desktop computer, laptop computer, notebook computer, netbook computer, tablet computer, personal communication system (PCS) device, personal navigation device, personal digital assistant (PDA), audio / video player, digital camera / camcorder, positioning device, television receiver, radio broadcast receiver, electronic book device, game device or any combination thereof, including accessories and peripherals of these devices or any combination thereof. It will be envisioned that computing device 3300 can support any type of interface to the user (such as "wearable" circuit, etc.).
[0219] Processing unit 3310 may be a physical or virtual processor and may implement various processes based on programs stored in memory 3320. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capabilities of computing device 3300. Processing unit 3310 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.
[0220] The computing device 3300 typically includes various computer storage media. Such media can be any media accessible to the computing device 3300, including but not limited to volatile and non-volatile media, or removable and non-removable media. The memory 3320 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (such as a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM) or flash memory), or any combination thereof. The storage unit 3330 can be any removable or non-removable medium, and can include a machine-readable medium that can be used to store information and / or data and can be accessed in the computing device 3300, such as a memory, a flash drive, a disk, or another other medium.
[0221] The computing device 3300 may also include additional removable / non-removable, volatile / non-volatile storage media. Fig.33 Although not shown in the figure, a magnetic disk drive for reading from and / or writing to a removable and nonvolatile magnetic disk and an optical disk drive for reading from and / or writing to a removable nonvolatile optical disk may be provided. In such a case, each drive may be connected to the bus (not shown) via one or more data media interfaces.
[0222] The communication unit 3340 communicates with another computing device via a communication medium. In addition, the functions of the components in the computing device 3300 can be implemented by a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 3300 can work in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general-purpose network nodes.
[0223] Input device 3350 may be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 3360 may be one or more of various output devices, such as a display, a loudspeaker, a printer, etc. Computing device 3300 may also communicate with one or more external devices (not shown) such as storage devices and display devices, with the aid of communication unit 3340, where one or more devices enable a user to interact with computing device 3300 or any device that enables computing device 3300 to communicate with one or more other computing devices (such as a network card, a modem, etc.), as desired. Such communication may be performed via an input / output (I / O) interface (not shown).
[0224] In some embodiments, some or all components of the computing device 3300 may also be arranged in a cloud computing architecture, rather than being integrated in a single device. In a cloud computing architecture, components may be provided remotely and work together to implement the functions described in the present disclosure. In some embodiments, cloud computing provides computing, software, data access, and storage services, which do not require end users to understand the physical location or configuration of the system or hardware that provides these services. In various embodiments, cloud computing provides services via a wide area network (such as the Internet) using a suitable protocol. For example, a cloud computing provider provides applications via a wide area network, which can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data may be stored on a server at a remote location. The computing resources in a cloud computing environment may be merged or distributed at a location in a remote data center. Cloud computing infrastructure may provide services through a shared data center, but they act as a single access point for users. Therefore, cloud computing architecture may be used to provide components and functions described herein from a service provider at a remote location. Alternatively, they may be provided from a conventional server or directly installed or otherwise installed on a client device.
[0225] The computing device 3300 may be used to implement video encoding / decoding in embodiments of the present invention. The memory 3320 may include one or more video encoding / decoding modules 3325 having one or more program instructions. These modules may be accessed and executed by the processing unit 3310 to perform the functions of various embodiments described herein.
[0226] In an example embodiment performing video encoding, input device 3350 may receive video data as input to be encoded 3370. For example, the video data may be processed by video codec module 3325 to generate an encoded bitstream. The encoded bitstream may be provided as output 3380 via output device 3360.
[0227] In an example embodiment performing video decoding, input device 3350 may receive an encoded bitstream as input 3370. The encoded bitstream may be processed, for example, by video codec module 3325 to generate decoded video data. The decoded video data may be provided as output 3380 via output device 3360.
[0228] Although the present disclosure has been specifically shown and described with reference to the preferred embodiments of the present disclosure, it will be appreciated by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present application as defined by the appended claims. These changes are intended to be encompassed by the scope of the present application. Therefore, the foregoing description of the embodiments of the present application is not intended to be limiting.
Claims
1. A video processing method, comprising: during a conversion between a video unit of a video and a bitstream of the video unit, determining whether at least one of a first set of samples or a second set of samples is outside a boundary associated with the video unit; applying a weighting process to the first sample set and the second sample set based on the determining; generating a prediction based on the weighted first sample set and the weighted second sample set; as well as The converting is performed based on the prediction.
2. The method of claim 1 , wherein applying the weighting process to the first sample set and the second sample set comprises: Based on the determination that the first sample set is outside the boundary and the second sample set is inside the boundary, mixing the first sample set based on a first weighting factor; as well as The second set of samples is mixed based on a second weighting factor, wherein the first weighting factor is different from the second weighting factor.
3. The method according to claim 2, wherein the first weighting factor of the first sample set for a motion compensated block or sub-block is set according to a predetermined rule. The method according to claim 3 , wherein the first weighting factor is set to a specific value.
5. The method of claim 2, wherein a third weighting factor for motion compensated samples around a bordering boundary of the first sample set and the second sample set is set according to a predetermined rule.
6. The method according to claim 5, wherein the predetermined rule comprises one of the following: increasing the weighting value as the distance from the abutment limit increases; or The weight value decreases as the distance from the butting limit increases. The method of claim 2 , wherein the first weighting factor and the second weighting factor are determined based on a distance to the boundary.
8. The method of claim 2, wherein the first weighting factor and the second weighting factor are determined based on distances to abutment boundaries of the first sample set and the second sample set.
9. The method of claim 1, wherein an outer boundary portion of a motion compensated block or sub-block is not used to generate the prediction if a reference block is partially or fully outside the boundary.
10. The method of claim 1, wherein whether to generate the prediction by applying the weighting process based on the determination is implicitly derived based on information of a codec of the video unit.
11. The method of claim 1, wherein whether to generate the prediction by applying the weighting process based on the determination is indicated at a video unit level.
12. The method of claim 1, wherein generating the prediction by applying the weighting process based on the determination is forced for a certain block without indication.
13. The method according to claim 12, wherein the certain block comprises at least one of the following: Multiple hypothesis blocks, or Bidirectional prediction block.
14. The method of claim 1, wherein if a reference block is partially or completely outside the boundary, an original motion compensated block or sub-block is modified to another motion compensated block or sub-block.
15. The method of claim 1, wherein if a reference block is partially or completely outside the boundary, an original motion vector used to generate the motion compensated block or sub-block is modified to another motion vector.
16. The method of claim 1, wherein the first sample set is in a first prediction and the second sample set is in a second prediction, and wherein applying the weighting process comprises: The first prediction and the second prediction are blended.
17. The method of claim 16, wherein at least one prediction block or sub-block is a motion compensated prediction block or sub-block.
18. The method of claim 16, wherein the first prediction is non-inter prediction and the second prediction is motion compensated prediction.
19. The method of claim 16, wherein the first prediction and the second prediction are both motion compensated predictions.
20. The method of claim 16, wherein the first prediction and the second prediction are generated from the same inter prediction direction.
21. The method of claim 16, wherein the first prediction and the second prediction are generated from different inter-prediction directions.
22. The method of claim 16, wherein blending the first prediction and the second prediction is applied to at least one of: a merge mode or a variant mode of the merge mode.
23. The method according to claim 22, wherein the variant mode of the merge mode comprises at least one of the following: Bilateral matching (BM) mode, Adaptive decoder side motion vector refinement (ADMVR) mode, Combined Inter and Intra Prediction (CIIP) mode, Template Matching (TM) mode, Merged Mode Vector Difference (MMVD) mode, Affine merge mode, Sub-block based temporal motion vector prediction (Sb TMVP) mode, Geometric Partitioning Mode (GPM) mode, GPM-MMVD mode, or GPM-TM mode.
24. The method of claim 16, wherein blending the first prediction and the second prediction is applied to at least one of: an Advanced Motion Vector Prediction (AMVP) mode or a variant of the AMVP mode.
25. The method of claim 24, wherein the variant of the AMVP mode comprises at least one of the following: Symmetric Motion Vector Difference (SMVD), AMVP merge mode, or Affine AMVP mode.
26. The method of claim 16, wherein blending the first prediction and the second prediction is enabled for prediction based on decoder-side motion vector refinement.
27. The method of claim 26, wherein mixing the first prediction and the second prediction is allowed for predictions generated by a decoder-side motion vector refinement related codec mode.
28. The method of claim 16, wherein blending the first prediction and the second prediction is not allowed for prediction based on decoder-side motion vector refinement.
29. The method of claim 16, wherein mixing the first prediction and the second prediction is not allowed for predictions generated by a decoder-side motion vector refinement related codec mode.
30. The method of claim 16, wherein blending the first prediction and the second prediction is allowed for predictions generated based on some kind of sub-block based prediction method.
31. The method of claim 16, wherein mixing the first prediction and the second prediction is allowed for some kind of coding unit / prediction unit (CU / PU) based prediction method.
32. The method of claim 16, wherein blending the first prediction and the second prediction is not allowed for predictions generated based on some kind of sub-block based prediction method.
33. The method of claim 16, wherein blending the first prediction and the second prediction does not coexist with some kind of sub-block based prediction method.
34. The method of claim 16, wherein mixing the first prediction and the second prediction is not allowed for certain kinds of CU / PU based prediction methods.
35. The method of claim 34, wherein hybridizing the first prediction and the second prediction does not coexist with some kind of CU / PU based prediction method.
36. The method of any one of claims 1-34, wherein the boundary is one of: Refer to the image border, Referring to the sub-picture boundaries, reference strip boundaries, or Reference tile boundaries.
37. The method of claim 36, wherein the identifier of the reference sub-picture is the same as the identifier of the current sub-picture.
38. The method of claim 36, wherein in case of a motion constrained tile set (MCTS), motion vectors are constrained to be within tiles having the same coordinates or position in the reference picture.
39. The method of claim 1, wherein if two motion compensated samples are outside the boundary, a final prediction value is generated without blending the motion compensated samples.
40. The method of claim 39, wherein a non-outer boundary sample is used to generate the final prediction value if the non-outer boundary sample is closer to the boundary.
41. The method of claim 39, wherein the final prediction value is generated according to a rule based on non-external boundary prediction samples inside the current mixed block.
42. The method of claim 39, wherein the final prediction value is generated by weighted blending in the same manner as samples inside the boundary are blended.
43. The method of claim 1, wherein the values of the outer boundary samples of the motion compensated block / sub-block are set according to a predetermined rule.
44. The method of claim 43, wherein the outer boundary samples refer to the outer boundary samples after a motion compensation process.
45. The method of claim 43, wherein the predetermined rule is based on non-external boundary motion compensated sample values inside the boundary.
46. The method of claim 45, wherein the non-exterior boundary motion compensated sample values at a first row located inside the boundary are copied above for an above exterior boundary sample.
47. The method of claim 45, wherein the non-exterior boundary motion compensated sample values at a first column located inside the boundary are duplicated on the left side for left exterior boundary samples.
48. The method of claim 45, wherein the non-exterior boundary motion compensated sample value at the upper left corner inside the boundary is copied for an upper left exterior boundary sample.
49. The method of claim 43, wherein the outer boundary samples refer to the outer boundary samples after bidirectional optical flow (BDOF) and before the weighting process.
50. The method of claim 43, wherein the rule is based on sample values of non-exterior boundary BDOF refinements that are interior to the boundary.
51. The method of claim 50, wherein sample values of the non-exterior boundary BDOF refinement at a first row located inside the boundary are copied above for upper exterior boundary samples.
52. The method of claim 50, wherein the sample values of the non-exterior boundary BDOF refinement at a first column located inside the boundary are duplicated on the left side for the left exterior boundary samples.
53. The method of claim 50, wherein the sample value of the non-exterior boundary BDOF refinement located at the upper left corner of the interior of the boundary is replicated for the upper left exterior boundary sample.
54. The method of claim 1, wherein if the prediction block or subblock pointed to by the motion vector is outside the boundary, a new prediction block or subblock is generated according to a predetermined rule.
55. The method of claim 54, wherein the new prediction block or sub-block is generated based on a zero motion vector.
56. The method of claim 54, wherein the new prediction block or sub-block is replaced by a co-located block.
57. The method of claim 54, wherein the new prediction block or sub-block is replaced by a non-external boundary prediction block or sub-block that is closest to the external boundary block or sub-block.
58. The method of claim 1, wherein the outer boundary check is based on motion vectors prior to decoder-side motion refinement.
59. The method of claim 58, wherein the decoder-side motion refinement comprises at least one of: Template matching based motion refinement, or Motion refinement based on bilateral matching.
60. The method of claim 1, wherein the outer boundary check is based on motion vectors after some stage of DMVR based motion refinement.
61. The method of claim 1, wherein outer boundary checking is based on motion vectors after all stages of DMVR based motion refinement.
62. The method of claim 1, wherein outer boundary checking is based on motion vectors after TM-based motion refinement.
63. The method of claim 1, wherein blending weights for outer boundary samples are determined based on the outer boundary check.
64. The method of claim 1, wherein whether to deblock is controlled at a codec tree unit (CTU) or codec unit (CU) level.
65. The method of claim 1, wherein the intra MTS type is determined based on a syntax element indicated at a picture parameter set (PPS) level.
66. The method of claim 1, wherein adaptive mixing weights are used for OBMC codec blocks, wherein the adaptive mixing weights are based on codec information other than neighboring reconstructed / predicted samples.
67. The method of claim 66, wherein the adaptive blending weights are determined by temporally encoded information in a reference picture.
68. The method of claim 66, wherein the adaptive blending weights are selected based on more than one lookup table.
69. A method according to claim 68, wherein a weight index is derived, or Wherein the weight index is indicated.
70. The method of any one of claims 1-69, wherein an indication of whether and / or how to apply the weighting process to the first sample set and the second sample set based on the determination is indicated in one of: Sequence level, Picture group level, Picture level, Stripe level, or Tile group level.
71. The method of any one of claims 1-69, wherein an indication of whether and / or how to apply the weighting process to the first sample set and the second sample set based on the determination is indicated in one of: Sequence header, Picture header, Sequence Parameter Set (SPS), Video Parameter Set (VPS), Dependency Parameter Set (DPS), Decoding Capability Information (DCI), Picture Parameter Set (PPS), Adaptive Parameter Set (APS), Strip header, or Tile group header.
72. The method of any one of claims 1-69, wherein an indication of whether and / or how to apply the weighting process to the first sample set and the second sample set based on the determination is included in one of: Prediction Block (PB), Transform Block (TB), Codec Block (CB), Prediction Unit (PU), Transformation Unit (TU), Codec Unit (CU), Virtual Pipeline Data Unit (VPDU), Codec Tree Unit (CTU), CTU line, Strips, Tile, sub-image, or An area that contains more than one sample or pixel.
73. The method of any one of claims 1-69, further comprising: Determine, based on coded information of the video unit, whether and / or how to apply the weighting process to the first sample set and the second sample set based on the determination, wherein the coded information includes at least one of the following: Block size, Color format, Single-tree partitioning and / or dual-tree partitioning, Color components, Strip type, or Image type.
74. The method of any one of claims 1-73, wherein the converting comprises encoding the video unit into the bitstream.
75. The method of any one of Claims 1-73, wherein the converting comprises decoding the video unit from the bitstream.
76. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform a method according to any one of claims 1-105.
77. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform a method according to any one of claims 1-105.
78. A non-transitory computer-readable recording medium storing a bit stream of a video generated by a method performed by a video processing device, wherein the method comprises: determining whether at least one of the first sample set or the second sample set is outside a boundary associated with a video unit of the video; applying a weighting process to the first sample set and the second sample set based on the determining; generating a prediction based on the weighted first sample set and the weighted second sample set; as well as A bitstream for the video unit is generated based on the prediction.
79. A method for storing a bitstream of a video, comprising: determining whether at least one of the first sample set or the second sample set is outside a boundary associated with a video unit of the video; applying a weighting process to the first sample set and the second sample set based on the determining; generating a prediction based on the weighted first sample set and the weighted second sample set; generating a bitstream for the video unit based on the prediction; as well as The bit stream is stored in a non-transitory computer-readable recording medium.
