Method and device for video processing and medium

Abstract

Embodiments of the present disclosure provide a solution for video processing. A method for video processing is presented. The method includes applying, for a conversion between a video unit of a video and a bitstream of the video unit, surround motion compensation (WAMC) during derivation of motion information for the video unit; and performing a conversion based on the derived motion information.

Description

Technical Field

[0001] Embodiments of the present disclosure relate generally to video processing techniques, and more particularly, to surround motion compensation. Background Art

[0002] Nowadays, digital video capabilities are being used in all aspects of people's lives. Various types of video compression technologies, such as MPEG-2, MPEG-4, ITU-TH. 263, ITU-TH.264 / MPEG-4 Part 10 Advanced Video Codec (AVC), ITU-TH.265 High Efficiency Video Codec (HEVC) standard, and Versatile Video Codec (VVC) standard have been proposed for video encoding / decoding. However, it is generally expected to further improve the encoding and decoding efficiency of video encoding and decoding technologies. Summary of the invention

[0003] The embodiments of the present disclosure provide a video processing solution.

[0004] In a first aspect, a method for video processing is proposed. The method includes: for conversion between a video unit of a video and a bitstream of the video unit, applying surround motion compensation (WAMC) during derivation of motion information for the video unit; and performing the conversion based on the derived motion information. In this way, codec efficiency and codec performance can be improved.

[0005] In a second aspect, another method for video processing is proposed. The method includes: for conversion between a video unit of a video and a bitstream of the video unit, determining whether a border filling process is allowed for the video unit based on whether surround motion compensation (WAMC) is enabled; and performing conversion based on the determination. In this way, codec efficiency and codec performance can be improved.

[0006] In a third aspect, a device for video processing is provided. The device includes a processor and a non-volatile memory having instructions thereon. The instructions, when executed by the processor, cause the processor to perform the method according to the first aspect of the present disclosure.

[0007] In a fourth aspect, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores instructions for causing a processor to execute the method according to the first aspect of the present disclosure.

[0008] In a fifth aspect, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by an apparatus for video processing. The method includes: applying surround motion compensation (WAMC) during derivation of motion information for the video unit; and performing conversion based on the derived motion information.

[0009] In a sixth aspect, a method for storing a bitstream of a video is provided. The method comprises: applying surround motion compensation (WAMC) during derivation of motion information for the video unit; performing conversion based on the derived motion information; and storing the bitstream in a non-transitory computer-readable recording medium.

[0010] In a seventh aspect, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by an apparatus for video processing. The method includes: determining whether a border filling process is allowed for a video unit based on whether surround motion compensation (WAMC) is enabled; and performing a conversion based on the determination.

[0011] In an eighth aspect, a method for storing a bitstream of a video is provided. The method includes: determining whether a border filling process is allowed for the video unit based on whether surround motion compensation (WAMC) is enabled; generating a bitstream based on the determination; and storing the bitstream in a non-transitory computer-readable recording medium.

[0012] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become more apparent through the following detailed description with reference to the accompanying drawings.In the exemplary embodiments of the present disclosure, the same reference numerals generally refer to the same components.

[0014] Figure 1 illustrates a block diagram of an example video encoding and decoding system according to some embodiments of the present disclosure;

[0015] Figure 2 illustrates a block diagram of a first example video encoder according to some embodiments of the present disclosure;

[0016] Figure 3 illustrates a block diagram of an example video decoder according to some embodiments of the present disclosure;

[0017] Figure 4 illustrates an example of an encoder block diagram;

[0018] Figure 5 67 intra prediction modes are illustrated;

[0019] Figure 6 The reference samples used for wide-angle intra prediction are shown;

[0020] Figure 7 The diagram shows the problem of discontinuity in the case of orientations exceeding 45°;

[0021] Figure 8 illustrates horizontal surround motion compensation in VVC;

[0022] Fig. 9 An example of 3x2 frame packing is shown;

[0023] Fig.10 The MMVD search point is illustrated;

[0024] Fig.11 is a graphic representation of the symmetric MVD pattern;

[0025] Fig.12 The figure shows the extended CU area used in BDOF;

[0026] Fig.13 The diagram shows the affine motion model based on control points;

[0027] Fig.14 The affine MVF of each sub-block is shown;

[0028] Fig.15 The diagram shows the position of the inherited affine motion predictor;

[0029] Fig.16 Illustration of control point motion vector inheritance;

[0030] Fig.17 illustrates the locations of candidate positions for a constructed affine merge pattern;

[0031] Fig.18 is a diagram of the use of motion vectors for the proposed combination method;

[0032] Fig.19 Sub-block MV VSB and pixel Δv(i,j) are shown;

[0033] Fig. 20A shows the spatial neighboring blocks used by ATVMP, and Fig. 20B shows the derivation of a sub-CU motion field by applying motion shifts from spatial neighbors and scaling motion information from corresponding collocated sub-CUs;

[0034] Fig.21 Positional illumination compensation is shown;

[0035] Fig. 22 The diagram shows no subsampling for the short edges;

[0036] Fig.23illustrates decoding side motion vector refinement;

[0037] Fig.24 A diamond-shaped area in the search area is shown;

[0038] Fig.25 The positions of the spatial merging candidates are shown;

[0039] Fig.26 illustrates candidate pairs considered for redundancy check of spatial merging candidates;

[0040] Fig. 27 is an illustration of motion vector scaling for temporal merging candidates;

[0041] Fig.28 Candidate positions for time domain merging candidates C0 and C1 are shown;

[0042] Fig.29 The VVC spatial domain neighboring blocks of the current block are shown;

[0043] Fig.30 is a diagram of the virtual blocks in the i-th search round;

[0044] Fig.31 An example of GPM segmentation grouped at the same angle is shown;

[0045] Fig.32 Unidirectional prediction MV selection for geometric partitioning mode is shown;

[0046] Fig.33 An example generation of warp weights w0 using a geometric partitioning pattern is shown;

[0047] Fig.34 The figure illustrates the spatial neighboring blocks used to derive spatial merge candidates;

[0048] Fig.35 The diagram shows that template matching is performed on a search area around the initial MV;

[0049] Fig.36 is a diagram of the sub-blocks of the OBMC application;

[0050] Fig.37 The SBT position, type and transformation type are illustrated;

[0051] Fig.38 The figure shows the neighboring samples used to calculate the SAD;

[0052] Fig.39 The figure illustrates the neighboring samples used to calculate the SAD of the sub-CU level motion information;

[0053] Fig.40 The sorting process is shown;

[0054] Fig.41 shows the reordering process in the encoder;

[0055] Fig.42 shows the reordering process in the decoder;

[0056] Fig.43 Bidirectional prediction with OOB is shown;

[0057] Fig.44 The MC boundary filling method is shown;

[0058] Fig.45 An example of deriving an M×4 padding block with a left padding direction is shown;

[0059] Fig.46 A flowchart of a method for video processing according to an embodiment of the present disclosure is illustrated;

[0060] Fig.47 A flowchart of a method for video processing according to an embodiment of the present disclosure is illustrated; and

[0061] Fig.48 Illustrated is a block diagram of a computing device in which various embodiments of the present disclosure may be implemented.

[0062] Throughout the drawings, same or similar reference numbers generally refer to same or similar elements. DETAILED DESCRIPTION

[0063] The principle of the present disclosure is now described in conjunction with some embodiments. It should be understood that these embodiments are described only for illustrative purposes to help those skilled in the art understand and implement the present disclosure without any limitation to the scope of the present disclosure. The disclosure described herein can be implemented in various ways except for the manner described below.

[0064] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0065] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes the particular feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an example embodiment, it is considered to be within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in other embodiments, whether or not explicitly described.

[0066] It should be understood that although the terms "first" and "second" etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of the exemplary embodiments, a first element may be referred to as a second element, and similarly a second element may be referred to as a first element. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0067] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms as well. It should also be understood that when used herein, the terms "comprises", "comprising", "has", "having", "includes", and / or "including" specify the presence of stated features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof. Example Environment

[0068] Figure 1 1 is a block diagram illustrating an example video codec system 100 that may utilize the techniques of the present disclosure. As shown, the video codec system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a video encoding device, and the destination device 120 may also be referred to as a video decoding device. In operation, the source device 110 may be configured to generate encoded video data, and the destination device 120 may be configured to decode the encoded video data generated by the source device 110. The source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0069] The video source 112 may include a source such as a video capture device. Examples of a video capture device include, but are not limited to, an interface for receiving video data from a video content provider, a computer graphics system for generating video data, and / or a combination thereof.

[0070] The video data may include one or more pictures. The video encoder 114 encodes the video data from the video source 112 to generate a bit stream. The bit stream may include a bit sequence that forms a coded representation of the video data. The bit stream may include coded pictures and associated data. The coded pictures are coded representations of pictures. The associated data may include sequence parameter sets, picture parameter sets and other grammatical structures. The I / O interface 116 may include a modulator / demodulator and / or a transmitter. The coded video data may be directly transmitted to the destination device 120 via the network 130A via the I / O interface 116. The coded video data may also be stored on a storage medium / server 130B for access by the destination device 120.

[0071] The destination device 120 may include an I / O interface 126, a video decoder 124, and a display device 122. The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain encoded video data from the source device 110 or the storage medium / server 130B. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to the user. The display device 122 may be integrated with the destination device 120, or may be outside the destination device 120, which is configured to be connected to an external display device interface.

[0072] The video encoder 114 and the video decoder 124 may operate according to a video compression standard, such as the High Efficiency Video Codec (HEVC) standard, the Versatile Video Codec (VVC) standard, and other existing and / or future standards.

[0073] Figure 2 is a block diagram showing an example of a video encoder 200 according to some embodiments of the present disclosure, which may be Figure 1 An example of a video encoder 114 in the system 100 is shown.

[0074] Video encoder 200 may be configured to implement any or all of the techniques of this disclosure. Figure 2 In the example of , video encoder 200 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video encoder 200. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0075] In some embodiments, the video encoder 200 may include a partitioning unit 201, a prediction unit 202, a residual generation unit 207, a transformation unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transformation unit 211, a reconstruction unit 212, a buffer 213 and an entropy coding unit 214, and the prediction unit 202 may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205 and an intra-frame prediction unit 206.

[0076] In other examples, the video encoder 200 may include more, fewer, or different functional components. In one example, the prediction unit 202 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode, where at least one reference picture is a picture where the current video block is located.

[0077] Furthermore, although some components (such as the motion estimation unit 204 and the motion compensation unit 205) may be integrated, for the purpose of explanation, these components are described in detail below. Figure 2 are shown separately in the example.

[0078] The partitioning unit 201 may partition a picture into one or more video blocks. The video encoder 200 and the video decoder 300 may support various video block sizes.

[0079] The mode selection unit 203 may select one of a plurality of coding modes (intra or inter) based on the error result, for example, and provide the resulting intra-coded block or inter-coded block to the residual generation unit 207 to generate residual block data, and to the reconstruction unit 212 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 203 may select a combination of intra and inter prediction (CIIP) modes, where the prediction is based on an inter prediction signal and an intra prediction signal. In the case of inter prediction, the mode selection unit 203 may also select a resolution for the motion vector (e.g., sub-pixel precision or integer pixel precision) for the block.

[0080] To perform inter-frame prediction on the current video block, the motion estimation unit 204 may generate motion information for the current video block by comparing the current video block with one or more reference frames from the buffer 213. The motion compensation unit 205 may determine a predicted video block for the current video block based on the motion information and decoded samples of pictures from the buffer 213 other than the picture associated with the current video block.

[0081] The motion estimation unit 204 and the motion compensation unit 205 may perform different operations on the current video block, for example, depending on whether the current video block is in an I slice, a P slice, or a B slice. As used herein, an "I slice" may refer to a portion of a picture consisting of macroblocks, all of which are based on macroblocks within the same picture. Furthermore, as used herein, in some aspects, a "P slice" and a "B slice" may refer to a portion of a picture consisting of macroblocks that are independent of macroblocks in the same picture.

[0082] In some examples, the motion estimation unit 204 may perform unidirectional prediction on the current video block, and the motion estimation unit 204 may search the reference pictures of list 0 or list 1 to find the reference video block for the current video block. The motion estimation unit 204 may then generate a reference index and a motion vector, the reference index indicating the reference picture in list 0 or list 1 containing the reference video block, and the motion vector indicating the spatial displacement between the current video block and the reference video block. The motion estimation unit 204 may output the reference index, the prediction direction indicator, and the motion vector as the motion information of the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0083] Alternatively, in other examples, the motion estimation unit 204 may perform bidirectional prediction on the current video block. The motion estimation unit 204 may search the reference pictures in list 0 for a reference video block for the current video block, and may also search the reference pictures in list 1 for another reference video block for the current video block. The motion estimation unit 204 may then generate a plurality of reference indexes and a plurality of motion vectors, the plurality of reference indexes indicating a plurality of reference pictures in list 0 and list 1 containing a plurality of reference video blocks, and the plurality of motion vectors indicating a plurality of spatial displacements between the plurality of reference video blocks and the current video block. The motion estimation unit 204 may output the plurality of reference indexes and the plurality of motion vectors of the current video block as motion information of the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the plurality of reference video blocks indicated by the motion information of the current video block.

[0084] In some examples, motion estimation unit 204 may output a complete set of motion information for use in a decoding process by a decoder. Alternatively, in some embodiments, motion estimation unit 204 may signal motion information of a current video block with reference to motion information of another video block. For example, motion estimation unit 204 may determine that motion information of a current video block is sufficiently similar to motion information of an adjacent video block.

[0085] In one example, motion estimation unit 204 may indicate to video decoder 300 a value in a syntax structure associated with the current video block that indicates that the current video block has the same motion information as another video block.

[0086] In another example, the motion estimation unit 204 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 300 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0087] As discussed above, the video encoder 200 may signal motion vectors in a predictive manner.Two examples of prediction signaling techniques that may be implemented by the video encoder 200 include Advanced Motion Vector Prediction (AMVP) and merge mode signaling.

[0088] The intra prediction unit 206 may perform intra prediction on the current video block. When the intra prediction unit 206 performs intra prediction on the current video block, the intra prediction unit 206 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a prediction video block and various syntax elements.

[0089] The residual generation unit 207 may generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) the prediction video block(s) of the current video block from the current video block. The residual data of the current video block may include residual video blocks corresponding to different sample portions of samples in the current video block.

[0090] In other examples, such as in skip mode, there may be no residual data for the current video block, and the residual generation unit 207 may not perform a subtraction operation.

[0091] Transform processing unit 208 may generate one or more transform coefficient video blocks for a current video block by applying one or more transforms to the residual video block associated with the current video block.

[0092] After transform processing unit 208 generates a transform coefficient video block associated with the current video block, quantization unit 209 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0093] The inverse quantization unit 210 and the inverse transform unit 211 may apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct a residual video block from the transform coefficient video block. The reconstruction unit 212 may add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by the prediction unit 202 to generate a reconstructed video block associated with the current video block for storage in the buffer 213.

[0094] After reconstruction unit 212 reconstructs the video block, a loop filtering operation may be performed to reduce video blocking artifacts in the video block.

[0095] The entropy coding unit 214 may receive data from other functional components of the video encoder 200. When the entropy coding unit 214 receives the data, the entropy coding unit 214 may perform one or more entropy coding operations to generate entropy-coded data and output a bitstream including the entropy-coded data.

[0096] Figure 3 is a block diagram showing an example of a video decoder 300 according to some embodiments of the present disclosure, which may be Figure 1 An example of a video decoder 124 in the system 100 is shown.

[0097] Video decoder 300 may be configured to perform any or all of the techniques of this disclosure. Figure 3 In the example of , video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video decoder 300. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0098] exist Figure 3 In the example of , the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, and a reconstruction unit 306 and a buffer 307. In some examples, the video decoder 300 can perform a decoding process that is generally opposite to the encoding process described with respect to the video encoder 200.

[0099] The entropy decoding unit 301 can retrieve the encoded bitstream. The encoded bitstream may include entropy-encoded video data (e.g., encoded video data blocks). The entropy decoding unit 301 can decode the entropy-encoded video data, and the motion compensation unit 302 can determine motion information from the entropy-decoded video data, the motion information including motion vectors, motion vector precision, reference picture list indexes, and other motion information. The motion compensation unit 302 can determine the information, for example, by performing AMVP and merge mode. AMVP is used, including deriving several most likely candidates based on data and reference pictures from adjacent PBs. The motion information typically includes horizontal and vertical motion vector displacement values, one or two reference picture indexes, and in the case of prediction areas in B strips, also includes an identification of which reference picture list is associated with each index. As used herein, in some aspects, "merge mode" may refer to deriving motion information from blocks that are adjacent in the spatial or temporal domain.

[0100] The motion compensation unit 302 may generate a motion compensated block, possibly performing interpolation based on interpolation filtering.An identifier for the interpolation filter used with sub-pixel precision may be included in the syntax element.

[0101] The motion compensation unit 302 may calculate interpolated values ​​for sub-integer pixels of a reference block using interpolation filtering used by the video encoder 200 during encoding of the video block. The motion compensation unit 302 may determine the interpolation filter used by the video encoder 200 based on received syntax information, and the motion compensation unit 302 may use the interpolation filter to generate a prediction block.

[0102] The motion compensation unit 302 may use at least part of the syntax information to determine the size of blocks used to encode (multiple) frames and / or (multiple) slices of the encoded video sequence, partition information describing how each macroblock of a picture of the encoded video sequence is partitioned, a mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-coded block, and other information for decoding the encoded video sequence. As used herein, in some aspects, a "slice" may refer to a data structure that can be decoded independently of other slices of the same picture in terms of entropy coding and decoding, signal prediction, and residual signal reconstruction. A slice may be an entire picture, or it may be a region of a picture.

[0103] The intra prediction unit 303 may use, for example, an intra prediction mode received in the bitstream to form a prediction block from spatially neighboring blocks. The inverse quantization unit 303 inversely quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 303 applies an inverse transform.

[0104] The reconstruction unit 306 may obtain the decoded block, for example, by adding the residual block to the corresponding prediction block generated by the motion compensation unit 202 or the intra prediction unit 303. If necessary, a deblocking filter may also be applied to filter the decoded block in order to remove blocking artifacts. The decoded video block is then stored in a buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction, and the buffer 307 also generates the decoded video for presentation on a display device.

[0105] Some exemplary embodiments of the present disclosure will be described in detail below. It should be noted that the section titles used in this document are for ease of understanding, rather than limiting the embodiments disclosed in the section to that section. In addition, although some embodiments are described with reference to general video codecs or other specific video codecs, the disclosed technology is also applicable to other video codec technologies. In addition, although some embodiments describe the video coding and decoding steps in detail, it should be understood that the corresponding decoding steps of the cancellation codec will be implemented by the decoder. In addition, the term video processing includes video coding or compression, video decoding or decompression, and video transcoding, in which video pixels are represented from one compression format to another compression format or at different compression bit rates. 1. Brief Overview The present disclosure relates to video coding technology. Specifically, it relates to intra-frame block replication in video coding. It can be applied to existing video coding standards, such as HEVC or VCC (universal video codec) standards, or all other video coding standards. It can also be applied to future video coding standards or video codecs. 2. Introduction Video codec standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T developed H.261 and H.263, ISO / IEC developed MPEG-1 and MPEG-4 Visual, and the two organizations jointly developed the H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Codec (AVC) and H.265 / HEVC standards. Since H.262, video codec standards have been based on a hybrid video codec structure that utilizes temporal prediction plus transform codec. In order to explore future video codec technologies beyond HEVC, VCEG and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and put them into reference software called the Joint Video Exploration Model (JEM). In April 2018, the Joint Video Experts Team (JVET) between VCEG (Q6 / 16) and ISO / IEC JTC1SC29 / WG11 (MPEG) was formed to work on the VVC standard with the goal of a 50% bitrate reduction compared to HEVC. ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC1 / SC29 / WG5) are studying the potential need for standardization of future video coding technologies with significantly greater compression capabilities than the current VVC standard. The two groups are jointly conducting this exploration activity through a joint collaborative project called the Joint Video Exploration Team (JVET) to evaluate the compression technology designs proposed by their respective experts in this area. New coding features and coding methods implemented in the Enhanced Compression Model (ECM) software are being coordinated and explored by the ITU-T VCEG and ISO / IEC MPEG Joint Video Exploration Team (JVET) as potential enhanced video coding technologies beyond the VVC features. 2.1. Encoding and decoding process of typical video codecs Figure 4 An example of an encoder block diagram for VVC is shown, which contains three loop filter blocks: deblocking filter (DF), sample adaptive offset (SAO) and ALF. Unlike DF, which uses a predefined filter, SAO and ALF use the original samples of the current picture to reduce the mean square error between the original samples and the reconstructed samples by adding an offset and applying a finite impulse response (FIR) filter, with the offset and filter coefficients respectively signaled via the codec side information. ALF is located at the last processing stage of each picture and can be seen as a tool that attempts to capture and repair artifacts created by previous stages. 2.2. Intra-mode codec with 67 intra-prediction modes To capture arbitrary edge directions present in natural videos, such as used in HEVC, the number of directional intra modes is extended from 33 to 65, as Figure 5 , and planar and DC modes remain the same. These more dense directional intra prediction modes apply to all block sizes and luma and chroma intra prediction. In HEVC, each intra-coded block has a square shape with the length of each side being a power of 2. Therefore, no division operation is required to generate intra predictors using DC mode. In VVC, in general, blocks can have a rectangular shape requiring the use of division operations for each block. To avoid division operations for DC prediction, only the longer sides are used to calculate the average for non-square blocks. 2.2.1. Wide-angle intra prediction Although 67 modes are defined in VVC, the exact prediction direction for a given intra prediction mode index also depends on the block shape. The normal angle intra prediction direction is defined to be from 45 degrees to -135 degrees in a clockwise direction. In VVC, several normal angle intra prediction modes are adaptively replaced with wide angle intra prediction modes for non-square blocks. The replaced mode is signaled using the original mode index, which is remapped to the index of the wide angle mode after parsing. The total number of intra prediction modes remains unchanged, i.e. 67, and the intra mode encoding and decoding method remains unchanged. To support these prediction directions, an upper reference of length 2W+1 and a left reference of length 2H+1 are defined as Figure 6 shown. The number of replaced modes in the wide direction mode depends on the aspect ratio of the block. The replaced intra prediction modes are shown in Table 2-1. Table 2-1 - Intra-prediction modes replaced by wide-angle mode like Figure 7 As shown, in the case of wide-angle intra prediction, two vertically adjacent predicted samples can use two non-adjacent reference samples. Therefore, a low-pass reference sample filter and edge smoothing are applied to wide-angle prediction to reduce the increase in the gap Δp α negative impact. If the wide angle mode represents a non-fractional offset. There are 8 modes in the wide angle mode that meet this condition, which are [-14, -12, -10, -6, 72, 76, 78, 80]. When a block is predicted by these modes, the samples in the reference buffer are directly copied without applying any interpolation. By this modification, the number of samples that need to be smoothed is reduced. In addition, it aligns the design of non-fractional modes in the conventional prediction mode and the wide angle mode. In VVC, 4:2:2 and 4:4:4 as well as 4:2:0 chroma formats are supported. The chroma derivation mode (DM) derivation table for the 4:2:2 chroma format was originally ported from HEVC, expanding the number of entries from 35 to 67 to align with the expansion of intra prediction modes. Since the HEVC specification does not support prediction angles below -135 degrees and above 45 degrees, the luma intra prediction modes ranging from 2 to 5 are mapped to 2. Therefore, the chroma DM derivation table for the 4:2:2: chroma format is updated by replacing some values ​​of the entries of the mapping table to more accurately convert the prediction angles for the chroma blocks. 2.3. Inter-frame prediction For each inter-predicted CU, the motion parameters consist of: motion vector, reference picture index and reference picture list usage index, as well as additional information required by the new codec features of VVC for inter-predicted sample generation. The motion parameters can be signaled in an explicit or implicit way. When the CU is encoded and decoded in skip mode, the CU is associated with one PU and the CU has no significant residual coefficients, nor decoded motion vector delta or reference picture index. The merge mode is specified as the motion parameters for the current CU are obtained from neighboring CUs, including spatial and temporal candidates, and additional schedules introduced in VVC. The merge mode can be applied to any inter-predicted CU, not just skip mode. Another alternative to the merge mode is the explicit transmission of motion parameters, where the motion vector for each reference picture list, the corresponding reference picture index and reference picture list usage flag, and other required information are explicitly signaled for each CU. 2.4.360-degree video encoding and decoding tool 2.4.1. Horizontal surround motion compensation Horizontal surround motion compensation is a 360-degree specific codec tool designed to improve the visual quality of reconstructed 360-degree videos in the equidistant cylindrical (ERP) projection format. In conventional motion compensation, when a motion vector refers to a sample that exceeds the picture boundary of a reference picture, repeated padding is applied to derive the value of the out-of-bounds sample by copying from those nearest neighbors on the corresponding picture boundary. For 360-degree videos, this method of repeated padding is not suitable and may result in visual artifacts known as "seam artifacts" in the reconstructed viewpoint video. Because 360-degree videos are captured on a sphere and inherently have no "boundaries", the out-of-bounds reference samples of the reference picture in the projected domain can always be obtained from neighboring samples in the spherical domain. For general projection formats, it may be difficult to derive corresponding neighboring samples in the spherical domain because it involves 2D to 3D and 3D to 2D coordinate conversions, as well as sample interpolation for fractional sample positions. For the left and right boundaries of the ERP projection format, the problem is simpler because the spherical neighborhood outside the left picture boundary can be obtained from the samples within the right picture boundary, and vice versa. Considering the widespread use of the ERP projection format and the relative ease of implementation, horizontal surround motion compensation is adopted in VVC to improve the visual quality of 360 videos encoded and decoded in the ERP projection format. The horizontal surround motion compensation process is depicted in Figure 8 When a portion of the reference block is outside the left (or right) boundary of the reference picture in the projected domain, the "out-of-bounds" portion is taken from the corresponding spherical neighbor located in the reference picture towards the right (or left) boundary in the projected domain instead of repetitive padding. Repetitive padding is only used for top and bottom picture boundaries. Figure 8 As depicted in , horizontal surround motion compensation can be combined with the non-normalized padding method commonly used in 360-degree video codecs (see padded ERP). In VVC, this is achieved by signaling a high-level syntax element to indicate a surround offset, which should be set to the width of the padding applied to the ERP picture; the syntax is used to adjust the position of the horizontal surround accordingly. The syntax is not affected by the specific amount of padding on the left and right picture boundaries, and therefore naturally supports asymmetric padding of ERP pictures, i.e., when the left and right padding are different. Horizontal surround motion compensation provides more meaningful information for motion compensation when the reference sample is outside the left and right boundaries of the reference picture. Under 360 video CTC, this tool improves compression performance not only in terms of rate-distortion performance, but also in terms of reducing seam artifacts and improving the subjective quality of the reconstructed 360-degree video. Horizontal surround motion compensation can also be used for other single-sided projection formats with constant sampling density in the horizontal direction, such as the adjusted equal-area projection in 360Lib. An example of the difference between the interpolation process by horizontal surround motion compensation and the interpolation process by repetition is shown below: The interpolation process of luma or chroma samples is repeated by padding: xInt i =Clip3(0, picW-1, xInt i ) (2-1) yInt i =Clip3(0, picH_1, yInt i ) (2-2) The process of interpolating luminance or chrominance samples through horizontal surround motion compensation: xInt=Clip3(0,picW-1,refWraparoundEnabledFlag?ClipH(o,picW,xIntL):xIntL) yInt i =Clip3(0, picH_1, yInt i ) (2-4) 2.4.2. Loop Filter Disabled Across Virtual Boundaries For projection formats consisting of multiple faces, no matter what type of compact frame packing arrangement is used, discontinuities occur between two or more adjacent faces in a frame packed picture. For example, consider Fig. 9 In the 3x2 frame packing configuration depicted in , the upper and lower halves of the frame packing picture are discontinuous in 3D geometry. If a loop filtering operation is performed across such a discontinuity, surface seam artifacts may become visible in the reconstructed video. To mitigate face seam artifacts and improve the subjective quality of 360-degree video, in-loop filtering operations should be disabled when discontinuities in pictures are packed across frames. In VVC, loop filtering operations across vertical and / or horizontal virtual boundaries are disabled, and the positions of these boundaries are signaled in the SPS or picture header. Compared to using two tiles (one for each continuous face set) and disabling cross-tile loop filtering operations, this technique is more flexible because it does not require the face size to be a multiple of the CTU size. The signaling is designed to be generic and applicable to other non-360-degree video use cases. The maximum number of vertical virtual boundaries is 3, and the maximum number of horizontal virtual boundaries is also 3. The distance between two virtual boundaries is greater than or equal to the CTU size, and the virtual boundary granularity is 8 luminance samples. Virtual boundaries can also be used in Progressive Decoding Refresh (GDR) or Progressive Intra Refresh (PIR), which are techniques to limit large bit rate variations between Intra (I) frames and Inter (P or B) frames while maintaining the same random access period. For this application, the position of the virtual boundary is signaled in the picture header, as the position can change from frame to frame. 2.5. Intra-block copy (IBC) Intra-block copying (IBC) is a tool adopted in the HEVC extension on SCC. It is well known that it significantly improves the encoding and decoding efficiency of screen content materials. Since the IBC mode is implemented as a block-level codec mode, block matching (BM) is performed at the encoder to find the best block vector (or motion vector) for each CU. Here, the block vector is used to represent the displacement from the current block to the reference block, which has been reconstructed in the current picture. The luminance block vector of the CU encoded and decoded by IBC is integer precision. The chrominance block vector is also rounded to integer precision. When combined with AMVR, the IBC mode can switch between 1-pixel and 4-pixel motion vector precision. The CU encoded and decoded by IBC is regarded as a third prediction mode in addition to the intra prediction mode and the inter prediction mode. The IBC mode is applicable to CUs whose width and height are both less than or equal to 64 luminance samples. On the encoder side, hash-based motion estimation is performed for IBC. The encoder performs RD checks on blocks with width or height no greater than 16 luma samples. For non-merge mode, a block vector search is first performed using a hash-based search. If the hash search does not return a valid candidate, a local search based on block matching is performed. In hash-based search, the hash key matching (32-bit CRC) between the current block and the reference blocks is extended to all allowed block sizes. The hash key calculation for each position in the current picture is based on 4x4 sub-blocks. For larger-sized current blocks, the hash key is determined to match the hash key of the reference block when the hash keys of all 4×4 sub-blocks match the hash keys of the corresponding reference positions. If the hash keys of multiple reference blocks are found to match the hash key of the current block, the block vector cost of each matching reference block is calculated, and the one with the smallest cost is selected. In the block matching search, the search range is set to cover the previous CTU and the current CTU. At CU level, the IBC mode is signaled using a flag and can be signaled as IBC AMV mode or IBC Skip / Merge mode as follows: - IBC skip / merge mode: The merge candidate index is used to indicate which block vector in the IBC-coded block from the neighboring candidates in the list is used to predict the current block. The merge list consists of spatial candidates, HMVP candidates and pairwise candidates. – IBC AMVP mode: Block vector differences are encoded in the same way as motion vector differences. The block vector prediction method uses two candidates as predictors, one from the left neighbor and one from the top neighbor (if encoded by IBC). When no neighbor is available, the default block vector will be used as the predictor. A flag is signaled to indicate the block vector predictor index. 2.6. Merge Mode with MVD (MMVD) In addition to the merge mode, a merge mode with motion vector difference (MMVD) is introduced in VCC, where implicitly derived motion information is directly used for prediction sample generation for the current CU. The MMVD flag is signaled immediately after sending the regular merge flag to specify whether the MMVD mode is used for the CU. In MMVD, after a merge candidate is selected, it is further refined by MVD information that is signaled. Further information includes a merge candidate flag, an index for specifying the magnitude of motion, and an index for indicating the direction of motion. In MMVD mode, one of the first two candidates in the merge list is selected to be used as the MV basis. The MMVD candidate flag is signaled to specify which of the first and second merge candidates is used. The distance index specifies the motion magnitude information and indicates a predefined offset from the starting point. Fig.10 As shown, the offset is added to the horizontal component or the vertical component of the starting MV. The relationship between the distance index and the predefined offset is specified in Table 2-2. Table 2-2 - Relationship between distance index and predefined offset The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate four directions, as shown in Table 2-3. It should be noted that the meaning of the MVD symbol may vary depending on the information of the starting MV. When the starting MV is a non-predicted MV or a bidirectionally predicted MV with two lists pointing to the same side of the current picture (i.e., the POCs of both references are greater than the POC of the current picture, or both are less than the POC of the current picture), the symbol in Table 2-3 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectionally predicted MV with two MVs pointing to different sides of the current picture (i.e., the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture) and the difference in POC in List 0 is greater than the difference in POC in List 1, the symbol in Table 2-3 specifies the sign of the MV offset added to the List 0 MV component of the starting MV, and the symbol for List 1 MV has an opposite value. Otherwise, if the POC difference in List 1 is greater than List 0, the sign in Table 2-3 specifies the sign of the MV offset added to the List 1 MV component of the starting MV, and the sign for List 0 MV has the opposite value. The MVD is scaled according to the difference in POC in each direction. If the difference in POC in both lists is the same, no scaling is required. Otherwise, if the difference in POC in list 0 is greater than the difference in POC in list 1, the MVD for list 1 is scaled by defining the difference in POC of L0 as td and the difference in POC of L1 as tb. If the difference in POC of L1 is greater than the difference in POC of L0, the MVD for list 0 is scaled in the same way. If the starting MV is unidirectionally predicted, the MVD is added to the available MVs. Table 2-3 - Signs of MV offsets specified by direction index Directions to IDX 00 01 10 11 x-axis + - N / A N / A y-axis N / A N / A + - 2.7. Symmetric MVD encoding and decoding In VVC, in addition to normal uni-prediction and bi-prediction mode MVD signaling, symmetric MVD mode for bi-prediction MVD signaling is applied. In symmetric MVD mode, motion information including reference picture indexes of both list 0 and list 1 and MVD of list 1 are not signaled but derived. The decoding process of the symmetric MVD mode is as follows: 1. At the stripe level, the variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1 are derived as follows: – If mvd_l1_zero_flag is 1, BiDirPredFlag is set equal to 0. – Otherwise, if the nearest reference picture in list-0 and the nearest reference picture in list-1 form a forward and backward reference pair of reference pictures or a backward and forward reference pair of reference pictures, then BiDirPredFlag is set to 1, and both list-0 and list-1 reference pictures are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0. 2. At the CU level, if the CU is bidirectionally predicted and BiDirPredFlag is equal to 1, a symmetric mode flag indicating whether the symmetric mode is used is explicitly signaled. When the symmetric mode flag is true, only mvp_l0_flag, mvp_l1_flag and MVD0 are explicitly signaled. The reference index for list-0 and list-1 is set equal to the reference picture pair, respectively. MVD1 is set equal to (-MVD0). The final motion vector is shown in the following equation. Fig.11 is an illustration for the symmetric MVD mode. In the encoder, symmetric MVD motion estimation starts with an initial MV evaluation. A set of initial MV candidates includes MVs obtained from unidirectional prediction search, MVs obtained from bidirectional prediction search, and MVs from the AMVP list. The MV with the lowest rate-distortion cost is selected as the initial MV for the symmetric MVD motion search. 2.8. Bidirectional Optical Flow (BDOF) The Bidirectional Optical Flow (BDOF) tool is included in VVC. BDOF, previously known as BIO, is included in JEM. BDOF in VVC is a simpler version that requires less computation, especially in terms of the number of multiplications and the size of the multipliers, compared to the JEM version. BDOF is used to refine the bidirectional prediction signal of a CU at the 4x4 sub-block level. BDOF is applied to a CU if the CU meets all of the following conditions: – the CU is encoded using “true” bi-prediction mode, i.e., one of the two reference pictures precedes the current picture in display order and the other follows the current picture in display order; – The distances from the two reference pictures to the current picture (i.e., POC differences) are the same; – Both reference images are short-term reference images; –CU is not encoded or decoded using affine mode or SbTMVP merge mode; – CU has more than 64 luma samples; – CU height and CU width are both greater than or equal to 8 luma samples; – BCW weight index indicates equal weight; – For the current CU, WP is not enabled; –CIIP mode is not used for the current CU. BDOF is only applied to the luminance component. As the name suggests, BDOF mode is based on the concept of optical flow and assumes that the motion of objects is smooth. For each 4x4 sub-block, motion refinement (v x ,v y ) is calculated by minimizing the difference between L0 and L1 predicted samples. Then motion refinement is used to adjust the bidirectionally predicted sample values ​​in the 4x4 sub-block. The following steps are applied in the BDOF process. First, the horizontal and vertical gradients of the two prediction signals, and It is calculated by directly computing the difference between two adjacent samples, i.e., Among them I (k) (i,j) is the sample value at coordinate (i,j) of the prediction signal in list k, k=0,1, and shift1 is calculated based on the luma bit depth bitDepth, because shift1=max(6,bitDepth-6). Then, the autocorrelations and cross-correlations of the gradients, S1, S2, S3, S5, and S6, are calculated as in where Ω is a 6×6 window around the 4×4 sub-block, and n a and n b The values ​​of are set equal to min(1, bitDepth-11) and min(4, bitDepth-8) respectively. Then the motion refinement (v x ,v y ) The autocorrelation and cross-correlation terms are derived using the following equations: in th′ BIO =2 max(5,BD-7) . [·] is the floor function, and Based on the motion refinement and gradients, the following adjustments are calculated for each sample in the 4x4 sub-block: Finally, the BDOF samples of the CU are calculated by adjusting the bidirectional prediction samples as follows: pred BDOF (x,y)=(I (0)(x,y)+I (1) (x,y)+b(x,y)+o offset )>>shift (2-12) These values ​​are chosen so that the multipliers in the BDOF process do not exceed 15 bits and the maximum bit width of the intermediate parameters in the BDOF process is kept within 32 bits. In order to derive the gradient value, some prediction samples I in the list k (k = 0, 1) outside the current CU boundary are (k) (i,j) needs to be generated. Fig.12 As depicted, BDOF in VVC uses an extended row / column around the CU boundary. To control the computational complexity of generating out-of-bounds prediction samples, prediction samples within the extended area (white positions) are generated by directly obtaining reference samples at nearby integer positions (using floor() operations on the coordinates) without interpolation, while a normal 8-tap motion compensated interpolation filter is used to generate prediction samples within the CU (gray positions). These extended sample values ​​are only used for gradient calculations. For the rest of the steps in the BDOF process, if any samples and gradient values ​​outside the CU boundary are needed, they are filled (i.e., repeated) from their nearest neighbors. When the width and / or height of a CU is greater than 16 luma samples, it will be divided into sub-blocks with a width and / or height equal to 16 luma samples, and the sub-block boundaries are considered as CU boundaries in the BDOF process. The maximum unit size for the BDOF process is limited to 16x16. For each sub-block, the BDOF process can be skipped. When the SAD between the initial L0 and L1 prediction samples is less than a threshold, the BDOF process is not applied to the sub-block. The threshold is set equal to (8*W*(H>>1), where W indicates the sub-block width, and H indicates the sub-block height. In order to avoid the additional complexity of the SAD calculation, the SAD between the initial L0 and L1 prediction samples calculated here in the DVMR process is reused. If BCW is enabled for the current block, i.e., the BCW weight index indicates unequal weights, then bidirectional optical flow is disabled. Similarly, if WP is enabled for the current block, i.e., luma_weight_lx_flag is 1 for either of the two reference pictures, then BDOF is also disabled. BDOF is also disabled when the CU is encoded and decoded in symmetric MVD mode or CIIP mode. 2.9. Combined Inter and Intra Prediction (CIIP) 2.10. Affine Motion Compensated Prediction In HEVC, only the translational motion model is applied to motion compensated prediction (MCP). In the real world, there are many kinds of motions, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, block-based affine transformation motion compensated prediction is applied. Fig.13 As shown, the affine motion field of a block is described by the motion information of two control points (4-parameters) or three control point motion vectors (6-parameters). For the 4-parameter affine motion model, the motion vector at a sample position (x, y) in a block is derived as: For the 6-parameter affine motion model, the motion vector at a sample position (x, y) in a block is derived as: Where (mv 0x ,mv 0y ) is the motion vector of the upper left control point, (mv 1x ,mv 1y ) is the motion vector of the upper right control point, and (mv 2x ,mv 2y ) is the motion vector of the lower left control point. To simplify motion compensated prediction, block-based affine transform prediction is applied. To derive the motion vector for each 4x4 luma subblock, the motion vector of the center sample of each subblock is Fig.14 As shown, it is calculated according to the above equation and rounded to 1 / 16 fractional precision. The motion compensation interpolation filter is then applied to generate a prediction for each subblock with a derived motion vector. The subblock size of the chroma component is also set to 4x4. The MV of the 4×4 chroma subblock is calculated as the average of the MVs of the four corresponding 4x4 luminance subblocks. Just like translational motion inter prediction, there are two affine motion inter prediction modes: affine merge mode and affine AMVP mode. 2.10.1. Affine Merge Prediction AF_MERGE mode can be applied to CUs with width and height greater than or equal to 8. In this mode, the CPMV of the current CU is generated based on the motion information of the spatially adjacent CUs. There can be up to five CPMVP candidates, and the index is signaled to indicate the candidate to be used for the current CU. The following three types of CPMV candidates are used to form the affine merge candidate list: – Inherited affine merge candidates inferred from the CPMV of neighboring CUs – Constructed affine merge candidate CPMVP derived using the translation MV of the neighboring CU –Zero MV In VVC, there are two maximum inherited affine candidates derived from the affine motion model of the neighboring blocks, one from the left neighboring CU and one from the upper neighboring CU. Fig.15 is shown in . Fig.16 The control point motion vector inheritance is shown. For the left predictor, the scanning order is A0->A1, and for the upper predictor, the scanning order is B0->B1->B2. Only the first inherited candidate from each side is selected. Pruning check is not performed between two inherited candidates. When a neighboring affine CU is identified, its control point motion vector is used to derive the CPMVP candidate in the affine merge list of the current CU. As shown in the figure, if the neighboring lower left block A is encoded and decoded in affine mode, the motion vectors v2, v3 and v4 of the upper left, upper right and lower left corners of the CU containing block A are obtained. When block A is encoded and decoded by a 4-parameter affine model, the two CPMVs of the current CU are calculated based on v2 and v3. In the case where block A is encoded and decoded by a 6-parameter affine model, the three CPMVs of the current CU are calculated based on v2, v3 and v4. The constructed affine candidates are candidates constructed by combining the neighboring translation motion information of each control point. The motion information of the control point is obtained from Fig.17 The CPMV is derived from the specified spatial and temporal neighbors shown in . k (k=1, 2, 3, 4) represents the kth control point. For CPMV1, check B2->B3->A2 blocks and use the MV of the first available block. For CPMV2, check B1->B0 blocks, and for CPMV3, check A1->A0 blocks. For TMVP, if available, TMVP is used as CPMV4. After obtaining the MVs of the four control points, an affine merge candidate is constructed based on the motion information. The following combinations of control point MVs are used for construction in order: {CPMV1,CPMV2,CPMV3},{CPMV1,CPMV2,CPMV4},{CPMV1,CPMV3,CPMV4}, {CPMV2,CPMV3,CPMV4},{CPMV1,CPMV2},{CPMV1,CPMV3} Combinations of 3 CPMVs construct 6-parameter affine merge candidates, and combinations of 2 CPMVs construct 4-parameter affine merge candidates. To avoid the motion scaling process, the relevant combinations of control point MVs are discarded if the reference indices of the control points are different. After the inherited affine merge candidates and the constructed affine merge candidates are checked, if the list is still not full, a zero MV is inserted at the end of the list. 2.10.2. Affine AMVP Prediction Affine AMVP mode can be applied to CUs with width and height both greater than or equal to 16. An affine flag in the CU level in the bitstream is signaled to indicate whether the affine AMVP mode is used, and then another flag is signaled to indicate 4-parameter affine or 6-parameter affine. In this mode, the difference between the CPMV of the current CU and its predicted value CPMVP is signaled in the bitstream. The affine AVMP candidate list size is 2, which is generated by using the following four types of CPMV candidates in order: – Inherited affine AMVP candidates inferred from the CPMV of neighboring CUs – Constructed affine AMVP candidate CPMVP derived using the translation MV of the neighboring CU –Translated MV from neighboring CU – Zero MV. The order of checking inherited affine AMVP candidates is the same as that of checking inherited affine merge candidates. The only difference is that for AVMP candidates, only affine CUs with the same reference picture as in the current block are considered. When the inherited affine motion prediction value is inserted into the candidate list, the pruning process is not applied. The constructed AMVP candidates are from Fig.17 The specified spatial neighbors shown in are derived. The same check order as for affine merge candidate construction is used. In addition, the reference picture indexes of neighboring blocks are also checked. The first block in the check order that is inter-coded and has the same reference picture as the current CU is used. Only when the current CU is encoded and decoded by the 4-parameter affine mode and both mv0 and mv1 are available, they are added to the affine AMVP list as a candidate. If the current CU is encoded and decoded by the 6-parameter affine mode and all three CPMVs are available, they are added to the affine AMVP list as a candidate. Otherwise, the constructed AMVP candidate is set to unavailable. If the affine AMVP list candidate is still less than 2 after the inherited affine AMVP candidate and the constructed AMVP candidate are checked, mv0, mv1, and mv2 will be added in order as translation MVs to predict all control point MVs of the current CU when available. Finally, if the affine AMVP list is still not full, zero MVs are used to fill the affine AMVP list. 2.10.3. Affine motion information storage In VVC, the CPMV of an affine CU is stored in a separate buffer. The stored CPMV is only used to generate the inherited CPMVP in affine merge mode and affine AMVP mode for the most recently decoded CU. The sub-block MV derived from the CPMV is used for motion compensation, merging of translation MVs / MV derivation of AMVP lists, and deblocking. To avoid picture row buffering for additional CPMV, affine motion data inherited from CUs of upper CTUs is handled differently from inheritance from normal neighboring CUs. If the candidate CU for affine motion data inheritance is in the upper CTU row, the lower left and lower right sub-block MVs in the row buffer are used for affine MVP derivation instead of the CPMVs. In this way, the CPMVs are only stored in the local buffer. If the candidate CU is 6-parameter affine coded, the affine model is downgraded to a 4-parameter model. Fig.18 As shown, along the upper CTU boundary, the lower left and lower right sub-block motion vectors of the CU are used for affine inheritance of the CU in the lower CTU. 2.10.4. Prediction Refinement with Optical Flow for Affine Mode Compared with pixel-based motion compensation, sub-block-based affine motion compensation can save memory access bandwidth and reduce computational complexity at the expense of prediction accuracy loss. In order to achieve finer granularity of motion compensation, prediction refinement with optical flow (PROF) is used to refine sub-block-based affine motion compensated predictions without increasing the memory access bandwidth for motion compensation. In VVC, after sub-block-based affine motion compensation is performed, the luma prediction samples are refined by adding the differences derived from the optical flow equation. PROF is described as the following four steps: Step 1) Sub-block based affine motion compensation is performed to generate a sub-block prediction I(i,j). Step 2) Spatial gradient g of sub-block prediction x (i,j) and g y (i,j) is calculated at each sample position using a 3-tap filter [-1,0,1]. The gradient calculation is exactly the same as in BDOF. g x (i,j)=(I(i+1,j)>>shift1)-(I(i-1,j)>>shift1) (2-15) g y (i,j)=(I(i,j+1)>>shift1)-(I(i,j-1)>>shift1) (2-16) Shift1 is used to control the precision of the gradient. For gradient calculation, the sub-block (i.e., 4x4) prediction is extended by one sample on each side. To avoid additional memory bandwidth and additional interpolation calculations, those extended samples on the extended boundary are copied from the nearest integer pixel position in the reference picture. Step 3) The brightness prediction refinement is calculated by the following optical flow equation. ΔI(i,j)=g x (i,j)*Δv x (i,j)+g y (i,j)*Δv y (i,j) (2-17) where Δv(i,j) is the difference between the sample MV calculated for the sample position (i,j) (denoted as v(i,j)) and the sub-block MV of the sub-block to which the sample (i,j) belongs, as Fig.19 As shown in . Δv(i,j) is quantized in units of 1 / 32 brightness sample precision. Since the affine model parameters and the sample positions relative to the sub-block center do not change from sub-block to sub-block, Δv(i,j) can be calculated for the first sub-block and reused for other sub-blocks in the same CU. Let dx(i,j) and dy(i,j) be the distance from the sample position (i,j) to the center (x SB ,y SB ), Δv(i,j) can be derived by the following equations: To maintain accuracy, the center of the sub-block (x SB ,y SB ) is calculated as ((W SB -1) / 2, (H SB -1) / 2), where W SB and H SB are the sub-block width and height respectively. For the 4-parameter affine model, For the 6-parameter affine model, Among them, (v 0x ,v 0y )、(v 1x ,v 1y )、(v 2x ,v 2y ) are the upper left, upper right and lower left control point motion vectors, w and h are the width and height of the CU. Step 4) Finally, the luma prediction refinement ΔI(i,j) is added to the sub-block prediction I(i,j). The final prediction I' is generated according to the following equation. I′(i,j)=I(i,j)+ΔI(i,j) (2-22) PROF is not applied to affine-coded CUs in two cases: 1) all control point MVs are the same, indicating that the CU has only translational motion; 2) affine motion parameters are greater than a certain limit, because sub-block based affine MC is downgraded to CU based MC to avoid large memory access bandwidth requirements. A fast codec method is applied to reduce the coding complexity of affine motion estimation with PROF. PROF is not applied in the following two cases during the affine motion estimation stage: a) If the CU is not a root block and its parent block does not select the affine mode as its best mode, PROF is not applied because the possibility of selecting the affine mode as the best mode for the current CU is low; b) If the magnitudes of the four affine parameters (C, D, E, F) are all less than a predetermined threshold and the current picture is not a low-latency picture, PROF is not applied because the improvement introduced by PROF for this case is small. In this way, affine motion estimation with PROF can be accelerated. 2.11. Sub-block based temporal motion vector prediction (SbTMVP) VVC supports a sub-block based temporal motion vector prediction (Sb TMVP) method. Similar to the temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses the motion field in the co-located picture to improve the motion vector prediction and merge mode for the CU in the current picture. The same co-located pictures used by TMVP are used for SbTVMP. SbTMVP differs from TMVP in the following two main aspects: –TMVP predicts motion at CU level, but SbTMVP predicts motion at sub-CU level; –Whereas TMVP extracts the temporal motion vector from the co-located block in the co-located picture (the co-located block is the bottom-right or center block relative to the current CU), SbTMVP applies motion shifting before extracting the temporal motion information from the co-located picture, where the motion shifting is obtained from the motion vector of one of the spatial neighboring blocks of the current CU. The SbTVMP process Fig. 20A and Fig. 20B SbTMVP predicts the motion vector of the sub-CU within the current CU in two steps. In the first step, Fig. 20A The spatial neighbor A1 in is checked. If A1 has a motion vector that uses the co-located picture as its reference picture, then this motion vector is selected as the motion shift to be applied. If no such motion is identified, the motion shift is set to (0,0). In the second step, the motion offset identified in step 1 is applied (i.e., added to the coordinates of the current block) to obtain Fig. 20B The co-located picture shown obtains sub-CU level motion information (motion vector and reference index). Fig. 20B The example in assumes that the motion shift is set to the motion of block A1. Then, for each sub-CU, the motion information of its corresponding block (the minimum motion grid covering the center sample) in the co-located picture is used to derive the motion information for the sub-CU. After the motion information of the co-located sub-CU is identified, it is converted into the motion vector and reference index of the current sub-CU in a manner similar to the TMVP process of HEVC, where temporal motion scaling is applied to align the reference picture of the temporal motion vector with the reference picture of the current CU. In VVC, a combined sub-block based merge list containing SbTVMP candidates and affine merge candidates is used for signaling of sub-block based merge mode. SbTVMP mode is enabled / disabled by a sequence parameter set (SPS) flag. If SbTMVP mode is enabled, the SbTMVP prediction value is added as the first entry in the list of sub-block based merge candidates, and then the affine merge candidates. The size of the sub-block based merge list is signaled in the SPS, and the maximum allowed size of the sub-block based merge list in VVC is 5. The sub-CU size used in SbTMVP is fixed to 8x8, and like the affine merge mode, the SbTMVP mode is only applicable to CUs whose width and height are both greater than or equal to 8. The encoding logic of the additional SbTMVP merge candidate is the same as other merge candidates, ie, for each CU in a P or B slice, an additional RD check is performed to decide whether to use the SbTMVP candidate. 2.12. Adaptive Motion Vector Resolution (AMVR) In HEVC, when use_integer_mv_flag is equal to 0 in the slice header, the motion vector difference (MVD) (between the motion vector of the CU and the predicted motion vector) is signaled in units of quarter luma samples. In VVC, the CU-level adaptive motion vector resolution (AMVR) scheme is introduced. AMVR allows the MVD of a CU to be encoded and decoded with different precisions. Depending on the mode for the current CU (normal AMVP mode or affine AMVP mode), the MVD of the current CU can be adaptively selected as follows: – Normal AMVP mode: quarter brightness samples, half brightness samples, integer brightness samples, or quad brightness samples. – Affine AMVP mode: quarter luma samples, integer luma samples, or 1 / 16 luma samples. The CU level MVD resolution indication is conditionally signaled if the current CU has at least one non-zero MVD component.If all MVD components (ie both horizontal and vertical MVD for reference list L0 and reference list L1) are zero, then quarter luma sample MVD resolution is inferred. For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter luma sample MVD precision is used for the CU. If the first flag is 0, no additional signaling is required and quarter luma sample MVD precision is used for the current CU. Otherwise, a second flag is signaled to indicate that half luma samples or other MVD precision (integer or four luma samples) are used for normal AMVP CUs. In the case of half luma samples, a 6-tap interpolation filter is used for half luma sample positions instead of the default 8-tap interpolation filter. Otherwise, a third flag is signaled to indicate whether integer luma samples or four luma sample MVD precision is used for normal AMVP CUs. In the case of an affine AMVP CU, a second flag is used to indicate whether integer luma samples or 1 / 16 luma sample MVD precision is used. In order to ensure that the reconstructed MV has the expected precision (quarter luma samples, half luma samples, integer luma samples, or four luma samples), the motion vector predictor for the CU will be rounded to the same precision as the MVD before being added with the MVD. Motion vector prediction values ​​are rounded towards zero (i.e., negative motion vector prediction values ​​are rounded towards positive infinity, and positive motion vector prediction values ​​are rounded towards negative infinity). The encoder uses RD check to determine the motion vector resolution for the current CU. In order to avoid always performing four CU-level RD checks for each MVD resolution, in VTM11, the RD check of MVD accuracy other than quarter brightness samples is only conditionally called. For the normal AVMP mode, the RD cost of quarter brightness sample MVD accuracy and integer brightness sample MV accuracy is first calculated. Then, the RD cost of integer brightness sample MVD accuracy is compared with the RD cost of quarter brightness sample MVD accuracy to determine whether it is necessary to further check the RD cost of four brightness sample MVD accuracy. When the RD cost of quarter brightness sample MVD accuracy is much smaller than the RD cost of integer brightness sample MVD accuracy, the RD check of four brightness sample MVD accuracy is skipped. Then, if the RD cost of integer brightness sample MVD accuracy is significantly greater than the best RD cost of the previously tested MVD accuracy, the check of half-block sample MVD accuracy is skipped. For the affine AMVP mode, if the affine inter mode is not selected after checking the rate-distortion cost of the affine merge / skip mode, merge / skip mode, quarter luma sample MVD accuracy normal AMVP mode, and quarter luma sample MVD accuracy affine AMVP mode, the 1 / 16 luma sample MV accuracy and 1 pixel MV accuracy affine inter modes are not checked. In addition, the affine parameters obtained in the quarter luma sample MV accuracy affine inter mode are used as the starting search points in the 1 / 16 luma sample and quarter luma sample MV accuracy affine inter modes. 2.13. Bidirectional Prediction with CU Level Weight (BCW) In HEVC, a bidirectional prediction signal is generated by averaging two prediction signals obtained from two different reference pictures and / or using two different motion vectors. In VVC, the bidirectional prediction mode is extended beyond simple averaging to allow a weighted average of the two prediction signals. P bi-pred =((8-w)*P0+w*P1+4)>>3 (2-23) In weighted average bidirectional prediction, five weights are allowed, w∈{-2,3,4,5,10}. For each bidirectional prediction CU, the weight w is determined in one of two ways: 1) For non-merged CUs, the weight index is signaled after the motion vector difference; 2) For merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index weights. BCW is only applied to CUs with 256 or more luma samples (i.e., CU width multiplied by CU height is greater than or equal to 256). For low-latency pictures, all 5 weights are used. For non-low-latency pictures, only 3 weights (w∈{3,4,5}) are used. – At the encoder, a fast search algorithm is applied to find the weight index without significantly increasing the encoder complexity. These algorithms are summarized below. For further details the reader is referred to the VTM software. When combined with AMVR, unequal weights for 1 pixel and 4 pixel motion vector precision are only conditionally checked if the current picture is a low-latency picture. – When combined with affine, affine ME is performed for unequal weights if and only if the affine mode is selected as the current best mode. – Unequal weights are only conditionally checked when the two reference pictures in bidirectional prediction are the same. – Unequal weights are not searched when certain conditions are met, which depend on the POC distance between the current picture and its reference pictures, the codec QP, and the temporal level. The BCW weight index is encoded using one context-encoded binary bit followed by a bypass-encoded binary bit. The first context-encoded binary bit indicates whether equal weights are used; and if unequal weights are used, an additional binary bit is signaled using bypass encoding to indicate that unequal weights are used. Weighted prediction (WP) is a codec tool supported by the H264 / AVC and HEVC standards for efficiently encoding and decoding video content with attenuation. Support for WP has also been added to the VVC standard. WP allows weighting parameters (weights and offsets) for each reference picture in each item in the reference picture lists L0 and L1 to be transmitted with a signal. Then, during motion compensation, the weights and offsets of the corresponding reference pictures are applied. WP and BCW are designed for different types of video content. In order to avoid the interaction between WP and BCW to complicate the VVC decoder design, if the CU uses WP, the BCW weight index is not transmitted with a signal, and w is inferred to be 4 (ie, equal weights are applied). For the merged CU, the weight index is inferred from the neighboring block based on the merge candidate index. This can be applied to the normal merge mode and the inherited affine merge mode. For the constructed affine merge mode, the affine motion information is constructed based on the motion information of up to 3 blocks. The BCW index for the CU using the constructed affine merge mode is simply set to be equal to the BCW index of the first control point MV. In VVC, CIIP and BCW cannot be applied to a CU together. When a CU is encoded and decoded using CIIP mode, the BCW index of the current CU is set to 2, for example, equal weight. 2.14. Local Illumination Compensation (LIC) Local illumination compensation (LIC) is a codec tool used to address the problem of local illumination changes between the current picture and its temporal reference picture. LIC is based on a linear model where a scaling factor and an offset are applied to the reference sample to obtain the predicted sample of the current block. Specifically, LIC can be modeled by the following equation: P(x,y)=α·P r (x+v x ,y+v y )+β Among them, P(x,y) is the prediction signal of the current block at the coordinate (x,y); P r (x+v x ,y+v y ) is the motion vector (v x ,v y ) points to the reference block; and α and β are the corresponding scaling factors and offsets applied to the reference block. Fig.21 The LIC process is shown. Fig.21 In , when LIC is applied to a block, by minimizing the neighboring samples of the current block (i.e., Fig.21 The difference between the template T in the temporal reference picture and its corresponding reference sample in the temporal reference picture (i.e., Fig.21 The minimum mean square error (LMSE) method is adopted to derive the values ​​of the LIC parameters (i.e., α and β) in T0 or T1. In addition, in order to reduce the computational complexity, both the template samples and the reference template samples are subsampled (adaptive subsampling) to derive the LIC parameters, i.e., only Fig.21 The shadow samples in are used to derive α and β. To improve the encoding and decoding performance, no subsampling is performed on the short edge, such as Fig. 22 shown. 2.15. Decoder-side Motion Vector Refinement (DMVR) To improve the accuracy of MV in merge mode, decoder-side motion vector refinement based on bilateral matching (BM) is applied in VVC. In bidirectional prediction operation, the refined MV is searched around the initial MV in reference picture list L0 and reference picture list L1. The BM method calculates the distortion between 2 candidate blocks in reference picture list L0 and list L1. Fig.23 As shown in , the SAD between two blocks based on each MV candidate (eg, MV0' and MV1') around the initial MV is calculated. The MV candidate with the lowest SAD becomes the refined MV and is used to generate the bidirectional prediction signal. In VVC, the application of DMVR is restricted and is only applied to CUs that are coded and decoded with the following modes and features: – CU level merge mode with bi-predictive MV - Relative to the current picture, one reference picture is in the past and the other reference picture is in the future. – The distances from the two reference pictures to the current picture (ie, the POC difference) are the same. – Both reference images are short-term reference images. –CU has more than 64 luma samples – CU height and CU width are both greater than or equal to 8 luma samples – BCW weight index indicates equal weight – WP is not enabled for the current block. –CIIP mode is not used for the current block. The refined MV derived by the DMVR process is used to generate inter-frame prediction samples and is also used in temporal motion vector prediction for future picture coding, while the original MV is used in the deblocking process and is also used in spatial motion vector prediction for future picture coding. Additional features of DMVR are mentioned in the following sub-clauses. 2.15.1. Search solution In DVMR, the search point is around the initial MV and the MV offset follows the MV difference mirror rule. In other words, any point checked by DMVR is recorded as a candidate MV pair (MV0, MV1), which follows the following two equations: MV0′=MV0+MV_offset (2-24) MV1′=MV1-MV_offset (2-25) Where MV_offset represents the refinement offset between the initial MV and the refined MV in one of the reference pictures. The refinement search range is two integer luma samples from the initial MV. The search includes an integer sample offset search stage and a fractional sample refinement stage. A 25-point full search is applied to the integer sample offset search. The SAD of the initial MV pair is first calculated. If the SAD of the initial MV pair is less than a threshold, the integer sampling phase of DMVR is terminated. Otherwise, the SAD of the remaining 24 points is calculated and checked in raster scan order. The point with the smallest SAD is selected as the output of the integer sample offset search phase. In order to reduce the impact of the uncertainty of DMVR refinement, it is proposed to give priority to the original MV during the DMVR process. The SAD between the reference blocks referenced by the initial MV candidates is reduced by 1 / 4 of the SAD value. The integer sample search is followed by fractional sample refinement. To save computational complexity, fractional sample refinement is derived by using the parameter error surface equation instead of using an additional search with SAD comparison. Fractional sample refinement is conditionally called based on the output of the integer sample search stage. When the integer sample search stage is terminated when the center has the minimum SAD in the first or second iteration search, fractional sample refinement is further applied. In the parametric error surface based sub-pixel offset estimation, the center location cost and the costs at four neighboring locations from the center are used to fit a 2-D parabolic error surface equation of the following form: E(x,y)=A(x min ) 2 +B(yy min ) 2 +C (2-26) Where (x min ,y min ) corresponds to the fractional position with the minimum cost and C corresponds to the minimum cost value. By solving the above equation using the cost values ​​of the five search points, (x min ,y min ) is calculated as: x min =(E(-1,0)-E(1,0)) / (2(E(-1,0)+E(1,0)-2E(0,0))) (2-27) y min =(E(0,-1)-E(0,1)) / (2((E(0,-1)+E(0,1)-2E(0,0))) (2-28) Since all cost values ​​are positive and the minimum is E(0,0), x min and min The value of is automatically constrained to be between -8 and 8. This corresponds to a half-pixel offset with 1 / 16 pixel MV precision in VVC. The calculated fraction (x min ,y min ) is added to the integer distance refinement MV to obtain the sub-pixel accurate refinement ΔMV. 2.15.2. Bilinear interpolation and sample padding In VVC, the resolution of MV is 1 / 16 luma sample. Samples at fractional positions are interpolated using an 8-tap interpolation filter. In DMVR, the search points are around the initial fractional pixel MV with an integer sample offset, so samples at those fractional positions need to be interpolated for the DMVR search process. In order to reduce computational complexity, a bilinear interpolation filter is used to generate fractional samples for the search process in DMVR. Another important effect is that by using a bilinear filter with a 2-sample search range, DVMR does not access more reference samples than the ordinary motion compensation process. After obtaining the refined MV through the DMVR search process, an ordinary 8-tap interpolation filter is applied to generate the final prediction. In order not to access more reference samples than the ordinary MC process, samples that are not required for the interpolation process based on the original MV but required for the interpolation process based on the refined MV will be filled from those available samples. 2.15.3. Maximum DMVR processing unit When the width and / or height of a CU is greater than 16 luma samples, it will be further partitioned into sub-blocks with width and / or height equal to 16 luma samples. The maximum unit size for the DMVR search process is limited to 16x16. 2.16. Multi-pass decoder-side motion vector refinement In this paper, multiple passes of decoder side motion vector refinement are applied instead of DMVR. In the first pass, bilateral matching (BM) is applied to the codec block. In the second pass, BM is applied to each 16x16 sub-block within the codec block. In the third pass, the MV in each 8x8 sub-block is refined by applying bidirectional optical flow (BDOF). The refined MVs are stored for both spatial and temporal motion vector prediction. 2.16.1. First pass - Block-based bilateral matching MV refinement In the first pass, the refined MV is derived by applying BM to the codec block. Similar to decoder side motion vector refinement (DMVR), the refined MV is searched around the two initial MVs (MV0 and MV1) in the reference picture lists L0 and L1. The refined MVs (MV0_pass1 and MV1_pass1) are derived around the starting MV based on the minimum bilateral matching cost between the 2 reference blocks in L0 and L1. BM performs a local search to derive integer sample precision intDeltaMV and half-pixel sample precision halfDeltaMv. The local search applies a 3x3 square search pattern to cycle through the search range [-sHor, sHor] in the horizontal direction and [-sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block size, and the maximum values ​​of sHor and sVer are 8. The bilateral matching cost is calculated as: bilCost = mvDistanceCost + sadCost. When the block size cbW*cbH is greater than 64, the MRSAD cost function is applied to remove the DC effect of the distortion between the reference blocks. When the bilCost at the center point of the 3x3 search pattern has the minimum cost, the intDeltaMV or halfDeltaMV local search is terminated. Otherwise, the current minimum cost search point becomes the new center point of the 3x3 search pattern, and the search for the minimum cost continues until it reaches the end of the search range. The existing fractional sample refinement is further applied to derive the final deltaMV. The refined MV after the first pass is derived as: MV0_pass1 = MV0 + deltaMV MV1_pass1 = MV1-deltaMV 2.16.2. Second pass - sub-block based bilateral matching MV refinement In the second pass, the refined MVs are derived by applying BM to the 16x16 grid sub-blocks. For each sub-block, the refined MVs are searched around the 2 MVs (MV0_pass1 and MV1_pass1) for reference picture lists L0 and L1, which are obtained in the first pass for reference picture lists L0 and L1. The refined MVs (MV0_pass2(sbIdx2) and MV1_pass2(sbIdx2)) are derived based on the minimum bilateral matching cost between the 2 reference sub-blocks in L0 and L1. For each subblock, BM performs a full search to derive integer sample precision intDeltaMV. The full search has a search range of [-sHor, sHor] in the horizontal direction and [-sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block size and the maximum values ​​of sHor and sVer are 8. The bilateral matching cost is calculated by applying the cost factor to the SATD cost between the two reference sub-blocks, such as: bilCost = satdCost * costFactor. The search area (2*sHor+1)*(2*sVer+1) is divided into Fig.24There are up to 5 diamond-shaped search regions shown. Each search region is assigned a costFactor, which is determined by the distance (intDeltaMV) between each search point and the starting MV, and each diamond-shaped region is processed in order starting from the center of the search region. In each region, the search points are processed in raster scan order, starting from the upper left to the lower right corner of the region. When the minimum bilCost in the current search region is less than the threshold and equal to sbW*sbH, the int-pel full search is terminated, otherwise the int-pel full search continues to the next search region until all search points are checked. BM performs a local search to derive the half-sample accuracy halfDeltaMv. The search pattern and cost function are the same as defined in 2.9.1. The existing VVC DMVR fractional sample refinement is further applied to derive the final deltaMV (sbIdx2). The refined MV in the second pass is then derived as: ·MV0_pass2(sbIdx2)=MV0_pass1+deltaMV(sbIdx2) ·MV1_pass2(sbIdx2)=MV1_pass1-deltaMV(sbIdx2) 2.16.3. Third pass - sub-block-based bidirectional optical flow MV refinement In the third pass, the refined MV is derived by applying BDOF to the 8x8 grid sub-blocks. For each 8x8 sub-block, BDOF refinement is applied to derive scaled Vx and Vy without clipping from the refined MV of the parent sub-block of the second pass. The derived bioMv(Vx, Vy) is rounded to 1 / 16 sample precision and clipped between -32 and 32. The refined MVs in the third pass (MV0_pass3(sbIdx3) and MV1_pass3(sbIdx3)) are derived as: ·MV0_pass3(sbIdx3)=MV0_pass2(sbIdx2)+bioMv ·MV1_pass3(sbIdx3)=MV0_pass2(sbIdx2)-bioMv 2.17. Sample-based BDOF In sample-based BDOF, motion refinement (Vx, Vy) is not derived on a block basis, but is performed per sample. The codec block is divided into 8×8 sub-blocks. For each sub-block, it is determined whether to apply BDOF by checking the SAD between two reference sub-blocks against a threshold. If it is decided to apply BDOF to a sub-block, for each sample in the sub-block, a sliding 5x5 window is used and for each sliding window the existing BDOF process is applied to derive Vx and Vy. The derived motion refinement (Vx, Vy) is applied to adjust the bidirectional prediction sample value for the center sample of the window. 2.18. Extended Merge Prediction In VVC, the merge candidate list is constructed by including the following five types of candidates in order: (1) Spatial MVP from the spatially adjacent CU (2) Temporal MVP from the co-located CU (3) History-based MVP from FIFO table (4) Paired Average MVP (5) Zero MV. The size of the merge list is signaled in the sequence parameter set header, and the maximum allowed merge list size is 6. For each CU code in merge mode, the index of the best merge candidate is encoded using truncated unary binarization (TU). The first binary bit of the merge index is coded with context, while bypass coding is used for the other binary bits. A derivation process for each category of merge candidates in this session is provided. As done in HEVC, VVC also supports parallel derivation of merge candidate lists for all CUs within a region of a certain size. 2.18.1. Spatial Candidate Export The derivation of spatial merge candidates in VVC is the same as in HEVC, except that the positions of the first two merge candidates are swapped. Up to four merge candidates are selected from the candidates located in the depicted positions. The order of derivation is B0, A0, B1, A1, and B2. Position B2 is considered only when one or more CUs at positions B0, A0, B1, A1 are not available (for example, because they belong to another slice or tile) or when intra-coded. After the candidate at position A1 is added, the addition of the remaining candidates is subject to a redundancy check, which ensures that candidates with the same motion information are excluded from the list, so that the coding efficiency is improved. Fig.25 shows the positions of the spatial merging candidates. In order to reduce the computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only the candidate pairs with Fig.26 The arrows in are linked to pairs, and a candidate is only added to the list if the corresponding candidate for redundancy check does not have the same motion information. 2.18.2. Time Domain Candidate Derivation In this step, only one candidate is added to the list. Specifically, when deriving this time domain merge candidate, the scaled motion vector is derived based on the co-located CU belonging to the co-located reference picture. The reference picture list used to derive the co-located CU is explicitly signaled in the slice header. Fig. 27 As shown by the dashed line in , the scaled motion vector for the temporal merge candidate is obtained, which is scaled from the motion vector of the co-located CU using the POC distance, tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal merge candidate is set equal to zero. The position for the time domain candidate is selected between candidates C0 and C1, such as Fig.28 If the CU at position C0 is not available, intra-coded, or outside the current row of CTU, position C1 is used. Otherwise, position C0 is used to derive the temporal merge candidate. 2.18.3. History-based Merge Candidate Export After spatial MVP and TMVP, history-based MVP (HMVP) merge candidates are added to the merge list. In this method, the motion information of previously coded blocks is stored in a table and is used as the MVP of the current CU. A table with multiple HMVP candidates is maintained during the encoding / decoding process. The table is reset (cleared) when a new CTU row is encountered. Whenever there is a CU that is non-subblock inter-coded, the associated motion information is added to the last entry of the table as a new HMVP candidate. The HMVP table size S is set to 6, which indicates that up to 6 history-based MVP (HMVP) candidates can be added to the table. When a new motion candidate is inserted into the table, a constrained first-in-first-out (FIFO) rule is utilized, where a redundancy check is first applied to find if the same HMVP exists in the table. If found, the same HMVP is removed from the table and all HMVP candidates are moved forward. HMVP candidates can be used in the merge candidate list construction process. The latest HMVP candidates in the table are checked in order and inserted into the candidate list after the TMVP candidate. Redundancy check is applied to the HMVP candidate to merge the candidate in the spatial or temporal domain. In order to reduce the number of redundancy check operations, the following simplifications are introduced: The number of HMPV candidates used for merge list generation is set to (N<=4)?M:(8-N), where N indicates the number of existing candidates in the merge list and M indicates the number of HMVP candidates available in the table. Once the total number of available merge candidates reaches the maximum allowed merge candidates minus 1, the merge candidate list construction process from HMVP is terminated. 2.18.4. Pairwise Average Merge Candidate Derivation Pairwise average candidates are generated by averaging predefined pairs of candidates in the existing merge candidate list, and the predefined pairs are defined as {(0,1),(0,2),(1,2),(0,3),(1,3),(2,3)}, where the number represents the merge index to the merge candidate list. The average motion vector is calculated separately for each reference list. If both motion vectors in a list are available, they are averaged even if they point to different reference pictures; if only one motion vector is available, it is used directly; if no motion vector is available, the list is kept invalid. When the merge list is not full after pairwise average merge candidates are added, zero MVPs are inserted at the end until the maximum merge candidates are reached. 2.18.5. Merging Estimated Regions The merge estimation region (MER) allows independent derivation of merge candidate lists for CUs in the same merge estimation region (MER). For the generation of the merge candidate list for the current CU, candidate blocks within the same MER as the current CU are not included. In addition, the update process for the history-based motion vector prediction candidate list is updated only when (xCb+cbWidth)>>Log2ParMrgLevel is greater than xCb>>Log2ParMrgLevel and (yCb+cbHeight)>>Log2ParMrgLevel is greater than (yCb>>Log2ParMrgLevel) and where (xCb, yCb) is the top left luma sample position of the current CU in the picture and (cbWidth, cbHeight) is the CU size. The MER size is selected at the encoder side and is signaled in the sequence parameter set as log2_parallel_merge_level_minus2. 2.19. New Merge Candidates 2.19.1. Non-adjacent merge candidate export In VVC, Fig.29 The five spatial neighboring blocks shown and one temporal neighbor are used to derive the merge candidate. Fig.30 The relationship between the virtual block and the current block is shown. It is proposed to derive additional merge candidates from positions that are not adjacent to the current block using the same pattern as in VVC. To achieve this, for each search round i, a virtual block is generated based on the current block as follows: First, the relative position of the virtual block to the current block is calculated as follows: Offsetx=-i×gridX, Offsetty=-i×gridY Where Offsetx and Offsetty represent the offset of the upper left corner of the virtual block relative to the upper left corner of the current block, and gridX and gridY are the width and height of the search grid. Second, the width and height of the virtual block are calculated by: newWidth=i×2×gridX+currWidth newHeight=i×2×gridY+currHeight. Where currWidth and currHeight are the width and height of the current block. newWidth and newHeight are the width and height of the new virtual block. gridX and gridY are currently set to currWidth and currHeight respectively. After generating the virtual block, block A i , B i , C i , D i and E i can be considered as VVC spatial neighbors of the virtual block and their positions are obtained by the same pattern as in VVC. Obviously, if search round i is 0, the virtual block is the current block. In this case, block A i , B i , C i , D i and E i It is the spatial neighboring block used in VVC merge mode. When building the merge candidate list, pruning is performed to ensure that each element in the merge candidate list is unique. The maximum search loop is set to 1, which means that five non-adjacent spatial neighboring blocks are used. Non-adjacent spatial merging candidates are inserted into the merging list in the order of B1->A1->C1->D1->E1 after the temporal merging candidates. 2.19.2.STMVP It is proposed to use three spatial merging candidates and one temporal merging candidate to derive the average candidate as the STMVP candidate. The STMVP is inserted before the top left spatial merge candidate. The STMVP candidates are pruned using all previous merge candidates in the merge list. For spatial candidates, the first three candidates in the current merge candidate list are used. For the time domain candidates, the same positions as the VTM / HEVC co-location positions are used. For spatial candidates, the first, second and third candidates inserted into the current merge candidate list before STMVP are denoted as F, S and T. The temporal candidate having the same position as the VTM / HEVC co-location position used in TMVP is denoted as Col. The motion vector of the STMVP candidate in prediction direction X (denoted as mvLX) is derived as follows: 1) If the reference indices of the four merge candidates are all valid in prediction direction X (X=0 or 1) and are all equal to zero, then mvLX=(mvLX_F+mvLX_S+mvLX_T+mvLX_Col)>>2 2) If the reference indices of three of the four merge candidates are all valid and equal to zero in prediction direction X (X=0 or 1), then mvLX=(mvLX_F×3+mvLX_S×3+mvLX_Col×2)>>3or mvLX=(mvLX_F×3+mvLX_T×3+mvLX_Col×2)>>3or mvLX=(mvLX_S×3+mvLX_T×3+mvLX_Col×2)>>3 3) If the reference indexes of two of the four merge candidates are valid in prediction direction X (X=0 or 1) and equal to zero, then mvLX=(mvLX_F+mvLX_Col)>>1 or mvLX=(mvLX_S+mvLX_Col)>>1 or mvLX=(mvLX_T+mvLX_Col)>>1 NOTE: If time domain candidates are not available, STMVP mode is turned off. 2.19.3. Merge list size If both non-adjacent and STMVP merging candidates are considered, the size of the merging list is signaled in the sequence parameter set header, and the maximum allowed size of the merging list is 8. 2.20. Geometric Partitioning Model (GPM) In VVC, geometric partitioning modes are supported for inter prediction. The geometric partitioning mode is signaled using a CU level flag as a type of merge mode, where other merge modes include normal merge mode, MMVD mode, CIIP mode, and sub-block merge mode. In a total of 64 partitions, for each possible CU size w×h=2 with exclusion of 8x64 and 64x8 m ×2 nwhere m,n∈{3…6}, supported by geometric partitioning schemes. When this mode is used, the CU passes through a geometrically positioned straight line ( Fig.31 ) is split into two parts. The position of the partition line is mathematically derived from the angle and offset parameters of the specific partition. Each part of the geometric partition in the CU is inter-predicted using its own motion; only unidirectional prediction is allowed for each partition, that is, each part has one motion vector and one reference index. Unidirectional prediction motion constraints are applied to ensure that only two motion compensated predictions are required for each CU, the same as conventional bidirectional prediction. The unidirectional prediction motion of each partition is derived using the process described in 2.20.1. If a geometric partitioning mode is used for the current CU, a geometric partitioning index indicating the partitioning mode (angle and offset) of the geometric partitioning and two merge indices (one for each partition) are further signaled. The number of maximum GPM candidate sizes is explicitly signaled in the SPS and the syntax binarization for the GPM merge indices is specified. After predicting each of the parts of the geometric partitioning, the sample values ​​along the edges of the geometric partitioning are adjusted using a hybrid process with adaptive weights as in 2.20.2. This is the prediction signal for the entire CU, and the transform and quantization processes will be applied to the entire CU as in other prediction modes. Finally, the motion field of the CU predicted using the geometric partitioning mode is stored as in 2.20.3. 2.20.1. One-way prediction candidate list construction In 2.18 the unidirectional prediction candidate list is derived directly from the merge candidate list constructed according to the extended merge prediction process. Denote n as the index of the unidirectional prediction motion in the geometric unidirectional prediction candidate list. The LX motion vector (x equals parity of n) of the nth extended merge candidate is used as the nth unidirectional prediction motion vector for the geometric partitioning mode. These motion vectors are in Fig.32 In the case where the corresponding LX motion vector of the n-th extended merge candidate does not exist, the L(1-X) motion vector of the same candidate is used instead of the unidirectional prediction motion vector for the geometric partitioning mode. 2.20.2. Blending Along Geometric Partition Edges After predicting each part of the geometric partition using its own motion, blending is applied to the two prediction signals to derive samples around the geometric partition edge. A blending weight is derived for each position of the CU based on the distance between the respective position and the partition edge. The distance from position (x,y) to the partition edge is derived as: where i,j are the indices for the angle and offset of the geometric partition, which depend on the geometric partition index that is signaled. x,j and ρ y,j The sign of depends on the angle index i. The weights for each part of the geometric partition are derived as follows: wIdxL(x,y)=partIdx? 32+d(x,y):32-d(x,y) (2-33) w1(x,y)=1-w0(x,y) (2-35) partIdx depends on the angle index i. An example of weight w0 is Fig.33 is shown in . 2.20.3. Motion Field Storage for Geometric Partitioning Mv1 from the first part of the geometric partition, Mv2 from the second part of the geometric partition, and a combination Mv of Mv1 and Mv2 are stored in the motion field of the CU coded by the geometric partition mode. The type of motion vector stored for each individual position in the motion field is determined as: sType=abs(motionIdx)<32?2:(motionIdx≤0?(1-partIdx):partIdx) (2-36) Where motionIdx is equal to d(4x+2,4y+2), which is recalculated according to equation (2-18). partIdx depends on the angle index i. If sType is equal to 0 or 1, then Mv0 or Mv1 is stored in the corresponding motion field, otherwise if sType is equal to 2, the combined Mv from Mv0 and Mv2 is stored. The combined Mv is generated using the following process: 1) If Mv1 and Mv2 are from different reference picture lists (one from L0 and the other from L1), then Mv1 and Mv2 are simply combined to form a bi-directional prediction motion vector. Otherwise, if Mv1 and Mv2 are from the same list, only the unidirectional predicted motion Mv2 is stored. 2.21. Multi-hypothesis prediction In Multiple Hypothesis Prediction (MHP), above Inter AMVP mode, Normal Merge mode, Affine Merge and MMVD mode, up to two additional predictors are signaled. The resulting overall prediction signal is iteratively accumulated over each additional prediction signal. p n+1 =(1-α n+1 ) n+α n+1 h n+1 The weight factor α is specified according to the following Table 2-4: Table 2-4 - Weighting factors for MHP add_hyp_weight_idx α 0 1 / 4 1 -1 / 8 For inter-AMVP mode, if unequal weights in BCW are selected in bi-prediction mode, only MHP is applied. The additional assumption may be a merge mode or an AMVP mode. In the case of the merge mode, the motion information is indicated by a merge index, and the merge candidate list is the same as in the geometric partitioning mode. In the case of the AMVP mode, the reference index, the MVP index, and the MVD are transmitted with a signal. 2.22. Non-adjacent spatial domain candidates Non-adjacent spatial merge candidates are inserted after the TMVP in the regular merge candidate list. The mode of spatial merge candidates is Fig.34 The distance between the non-adjacent spatial candidates and the current codec block is based on the width and height of the current codec block. 2.23. Template Matching (TM) Template matching (TM) is a decoder-side MV derivation method to refine the motion information of the current CU by finding the closest match between the template in the current picture (i.e., the neighboring blocks above and / or to the left of the current CU) and the blocks in the reference picture (i.e., the same size as the template). Fig.35 As shown, a better MV will be searched around the initial motion of the current CU within the [-8, +8] pixel search range. Two modifications of template matching are proposed: the search step size is determined based on the AMVR mode, and the TM can be cascaded with the bilateral matching process in the merge mode. In AMVP mode, MVP candidates are determined based on template matching error to pick the MVP candidate that achieves the minimum difference between the current block template and the reference block template, and then TM is performed only for this specific MVP candidate for MV refinement. TM refines the MVP candidate by using iterative diamond search starting from full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode) within the [8, +8] pixel search range. The AMVP candidate can be further refined by using a cross search with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode), followed by a half-pixel and quarter-pixel sequential search depending on the AMVR mode as specified in Tables 2-5. This search process ensures that the MVP candidate still maintains the same MV accuracy as indicated by the AMVR mode after the TM process. Table 2-5 - Search mode for AMVR and merge mode with AMVR. In merge mode, a similar search method is applied to the merge candidate indicated by the merge index. As shown in Table 2-5, depending on the alternative interpolation filter (the interpolation filter used when AMVR is half-pixel mode) according to whether the merged motion information is used, TM can perform all the ways to 1 / 8 pixel MVD accuracy or skip the accuracy beyond half-pixel MVD accuracy. In addition, when TM mode is enabled, template matching can be used as an independent process or an additional MV refinement process between block-based and sub-block-based bilateral matching (BM) methods, depending on whether BM is enabled according to its enabling condition. 2.24. Overlapped Block Motion Compensation (OBMC) Overlapped block motion compensation (OBMC) has been previously used in H.263. In JEM, unlike H.263, OBMC can be turned on and off using CU-level syntax. When OBMC is used in JEM, OBMC is performed on all motion compensated (MC) block boundaries except the right and bottom boundaries of the CU. In addition, it is applied to both luminance and chrominance components. In JEM, an MC block corresponds to a codec block. When a CU is encoded and decoded in sub-CU mode (including sub-CU merging, affine, and FRUC modes), each sub-block of the CU is an MC block. In order to handle CU boundaries in a unified manner, OBMC is performed at the sub-block level for all MC block boundaries, where the sub-block size is set equal to 4x4, as shown in FIG. Fig.36 shown. When OBMC is applied to the current sub-block, in addition to the current motion vector, the motion vectors of four connected neighboring sub-blocks are also used to derive the prediction block of the current sub-block if they are available and different from the current motion vector. These multiple prediction blocks based on multiple motion vectors are combined to generate the final prediction signal of the current sub-block. The prediction block based on the motion vectors of neighboring sub-blocks is denoted as P N , where N indicates the index for the adjacent upper, lower, left and right sub-blocks, and the prediction block based on the motion vector of the current sub-block is represented as P C When P based on the motion information of neighboring sub-blocks N When the OBMC contains the same motion information as the current sub-block, the OBMC is not N Execute. Otherwise, P N Each sample of is added to P C In the same sample, that is, P N Four rows / columns are added to P C The weight factors {1 / 4, 1 / 8, 1 / 16, 1 / 32} are used for P N , weight factors {3 / 4, 7 / 8, 15 / 16, 31 / 32} are used for PC The exception is small MC blocks (ie, when the height or width of the codec block is equal to 4 or the CU is coded in sub-CU mode), for which only P N Add two rows / columns to P C In this case, the weight factors {1 / 4,1 / 8} are used for P N , and weight factors {3 / 4,7 / 8} are used for P C For P generated based on the motion vectors of vertically (horizontally) adjacent sub-blocks N , P N Samples in the same row (column) of are added to P with the same weight factor C middle. In JEM, for CUs with a size less than or equal to 256 luma samples, a CU level flag is signaled to indicate whether OBMC is applied to the current CU. For CUs with an application size greater than 256 luma samples or not encoded and decoded in AMVP mode, OBMC is applied by default. At the encoder, when OBMC is applied to a CU, its impact is taken into account during the motion estimation stage. The prediction signal formed by OBMC using the motion information of the upper and left neighboring blocks is used to compensate the upper and left boundaries of the original signal of the current CU, and then the normal motion estimation process is applied. 2.25. Multi-Transform Selection (MTS) for Core Transforms In addition to DCT-II, which has been adopted in HEVC, the multi-transform selection (MTS) scheme is used for residual coding of both inter-frame and intra-frame coded blocks. It uses multiple selected transforms from DCT8 / DST7. The newly introduced transform matrices are DST-VII and DCT-VIII. Table 2-6 shows the basis functions of the selected DST / DCT. Table 2-6 - Transform basis functions of DCT-II / VIII and DST-VII for N-point input To maintain the orthogonality of the transform matrix, the transform matrix is ​​quantized more accurately than in HEVC. To keep the intermediate values ​​of the transform coefficients within 16 bits, all coefficients will have 10 bits after horizontal and vertical transforms. To control the MTS scheme, separate enable flags are specified at the SPS level for intra and inter, respectively. When MTS is enabled at the SPS, a CU level flag is signaled to indicate whether MTS is applied. Here, MTS is applied only for luma. When one of the following conditions applies, MTS signaling is skipped. - The position of the last significant coefficient for the luma TB is less than 1 (ie, DC only). – The last significant coefficient of the luminance TB is located inside the MTS zeroing region. If the MTS CU flag is equal to zero, DCT2 is applied in both directions. However, if the MTS CU flag is equal to 1, two other flags are additionally signaled to indicate the transform type for the horizontal and vertical directions, respectively. The transform and signaling mapping table is shown in Table 2-7. Unified transform selection for ISP and implicit MTS is used by removing intra-mode and block shape dependencies. If the current block is in ISP mode or if the current block is an intra-block and both intra- and inter-frame explicit MTS are turned on, only DST7 is used for horizontal and vertical transform cores. When it comes to transform matrix precision, an 8-bit main transform core is used. Therefore, all transform cores used in HEVC are kept the same, including 4-point DCT-2 and DST-7, 8-point, 16-point and 32-point DCT-2. In addition, other transform cores including 64-point DCT-2, 4-point DCT-8, 8-point, 16-point, 32-point DST-7 and DCT-8 use 8-bit main transform cores. Table 2-7 - Transformation and signaling mapping table To reduce the complexity of large-sized DST-7 and DCT-8, high-frequency transform coefficients are zeroed for DST-7 and DCT-8 blocks with size (width or height, or both) equal to 32. Only coefficients within the 16x16 low-frequency region are retained. As in HEVC, the residual of a block can be coded using transform skip mode. To avoid syntax coding redundancy, the transform skip flag is not signaled when the CU level MTS_CU_flag is not equal to zero. Note that when LFNST or MIP is activated for the current CU, the implicit MTS transform is set to DCT2. When MTS is enabled for an inter-coded block, implicit MTS can still be enabled. 2.26. Sub-Block Transform (SBT) In VTM, sub-block transforms are introduced for inter-predicted CUs. In transform mode, only a sub-portion of the residual block is encoded and decoded for the CU. When an inter-predicted CU has cu_cbf equal to 1, cu_sbt_flag may be signaled to indicate whether the entire residual block or a sub-portion of the residual block is encoded and decoded. In the former case, the inter-MTS information is further parsed to determine the transform type of the CU. In the latter case, a portion of the residual block is encoded and decoded using the inferred adaptive transform and another portion of the residual block is zeroed. When SBT is used for an inter-coded CU, the SBT type and SBT position information are signaled in the bitstream. There are two SBT types and two SBT positions, such as Fig.37 As shown. For SBT-V (or SBT-H), the TU width (or height) can be equal to half of the CU width (or height) or 1 / 4 of the CU width (or height), resulting in 2:2 partitioning or 1:3 / 3:1 partitioning. 2:2 partitioning is similar to binary tree (BT) partitioning, while 1:3 / 3:1 partitioning is similar to asymmetric binary tree (ABT) partitioning. In ABT partitioning, only small areas contain non-zero residuals. If one dimension of the CU is 8 in luma samples, 1:3 / 3:1 partitioning along that dimension is not allowed. There are up to 8 SBT modes for a CU. Position dependent transform kernels are selected for luma transform blocks in SBT-V and SBT-H (chroma TBs always use DCT-2). The two positions of SBT-H and SBT-V are associated with different kernel transforms. More specifically, the horizontal and vertical transforms for each SBT position are in Fig.37 For example, the horizontal and vertical transforms for SBT-V position 0 are DCT-8 and DST-7, respectively. When a side of the residual TU is larger than 32, the transforms for both dimensions are set to DCT-2. Therefore, the sub-block transform jointly specifies the TU tile, cbf, and horizontal and vertical core transform types of the residual block. SBT is not applied to CUs coded in combined inter-intra modes. 2.27. Adaptive Merge Candidate Reordering Based on Template Matching In order to improve the encoding and decoding efficiency, after the merge candidate list is constructed, the order of each merge candidate is adjusted according to the template matching cost. The merge candidates are arranged in the list according to the template matching cost in ascending order. It is operated in the form of subgroups. The template matching cost is measured by the SAD (sum of absolute differences) between the neighboring samples of the current CU and its corresponding reference samples. If the merge candidate contains bidirectional prediction motion information, the corresponding reference sample is the average of the corresponding reference sample in reference list 0 and the corresponding reference sample in reference list 1, such as Fig.38 If the merge candidate contains sub-CU level motion information, the corresponding reference sample is composed of the neighboring samples of the corresponding reference sub-block, such as Fig.39 as shown in . The sorting process is operated in the form of subgroups, such as Fig.40 As shown in Figure 1, the first three merge candidates are sorted together. The last three merge candidates are sorted together. The template size (width of the left template or height of the upper template) is 1. The subgroup size is 3. 2.28. Adaptive Merge Candidate List We can assume that the number of merge candidates is 8. We take the first 5 merge candidates as the first subgroup and the last 3 merge candidates as the second subgroup (ie, the last subgroup). For the encoder, after the merge candidate list is constructed, some merge candidates are adaptively reordered according to the template matching cost in ascending order, such as Fig.41 shown. More specifically, the template matching costs for the merging candidates in all subgroups except the last subgroup are calculated; then the merging candidates are reordered in their own subgroups except the last subgroup; finally, a final merging candidate list will be obtained. For the decoder, after the merge candidate list is constructed, some or no merge candidates are adaptively reordered in ascending order of template matching cost, e.g. Fig.42 As shown. Fig.42 In , a subgroup that is among the selected (signaled) merging candidates is referred to as a selected subgroup. More specifically, if the selected merge candidate is located in the last subgroup, the merge candidate list construction process is terminated after the selected merge candidate is derived, reordering is not performed and the merge candidate list is not changed; otherwise, the execution process is as follows: The merge candidate list construction process is terminated after all merge candidates in the selected subgroup are derived; the template matching cost for the merge candidates in the selected subgroup is calculated; the merge candidates in the selected subgroup are reordered; finally, a new merge candidate list will be obtained. For both encoder and decoder: The template matching cost is derived as a function of T and RT, where T is the set of samples in the template and RT is the set of reference samples for the template. When deriving reference samples for the template of a merge candidate, the motion vector of the merge candidate is rounded to integer pixel precision. The reference samples for the template (RT) for bidirectional prediction are derived by weighted averaging the reference samples of the template in reference list 0 (RT0) and the reference samples of the template in reference list 1 (RT1) as follows: RT=((8-w)*RT0+w*RT1+4)>>3 (2-37) The weights of the reference templates in reference list 0 (8-w) and the weights of the reference templates in reference list 1 (w) are determined by the BCW index of the merge candidate. The BCW indexes equal to {0, 1, 2, 3, 4} correspond to w equal to {-2, 3, 4, 5, 10} respectively. If the local illumination compensation (LIC) flag of the merge candidate is true, the reference samples of the template are derived by the LIC method. The template matching cost is calculated based on the sum of absolute differences (SAD) of T and RT. The template size is 1. That is, the width of the left template and / or the height of the upper template is 1. If the codec mode is MMVD, the merge candidates used to derive the base merge candidate are not reordered. If the coding mode is GPM, the merge candidates used to derive the unidirectional prediction candidate list are not reordered. 2.29. Geometric Prediction Mode with Motion Vector Difference In the geometric prediction mode with motion vector difference (GMVD), each geometric partition in GPM can decide whether to use GMVD. If GMVD is selected for a geometric area, the MV of the area is calculated as the sum of the MV of the merge candidate and the MVD. All other processing remains the same as GPM. With GMVD, the MVD is signaled as a pair of direction and distance. There are nine candidate distances (1 / 4 pixel, 1 / 2 pixel, 1 pixel, 2 pixels, 3 pixels, 4 pixels, 6 pixels, 8 pixels, 16 pixels) and eight candidate directions (four horizontal / vertical directions and four diagonal directions). In addition, when pic_fpel_mmvd_enabled_flag is equal to 1, the MVD in GMVD is also left-shifted by 2, as in MMVD. 2.30. Affine MMVD In affine MMVD, affine merging candidates (which are referred to as basic affine merging candidates) are selected and the MVs of control points are further refined by the signaled MVD information. The MVD information for the MVs of all control points is the same in one prediction direction. When the starting MV is a bidirectional prediction MV with two MVs pointing to different sides of the current picture (i.e., the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture), the MV offset added to the list 0MV component of the starting MV and the MV offset for the list 1MV have opposite values; otherwise, when the starting MV is a bidirectional prediction MV with both lists pointing to the same side of the current picture (i.e., the POCs of the two references are both greater than the POC of the current picture, or both are less than the POC of the current picture), the MV offset added to the list 0MV component of the starting MV and the MV offset for the list 1MV are the same. 2.31. Adaptive Decoder-side Motion Vector Refinement (ADMVR) In ECM-2.0, a multi-pass decoder-side motion vector refinement (DMVR) method is applied in regular merge mode if the selected merge candidate meets the DMVR condition. In the first pass, bilateral matching (BM) is applied to the codec block. In the second pass, BM is applied to each 16x16 sub-block within the codec block. In the third pass, the MV in each 8x8 sub-block is refined by applying bidirectional optical flow (BDOF). The adaptive decoder-side motion vector refinement method consists of two new merge modes that are introduced to refine the MV in only one direction, L0 or L1, of bidirectional prediction for merge candidates that meet the DMVR conditions. A multi-pass DMVR process is applied to the selected merge candidates to refine the motion vector, however MVD0 or MVD1 is set to zero in the first pass (i.e., PU level) DMVR. Similar to the conventional merge mode, the merge candidates for the proposed merge mode are derived from blocks, TMVP, non-adjacent blocks, HMVP and paired candidates that are encoded by spatial neighbors. The difference is that only those that meet the DMVR condition are added to the candidate list. The same merge candidate list (i.e., ADMVR merge list) is used by both proposed merge modes and the merge index is encoded in the conventional merge mode. 2.32. Out-of-bounds (OOB) check for bidirectional motion compensation In ECM-4.0, due to the reference sample filling of the reference picture, it is possible that the inter-frame CU has a reference block that is partially or completely outside the reference picture, such as Fig.43 As shown in Fig.43 In , bidirectional motion compensation is performed to generate an inter-frame prediction block for the current block. In this example, the list 0 reference block is partially out of bounds (OOB), while the list 1 reference block is completely inside the reference picture. However, the OOB portion of the motion compensated block generally provides less prediction efficiency because the OOB portion is just repeated samples derived from boundary samples within the reference picture. When more than one prediction block is combined, the OOB prediction samples are discarded and only non-OOB prediction values ​​are used to generate the final prediction value. Specifically, assuming Pos_x i,j and Pos_y i,j Indicates the position of a prediction sample in a current block, and (x=0,1) represents the MV of the current block; Pos LeftBdry 、Pos RightBdry 、Pos TopBdry and Pos BottomBdry are the locations of the four boundaries of the picture. A prediction sample is considered OOB when at least one of the following conditions is met: Where half_pixel is equal to 8, which represents the half-pixel sample distance in 1 / 16 pixel sample accuracy. After checking the OOB condition for each sample, the final prediction sample of the bidirectional block is generated as follows: if is OOB and Non-OOB Otherwise, if it is Non-OOB and It's OOB otherwise Please note that the above OOB checking process also applies when BCW is enabled. 2.33. Motion Compensated Boundary Filling In ECM-4.0, the extended picture area is an area surrounding the picture with a size of (maxCUwidth+16) in each direction of the picture boundary. The pixels in the extended area are derived by repeated boundary padding. When the reference block is partially or completely out of bounds (OOB), the repeatedly padded pixels are used for motion compensation (MC), which provides less prediction efficiency. Fig.44 The MC boundary filling method is shown. In the proposed method, samples outside the image boundary are derived through motion compensation instead of using only repeat padding as in ECM. In the implementation, the total padding region size is increased to 64 compared to ECM to keep the non-normative nature of MV clipping to achieve repeat padding. For motion compensated padding, the MV of the 4x4 boundary block is used to derive the M×4 or 4×M padding block. The value M is derived as the distance from the reference block to the picture boundary, such as Fig.45 If the boundary block is intra-coded, MV is not available and M is set equal to 0. If M is less than 64, the rest of the padded area is filled with repeated padded samples. In case of bidirectional inter prediction, only one prediction direction is used in MC boundary filling, which prediction direction has a motion vector pointing to a pixel position in the reference picture in the filling direction away from the picture boundary. The pixels in the MC filling block are corrected by an offset equal to the difference between the DC value of the reconstructed boundary block and its corresponding reference block. When one reference block is outside the picture and the other is inside the picture for a bidirectionally predicted block, only the reference block inside the picture is used for inter prediction. Otherwise, when the reference block is outside the picture and is a unidirectionally predicted block, or when both reference blocks are outside the picture for a bidirectionally predicted block, MC-padded samples are used for inter prediction. 3. Question New codec tools in ECM are explored as potential enhanced video codec techniques that go beyond the capabilities of VVC. Horizontal surround motion compensation is designed for 360-degree video codecs. When a portion of a reference block is outside the left (or right) boundary of a reference picture in the projected domain, instead of repeated padding, the "out-of-bounds" portion is taken from the corresponding spherical neighbor located within the reference picture towards the right (or left) boundary in the projected domain. However, horizontal surround motion compensation is not considered in the new codec tools of ECM, which may limit the performance of ECM for 360-degree video codecs. 4. Detailed solution The following detailed embodiments should be considered as examples to explain the general concept. These embodiments should not be interpreted in a narrow manner. In addition, these embodiments can be combined in any way. In the present disclosure, wrap-around motion compensation (WAMC) may refer to a codec method in which, when a motion vector refers to samples beyond one or more picture boundaries of a reference picture, instead of padding the boundaries (e.g., repeat padding, mirror padding, motion compensated padding, or other padding methods) to derive values ​​for the out-of-bounds samples, the motion vector is adjusted within the picture with one or more wrap-around offsets. For example, when a motion vector exceeds the right picture boundary, a right surround offset may be subtracted from the motion vector to adjust the motion vector within the picture. For example, when the motion vector exceeds the left picture boundary, the left surround offset may be added to the motion vector to adjust the motion vector within the picture. The left surround offset may or may not be equal to the right surround offset. For example, when a motion vector exceeds a lower picture boundary, a lower surround offset may be subtracted from the motion vector to adjust the motion vector inside the picture. For example, when a motion vector exceeds an upper picture boundary, an upper surround offset is added to the motion vector to adjust the motion vector inside the picture. The upper surround offset may or may not be equal to the lower surround offset. In the present disclosure, a motion candidate list may refer to a list of one or more motion candidates used in the decoding / reconstruction process of a video unit, such as a motion candidate list for TM merge mode and / or normal merge mode, and / or adaptive DMVR, and / or CIIP with TM merge mode, and / or AMVP, and / or affine, and / or GPM (GEO), and / or TPM, and / or MMVD, and / or CIIP, and / or MHP, and / or AMVP-merge, affine AMVP, TM AMVP, GPM with MMVD, IBC merge, IBC AMVP, template IBC, intra-template matching and / or any inter-mode that uses a motion candidate list to derive prediction / reconstruction of a block. The motion candidate list may be a TM merge / AMVP list after block-based bilateral matching refinement and / or template matching refinement and / or sub-block-based bilateral matching refinement. The motion candidate list may be a regular merge / AMVP list after the DMVR / multi-pass DMVR process. The motion candidate list may be an ADMVR merged list after the DMVR / multi-pass DMVR process. The motion candidate list may be a GPM merge list (eg, AGPMList, LGPMList, or LAGPMList) after a template matching refinement process. Interaction between surround motion compensation and interframe codec tools 1. It is proposed that surround motion compensation can be used to derive motion information for a video unit, which is used to derive prediction / reconstruction of the video unit. a. Alternatively, WAMC should be disabled for specific codecs, such as LIC, TM, DMVR / Multi-pass DMVR, ADMVR, OBMC, BDOF / Sample-based BDOF, TM- Merge, TM-AMVP, MHP, ARMC, GPM MMVD, affine MMVD, GPM with inter and intra prediction, template / bilateral matching AMVP merge mode, or template-based codec tools such as TM-based reordering for MMVD and affine MMVD, MVD sign prediction, ARMC based on MV candidate type, CIIP-TM, GPM-TM, OOB check, motion compensated filling. b. In one example, when a motion refinement process is used in the derivation of motion information, WAMC may be used during the motion refinement process. i. In one example, when the motion refinement process is a bilateral matching based coding method, WAMC can be used to derive one or more prediction blocks for calculating the bilateral matching cost. 1) In one example, the encoding and decoding method based on bilateral matching may be referred to as decoder-side motion vector refinement (DMVR) or multi-pass DMVR. ii. In one example, when the motion refinement process is a codec method based on template matching, WAMC can be used to derive predictions of the template. c. In one example, WAMC may be used during motion list construction. i. In one example, WAMC may be used to determine whether one or more motion candidates are added to a motion list. 1) In one example, the determination of whether one or more motion candidates are added to a motion list may be within a specific motion candidate type. a) In one example, the motion candidate type may refer to spatially adjacent (adjacent and / or non-adjacent) motion candidates, temporally adjacent (adjacent and / or non-adjacent) motion candidates, sub-block based temporal motion candidates, historical motion candidates, paired motion candidates, and affine motion candidates. 2) In one example, when template matching is used in the determination of whether one or more motion candidates are added to the motion list and a motion candidate with a smaller template matching cost is added to the motion list, WAMC may be used to derive a prediction of the template. a) Alternatively, when bilateral matching is used for determination, WAMC can be used to derive predictions for video units. d. In one example, WAMC can be used in motion list reordering. i. In one example, during motion list reordering, the cost is calculated using the prediction of the template or the prediction of the video unit for each motion candidate in the motion list. WAMC can be used to derive the prediction of the template or the prediction of the video unit. e. In one example, when a template is used to refine or derive one or more motion vectors or motion vector differences, WAMC may be used to derive samples of the template. f. In one example, when a template is used to determine a reference index, WAMC may be used to derive samples of the template. g. In one example, when a template is used to determine one or more signs of a motion vector difference or residual coefficient, WAMC may be used to derive samples of the template. h. In one example, when a template is used to determine the partitioning of a video unit (eg, the partitioning in a GPM), WAMC may be used to derive samples of the template. i. In one example, WAMC can be used to derive prediction samples produced by OBMC. i. In one example, when a template is used to determine the mixing weights of OBMC, WAMC can be used to derive samples of the template. j. In one example, WAMC may be applied in different ways for different interpolation filters. i. For example, when the interpolation filter is 6-tap, 8-tap, or 12-tap, WAMC may be applied in different ways. k. In one example, WAMC can be applied to prediction samples generated by a multi-hypothesis prediction method. 2. When the prediction of the current video unit is adjusted by using one or more offsets, WAMC may be used to derive the one or more offsets. a. In one example, one or more offsets may be derived using a linear model based approach, such as local illumination compensation (LIC). i. In one example, WAMC may be used to derive a prediction of a template to compute linear parameters, which are used to derive one or more offsets. b. In one example, one or more offsets may be derived using an optical flow based approach, such as bidirectional optical flow (BDOF) or sample based BDOF, or prediction refinement with optical flow for affine mode (PROF). i. In one example, WAMC may be used to derive a prediction block of refined motion vectors for each sub-block in a BDOF or for each sample in a sample-based BDOF. ii. In one example, WAMC can be used to derive the predicted sub-blocks in PROF. c. In one example, a video unit may be adjusted using one or more offsets at a block level or a sub-block level or a sample level. Interaction between surround motion compensation and border filling 3. It is proposed that when WAMC is enabled, out-of-bounds (OOB) checking may not be allowed. a. Alternatively, when WAMC is enabled, the wraparound offset may be used in the OOB check. b. Alternatively, WAMC may be independent of OOB checking. 4. It is proposed that when WAMC is enabled, one or more directions (left, and / or right, and / or top, and / or bottom) of motion compensated border filling may not be allowed. a. In one example, when WAMC is horizontally wrapped, the left and right directions of motion compensated border filling may not be allowed. b. Alternatively, when WAMC wraps vertically, the up and down directions of motion compensated border filling may not be allowed. c. Alternatively, all directions of motion compensated border filling may not be allowed when WAMC is both horizontally and vertically wrapped. d. In one example, when WAMC is enabled and a padded area for one or more directions is generated by motion compensated boundary filling, the padded area may not be used. 5. The padded region size using motion compensated border filling in the clipping operation for sub-picture and / or surround motion vectors is proposed. a. In one example, the padded region size (S) can be used to determine a threshold to clip the motion vector. The horizontal maximum value of the picture boundary, the horizontal minimum value of the picture boundary, the vertical maximum value of the picture boundary, and the vertical minimum value of the picture boundary are represented as horMax, horMin, verMax, and verMin, respectively. i. In one example, when motion compensated boundary filling is used, horMax+S, horMin-S, verMax+S, verMin-S can be used as thresholds for clipping motion vectors. 6. In the above items, WAMC can refer to horizontal and / or vertical surround. General 7. In the above examples, a video unit may refer to a slice / a tile / a codec tree unit (CTU) / a CTU row / a group of CTUs / a codec unit (CU) / a prediction unit (PU) / a transform unit (TU) / a codec tree block (CTB) / a codec block (CB) / a prediction block (PB) / a transform block (TB) / a block / a sub-block of a block / a sub-region within a block / any other region containing more than one sample or pixel. 8. Whether and / or how to apply the methods disclosed above may be signaled at the sequence level / picture group level / picture level / slice level / tile group level, e.g. in a sequence header / picture header / SPS / VPS / DPS / DCI / PPS / PPS / slice header / tile group header. 5. Examples 5.1. Example 1 To address the ECM corruption issue for 360-degree video codecs, several fixes are proposed in this paper. Specifically, when surround motion compensation is enabled, clipMv() is replaced by wrapClipMv(). The proposed fixes are applied to several new adopted codec tools: LIC, ARMC, MVD symbol prediction, and multi-pass DMVR. In addition, the out-of-bounds check in ECM-5.0 is used to discard prediction samples outside the picture boundary for bidirectional prediction, because OOB prediction is derived using only repetitive padding from the picture boundary. However, when surround motion compensation is enabled, the motion vectors are adjusted inside the picture. Therefore, when surround motion compensation is enabled, the OOB check can be skipped directly.

[0106] As used herein, the term "video unit" or "video block" may be a sequence, a picture, a slice, a tile, a brick, a sub-picture, a codec tree unit (CTU) / codec tree block (CTB), a CTU / CTB row, one or more codec units (CU) / codec blocks (CB), one or more CTU / CTBs, one or more virtual pipeline data units (VPDUs), a sub-region within a picture / slice / tile / brick. The term "around motion compensation (WAMC)" as used herein may refer to a codec method when a motion vector refers to samples beyond one or more picture boundaries of a reference picture, instead of padding the boundaries (e.g., repeat padding, mirror padding, motion compensated padding, or other padding methods) to derive the value of the out-of-bounds samples, the motion vector is adjusted within the picture with one or more surrounding offsets.

[0107] Fig.46 FIG. 4 shows a flow chart of a method 4600 for video processing according to an embodiment of the present disclosure. The method 4600 is implemented during conversion between a video unit of a video and a bit stream of the video.

[0108] At block 4610, for conversion between a video unit of a video and a bitstream of the video unit, surround motion compensation (WAMC) is applied during derivation of motion information for the video unit. In some embodiments, WAMC refers to at least one of: horizontal surround and vertical surround.

[0109] At block 4620, conversion is performed based on the derived motion information. In some embodiments, conversion may include encoding the video unit into a bitstream. Alternatively or additionally, conversion may include decoding the video unit from the bitstream. In this way, coding efficiency and coding performance may be improved.

[0110] In some embodiments, WAMC may be applied to derive a prediction or reconstruction of a video unit. For example, surround motion compensation may be used to derive motion information for a video unit, which is used to derive a prediction / reconstruction of the video unit.

[0111] In some embodiments, if the template is used to refine or derive one or more motion vectors or one or more motion vector differences, WAMC may be applied to derive samples of the template. In some embodiments, if the template is used to determine a reference index, WAMC may be applied to derive samples of the template. In some embodiments, if the template is used to determine one or more signs of a motion vector difference or one or more residual coefficients, WAMC may be applied to derive samples of the template. In some embodiments, if the template is used to determine a partition of a video unit, WAMC may derive samples of the template.

[0112] In some embodiments, WAMC is applied to prediction samples produced by a multiple hypothesis prediction (MHP) process. In some embodiments, WAMC is applied to derive prediction samples produced by overlapped block motion compensation (OBMC). In some embodiments, if a template is used to determine the blending weights for OBMC, WAMC may be applied to derive samples for the template.

[0113] In some embodiments, if a motion refinement process is used for derivation of motion information, WAMC may be applied during the motion refinement process. In some embodiments, if the motion refinement process is a bilateral matching based codec process, WAMC is applied to derive one or more prediction blocks for determining bilateral matching costs. In some embodiments, the bilateral matching based codec process refers to decoder side motion vector refinement (DMVR) or multi-pass DMVR. In some embodiments, if the motion refinement process is a template matching based codec process, WAMC is applied to derive predictions for the template.

[0114] In some embodiments, WAMC may be applied during construction of a motion list for a video unit. In some embodiments, WAMC is used to determine whether to add one or more motion candidates to a motion list.

[0115] In some embodiments, the determination of whether one or more motion candidates are added to the motion list is within the motion candidate type. In some embodiments, the motion candidate type is one of: a spatially adjacent (adjacent and / or non-adjacent) motion candidate, a temporally adjacent (adjacent and / or non-adjacent) motion candidate, a sub-block based temporal motion candidate, a historical motion candidate, a paired motion candidate, or an affine motion candidate.

[0116] In some embodiments, if in determining whether one or more motion candidates are added to the motion list and the motion candidate with the smaller template matching cost is added to the motion list, WAMC is used to derive a prediction of the template. In some embodiments, if bilateral matching is used in the determination, WAMC is applied to derive a prediction of the template.

[0117] In some embodiments, WAMC may be applied during reordering of motion lists of video units. In some embodiments, during reordering of motion lists, a prediction of a template or a prediction of a video unit for each motion candidate in the motion list is used to calculate a cost, and WAMC is applied to derive a prediction of the template or a prediction of the video unit.

[0118] In some embodiments, WAMC is applied differently for different interpolation filters. In some embodiments, WAMC is applied differently when the interpolation filter is 6-tap, 8-tap, or 12-tap. In other words, when the interpolation filter is 6-tap, WAMC may be applied differently than when the interpolation filter is 8-tap.

[0119] In some embodiments, if one or more offsets are used to adjust the prediction of the video unit, WAMC may be applied to derive the one or more offsets. In some embodiments, the one or more offsets are derived using a linear model based process. In some embodiments, the linear model based process includes local illumination compensation (LIC). In some embodiments, WAMC is applied to derive the prediction of the template to determine the linear parameters.

[0120] In some embodiments, the one or more offsets are derived using an optical flow based process. In some embodiments, the optical flow based process comprises one of: bidirectional optical flow (BDOF), sample based BDOF, or prediction refinement with optical flow for affine mode (PROF).

[0121] In some embodiments, WAMC is applied to derive prediction blocks, which are used to derive refined motion vectors for each sub-block in a BDOF or each sample in a sample-based BDOF. In some embodiments, WAMC is applied to derive predicted sub-blocks in a PROF. In some embodiments, video units are adjusted using one or more offsets at a block level, or a sub-block level, or a sample level.

[0122] In some embodiments, WAMC is disabled for a specific codec or template-based codec. In some embodiments, the specific codec includes one of the following: LIC, template matching (TM), DMVR, multi-pass DMVR, adaptive decoder-side motion vector refinement (ADMVR), OBMC, BDOF, sample-based BDOF, TM merge, TM advanced motion vector prediction (AMVP), MHP, adaptive reordering of merge candidates (ARMC), geometric partitioning mode (GPM) merge mode with motion vector difference (MMVD), affine MMVD, GPM with inter-frame and intra-frame prediction, template matching AMVP-merge mode or bilateral matching AMVP-merge mode. In some embodiments, the template-based codec includes one of the following: TM-based reordering for MMVD and affine MMVD, motion vector difference (MVD) sign prediction, ARMC based on motion vector (MV) candidate type, combination of intra-frame and inter-frame prediction (CIIP)-TM, GPM-TM, out-of-bounds (OOB) check or motion compensated padding.

[0123] In some embodiments, a video unit comprises one of: a color component, a sub-picture, a slice, a tile, a codec tree unit (CTU), a CTU row, a group of CTUs, a codec unit (CU), a prediction unit (PU), a transform unit (TU), a coding tree block (CTB), a codec block (CB), a prediction block (PB), a transform block (TB), a block, a sub-block of a block, a sub-region within a block, or a region containing more than one sample or pixel.

[0124] In some embodiments, an indication of whether and / or how WAMC is applied to the derivation of motion information for a video unit is indicated at one of the following: sequence level, picture level group, picture level, slice level, or tile group level. In some embodiments, an indication of whether and / or how WAMC is applied to the derivation of motion information for a video unit is indicated in one of the following: sequence header, picture header, sequence parameter set (SPS), video parameter set (VPS), dependency parameter set (DPS), decoding capability information (DCI), picture parameter set (PPS), adaptation parameter set (APS), slice header, or tile group header.

[0125] According to another embodiment of the present disclosure, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by a device for video processing. The method includes: applying surround motion compensation (WAMC) during the derivation of motion information for the video unit; and performing conversion based on the derived motion information.

[0126] According to some further embodiments of the present disclosure, a method for storing a bitstream of a video is provided. The method includes: applying surround motion compensation (WAMC) during derivation of motion information for the video unit; performing conversion based on the derived motion information; and storing the bitstream in a non-transitory computer-readable recording medium.

[0127] Fig.47 A flow chart of a method 4700 for video processing according to an embodiment of the present disclosure is shown. The method 4700 is implemented during conversion between a video unit of a video and a bit stream of the video.

[0128] At block 4710, for conversion between a video unit of a video and a bitstream of the video unit, it is determined whether a border filling process is allowed for the video unit based on whether surround motion compensation (WAMC) is enabled. In some embodiments, WAMC refers to at least one of: horizontal surround or vertical surround.

[0129] At block 4720, conversion is performed based on the determination. In some embodiments, the conversion may include encoding the video unit into a bitstream. Alternatively or in addition, the conversion may include decoding the video unit from the bitstream. In this way, coding efficiency and coding performance may be improved.

[0130] In some embodiments, based on determining that WAMC is enabled, out of bounds (OOB) checks are not allowed. In some embodiments, if WAMC is enabled, wraparound offsets are used in OOB checks. In some embodiments, WAMC is independent of OOB checks.

[0131] In some embodiments, if WAMC is enabled, one or more directions of motion compensated border filling are prohibited. For example, when WAMC is enabled, one or more directions of motion compensated border filling (left, and / or right, and / or up, and / or down) may not be allowed.

[0132] In some embodiments, if WAMC is horizontal wrapping, the left and right directions of motion compensated border filling are disabled. In some embodiments, if WAMC is horizontal and vertical wrapping, all directions of motion compensated border filling are disabled. In some embodiments, WAMC is vertical wrapping, the top and bottom directions of motion compensated border filling are disabled. In some embodiments, WAMC is enabled and padded areas for one or more directions are generated by motion compensated border filling, the padded areas are not used.

[0133] In some embodiments, the filled region size of the motion compensated border filling is used in a clipping operation of a motion vector for at least one of a sub-picture or a surround. In some embodiments, the filled region size is used in determining a threshold for clipping motion vectors. In some embodiments, when motion compensated border filling is used, horMax+S, horMin-S, verMax+S, verMin-S are used as thresholds for clipping motion vectors, and wherein horMax represents the horizontal maximum value of the image boundary, horMin represents the horizontal minimum value of the image boundary, verMax represents the vertical maximum value of the image boundary, verMin represents the vertical minimum value of the image boundary, and S represents the filled region size.

[0134] In some embodiments, the video unit includes one of the following: a color component, a sub-picture, a slice, a tile, a codec tree unit (CTU), a CTU row, a group of CTUs, a codec unit (CU), a prediction unit (PU), a transform unit (TU), a coding tree block (CTB), a codec block (CB), a prediction block (PB), a transform block (TB), a block, a sub-block of a block, a sub-region within a block, or a region containing more than one sample or pixel. In some embodiments, an indication of whether and / or how to determine whether a boundary filling process is allowed is indicated at one of the following: a sequence level, a picture level group, a picture level, a slice level, or a tile group level. In some embodiments, an indication of whether and / or how to determine whether a boundary filling process is allowed is indicated in one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), a decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header.

[0135] According to another embodiment of the present disclosure, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by a device for video processing. The method includes: determining whether a boundary filling process is allowed for a video unit of the video based on whether surround motion compensation (WAMC) is enabled; and generating a bitstream based on the determination.

[0136] According to some other embodiments of the present disclosure, a method for storing a bitstream of a video is provided. The method includes: determining whether a border filling process is allowed for a video unit based on whether surround motion compensation (WAMC) is enabled; generating a bitstream based on the determination; and storing the bitstream in a non-transitory computer-readable recording medium.

[0137] Implementations of the present disclosure may be described in view of the following clauses, and features thereof may be combined in any reasonable manner.

[0138] Clause 1. A method of video processing, comprising: converting between a video unit of a video and a bitstream of the video unit, applying surround motion compensation (WAMC) during derivation of motion information for the video unit; and performing the converting based on the derived motion information.

[0139] Clause 2. The method of clause 1, wherein applying the WAMC during the derivation of the motion information comprises: applying the WAMC to derive a prediction or a reconstruction of the video unit.

[0140] Clause 3. A method according to clause 1, wherein applying the WAMC during the derivation of the motion information comprises: applying the WAMC to derive samples of the template based on determining that a template is used to refine or derive one or more motion vectors or one or more motion vector differences.

[0141] Clause 4. The method of clause 1, wherein applying the WAMC during the derivation of the motion information comprises: applying the WAMC to derive samples of the template based on determining that a template is used to determine a reference index.

[0142] Clause 5. A method according to clause 1, wherein applying the WAMC during the derivation of the motion information comprises: applying the WAMC to derive samples of the template based on determining one or more symbols or one or more residual coefficients of a template used to determine a motion vector difference.

[0143] Clause 6. The method of clause 1, wherein applying the WAMC during the derivation of the motion information comprises: based on determining that a template is used to determine the partitioning of the video unit, applying the WAMC to derive samples of the template.

[0144] Clause 7. The method of clause 1, wherein the WAMC is applied to prediction samples generated by a multiple hypothesis prediction (MHP) process.

[0145] Clause 8. The method of clause 1, wherein the WAMC is used to derive prediction samples generated by overlapped block motion compensation (OBMC).

[0146] Clause 9. The method of clause 8, wherein applying the WAMC during the derivation of the motion information comprises: applying the WAMC to derive samples of the template based on determining that a template is used to determine blending weights of OBMC.

[0147] Clause 10. The method of clause 1, wherein applying the WAMC during the derivation of the motion information comprises: applying the WAMC during the motion refinement process based on determining that a motion refinement process is used for the derivation of the motion information.

[0148] Clause 11. The method of clause 10, wherein if the motion refinement process is a bilateral matching based codec process, the WAMC is applied to derive one or more prediction blocks for determining a bilateral matching cost.

[0149] Clause 12. The method of clause 11, wherein the bilateral matching based encoding and decoding process refers to decoder-side motion vector refinement (DMVR) or multi-pass DMVR.

[0150] Clause 13. The method of clause 10, wherein if the motion refinement process is a template matching based codec process, the WAMC is applied to derive a prediction of the template.

[0151] Clause 14. The method of Clause 1, wherein applying the WAMC during the derivation of the motion information comprises: applying the WAMC during construction of a motion list for the video unit.

[0152] Clause 15. The method of clause 14, wherein the WAMC is used to determine whether one or more motion candidates are added to the motion list.

[0153] Clause 16. The method of Clause 15, wherein the determination of whether the one or more motion candidates are added to the motion list is within a motion candidate type.

[0154] Clause 17. The method of clause 16, wherein the motion candidate type is one of: a spatial neighboring motion candidate, a temporal neighboring motion candidate, a sub-block based temporal motion candidate, a historical motion candidate, a pairwise motion candidate, or an affine motion candidate.

[0155] Clause 18. A method according to clause 14, wherein the WAMC is applied to derive a prediction of the template based on determining whether template matching is used in the determination of whether one or more motion candidates are added to the motion list, and the motion candidate with a smaller template matching cost is added to the motion list.

[0156] Clause 19. The method of clause 18, wherein upon determining that bilateral matching is used in the determining, the WAMC is applied to derive the prediction of the template.

[0157] Clause 20. The method of Clause 1, wherein applying the WAMC during the derivation of the motion information comprises applying the WAMC during reordering of motion lists of the video units.

[0158] Clause 21. A method according to clause 20, wherein a cost is calculated using a prediction of a template for each motion candidate in the motion list or the prediction of the video unit during the reordering of the motion list, and the WAMC is applied to derive the prediction of the template or the prediction of the video unit.

[0159] Clause 22. The method of clause 1, wherein the WAMC is applied differently for different interpolation filters.

[0160] Clause 23. The method of clause 22, wherein the WAMC is applied differently when the interpolation filter is a 6-tap, 8-tap, or 12-tap interpolation filter.

[0161] Clause 24. The method of clause 1, wherein applying the WAMC during the derivation of the motion information comprises: applying the WAMC to derive the one or more offsets based on determining the prediction of the video unit to be adjusted using the one or more offsets.

[0162] Clause 25. The method of clause 24, wherein the one or more offsets are derived using a linear model-based process.

[0163] Clause 26. The method of clause 25, wherein the linear model based process comprises local illumination compensation (LIC).

[0164] Clause 27. The method of clause 25, wherein the WAMC is applied to a prediction of a derived template to determine linear parameters.

[0165] Clause 28. The method of clause 24, wherein the one or more offsets are derived using an optical flow based process.

[0166] Clause 29. The method of clause 28, wherein the optical flow based process comprises one of: bidirectional optical flow (BDOF), sample-based BDOF, or prediction refinement with optical flow for affine mode (PROF).

[0167] Clause 30. The method of clause 28, wherein the WAMC is applied to derive a prediction block used to derive a refined motion vector for each sub-block in a BDOF or each sample in a sample-based BDOF.

[0168] Clause 31. The method of clause 28, wherein the WAMC is applied to derive prediction sub-blocks in PROF.

[0169] Clause 32. The method of clause 24, wherein the video unit is adjusted using the one or more offsets at a block level, or a sub-block level, or a sample level.

[0170] Clause 33. The method of clause 1, wherein the WAMC is disabled for a specific codec or a template-based codec.

[0171] Clause 34. A method according to clause 33, wherein the specific codec tool includes one of the following: LIC, template matching (TM), DMVR, multi-pass DMVR, adaptive decoder-side motion vector refinement (ADMVR), OBMC, BDOF, sample-based BDOF, TM merge, TM advanced motion vector prediction (AMVP), MHP, adaptive reordering of merge candidates (ARMC), geometric partitioning mode (GPM) with motion vector difference (MMVD) merge mode, affine MMVD, GPM with inter-frame and intra-frame prediction, template matching AMVP-merge mode, or bilateral matching AMVP-merge mode.

[0172] Clause 35. A method according to clause 33, wherein the template-based encoding and decoding tool includes one of the following: TM-based reordering for MMVD and affine MMVD, motion vector difference (MVD) sign prediction, ARMC based on motion vector (MV) candidate type, combined intra-frame and inter-frame prediction (CIIP)-TM, GPM-TM, out-of-bounds (OOB) check, or motion compensated filling.

[0173] Clause 36. A method according to any one of clauses 1 to 35, wherein the video unit comprises one of the following: a color component, a sub-picture, a slice, a tile, a codec tree unit (CTU), a CTU row, a group of CTUs, a codec unit (CU), a prediction unit (PU), a transform unit (TU), a codec tree block (CTB), a codec block (CB), a prediction block (PB), a transform block (TB), a block, a sub-block of a block, a sub-region within a block, or a region containing more than one sample or pixel.

[0174] Clause 37. A method according to any one of clauses 1 to 35, wherein an indication of whether and / or how the WAMC is applied to the derivation of the motion information of the video unit is indicated at one of: a sequence level, a picture level group, a picture level, a slice level, or a tile group level.

[0175] Clause 38. A method according to any one of clauses 1 to 35, wherein an indication of whether and / or how the WAMC is applied to the derivation of the motion information of the video unit is indicated in one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header.

[0176] Clause 39. A method of video processing, comprising: for conversion between a video unit of a video and a bitstream of the video unit, determining whether a border filling process is allowed for the video unit based on whether surround motion compensation (WAMC) is enabled; and performing the conversion based on the determination.

[0177] Clause 40. The method of clause 39, wherein based on determining that the WAMC is enabled, out-of-bounds (OOB) checking is not allowed.

[0178] Clause 41. The method of clause 39, wherein based on determining that the WAMC is enabled, a wraparound offset is used in the OOB check.

[0179] Clause 42. The method of clause 39, wherein the WAMC is independent of the OOB check.

[0180] Clause 43. The method of clause 39, wherein based on determining that the WAMC is enabled, one or more directions of motion compensated border filling are disabled.

[0181] Clause 44. The method of clause 43, wherein based on determining that the WAMC is horizontal wrapping, a left direction and a right direction of the motion compensated border filling are disabled.

[0182] Clause 45. The method of clause 43, wherein based on determining that the WAMC is horizontally and vertically wrapped, all directions of the motion compensated border filling are disabled.

[0183] Clause 46. The method of clause 43, wherein based on determining that the WAMC is vertical wrapping, an upward direction and a downward direction of motion compensated border filling are disabled.

[0184] Clause 47. The method of clause 43, wherein based on determining that the WAMC is enabled and padded regions for one or more directions are generated by the motion compensated boundary filling, the padded regions are not used.

[0185] Clause 48. The method of clause 39, wherein a filled region size of the motion compensated border fill is used in a clipping operation of a motion vector for at least one of a sub-picture or a surround.

[0186] Clause 49. The method of clause 48, wherein the padded region size is used in determining a threshold for clipping the motion vector.

[0187] Clause 50. A method according to Clause 49, wherein when the motion compensation boundary filling is used, horMax+S, horMin-S, verMax+S, verMin-S are used as thresholds for limiting motion vectors, and wherein horMax represents the horizontal maximum value of the image boundary, horMin represents the horizontal minimum value of the image boundary, verMax represents the vertical maximum value of the image boundary, verMin represents the vertical minimum value of the image boundary, and S represents the size of the filled area.

[0188] Clause 51. A method according to any one of clauses 39 to 50, wherein the video unit comprises one of the following: a color component, a sub-picture, a slice, a tile, a codec tree unit (CTU), a CTU row, a group of CTUs, a codec unit (CU), a prediction unit (PU), a transform unit (TU), a codec tree block (CTB), a codec block (CB), a prediction block (PB), a transform block (TB), a block, a sub-block of a block, a sub-region within a block, or a region containing more than one sample or pixel.

[0189] Clause 52. A method according to any one of clauses 39 to 50, wherein an indication of whether and / or how to determine whether the boundary filling process is allowed is indicated at one of the following: sequence level, picture level group, picture level, slice level, or tile group level.

[0190] Clause 53. A method according to any one of clauses 39 to 50, wherein an indication of whether and / or how to determine whether the boundary filling process is allowed is indicated in one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header.

[0191] Clause 54. The method of any one of clauses 1 to 53, wherein the WAMC refers to at least one of horizontal wrapping or vertical wrapping.

[0192] Clause 55. The method of any one of clauses 1 to 54, wherein the converting comprises encoding the video unit into the bitstream.

[0193] Clause 56. The method of any one of clauses 1 to 54, wherein the converting comprises decoding the video unit from the bitstream.

[0194] Clause 57. An apparatus for video processing, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any of clauses 1-56.

[0195] Clause 58. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method of any of clauses 1-56.

[0196] Clause 59. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing device, wherein the method includes: applying surround motion compensation (WAMC) during the derivation of motion information for a video unit of the video; and generating a bitstream based on the derived motion information.

[0197] Clause 60. A method for storing a bitstream of a video, comprising: applying surround motion compensation (WAMC) during derivation of motion information of a video unit of the video; generating a bitstream based on the derived motion information; and storing the bitstream in a non-transitory computer-readable recording medium.

[0198] Clause 61. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing device, the method comprising: determining whether a boundary filling process is allowed for a video unit of the video based on whether surround motion compensation (WAMC) is enabled; and generating a bitstream based on the determination.

[0199] Clause 62. A method for storing a bitstream of a video, comprising: determining whether a border filling process is allowed for a video unit of the video based on whether surround motion compensation (WAMC) is enabled; generating a bitstream based on the determination; and storing the bitstream in a non-transitory computer-readable recording medium. Example Device

[0200] Fig.48 A block diagram of a computing device 4800 in which various embodiments of the present disclosure may be implemented is shown. The computing device 4800 may be implemented as a source device 110 (or video encoder 114 or 200) or a destination device 120 (or video decoder 124 or 300), or may be included in a source device 110 (or video encoder 114 or 200) or a destination device 120 (or video decoder 124 or 300).

[0201] It should be understood that Fig.48 The computing device 4800 shown in the figure is for illustrative purposes only and is not intended to in any way imply any limitation on the functionality and scope of the embodiments of the present disclosure.

[0202] like Fig.48 As shown, computing device 4800 includes a general computing device 4800. Computing device 4800 may include at least one or more processors or processing units 4810, memory 4820, storage unit 4830, one or more communication units 4840, one or more input devices 4850, and one or more output devices 4860.

[0203] In some embodiments, the computing device 4800 can be implemented as any user terminal or server terminal with computing power. The server terminal can be a server provided by a service provider, a large computing device, etc. The user terminal can be, for example, any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet computer, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a gaming device, or any combination thereof, and includes the accessories and peripherals of these devices, or any combination thereof. It is conceivable that the computing device 4800 can support any type of interface to the user (such as a "wearable" circuit device, etc.).

[0204] The processing unit 4810 may be a physical processor or a virtual processor and may implement various processes based on a program stored in the memory 4820. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the computing device 4800. The processing unit 4810 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0205] The computing device 4800 typically includes various computer storage media. Such media can be any media accessible by the computing device 4800, including but not limited to volatile media and non-volatile media, or removable media and non-removable media. The memory 4820 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (such as a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM) or flash memory) or any combination thereof. The storage unit 4830 can be any removable or non-removable medium, and can include machine-readable media, such as a memory, a flash drive, a disk, or other media that can be used to store information and / or data and can be accessed in the computing device 4800.

[0206] The computing device 4800 may also include additional removable / non-removable storage media, volatile / non-volatile storage media. Fig.48 Although not shown in the figure, a magnetic disk drive for reading from and / or writing to a removable nonvolatile magnetic disk and an optical disk drive for reading from and / or writing to a removable nonvolatile optical disk may be provided. In this case, each drive may be connected to the bus (not shown) via one or more data medium interfaces.

[0207] The communication unit 4840 communicates with another computing device via a communication medium. In addition, the functions of the components in the computing device 4800 can be implemented by a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 4800 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general network nodes.

[0208] Input device 4850 may be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 4860 may be one or more of various output devices, such as a display, speaker, printer, etc. With the aid of communication unit 4840, computing device 4800 may also communicate with one or more external devices (not shown), such as storage devices and display devices, and computing device 4800 may also communicate with one or more devices that enable a user to interact with computing device 4800, or any device that enables computing device 4800 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.), if necessary. Such communication may be performed via an input / output (I / O) interface (not shown).

[0209] In some embodiments, some or all components of the computing device 4800 may also be arranged in a cloud computing architecture rather than being integrated in a single device. In a cloud computing architecture, components may be provided remotely and work together to implement the functions described in the present disclosure. In some embodiments, cloud computing provides computing, software, data access and storage services, which will not require the end user to know the physical location or configuration of the system or hardware that provides these services. In various embodiments, cloud computing provides services via a wide area network (such as the Internet) using a suitable protocol. For example, a cloud computing provider provides applications via a wide area network, which can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data may be stored on a remote server. The computing resources in a cloud computing environment may be merged or distributed at the location of a remote data center. Cloud computing infrastructures may provide services through a shared data center, although they appear as a single access point for users. Therefore, cloud computing architectures may be used to provide components and functions described herein from a service provider at a remote location. Alternatively, they may be provided by a conventional server, or installed directly or otherwise on a client device.

[0210] In an embodiment of the present disclosure, the computing device 4800 may be used to implement video encoding / decoding. The memory 4820 may include one or more video codec modules 4825 having one or more program instructions. These modules can be accessed and executed by the processing unit 4810 to perform the functions of the various embodiments described herein.

[0211] In an example embodiment of performing video encoding, an input device 4850 may receive video data as input 4870 to be encoded. The video data may be processed by, for example, a video codec module 4825 to generate an encoded bitstream. The encoded bitstream may be provided as output 4880 via an output device 4860.

[0212] In an example embodiment of performing video decoding, the input device 4850 may receive an encoded bitstream as input 4870. The encoded bitstream may be processed by, for example, a video codec module 4825 to generate decoded video data. The decoded video data may be provided as output 4880 via an output device 4860.

[0213] Although the present disclosure has been specifically shown and described with reference to the preferred embodiments of the present disclosure, it will be understood by those skilled in the art that various changes can be made in form and detail without departing from the spirit and scope of the present application as defined by the appended claims. These changes are intended to be encompassed by the scope of the present application. Therefore, the foregoing description of the embodiments of the present application is not intended to be limiting.

Claims

1. A video processing method, comprising: applying surround motion compensation (WAMC) during derivation of motion information for a video unit of a video and a bitstream for the video unit; as well as The converting is performed based on the derived motion information.

2. The method of claim 1 , wherein applying the WAMC during the derivation of the motion information comprises: The WAMC is applied to derive a prediction or a reconstruction of the video unit.

3. The method of claim 1 , wherein applying the WAMC during the derivation of the motion information comprises: Based on determining that a template is used to refine or derive one or more motion vectors or one or more motion vector differences, the WAMC is applied to derive samples of the template.

4. The method of claim 1 , wherein applying the WAMC during the derivation of the motion information comprises: Based on determining that a template is used to determine a reference index, the WAMC is applied to derive samples of the template.

5. The method of claim 1 , wherein applying the WAMC during the derivation of the motion information comprises: Based on determining that a template is used for determining one or more symbols or one or more residual coefficients of a motion vector difference, the WAMC is applied to derive samples of the template.

6. The method of claim 1 , wherein applying the WAMC during the derivation of the motion information comprises: Based on determining that a template is used to determine the partitioning of the video unit, the WAMC is applied to derive samples of the template.

7. The method of claim 1, wherein the WAMC is applied to prediction samples generated by a multiple hypothesis prediction (MHP) process.

8. The method of claim 1, wherein the WAMC is used to derive prediction samples generated by overlapped block motion compensation (OBMC).

9. The method of claim 8, wherein applying the WAMC during the derivation of the motion information comprises: Based on determining that a template is used to determine the mixing weights of the OBMC, the WAMC is applied to derive samples of the template.

10. The method of claim 1, wherein applying the WAMC during the derivation of the motion information comprises: Based on determining that a motion refinement process is used for the derivation of the motion information, the WAMC is applied during the motion refinement process.

11. The method of claim 10, wherein if the motion refinement process is a bilateral matching based codec process, the WAMC is applied to derive one or more prediction blocks for determining a bilateral matching cost.

12. The method of claim 11, wherein the bilateral matching based encoding and decoding process refers to decoder side motion vector refinement (DMVR) or multi-pass DMVR.

13. The method of claim 10, wherein if the motion refinement process is a codec process based on template matching, the WAMC is applied to derive a prediction of the template.

14. The method of claim 1, wherein applying the WAMC during the derivation of the motion information comprises: The WAMC is applied during construction of a motion list for the video unit.

15. The method of claim 14, wherein the WAMC is used to determine whether one or more motion candidates are added to the motion list. 16 . The method of claim 15 , wherein the determination of whether the one or more motion candidates are added to the motion list is within a motion candidate type.

17. The method of claim 16, wherein the motion candidate type is one of: Spatial neighboring motion candidates, Temporal neighbor motion candidates, Sub-block based temporal motion candidates, Historical motion candidates, paired motion candidates, or Affine motion candidates.

18. The method of claim 14, wherein the WAMC is applied to derive a prediction of the template based on determining whether template matching is used in the determination of whether the one or more motion candidates are added to the motion list, and the motion candidate with a smaller template matching cost is added to the motion list.

19. The method of claim 18, wherein upon determining that bilateral matching is used in the determining, the WAMC is applied to derive the prediction of the template.

20. The method of claim 1, wherein applying the WAMC during the derivation of the motion information comprises: The WAMC is applied during reordering of the motion list of the video unit.

21. The method of claim 20, wherein a cost is calculated using a prediction of a template for each motion candidate in the motion list or the prediction of the video unit during the reordering of the motion list, and the WAMC is applied to derive the prediction of the template or the prediction of the video unit.

22. The method of claim 1, wherein the WAMC is applied in different ways for different interpolation filters.

23. The method of claim 22, wherein when the interpolation filter is a 6-tap, 8-tap, or 12-tap interpolation filter, the WAMC is applied in a different manner.

24. The method of claim 1, wherein applying the WAMC during the derivation of the motion information comprises: Based on determining the prediction of the video unit to be adjusted using one or more offsets, applying the WAMC to derive the one or more offsets.

25. The method of claim 24, wherein the one or more offsets are derived using a linear model based process.

26. The method of claim 25, wherein the linear model based process comprises local illumination compensation (LIC).

27. The method of claim 25, wherein the WAMC is applied to derive predictions of templates to determine linear parameters.

28. The method of claim 24, wherein the one or more offsets are derived using an optical flow based process.

29. The method of claim 28, wherein the optical flow based process comprises one of the following: Bidirectional Optical Flow (BDOF), Sample-based BDOF, or Prediction refinement with optical flow for affine mode (PROF).

30. The method of claim 28, wherein the WAMC is applied to derive a prediction block used to derive a refined motion vector for each sub-block in a BDOF or each sample in a sample-based BDOF.

31. The method of claim 28, wherein the WAMC is applied to derive predicted sub-blocks in PROF.

32. The method of claim 24, wherein the video unit is adjusted using the one or more offsets at a block level, or a sub-block level, or a sample level.

33. The method of claim 1, wherein the WAMC is disabled for a specific codec or a template-based codec.

34. The method of claim 33, wherein the specific codec tool comprises one of the following: LIC, Template Matching (TM), DMVR, Multi-pass DMVR, Adaptive decoder-side motion vector refinement (ADMVR), OBMC, BDOF, Sample-based BDOF, TM merger, TM Advanced Motion Vector Prediction (AMVP), MHP, Adaptive Reordering of Merge Candidates (ARMC), Geometric Partitioning Pattern (GPM) merge mode with Motion Vector Difference (MMVD), Affine MMVD, GPM with inter- and intra-frame prediction, Template matching AMVP-Merge mode, or Bilateral matching AMVP-merge mode.

35. The method of claim 33, wherein the template-based encoding and decoding tool comprises one of: TM-based reordering for MMVD and affine MMVD, Motion Vector Difference (MVD) sign prediction, ARMC based on motion vector (MV) candidate type, Combined Intra and Inter Prediction (CIIP)-TM, GPM-TM, Out-of-bounds (OOB) checking, or Motion compensated padding.

36. The method of any one of claims 1 to 35, wherein the video unit comprises one of: a color component, Sub-picture, Strips, Tile, Codec Tree Unit (CTU), CTU line, A group of CTUs, Codec Unit (CU), Prediction Unit (PU), Transformation Unit (TU), Codec Tree Block (CTB), Codec Block (CB), Prediction Block (PB), Transform Block (TB), piece, Sub-blocks of a block, a sub-region within a block, or An area that contains more than one sample or pixel.

37. The method of any one of claims 1 to 35, wherein an indication of whether and / or how to apply the WAMC to the derivation of the motion information of the video unit is indicated at one of: Sequence level, Picture level group, Picture level, Stripe level, or Tile group level.

38. The method of any one of claims 1 to 35, wherein an indication of whether and / or how to apply the WAMC to the derivation of the motion information of the video unit is indicated in one of: Sequence header, Picture header, Sequence Parameter Set (SPS), Video Parameter Set (VPS), Dependency Parameter Set (DPS), Decoding Capability Information (DCI), Picture Parameter Set (PPS), Adaptive Parameter Set (APS), Strip header, or Tile group header.

39. A method for video processing, comprising: For conversion between a video unit of a video and a bitstream of the video unit, determining whether a border filling process is allowed for the video unit based on whether surround motion compensation (WAMC) is enabled; as well as The converting is performed based on the determination.

40. The method of claim 39, wherein based on determining that the WAMC is enabled, out-of-bounds (OOB) checking is not allowed.

41. The method of claim 39, wherein based on determining that the WAMC is enabled, a wraparound offset is used in the OOB check.

42. The method of claim 39, wherein the WAMC is independent of the OOB check.

43. The method of claim 39, wherein based on determining that the WAMC is enabled, one or more directions of motion compensated border filling are disabled.

44. The method of claim 43, wherein based on determining that the WAMC is horizontal wrapping, a left direction and a right direction of the motion compensated border filling are disabled.

45. The method of claim 43, wherein based on determining that the WAMC is horizontally and vertically wrapping, all directions of the motion compensated border filling are disabled.

46. ​​The method of claim 43, wherein based on determining that the WAMC is vertical wrapping, an upward direction and a downward direction of motion compensated border filling are disabled.

47. The method of claim 43, wherein based on determining that the WAMC is enabled and padded regions for one or more directions are generated by the motion compensated boundary filling, the padded regions are not used.

48. The method of claim 39, wherein a filled region size of a motion compensated border fill is used in a clipping operation of a motion vector for at least one of a sub-picture or a surround.

49. The method of claim 48, wherein the padded region size is used in determining a threshold for clipping the motion vectors.

50. The method of claim 49, wherein when the motion compensated border filling is used, horMax+S, horMin-S, verMax+S, verMin-S are used as thresholds for limiting motion vectors, and wherein horMax represents the horizontal maximum value of the image boundary, horMin represents the horizontal minimum value of the image boundary, verMax represents the vertical maximum value of the image boundary, verMin represents the vertical minimum value of the image boundary, and S represents the size of the filled area.

51. The method of any one of claims 39 to 50, wherein the video unit comprises one of: Color components, Sub-picture, Strips, Tile, Codec Tree Unit (CTU), CTU line, A group of CTUs, Codec Unit (CU), Prediction Unit (PU), Transformation Unit (TU), Codec Tree Block (CTB), Codec Block (CB), Prediction Block (PB), Transform Block (TB), piece, Sub-blocks of a block, a sub-region within a block, or An area that contains more than one sample or pixel.

52. A method according to any one of claims 39 to 50, wherein an indication of whether and / or how to determine whether the border filling process is allowed is indicated at one of: Sequence level, Picture level group, Picture level, Stripe level, or Tile group level.

53. A method according to any one of claims 39 to 50, wherein an indication of whether and / or how to determine whether the border filling process is allowed is indicated in one of the following: Sequence header, Picture header, Sequence Parameter Set (SPS), Video Parameter Set (VPS), Dependency Parameter Set (DPS), Decoding Capability Information (DCI), Picture Parameter Set (PPS), Adaptive Parameter Set (APS), Strip header, or Tile group header.

54. The method of any one of claims 1 to 53, wherein the WAMC refers to at least one of horizontal wrapping or vertical wrapping.

55. The method of any one of claims 1 to 54, wherein the converting comprises encoding the video unit into the bitstream.

56. The method of any one of claims 1 to 54, wherein the converting comprises decoding the video unit from the bitstream.

57. An apparatus for video processing, comprising a processor and a non-volatile memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1-56.

58. A non-transitory computer-readable storage medium storing instructions, the instructions causing a processor to execute the method according to any one of claims 1-56.

59. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing device, wherein the method comprises: applying surround motion compensation (WAMC) during derivation of motion information for a video unit of the video; as well as A bitstream is generated based on the derived motion information.

60. A method for storing a bitstream of a video, comprising: applying surround motion compensation (WAMC) during derivation of motion information for a video unit of the video; generating a bitstream based on the derived motion information; as well as The bit stream is stored in a non-transitory computer-readable recording medium.

61. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing device, the method comprising: determining whether a border filling process is enabled for a video unit of the video based on whether surround motion compensation (WAMC) is enabled; as well as A bitstream is generated based on the determination.

62. A method for storing a bitstream of a video, comprising: determining whether a border filling process is enabled for a video unit of the video based on whether surround motion compensation (WAMC) is enabled; generating a bitstream based on the determination; as well as The bit stream is stored in a non-transitory computer-readable recording medium.

Images