Method and device for video processing and medium

Abstract

The embodiment of the invention provides a scheme for video processing. A method for video processing is provided. The method includes determining that an Intra Block Copy (IBC) and Intra Prediction Combination (CIBCIP) mode is applied to a target block in a conversion process between the target block of a video and a bitstream of the target block; obtaining an IBC prediction signal and an intra-frame prediction signal based on a CIBCIP mode; deriving a prediction or reconstruction of the target block by combining the IBC prediction signal and the intra prediction signal; and performing conversion based on the prediction or reconstruction of the target block.

Description

Technical Field

[0001] Embodiments of the present disclosure relate generally to video coding techniques, and more particularly, to intra prediction with combined intra block copying. Background Art

[0002] Nowadays, digital video capabilities are being used in all aspects of people's lives. A variety of video compression technologies, such as MPEG-2, MPEG-4, ITU-TH.263, ITU-TH.264 / MPEG-4 Part 10 Advanced Video Codec (AVC), ITU-TH.265 High Efficiency Video Codec (HEVC) standard, and Versatile Video Codec (VVC) standard have been proposed for video encoding / decoding. However, the encoding and decoding efficiency of traditional video coding and decoding technologies is usually very low, which is not ideal. Summary of the invention

[0003] An embodiment of the present disclosure provides a solution for video processing.

[0004] In a first aspect, a method for video processing is proposed. The method includes: in the conversion process between a target block of a video and a bit stream of the target block, determining that an intra block copy (IBC) and intra prediction combination (CIBCIP) mode is applied to the target block; obtaining an IBC prediction signal and an intra prediction signal based on the CIBCIP mode; deriving a prediction or reconstruction of the target block by combining the IBC prediction signal and the intra prediction signal; and performing conversion based on the prediction or reconstruction of the target block. Compared with traditional schemes, the provided method can effectively improve encoding and decoding efficiency and performance.

[0005] In a second aspect, another method for video processing is provided. The method includes: during a conversion process between a target block of a video and a bitstream of the target block, determining at least one of the following: whether to apply an intra-block copy (IBC) and intra-frame prediction combination (CIBCIP) mode to the target block, or applying the CIBCIP mode to the target block based on codec information; and performing the conversion based on the determination. Compared with the traditional scheme, the provided method can effectively improve the codec efficiency and performance.

[0006] In a third aspect, another method for video processing is proposed. The method includes: during conversion between a target block of a video and a bitstream of the target block, determining whether a combined intra block copy (IBC) and intra prediction (CIBCIP) mode is applied to the target block based on an indication associated with a CIBCIP mode; and performing conversion based on the determination. Compared with traditional schemes, the provided method can effectively improve coding efficiency and performance.

[0007] In a fourth aspect, a device for processing video data is provided. The device for processing video data comprises a processor and a non-volatile memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform a method according to any one of the first aspect, the second aspect or the third aspect.

[0008] In a fifth aspect, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores instructions for causing a processor to execute the method according to any one of the first aspect, the second aspect, or the third aspect.

[0009] In a sixth aspect, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by a video processing device, wherein the method includes: determining that an intra-block copy (IBC) and intra-frame prediction combination (CIBCIP) mode is applied to a target block; obtaining an IBC prediction signal and an intra-frame prediction signal based on the CIBCIP mode; deriving a prediction or reconstruction of the target block by combining the IBC prediction signal and the intra-frame prediction signal; and generating a bitstream of the target block based on the prediction or reconstruction of the target block.

[0010] In a seventh aspect, a method for storing a bitstream of a video includes: determining that an intra block copy (IBC) combined with intra prediction (CIBCIP) mode is applied to a target block; obtaining an IBC prediction signal and an intra prediction signal based on the CIBCIP mode; deriving a prediction or reconstruction of the target block by combining the IBC prediction signal and the intra prediction signal; generating a bitstream of the target block based on the prediction or reconstruction of the target block; and storing the bitstream in a non-transitory computer-readable recording medium.

[0011] In an eighth aspect, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by a video processing device, wherein the method includes: determining at least one of the following: whether to apply an intra-block copy (IBC) and intra-frame prediction combination (CIBCIP) mode to a target block, or a process of applying the CIBCIP mode to the target block based on codec information; and generating a bitstream of the target block based on the determination.

[0012] In a ninth aspect, a method for storing a bitstream of a video includes: determining at least one of the following: whether to apply an intra block copy (IBC) and intra prediction combination (CIBCIP) mode to a target block, or a process of applying the CIBCIP mode to the target block based on codec information; generating a bitstream of the target block based on the determination; and storing the bitstream in a non-transitory computer-readable recording medium.

[0013] In a tenth aspect, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by a video processing device, wherein the method includes: determining whether a combined intra block copy (IBC) and intra prediction (CIBCIP) mode is applied to a target block based on an indication associated with the CIBCIP mode; and generating a bitstream of the target block based on the determination.

[0014] In an eleventh aspect, a method for storing a bitstream of a video includes: determining whether an intra block copy (IBC) and intra prediction combination (CIBCIP) mode is applied to a target block based on an indication associated with the CIBCIP mode; generating a bitstream of the target block based on the determination; and storing the bitstream in a non-transitory computer-readable recording medium.

[0015] This disclosure is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This disclosure is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other objects, features and advantages of example embodiments of the present disclosure will become more apparent from the following detailed description with reference to the accompanying drawings, in which like reference numerals generally refer to like components throughout the example embodiments of the present disclosure.

[0017] Figure 1 A block diagram illustrating an example video encoding and decoding system according to some embodiments of the present disclosure is shown;

[0018] Figure 2 shows a block diagram illustrating a first example video encoder according to some embodiments of the present disclosure;

[0019] Figure 3 shows a block diagram illustrating an example video decoder according to some embodiments of the present disclosure;

[0020] Figure 4 An example diagram showing example locations of spatial merging candidates is shown;

[0021] Figure 5 An example diagram for showing candidate pairs of redundant checks being considered for spatial merging candidates is shown;

[0022] Figure 6 An example diagram showing example motion vector scaling for a temporal merging candidate is shown;

[0023] Figure 7 An example diagram for displaying candidate positions for time domain merging candidates C0 and C1 is shown;

[0024] Figure 8 An example diagram for displaying VVC spatial neighboring blocks of a current block is shown;

[0025] Fig. 9 An example diagram for displaying the relationship between a virtual block and a current block is shown.

[0026] Fig.10 An example diagram showing spatially neighboring blocks used to derive spatial merging candidates is shown;

[0027] Fig.11A and Fig. 11B The SbTMVP process in VVC is shown;

[0028] Fig.12 A-12D shows the current CTU processing order and the available samples in the current and left CTUs;

[0029] Fig.13 Example neighboring samples used to calculate SAD are shown;

[0030] Fig.14 shows example neighboring samples for calculating SAD of sub-CU level motion information;

[0031] Fig.15 An example diagram for displaying a sorting process is shown;

[0032] Fig.16 An example diagram showing a reordering process in an encoder;

[0033] Fig.17 An example diagram showing a reordering process in a decoder;

[0034] Fig.18 An example diagram for illustrating performing template matching on a search area around an initial MV is shown;

[0035] Fig.19 An example graph for displaying template matching predictions is shown;

[0036] Fig. 20 An example diagram showing an intra-frame template matching search area used is shown;

[0037] Fig.21 An example diagram showing a template used in TIMD and its reference sample is shown;

[0038] Fig. 22 An example diagram for displaying a template and a reference sample of the template is shown;

[0039] Fig.23An example diagram for displaying templates and reference samples of templates in reference list 0 and reference list 1 is shown;

[0040] Fig.24 An example diagram showing a template and a reference sample of the template for a block having sub-block motion using motion information of a sub-block of a current block is shown;

[0041] Fig.25 An example diagram showing a template and a reference sample of the template for a block having sub-block motion using motion information of each sub-template is shown;

[0042] Fig.26 An example diagram showing a template and a reference sample of the template for a block with OBMC is shown;

[0043] Fig. 27 An example diagram showing motion estimation for a rectangular block with hash values ​​for square sub-blocks is shown;

[0044] Fig.28 An example luminance mapping with a chroma scaling architecture is shown;

[0045] Fig.29 An example diagram showing MMVD search points is shown;

[0046] Fig.30 An example diagram showing an example of inter-frame prediction based on triangulation is shown;

[0047] Fig.31 An example of unidirectional prediction MV selection for triangulation mode is shown;

[0048] Fig.32 Example weights used in the blending process are shown;

[0049] Figures 33A-33C Three MV storage areas with triangleDir equal to 0 are shown;

[0050] Fig.34 An example of GPM segmentation grouped by the same angle is shown;

[0051] Fig.35 An example of unidirectional prediction MV selection for geometric partitioning mode is shown;

[0052] Fig.36 An example of using the geometric partitioning mode to exemplarily generate the warp weight w_0 is shown;

[0053] Fig.37 shows examples of top and left neighboring blocks used in CIIP weight derivation;

[0054] Figures 38A-38BExamples of candidate positions for spatial candidates and temporal candidates are shown;

[0055] Fig.39 shows an example of deriving a sub-CU bv motion field from a corresponding co-located sub-CU by applying motion shifts from spatial neighbors;

[0056] Fig.40 An example of intra-frame template matching is shown;

[0057] Fig.41 shows an example of adjacent spatially neighboring blocks used in accordance with some embodiments of the present disclosure;

[0058] Fig.42 shows an example of top and left neighboring blocks used in CIIP_N1 and CIIP_N2 weight derivation according to some embodiments of the present disclosure;

[0059] Fig.43 An example of triangulation-based IBC prediction according to some embodiments of the present disclosure is shown;

[0060] Figure 44A-Figure 44B Two example search modes are shown according to some embodiments of the present disclosure;

[0061] Fig.45 A flowchart of a method for video processing according to some embodiments of the present disclosure is shown;

[0062] Fig.46 A flowchart of a method for video processing according to some embodiments of the present disclosure is shown;

[0063] Fig.47 A flowchart showing a method for video processing according to some embodiments of the present disclosure; and

[0064] Fig.48 A block diagram of a computing device is shown in which various embodiments of the present disclosure may be implemented.

[0065] Throughout the drawings, the same or similar reference numbers generally refer to the same or similar elements. DETAILED DESCRIPTION

[0066] The principle of the present disclosure is now described in conjunction with some embodiments. It should be understood that these embodiments are described only for illustrative purposes to help those skilled in the art understand and implement the present disclosure, and do not constitute any limitation to the scope of the present disclosure. The disclosure described herein can be implemented in various ways except for the way described below.

[0067] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0068] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but it is not necessary that every embodiment include the particular feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an example embodiment, it is considered that structures or features related to other embodiments that affect that feature are within the knowledge of those skilled in the art, whether or not explicitly described.

[0069] It should be understood that although the terms "first" and "second" etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0070] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "comprise", "include", "have", "have", "include" and / or "include", when used herein, specify the presence of stated features, elements and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components and / or combinations thereof.

[0071] Example Environment

[0072] Figure 1 1 is a block diagram illustrating an example video codec system 100 that may utilize the techniques of the present disclosure. As shown, the video codec system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a video encoding device, and the destination device 120 may also be referred to as a video decoding device. In operation, the source device 110 may be configured to generate encoded video data, and the destination device 120 may be configured to decode the encoded video data generated by the source device 110. The source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0073] Video source 112 may include a source such as a video capture device. Examples of a video capture device include, but are not limited to, an interface that receives video data from a video content provider, a computer graphics system for generating video data, and / or combinations thereof.

[0074] The video data may include one or more pictures. The video encoder 114 encodes the video data from the video source 112 to generate a bitstream. The bitstream may include a bit sequence that forms a codec representation of the video data. The bitstream may include a codec picture and associated data. The codec picture is a codec representation of the picture. The associated data may include a sequence parameter set, a picture parameter set, and other grammatical structures. The I / O interface 116 may include a modulator / demodulator and / or a transmitter. The encoded video data may be directly transmitted to the destination device 120 via the network 130A via the I / O interface 116. The encoded video data may also be stored on a storage medium / server 130B for access by the destination device 120.

[0075] The destination device 120 may include an I / O interface 126, a video decoder 124, and a display device 122. The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain encoded video data from the source device 110 or the storage medium / server 130B. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to the user. The display device 122 may be integrated with the destination device 120, or may be outside the destination device 120, which is configured to be connected to an external display device interface.

[0076] The video encoder 114 and the video decoder 124 may operate according to a video compression standard, such as the High Efficiency Video Codec (HEVC) standard, the Versatile Video Codec (VVC) standard, and other existing and / or future standards.

[0077] Figure 2 is a block diagram showing an example of a video encoder 200 according to some embodiments of the present disclosure, which may be Figure 1 An example of a video encoder 114 in the system 100 is shown.

[0078] Video encoder 200 may be configured to implement any or all of the techniques of this disclosure. Figure 2 In the example of , video encoder 200 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video encoder 200. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0079] In some embodiments, the video encoder 200 may include a partitioning unit 201, a prediction unit 202, a residual generation unit 207, a transformation unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transformation unit 211, a reconstruction unit 212, a buffer 213 and an entropy coding unit 214, and the prediction unit 202 may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205 and an intra-frame prediction unit 206.

[0080] In other examples, the video encoder 200 may include more, fewer, or different functional components. In one example, the prediction unit 202 may include an intra block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode, where at least one reference picture is a picture in which the current video block is located.

[0081] Furthermore, although some components, such as the motion estimation unit 204 and the motion compensation unit 205, may be integrated, for purposes of explanation, these components are described in detail in the following sections. Figure 2 are shown separately in the example.

[0082] The partitioning unit 201 may partition a picture into one or more video blocks. The video encoder 200 and the video decoder 300 may support various video block sizes.

[0083] The mode selection unit 203 may select one of a plurality of codec modes (INTRA codec or INTER codec) based on the error result, for example, and provide the generated INTRA codec block or INTER codec block to the residual generation unit 207 to generate residual block data, and to the reconstruction unit 212 to reconstruct the coding block for use as a reference picture. In some examples, the mode selection unit 203 may select a combination of intra and inter prediction (CIIP) modes, where the prediction is based on an inter prediction signal and an intra prediction signal. In the case of inter prediction, the mode selection unit 203 may also select a resolution for a motion vector (e.g., sub-pixel precision or integer pixel precision) for the block.

[0084] To perform inter-frame prediction on the current video block, the motion estimation unit 204 may generate motion information for the current video block by comparing the current video block with one or more reference frames from the buffer 213. The motion compensation unit 205 may determine a predicted video block for the current video block based on the motion information and decoded samples of pictures from the buffer 213 other than the picture associated with the current video block.

[0085] The motion estimation unit 204 and the motion compensation unit 205 may perform different operations on the current video block, for example, depending on whether the current video block is in an I slice, a P slice, or a B slice. As used herein, an "I slice" may refer to a portion of a picture consisting of macroblocks, all of which are based on macroblocks within the same picture. Furthermore, as used herein, in some aspects, a "P slice" and a "B slice" may refer to a portion of a picture consisting of macroblocks that are independent of macroblocks in the same picture.

[0086] In some examples, the motion estimation unit 204 may perform unidirectional prediction on the current video block, and the motion estimation unit 204 may search the reference pictures of list 0 or list 1 to find the reference video block for the current video block. The motion estimation unit 204 may then generate a reference index and a motion vector, the reference index indicating the reference picture in list 0 or list 1 containing the reference video block, and the motion vector indicating the spatial displacement between the current video block and the reference video block. The motion estimation unit 204 may output the reference index, the prediction direction indicator, and the motion vector as the motion information of the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0087] Alternatively, in other examples, the motion estimation unit 204 may perform bidirectional prediction on the current video block. The motion estimation unit 204 may search the reference pictures in list 0 for a reference video block for the current video block, and may also search the reference pictures in list 1 for another reference video block for the current video block. The motion estimation unit 204 may then generate a plurality of reference indexes and a plurality of motion vectors, the plurality of reference indexes indicating a plurality of reference pictures in list 0 and list 1 containing a plurality of reference video blocks, and the plurality of motion vectors indicating a plurality of spatial displacements between the plurality of reference video blocks and the current video block. The motion estimation unit 204 may output the plurality of reference indexes and the plurality of motion vectors of the current video block as motion information of the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the plurality of reference video blocks indicated by the motion information of the current video block.

[0088] In some examples, motion estimation unit 204 may output a complete set of motion information for use in a decoding process by a decoder. Alternatively, in some embodiments, motion estimation unit 204 may signal motion information of a current video block with reference to motion information of another video block. For example, motion estimation unit 204 may determine that motion information of a current video block is sufficiently similar to motion information of an adjacent video block.

[0089] In one example, motion estimation unit 204 may indicate to video decoder 300 a value in a syntax structure associated with the current video block that indicates that the current video block has the same motion information as another video block.

[0090] In another example, the motion estimation unit 204 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 300 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0091] As discussed above, the video encoder 200 may signal motion vectors in a predictive manner.Two examples of prediction signaling techniques that may be implemented by the video encoder 200 include Advanced Motion Vector Prediction (AMVP) and merge mode signaling.

[0092] The intra prediction unit 206 may perform intra prediction on the current video block. When the intra prediction unit 206 performs intra prediction on the current video block, the intra prediction unit 206 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a prediction video block and various syntax elements.

[0093] The residual generation unit 207 may generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) the prediction video block(s) of the current video block from the current video block. The residual data of the current video block may include residual video blocks corresponding to different sample portions of samples in the current video block.

[0094] In other examples, such as in skip mode, there may be no residual data for the current video block, and the residual generation unit 207 may not perform a subtraction operation.

[0095] Transform processing unit 208 may generate one or more transform coefficient video blocks for a current video block by applying one or more transforms to the residual video block associated with the current video block.

[0096] After transform processing unit 208 generates a transform coefficient video block associated with the current video block, quantization unit 209 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0097] The inverse quantization unit 210 and the inverse transform unit 211 may apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct a residual video block from the transform coefficient video block. The reconstruction unit 212 may add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by the prediction unit 202 to generate a reconstructed video block associated with the current video block for storage in the buffer 213.

[0098] After reconstruction unit 212 reconstructs the video block, a loop filtering operation may be performed to reduce video blocking artifacts in the video block.

[0099] The entropy coding unit 214 may receive data from other functional components of the video encoder 200. When the entropy coding unit 214 receives the data, the entropy coding unit 214 may perform one or more entropy coding operations to generate entropy-coded data and output a bitstream including the entropy-coded data.

[0100] Figure 3 is a block diagram showing an example of a video decoder 300 according to some embodiments of the present disclosure, which may be Figure 1 An example of a video decoder 124 in the system 100 is shown.

[0101] Video decoder 300 may be configured to perform any or all of the techniques of this disclosure. Figure 3 In the example of , video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video decoder 300. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0102] exist Figure 3 In the example of , the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, and a reconstruction unit 306 and a buffer 307. In some examples, the video decoder 300 can perform a decoding process that is generally opposite to the encoding process described with respect to the video encoder 200.

[0103] The entropy decoding unit 301 can retrieve the encoded bitstream. The encoded bitstream may include entropy-encoded video data (e.g., encoded video data blocks). The entropy decoding unit 301 can decode the entropy-encoded video data, and the motion compensation unit 302 can determine motion information from the entropy-encoded video data, the motion information including motion vectors, motion vector precision, reference picture list indexes, and other motion information. The motion compensation unit 302 can determine the information, for example, by performing AMVP and merge mode. AMVP is used, including deriving several best candidates based on data and reference pictures of adjacent PBs. The motion information typically includes horizontal and vertical motion vector displacement values, one or two reference picture indexes, and in the case of prediction areas in B strips, also includes an identification of which reference picture list is associated with each index. As used herein, in some aspects, "merge mode" may refer to deriving motion information from blocks adjacent in space or time.

[0104] The motion compensation unit 302 may generate a motion compensated block, which may perform interpolation based on interpolation filtering. An identifier for the interpolation filter used with sub-pixel precision may be included in a syntax element.

[0105] The motion compensation unit 302 may calculate interpolated values ​​for sub-integer pixels of a reference block using interpolation filtering used by the video encoder 200 during encoding of the video block. The motion compensation unit 302 may determine the interpolation filtering used by the video encoder 200 based on received syntax information, and the motion compensation unit 302 may use the interpolation filtering to generate a prediction block.

[0106] The motion compensation unit 302 may use at least part of the syntax information to determine the size of blocks used to encode (multiple) frames and / or (multiple) slices of the encoded video sequence, partition information describing how each macroblock of a picture of the encoded video sequence is partitioned, a mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-coded block, and other information for decoding the encoded video sequence. As used herein, in some aspects, a "slice" may refer to a data structure that can be decoded independently of other slices of the same picture in terms of entropy coding and decoding, signal prediction, and residual signal reconstruction. A slice may be an entire picture, or it may be a region of a picture.

[0107] The intra prediction unit 303 may use, for example, an intra prediction mode received in the bitstream to form a prediction block from spatially neighboring blocks. The inverse quantization unit 303 inversely quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 303 applies an inverse transform.

[0108] The reconstruction unit 306 may obtain the decoded block, for example, by adding the residual block to the corresponding prediction block generated by the motion compensation unit 202 or the intra prediction unit 303. If necessary, a deblocking filter may also be applied to filter the decoded block to remove blocking artifacts. The decoded video block is then stored in a buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction, and the buffer 307 also produces the decoded video for presentation on a display device.

[0109] Some exemplary embodiments of the present disclosure will be described in detail below. It should be noted that the section titles used in this document are for ease of understanding, rather than limiting the embodiments disclosed in the section to that section. In addition, although some embodiments are described with reference to general video codecs or other specific video codecs, the disclosed technology is also applicable to other video codec technologies. In addition, although some embodiments describe the video coding and decoding steps in detail, it should be understood that the corresponding decoding steps of the cancellation codec will be implemented by the decoder. In addition, the term video processing includes video coding or compression, video decoding or decompression, and video transcoding, in which video pixels are represented from one compression format to another compression format or at different compression bit rates.

[0110] 1. Brief Overview

[0111] The present disclosure relates to video coding and decoding technology, and more specifically, to IBC prediction and related technologies in video coding and decoding, which can be applied to existing HEVC, VVC and other video coding and decoding standards, and can also be applied to future video coding and decoding standards or video codecs.

[0112] 2. Introduction

[0113] Video codec standards have evolved mainly through the development of the well-known ITU-T and ISO / IEC standards. ITU-T produced H.261 and H.263, ISO / IEC produced MPEG-1 and MPEG-4 Visual, and the two organizations jointly produced H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Codec (AVC) and H.265 / HEVC. Since H.262, video codec standards are based on a hybrid video codec structure, which utilizes temporal prediction plus transform codec. In order to explore future video codec technologies beyond HEVC, the Joint Video Exploration Team (JVET) was jointly established by VCEG and MPEG in 2015. JVET meetings are held simultaneously every quarter, and the new video codec standard is officially named Versatile Video Codec (VVC). At the JVET meeting in April 2018, the first version of the VVC Test Model (VTM) was released at that time. The VVC working draft and the test model VTM were subsequently updated after each meeting. The VVC project achieved technical completion (FDIS) at the July 2020 meeting.

[0114] 2.1 Extended Merger Forecast

[0115] In VVC, the merged candidate list includes the following five categories of candidates in order:

[0116] 1) Spatial MVP from spatially adjacent CUs

[0117] 2) Time domain MVP from the same CU

[0118] 3) History-based MVP in FIFO table

[0119] 4) Paired Average MVP

[0120] 5) Zero MV.

[0121] The size of the merge list is signaled in the sequence parameter set header, and the maximum allowed size of the merge list is 6. For each CU code in merge mode, the best merge candidate index is encoded using truncated unary binarization (TU). The first bin of the merge index uses context codec, and the other bins use bypass codec.

[0122] This section provides the export process of various merge candidates. As HEVC does, VVC also supports parallel export of merge candidate lists for all CUs in a certain size area.

[0123] 2.1.1 Spatial Candidate Export

[0124] Figure 4An example diagram 400 shows example positions of spatial merge candidates. The derivation of spatial merge candidates in VVC is the same as in HEVC, except that the positions of the first two merge candidates are swapped. Figure 4 Among the candidates at the positions shown, up to four merge candidates are selected. The order of export is B 1 , A 1 , B 0 , A 0 , and B 2 . Position B2 is considered only when one or more CUs of positions B0, A0, B1, A1 are not available (for example, because it belongs to another slice or tile) or when intra-coding. After adding the candidate of position A1, the addition of the remaining candidates will be subject to a redundancy check, which ensures that candidates with the same motion information are excluded from the list so that the coding efficiency is improved. In order to reduce computational complexity, not all possible candidate pairs are considered in the redundancy check.

[0125] Figure 5 An example diagram 500 is shown of candidate pairs considered for redundancy checking of spatial merging candidates. Figure 5 The arrows in are linked pairs, and a candidate is added to the list only if the corresponding candidates used for redundancy checking do not have the same motion information.

[0126] 2.1.2 Time Domain Candidate Derivation

[0127] In this step, only one candidate is added to the list. In particular, in the derivation of this temporal merge candidate, the scaled motion vector is derived based on the co-located CU belonging to the co-located reference picture. The reference picture list used to derive the co-located CU is explicitly signaled in the slice header. Figure 6 An example motion vector scaling 600 for a temporal merge candidate is shown. The scaled motion vector of the temporal merge candidate is as follows: Figure 6 The scaled motion vector is obtained as shown by the dotted line in , which is scaled from the motion vector of the co-located CU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal merge candidate is set to zero.

[0128] Figure 7 An example diagram 700 is shown of candidate positions for temporal merging candidates C0 and C1. The position of the temporal candidate is selected between candidates C0 and C1, such as Figure 7 As shown. If the CU at position C0 is not available, is intra-coded, or is outside the current CTU row, position C1 is used. Otherwise, position C0 is used in the derivation of the temporal merge candidate.

[0129] 2.1.3 History-based Merge Candidate Export

[0130] History-based MVP (HMVP) merge candidates are added to the merge list after spatial MVP and TMVP. In this method, the motion information of the previous codec block is stored in a table and used as the MVP of the current CU. A table with multiple HMVP candidates is maintained during the encoding / decoding process. The table is reset (cleared) when a new CTU row is encountered. Whenever there is a non-sub-block inter-codec CU, the relevant motion information is added to the last entry of the table as a new HMVP candidate.

[0131] The HMVP table size S is set to 6, which means that up to 6 history-based MVP (HMVP) candidates can be added to the table. When inserting a new motion candidate into the table, a constrained first-in-first-out (FIFO) rule is used, where a redundancy check is first applied to find if the same HMVP exists in the table. If found, the same HMVP is deleted from the table and then all HMVP candidates are moved forward.

[0132] HMVP candidates can be used in the construction process of the merge candidate list. The nearest HMVP candidates in the table are checked in order and inserted after the TMVP candidate in the candidate list. Redundancy check is applied to HMVP candidates for spatial or temporal merge candidates.

[0133] To reduce the number of redundant checking operations, the following simplifications are introduced:

[0134] 1. The number of HMPV candidates for merge list generation is set to (N<=4)?M:(8-N), where N represents the number of existing candidates in the merge list and M represents the number of available HMVP candidates in the table.

[0135] 2. Once the total number of available merge candidates reaches the maximum allowed merge candidate minus 1, the merge candidate list construction process from HMVP is terminated.

[0136] 2.1.4 Pairwise Average Merging Candidate Derivation

[0137] Pairwise average candidates are generated by averaging predefined candidate pairs in the existing merge candidate lists, where the predefined pairs are defined as {(0,1), (0,2), (1,2), (0,3), (1,3), (2,3)}, where the number represents the merge index of the merge candidate list. The averaged motion vector is calculated separately for each reference list. If two motion vectors are available in one list, they are averaged even if they point to different reference pictures; if only one motion vector is available, it is used directly; if no motion vector is available, this list is kept invalid.

[0138] When the merge list is not full after adding pairwise average merge candidates, zero MVPs are inserted to the end until the maximum number of merge candidates is encountered.

[0139] 2.2 New Merge Candidates

[0140] 2.2.1 Interaction between codec tools

[0141] Figure 8 An example diagram 800 is shown for showing VVC spatial neighboring blocks of a current block. In VVC, Figure 8 Five spatial neighboring blocks and one temporal neighboring block are shown for deriving merge candidates.

[0142] It is proposed to derive additional merge candidates from locations that are not adjacent to the current block using the same style as in VVC. To achieve this, for each search round i, a virtual block is generated based on the current block as follows:

[0143] First, the relative position of the virtual block to the current block is calculated using the following formula:

[0144] Offsetx=-i×gridX, Offsety=-i×gridY

[0145] Where Offsetx and Offsetty represent the offset of the upper left corner of the virtual block relative to the lower left corner of the current block, and gridX and gridY are the width and height of the search grid.

[0146] Second, the width and height of the virtual block are calculated using the following formula:

[0147] newWidth=i×2×gridX+currWidth

[0148] newHeight=i×2×gridY+currHeight.

[0149] Where currWidth and currHeight are the width and height of the current block. newWidth and newHeight are the width and height of the new virtual block.

[0150] gridX and gridY are currently set to currWidth and currHeight respectively.

[0151] Fig. 9 A schematic diagram 900 for displaying the relationship between a virtual block and a current block is shown. Fig. 9 The virtual blocks in the ith search round are also shown.

[0152] After generating the virtual block, block A i , Bi , C i , D i and E i can be considered as the VVC spatial neighbors of the virtual block, and their positions are obtained using the same pattern in VVC. Obviously, if the search round i is 0, the virtual block is the current block. In this case, block A i , B i , C i , D i and E i is the spatial neighboring block used in VVC merge mode.

[0153] When constructing the merge candidate list, pruning is performed to ensure that each element in the merge candidate list is unique. The maximum search round is set to 1, which means that five non-adjacent spatial neighboring blocks are used.

[0154] Non-adjacent spatial merging candidates are inserted into the merging list after the temporal merging candidates in the order B1->A1->C1->D1->E1.

[0155] 2.2.2 Non-adjacent spatial candidates

[0156] Fig.10 An example diagram 1000 is shown, which shows spatially adjacent blocks used to derive spatial merge candidates. Non-adjacent spatial merge candidates are inserted into the regular merge candidate list after the TMVP. The pattern of spatial merge candidates is as follows: Fig.10 As shown in the figure, the distance between non-adjacent spatial candidates and the current codec block is based on the width and height of the current codec block. No line buffer limit is applied.

[0157] 2.2.3STMVP

[0158] It is proposed to use three spatial merging candidates and one temporal merging candidate to derive an average candidate as the STMVP candidate.

[0159] The STMVP is inserted before the top left spatial merge candidate.

[0160] The STMVP candidate is pruned along with all previous merge candidates in the merge list.

[0161] For spatial candidates, the first three candidates in the current merge candidate list are used.

[0162] For the temporal candidates, the same positions as the VTM / HEVC co-location positions are used.

[0163] For spatial candidates, the first, second, and third candidates inserted into the current merge candidate list before STMVP are denoted as F, S, and T.

[0164] The temporal candidate having the same position as the VTM / HEVC co-location position used in TMVP is denoted Col.

[0165] The motion vector of the STMVP candidate in prediction direction X (denoted as mvLX) is derived as follows:

[0166] 1) If the reference indices of the four merge candidates are all valid and equal to zero in the prediction direction X (X=0 or 1),

[0167] mvLX=(mvLX_F+mvLX_S+mvLX_T+mvLX_Col)>>2

[0168] 2) If the reference indexes of three of the four merge candidates are valid and equal to zero in the prediction direction X (X=0 or 1),

[0169] mvLX=(mvLX_F×3+mvLX_S×3+mvLX_Col×2)>>3 or

[0170] mvLX=mvLX_F×3+mvLX_T×3+mvLX_Col×2)>>3 or

[0171] mvLX=(mvLX_S×3+mvLX_T×3+mvLX_Col×2)>>3

[0172] 3) If the reference indexes of two of the four merge candidates are valid and equal to zero in the prediction direction X (X=0 or 1),

[0173] mvLX=(mvLX_F+mvLX_Col)>>1 or

[0174] mvLX=(mvLX_S+mvLX_Col)>>1 or

[0175] mvLX=(mvLX_T+mvLX_Col)>>1

[0176] NOTE: If time domain candidates are not available, STMVP mode is turned off.

[0177] 2.2.4 Merge List Size

[0178] If both non-adjacent merge candidates and STMVP merge candidates are considered, the size of the merge list is signaled in the sequence parameter set header, and the maximum allowed size of the merge list is increased (e.g., 8).

[0179] 2.3 Sub-block based temporal motion vector prediction (SbTMVP)

[0180] VVC supports the sub-block-based temporal motion vector prediction (SbTMVP) method. Similar to the temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses the motion field in the co-located picture to improve the motion vector prediction and merge mode of the CU in the current picture. The same co-located picture used by TMVP is used for SbTMVP. SbTMVP differs from TMVP in the following two main aspects:

[0181] -TMVP predicts motion at CU level, but SbTMVP predicts motion at sub-CU level;

[0182] -While TMVP pre-fetches the temporal motion vector from the co-located block in the co-located picture (the co-located block is the bottom-right or center block relative to the current CU), SbTMVP applies motion shifting before pre-fetching the temporal motion information from the co-located picture, where the motion shifting is obtained from the motion vector from one of the spatial neighboring blocks of the current CU.

[0183] Fig.11A and Fig. 11B The SbTMVP process in VVC is shown. Fig.11A An example diagram 1110 of spatial neighboring blocks used to illustrate SbTMVP is shown. SbTMVP predicts the motion vector of a sub-CU within the current CU in two steps. In the first step, check Fig.11A If A1 has a motion vector that uses the co-located picture as its reference picture, then that motion vector is selected as the motion offset to be applied. If no such motion is identified, then the motion offset is set to (0, 0).

[0184] Fig. 11B The sub-CU motion field of the current picture 1120 is shown to be derived by applying the motion shift from the spatial neighbor and scaling the motion information of the corresponding co-located sub-CU from the co-located picture 1122. In the second step, the motion shift identified in step 1 is applied (i.e., added to the coordinates of the current block) to obtain the sub-CU level motion information (motion vector and reference index) from the co-located picture, as shown in Fig. 11B shown. Fig. 11B The example in assumes that the motion shift is set to the motion of block A1. Then, for each sub-CU, the motion information of its corresponding block in the co-located picture (the minimum motion grid covering the center sample) is used to derive the motion information of the sub-CU. After the motion information of the co-located sub-CU is identified, it is converted into the reference index and motion vector of the current sub-CU in a manner similar to the TMVP process of HEVC, where temporal motion scaling is applied to align the reference picture of the temporal motion vector with the reference picture of the current CU.

[0185] In VVC, a combined sub-block-based merge list containing SbTMVP candidates and affine merge candidates is used for signaling based on sub-block merge mode. The SbTMVP mode is enabled / disabled by a sequence parameter set (SPS) flag. If the SbTMVP mode is enabled, the SbTMVP predictor is added as the first entry in the list of sub-block-based merge candidates, followed by the affine merge candidates. The size of the sub-block-based merge list is signaled in the SPS, and the maximum allowed size of the sub-block-based merge list in VVC is 5.

[0186] The sub-CU size used in SbTMVP is fixed to 8x8, and like the affine merge mode, the SbTMVP mode is only applicable to CUs whose width and height are both greater than or equal to 8.

[0187] The encoding logic of the additional SbTMVP merge candidate is the same as that of other merge candidates, that is, for each CU in a P or B slice, an additional RD check is performed to decide whether to use the SbTMVP candidate.

[0188] 2.4 Intra-Block Copy (IBC)

[0189] Intra-block copy (IBC) is a tool adopted in the HEVC extension on SCC. It is well known that it significantly improves the coding efficiency of screen content materials. Since the IBC mode is implemented as a block-level coding mode, block matching (BM) is performed at the encoder to find the best block vector (or motion vector) for each CU. Here, the block vector is used to indicate the displacement from the current block to the reference block, which has been reconstructed within the current picture. The luminance block vector of the IBC-encoded CU has integer precision. The chrominance block vector is also rounded to integer precision. When used in conjunction with AMVR, the IBC mode can switch between 1-pixel and 4-pixel motion vector precision. The IBC-encoded CU is regarded as a third prediction mode in addition to the intra or inter prediction mode. The IBC mode is applicable to CUs with a width and height that are both less than or equal to 64 luminance samples.

[0190] On the encoder side, hash-based motion estimation is performed for IBC. The encoder performs RD checks on blocks with width or height no greater than 16 luma samples. For non-merge mode, a block vector search is first performed using a hash-based search. If the hash search does not return a valid candidate, a local search based on block matching is performed.

[0191] In hash-based search, the hash key matching (32-bit CRC) between the current block and the reference blocks is extended to all allowed block sizes. The hash key calculation for each position in the current picture is based on 4×4 sub-blocks. For current blocks of larger size, the hash key is determined to match the hash key of the reference block when all hash keys in all 4×4 sub-blocks match the hash keys in the corresponding reference positions. If multiple reference blocks are found whose hash keys match the hash key of the current block, the block vector cost of each matching reference is calculated and the one with the minimum cost is selected.

[0192] In the block matching search, the search range is set to cover the previous CTU and the current CTU.

[0193] At the CU level, the IBC mode is signaled through a flag, which can be signaled as IBC AMVP mode or IBC Skip / Merge mode as shown below:

[0194] – IBC skip / merge mode: The merge candidate index is used to indicate which block vector from the list of neighboring candidate IBC codec blocks is used to predict the current block. The merge list includes spatial candidates, HMVP candidates, and pairwise candidates.

[0195] – IBC AMVP mode: Block vector differences are encoded in the same way as motion vector differences. The block vector prediction method uses two candidates as predictors, one from the left neighbor and one from the top neighbor (if IBC encoded). When either neighbor is not available, the default block vector will be used as predictor. A flag is signaled to indicate the block vector predictor index.

[0196] 2.4.1 Simplification of IBC vector prediction

[0197] The BV predictors for merge mode and AMVP mode in IBC will share a common predictor list, which consists of the following elements:

[0198] · Two adjacent spatial locations (such as Figure 4 A0, B0 in

[0199] 5 HMVP entries

[0200] Default is zero vector

[0201] For merge mode, at most the first 6 entries of this list will be used; for AMVP mode, the first 2 entries of this list will be used. And this list meets the shared merge list region requirement (sharing the same list within SMR).

[0202] 2.4.2 IBC Reference Area

[0203] To reduce memory consumption and decoder complexity, IBC in VVC only allows reconstruction of predefined areas, which include the area of ​​the current CTU and some areas of the left CTU. Fig.12 A-12D shows an example diagram for showing the current CTU processing order and the available samples in the current and left CTUs. For example, Fig.12 A-12D shows the reference area of ​​the IBC mode, where each block represents a 64x64 luma sample unit.

[0204] Depending on the location of the current codec CU position within the current CTU, the following applies:

[0205] -like Fig.12 As shown in Figure 1210 of A, if the current block falls into the 64×64 block at the upper left of the current CTU, in addition to the reconstructed samples in the current CTU, the CPR mode can also be used to refer to the reference samples in the lower right 64x64 block of the left CTU. The current block can also use the CPR mode to refer to the reference samples in the lower left 64×64 block of the left CTU and the reference samples in the upper right 64×64 block of the left CTU.

[0206] -like Fig.12 As shown in Figure 1230 of B, if the current block falls into the 64×64 block to the upper right of the current CTU, in addition to the samples that have been reconstructed in the current CTU, if the luminance position (0, 64) has not been reconstructed relative to the current CTU, the current block can also use the CPR mode and refer to the reference samples in the lower left 64×64 block and the lower right 64×64 block of the left CTU; otherwise, the current block can also refer to the reference sample in the lower right 64×64 block of the left CTU.

[0207] -like Fig.12 As shown in Figure 1250 of C, if the current block falls into the 64×64 block at the lower left of the current CTU, in addition to the samples that have been reconstructed in the current CTU, if the luma position (64, 0) has not been reconstructed relative to the current CTU, the current block can also use the CPR mode and reference samples in the upper right 64×64 block and the lower right 64×64 block of the left CTU. Otherwise, the current block can also use the CPR mode and reference samples in the lower right 64×64 block of the left CTU.

[0208] -like Fig.12 As shown in Figure 1270 of D, if the current block falls into the 64×64 block at the bottom right of the current CTU, only the CPR mode can be used to refer to the samples that have been reconstructed in the current CTU.

[0209] This restriction allows the IBC mode to be implemented using local on-chip memory for hardware implementation.

[0210] 2.4.3 Interaction between IBC and other encoding and decoding tools

[0211] The interaction between IBC mode and other inter-frame codec tools in VVC, such as pairwise merge candidates, history-based motion vector predictor (HMVP), combined intra / frame prediction modes (CIIP), merge mode with motion vector difference (MMVD), and geometric partition model (GPM), is as follows:

[0212] -IBC can be used with pairwise merge candidates and HMVP. A new pairwise IBC merge candidate can be generated by averaging two IBC merge candidates. For HMVP, IBC motions are inserted into the history buffer for future reference.

[0213] - IBC cannot be used in conjunction with the following interframe tools: Affine Motion, CIIP, MMVD, and GPM.

[0214] - When using DUAL_TREE partitioning, IBC is not allowed for chroma codec blocks.

[0215] Unlike in the HEVC screen content codec extension, the current picture is no longer included as one of the reference pictures in reference picture list 0 for IBC prediction. The motion vector derivation process for IBC mode excludes all neighboring blocks in inter mode and vice versa. The following IBC design aspects are applied:

[0216] - IBC shares the same process as regular MV merging, including pairwise merging candidates and history-based motion predictor, but TMVP and zero vectors are not allowed as they are invalid for IBC mode.

[0217] - Separate HMVP buffers (5 candidates each) for legacy MV and IBC.

[0218] - Block vector constraints are implemented as bitstream consistency constraints, where the encoder needs to ensure that there are no invalid vectors in the bitstream and that merges should not be used if the merge candidate is invalid (out of range or 0). This bitstream consistency constraint is represented by a virtual buffer as described below.

[0219] - For deblocking, IBC is handled as an inter-mode.

[0220] - If the current block is coded using IBC prediction mode, AMVR does not use quarter pixels; instead, AMVR is signaled to only indicate whether the MV is an inter pixel or 4 integer pixels.

[0221] - The number of IBC merge candidates may be signaled in the slice header separately from the number of normal, sub-block and geometric merge candidates.

[0222] The concept of virtual buffer is used to describe the allowable reference area of ​​IBC prediction mode and valid block vector. Let CTU size be ctbSize, and the width of virtual buffer ibcBuf is wIbcBuf=128x128 / ctbSize, and the height is hIbcBuf=ctbSize. For example, for CTU size of 128×128, the size of ibcBuf is also 128×128; for CTU size of 64×64, the size of ibcBuf is 256×64; for CTU size of 32×32, the size of ibcBuf is 512×32.

[0223] The size of each dimension of VPDU is min(ctbSize, 64), Wv = min(ctbSize, 64).

[0224] The virtual IBC buffer ibcBuf is maintained as follows.

[0225] - At the start of decoding each CTU line, flush the entire ibcBuf with an invalid value of -1.

[0226] -When starting to decode the VPDU (xVPDU, yVPDU) relative to the upper left corner of the picture, set ibcBuf[x][y] = -1, where x = xVPDU% wIbcBuf, ..., xVPDU% wIbcBuf + Wv-1; y = yVPDU% ctbSize, ..., yVPDU% ctbSize + Wv-1.

[0227] -After decoding, the CU contains the (x,y) relative to the upper left corner of the picture, set

[0228] –ibcBuf[x%wIbcBuf][y%ctbSize]=recSample[x][y]

[0229] For a block covering coordinates (x, y), it is valid if the following holds for the block vector bv = (bv[0], bv[1]); otherwise, it is invalid:

[0230] ibcBuf[(x+bv[0])%wIbcBuf][(y+bv[1])%ctbSize] should not be equal to -1.

[0231] 2.4.4IBC Virtual Buffer Test

[0232] The luma block vector bvL (a luma block vector with 1 / 16 fractional sample accuracy) should obey the following constraints:

[0233] -CtbSizeY is greater than or equal to ((yCb+(bvL[1]>>>>4))&(CtbSizeY-1))+cbHeight.

[0234] -For x=xCb..xCb+cbWidth-1 and y=yCb..yCb+cbHeight-1, it is -1, IbcVir-Buf[0][(x+(bvL[0]>>4))&(Ib-cBufWidthY-1)][(y+(bvL[1]>>4))&(CtbSizeY-1)] shall not be equal.

[0235] Otherwise, bvL is considered as an invalid bv.

[0236] Samples are processed in CTB units. The width and height of each luma CTB are arrays of size CtbSizeY (in samples).

[0237] -(xCb,yCb) is the luminance position of the top left sample of the current luminance codec block relative to the top left luminance sample of the current picture,

[0238] -cbWidth specifies the width of the current codec block in luma samples.

[0239] -cbHeight specifies the height of the current codec block in luma samples.

[0240] 2.5 Adaptive Merge Candidate Sorting Based on Template Matching

[0241] In order to improve the encoding and decoding efficiency, after building the merge candidate list, the order of each merge candidate is adjusted according to the template matching cost. The merge candidates are arranged in the list according to the template matching cost in ascending order. It operates in the form of subgroups.

[0242] Fig.13 An example diagram 1300 of neighboring samples used to calculate SAD is shown. The template matching cost is measured by the SAD (sum of absolute differences) between the neighboring samples of the current CU in the current picture 1810 and its corresponding reference samples. If the merge candidate includes bi-directional prediction motion information, the corresponding reference sample is the average of the corresponding reference sample in reference list 0 1320 and the corresponding reference sample in reference list 1 1330, as shown in FIG. Fig.13 shown. Fig.14 An example diagram 1400 is shown for displaying neighboring samples for calculating sub-CU level motion information. If the merge candidate contains sub-CU level motion information of the current picture 1410, the corresponding reference sample is composed of the neighboring samples of the corresponding reference sub-block in the reference picture 1420, such as Fig.14 shown.

[0243] Fig.15 FIG. 1 shows a sorting process of sorting the original merge candidate list 1510 to obtain the updated merge candidate list 1520. The sorting process is performed in the form of subgroups, such as Fig.15 As shown in Figure 1, the first three merge candidates are sorted together. The last three merge candidates are sorted together.

[0244] The template size (width of the left template or height of the upper template) is 1. The subgroup size is 3.

[0245] 2.6 Adaptive Merge Candidate List

[0246] Assume that the number of merging candidates is 8. The first 5 merging candidates are used as the first subgroup, and the following 3 merging candidates are used as the second subgroup (ie, the last subgroup).

[0247] Fig.16 An example diagram for illustrating the reordering process in an encoder is shown. For the encoder, after the merge candidate list is constructed in block 1602, some merge candidates are adaptively reordered in ascending order of merge candidate cost, such as Fig.16 shown.

[0248] More specifically, in box 1604 , the template matching costs of the merge candidates in all subgroups except the last subgroup are calculated; then in box 1606 , the merge candidates in its own subgroup except the last subgroup are reordered; finally, in box 1608 , a final merge candidate list is obtained.

[0249] Fig.17 An example diagram 1700 is shown for illustrating the reordering process in a decoder. For the decoder, after constructing the merge candidate list, some / no merge candidates are adaptively reordered in ascending order of merge candidate cost, such as Fig.17 As shown. Fig.17 In , the subgroup where the selected (signal) merging candidate is located is called the selected subgroup.

[0250] More specifically, if the merge candidate selected in block 1702 is located in the last subgroup, then in block 1704, the merge candidate list construction process is terminated after deriving the selected merge candidate, and in block 1706, no reordering is performed and the merge candidate list is not changed; otherwise, the execution process is as follows:

[0251] In box 1708, after all merge candidates in the selected subgroup are derived, the merge candidate list construction process is terminated; in box 1710, the template matching cost of the merge candidates in the selected subgroup is calculated; in box 1712, the merge candidates in the selected subgroup are reordered; finally, in box 1714, a new merge candidate list is obtained.

[0252] For encoder and decoder:

[0253] The template matching cost is derived as a function of T and RT, where T is the set of samples in the template and RT is the set of reference samples for the template.

[0254] When deriving the reference samples of the template of the merge candidate, the motion vector of the merge candidate is rounded to integer pixel precision. An 8-tap or 12-tap luma interpolation filter may also be used.

[0255] The reference samples for the template (RT) for bidirectional prediction are obtained by adding the following to the reference list 0 (RT 0 ) in the reference sample and reference list 1 (RT 1 ) is derived by weighted averaging the reference samples of the template.

[0256] RT=((8-w)*RT 0 +w*RT 1 +4)>>3

[0257] The weight of the reference template in reference list 0 (8-w) and the weight of the reference template in reference list 1 (w) are determined by the BCW index of the merge candidate. BCW indexes equal to {0, 1, 2, 3, 4} correspond to w equal to {-2, 3, 4, 5, 10} respectively.

[0258] If the local illumination compensation (LIC) flag of the merge candidate is true, the reference samples of the template are derived using the LIC method.

[0259] The template matching cost is calculated based on the sum of absolute differences (SAD) between T and RT.

[0260] The template size is 1. This means that the width of the left template and / or the height of the top template is 1.

[0261] If the codec mode is MMVD, the merge candidates used to derive the base merge candidate are not reordered.

[0262] If the coding mode is GPM, the merge candidates used to derive the unidirectional prediction candidate list are not reordered.

[0263] 2.7 Template Matching (TM)

[0264] Template Matching (TM) is a decoder-side MV derivation method used to refine the motion information of the current CU by finding the closest match between a template in the current picture (i.e., the top and / or left neighboring blocks of the current CU) and a block in a reference picture (i.e., the same size as the template). Fig.18 An example diagram 1800 is shown for illustrating template matching performed on a search area around an initial MV. Fig.18 As shown, in the [-8, +8] pixel search range, a better MV is searched around the initial motion of the current CU. This paper adopts template matching and makes two modifications: the search step size is determined based on the AMVR mode, and the TM can cascade the bilateral matching process in the merge mode.

[0265] In AMVP mode, MVP candidates are determined based on template matching error to pick the one that achieves the minimum difference between the current block template and the reference block template, and then TM performs MV refinement only on that specific MVP candidate. TM refines that MVP candidate by using an iterative diamond search, starting with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode) within a search range of [-8, +8] pixels. The AMVP candidate can be further refined by using a cross search with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode), followed by half-pixel and quarter-pixel, respectively, according to the AMVR mode specified in Table 1. This search process ensures that the MVP candidate still maintains the same MV accuracy as indicated by the AMVR mode after the TM process.

[0266] Table 1. Search patterns of AMVR and merge patterns with AMVR

[0267]

[0268] In merge mode, a similar search method is applied to the merge candidates indicated by the merge index. As shown in Table 1, TM can be performed all the way down to 1 / 8 pixel MVD accuracy, or skip accuracy beyond half-pixel MVD accuracy, depending on whether an alternative interpolation filter is used based on the merged motion information (used when AMVR is half-pixel mode). In addition, when TM mode is enabled, template matching can work as an independent process between block-based and sub-block-based bilateral matching (BM) methods or as an additional MV refinement process, depending on whether BM can be enabled based on its enabling condition check.

[0269] At the encoder side, the TM merge mode will perform MV refinement on each merge candidate.

[0270] 2.8 Intra-frame Template Matching for IBC (TM_IBC)

[0271] Template Matching Prediction (TMP) is a special intra prediction mode that copies the best prediction block from the reconstructed part of the current frame, whose L-shaped template is matched with the current template. Fig.19An example diagram 1900 is shown for illustrating template matching prediction. For a predefined search range, the encoder searches for the template that is most similar to the current template in the reconstructed part of the current frame and takes the corresponding block as the predicted block. The encoder then signals the use of this mode and performs the inverse operation on the decoder side.

[0272] Template matching prediction is a codec tool that is mainly applicable to screen content codecs. A prediction signal is generated on the decoder side by matching the L-shaped causal neighbors of the current block with another block in a predefined search area. Fig. 20 Shown in. Fig. 20 An example diagram 2000 is shown for displaying the used template matching search area. Specifically, the search range is divided into 3 areas:

[0273] R1: In the current CTU

[0274] R2: Upper left outside the current CTU

[0275] R3: Above the current CTU

[0276] R4: On the left side of the current CTU

[0277] In each region, the decoder searches for the template with the least SAD relative to the current region and uses its corresponding block as the prediction block.

[0278] The sizes of all regions (SearchRange_w, SearchRange_h) are set proportionally to the block size (BlkW, BlkH) so that there is a fixed number of SAD comparisons per pixel. That is:

[0279] SearchRange_w=a*BlkW

[0280] SearchRange_h=a*BlkH

[0281] where "a" is a constant that controls the gain / complexity tradeoff.

[0282] 2.9 Template-based Intra-mode Derivation Using MPM

[0283] The TIMD mode is derived from the MPM using a neighboring template. The TIMD mode is used as an additional intra prediction method for the CU. Fig.21 An example diagram 2100 is shown for displaying the templates used in TIMD and their reference samples. Fig.21As shown, the reference samples of the template for each candidate mode are used to generate the prediction samples of the template. The cost is calculated as the sum of absolute transform differences (SATD) between the prediction samples and the reconstructed samples of the template. The intra prediction mode with the minimum cost is selected as the TIMD mode and used for intra prediction of the CU.

[0284] 2.9.1 TIMD Mode Export

[0285] For each intra prediction mode in the MPM, the SATD between the prediction and reconstructed samples of the template is calculated. The intra prediction mode with the smallest SATD is selected as the TIMD mode for intra prediction of the current CU. The TIMD mode export supports position-dependent intra prediction combination (PDPC) and gradient PDPC.

[0286] 2.9.2TIMD Signaling

[0287] A flag is signaled in the sequence parameter set (SPS) to enable / disable TIMD. When the flag is true, a CU level flag is signaled to indicate whether TIMD is used for the CU. The TIMD flag is signaled immediately after the MIP flag. If the TIMD flag is equal to true, all other syntax elements related to luma intra prediction mode are skipped.

[0288] 2.9.3 Interaction with new codec tools in ECM-1.0

[0289] When the DIMD flag or the MIP flag is equal to true, the TIMD flag is not signaled and is set equal to false.

[0290] It is allowed to combine TIMD with ISP and MRL. When TIMD is combined with ISP or MRL and the TIMD flag is equal to true, the derived TIMD mode is used as the intra prediction mode for ISP or MRL.

[0291] When the secondary MPM is enabled, both the primary and secondary MPMs are used to derive the TIMD mode.

[0292] The 6-tap interpolation filter is not used in the export of TIMD mode.

[0293] 2.9.4 Modification of MPM table structure in TIMD mode export

[0294] During the construction of the MPM list, when neighboring blocks are inter-encoded, their intra prediction modes are derived as planes. In order to improve the accuracy of the MPM list, when neighboring blocks are inter-encoded, the propagated intra prediction modes are derived using motion vectors and reference pictures and used in the construction of the MPM list.

[0295] 2.10 Adaptive Merge Candidate List

[0296] In the following, a template is a set of reconstructed samples that are adjacent or non-adjacent to a current block. The reference samples of the template are derived according to the same motion information of the current block. For example, the reference samples of the template are a mapping of the template that depends on the motion information. In this case, the reference samples of the template are located by the motion vector of the motion information in the reference picture indicated by the reference index of the motion information. Fig. 22 An example diagram 2200 for displaying templates and reference samples of the templates is shown, where RT represents a reference sample in a reference picture 2220 of a template T in a current picture 2210 .

[0297] When the merge candidate uses bidirectional prediction, the reference sample of the template of the merge candidate is represented by RT, and RT can be derived from the RT of the reference picture in reference picture list 0. 0 and RT derived from the reference picture in reference picture list 1 1 In one example, RT 0 A reference sample set on a reference picture of a current block indicated by a reference index of a merge candidate that references a reference picture in reference list 0, wherein the MV of the merge candidate references reference list 0). In one example, RT 1 A reference sample set on a reference picture of a current block indicated by a reference index of a merge candidate that references a reference picture in reference list 1, wherein the MV of the merge candidate references reference list 1). Fig.23 A schematic diagram showing reference samples of templates in a current picture 2310 and templates in reference list 0 2320 and reference list 1 2330 is shown.

[0298] In one example, the reference samples (RT) of the templates used for bidirectional prediction are obtained by referencing the reference samples (RT) of the templates in reference list 0. 0 ) and the reference sample (RT 1 ) by taking an equally weighted average. An example is as follows:

[0299] RT=(RT 0 +RT 1 +1)>>1

[0300] In one example, the reference sample (RT) of the template used for bidirectional prediction bi-pred ) is obtained by comparing the reference sample (RT 0 ) and the reference sample (RT 1 ) is derived by taking a weighted average. An example is as follows:

[0301] RT=((2 N-w)*RT 0 +w*RT 1 +2 N-1 )>>N, for example, N=3.

[0302] In one example, the weight of the reference templates in reference list 0 such as (8-w) and the weight of the reference templates in reference list 1 such as (w) may be determined by the BCW index of the merge candidate.

[0303] The merge candidates can be divided into several groups according to some criteria. Each group is called a subgroup. For example, adjacent spatial and temporal merge candidates can be used as the first subgroup, and the remaining merge candidates can be used as the second subgroup; in another example, the first N (N≥2) merge candidates can also be used as the first subgroup, the following M (M≥2) merge candidates can be used as the second subgroup, and the remaining merge candidates can be used as the third subgroup. Note that the proposed method can be applied to the candidate list construction process of merging inter-frame codec blocks (e.g., translational motion), affine codec blocks; or other motion candidate list construction processes (e.g., AMVP list, IBC AMVP list, and IBC merge list).

[0304] W and H are the width and height of the current block (e.g., luminance block). The following description takes the merge candidate list construction process as an example:

[0305] 1. The merging candidates can be adaptively rearranged in the final merging candidate list according to one or some criteria.

[0306] a. In one example, firstly, part or all of the process of building the current merge candidate list is called, and then the candidates in the list are reordered.

[0307] Alternatively, the candidates in the first subgroup may be reordered and they should be added before those in the second subgroup, where the first subgroup is added before the second subgroup.

[0308] (i) In one example, multiple merge candidates of a first category may be first derived and then reordered within the first category; then merge candidates from the second category may be determined based on the reordered candidates in the first category (eg, how pruning is applied).

[0309] ii Alternatively, the first merge candidate in the first category may be compared with the second merge candidate in the second category to determine the order of the first or second merge candidate in the final merge candidate list.

[0310] b. In one example, merge candidates may be adaptively rearranged before retrieval.

[0311] i. In one example, the process of adaptively arranging merge candidates may be processed before obtaining merge candidates to be used in the motion compensation process.

[0312] c. In one example, if the width of the current block is greater than the height of the current block, the above candidate is added before the left candidate.

[0313] d. In one example, if the width of the current block is smaller than the height of the current block, the above candidate is added after the left candidate.

[0314] e. Whether to adaptively rearrange merge candidates may depend on the selected merge candidate or the selected merge candidate index.

[0315] i. In one example, if the selected merge candidate is in the last subgroup, the merge candidates are not adaptively re-arranged.

[0316] f. In one example, costs are assigned to the merge candidates, and the merge candidates are adaptively reordered in ascending order of their costs.

[0317] i. In one example, the cost of merging candidates may be a template matching cost.

[0318] ii. In one example, the template is a set of reconstructed samples that are adjacent or non-adjacent to the current block.

[0319] iii. The template matching cost is derived as a function of T and RT, where T is the set of samples in the template and RT is the set of reference samples for the template.

[0320] (i) How to obtain the reference sample of the template of the merge candidate may depend on the motion information of the merge candidate

[0321] a) In one example, when deriving the reference samples of the template, the motion vector of the merge candidate is rounded to integer pixel precision, where the integer motion vector may be its nearest integer motion vector.

[0322] b) In one example, when deriving the reference samples of the template, an N-tap interpolation filter is used to obtain the reference samples of the template at the sub-pixel position. For example, N can be 2, 4, 6 or 8.

[0323] c) In one example, when deriving reference samples for a template, the motion vectors of the merge candidates may be scaled to a given reference picture (eg, for each reference picture list, if available).

[0324] d) For example, on the reference picture of the current block indicated by the reference index of the merge candidate, obtain the reference sample of the template of the merge candidate, the reference picture having Fig. 22The merge candidate MV or modified MV shown (eg, according to clauses a)-b)).

[0325] e) For example, when the merge candidate utilizes bidirectional prediction, the reference sample of the template of the merge candidate is represented by RT, and RT can be derived from the RT of the reference picture in reference picture list 0. 0 and RT derived from the reference picture in reference picture list 1 1 And generated.

[0326] [1] In one example, RT 0 A reference sample set on a reference picture of a current block indicated by a reference index of a merge candidate that references a reference picture in reference list 0, wherein the MV of the merge candidate references reference list 0),

[0327] [2] In one example, RT 1 A reference sample set on a reference picture of a current block indicated by a reference index of a merge candidate that references a reference picture in reference list 1, where the MV of the merge candidate references reference list 1).

[0328] [3] For example Fig.23 shown.

[0329] f) In one example, the reference samples (RT) of the templates used for bidirectional prediction are obtained by comparing the reference samples (RT) of the templates in reference list 0. 0 ) and the reference sample (RT 1 ) are derived by taking an equally weighted average of . An example is as follows:

[0330] RT=(RT 0 +RT 1 +1)>>1

[0331] g) In one example, the reference sample (RT) of the template used for bidirectional prediction bi-pred ) is obtained by comparing the reference sample (RT 0 ) and the reference sample (RT 1 ) is derived by taking a weighted average. An example is as follows:

[0332] RT=((2 N -w)*RT 0 +w*RT 1 +2 N-1 )>N, for example, N=3.

[0333] h) In one example, the weights of reference templates in reference list 0 such as (8-w) and the weights of reference templates in reference list 1 such as (w) may be determined by the BCW index of the merge candidate.

[0334] [1] In one example, the BCW index is equal to 0 and w is set equal to -2.

[0335] [2] In one example, the BCW index is equal to 1 and w is set equal to 3.

[0336] [3] In one example, the BCW index is equal to 2 and w is set equal to 4.

[0337] [4] In one example, the BCW index is equal to 3 and w is set equal to 5.

[0338] [5] In one example, the BCW index is equal to 4 and w is set equal to 10.

[0339] i) In one example, if the local illumination compensation (LIC) flag of the merge candidate is true, the reference samples of the template are derived using the LIC method.

[0340] (ii) The cost can be calculated based on the sum of absolute differences (SAD) of T and RT.

[0341] a) Alternatively, the cost may be calculated based on the sum of absolute transformed differences (SATD) of T and RT.

[0342] b) Alternatively, the cost can be calculated based on the sum of squared differences (SSD) of T and RT.

[0343] c) Alternatively, the cost may be calculated based on weighted SAD / weighted SATD / weighted SSD.

[0344] (iii) In addition to the SAD calculated in (ii), the cost may also consider the continuity (Boundary_SAD) between RT and reconstructed samples adjacent or non-adjacent to T. For example, reconstructed samples to the left and / or above adjacent or non-adjacent to T are considered.

[0345] a) In one example, the cost may be calculated based on SAD and Boundary_SAD.

[0346] [1] In one example, the cost may be calculated as (SAD + w*Bound-ary_SAD). w may be predefined, signaled, or derived from decoded information.

[0347] 2. Whether and / or how to reorder merge candidates may depend on the category of the merge candidates.

[0348] aIn one example, only adjacent spatial and temporal merging candidates may be reordered.

[0349] b. In one example, only adjacent spatial merging candidates, STMVP merging candidates and temporal merging candidates may be reordered.

[0350] c. In one example, only adjacent spatial, STMVP, temporal, and non-adjacent spatial merging candidates may be reordered.

[0351] d. In one example, only adjacent spatial, STMVP, temporal, non-adjacent spatial, and HMVP merging candidates may be reordered.

[0352] e. In one example, only adjacent spatial, STMVP, temporal, non-adjacent spatial HMVP and pairwise average merging candidates may be reordered.

[0353] f. In one example, only adjacent spatial, temporal, HMVP and pairwise average merging candidates may be reordered.

[0354] g. In one example, only adjacent spatial, temporal and HMVP merging candidates may be reordered.

[0355] h. In one example, only adjacent spatial merging candidates may be reordered.

[0356] i. In one example, only the first subgroup can be reordered.

[0357] j. In one example, the last subgroup cannot be reordered.

[0358] k. In one example, only the top N merge candidates may be reordered.

[0359] i. In one example, N is set equal to 5.

[0360] 1. In one example, for candidates that are not reordered, they will be arranged in the merge candidate list according to the original order.

[0361] m. In one example, candidates that are not reordered may be placed after candidates to be reordered.

[0362] n. In one example, candidates that are not reordered may be placed before candidates to be reordered.

[0363] o. In one example, some combinations of the above items (a-k) can be reordered.

[0364] p. Different subgroups can be reordered individually.

[0365] q. Two candidates in different subgroups cannot be compared and / or re-ranked.

[0366] r. If the first subgroup is before the second subgroup, then after reordering, the first candidate in the first subgroup needs to be placed before the second candidate in the second subgroup.

[0367] 3. Whether and / or how to reorder merge candidates may depend on the codec mode.

[0368] a. In one example, if the coding mode is the normal merge mode, the merge candidates may be reordered.

[0369] b. In one example, if the codec mode is MMVD, the merge candidates used to derive the base merge candidate are not reordered.

[0370] i. Alternatively, the reordering method may be different for MMVD mode and other merging modes.

[0371] c. In one example, if the coding mode is CIIP, the merge candidate for combining with intra prediction is based on the reordered merge candidates.

[0372] i. Alternatively, the reordering method may be different for CIIP mode and other merge modes.

[0373] d. In one example, if the coding mode is GPM, the merge candidates used to derive the unidirectional prediction candidate list are not reordered.

[0374] i. Alternatively, the reordering method may be different for GPM mode and other merge modes.

[0375] e. In one example, if the coding mode is the triangulation mode, the merge candidates used to derive the unidirectional prediction candidate list are not reordered.

[0376] i. Alternatively, the reordering method may be different for triangle mode and other merging modes.

[0377] f. In one example, if the coding mode is a sub-block based merge mode, candidate merges based on part or all of the sub-blocks are reordered.

[0378] i. Alternatively, the reordering method may be different for the sub-block based merge mode and other merge modes.

[0379] ii. In one example, merging candidates based on unidirectional prediction sub-blocks are not reordered.

[0380] iii. In one example, SbTMVP candidates are not reordered.

[0381] iv. In one example, the constructed affine candidates are not reordered.

[0382] v. In one example, zero-filled affine candidates are not reordered.

[0383] 4. Whether and / or how to reorder merging candidates may depend on the available number of neighboring spatial and / or STMVP and / or temporal merging candidates.

[0384] 5. Whether the merge candidates need to be reordered may depend on decoded information (eg, the width and / or height of the CU).

[0385] a. In one example, if the height is greater than or equal to M, the width is greater than or equal to N, and the width*height is greater than or equal to R, the merge candidates may be reordered.

[0386] i. In one example, M, N, and R are set equal to 8, 8, and 128.

[0387] ii. In one example, M, N, and R are set equal to 16, 16, and 512.

[0388] b. In one example, if the height is greater than or equal to M and the width is greater than or equal to N, the merge candidates may be reordered.

[0389] i. In one example, M and N are set equal to 8 and 8.

[0390] ii. In one example, M and N are set equal to 16 and 16.

[0391] 6. The subgroup size can be adaptive.

[0392] a. In one example, the subgroup size is determined according to the available number of neighboring spatial and / or STMVP and / or temporal merging candidates denoted as N.

[0393] i. In one example, if N is less than M and greater than Q, the subgroup size is set to N;

[0394] ii. In one example, if N is less than or equal to Q, no reordering is performed;

[0395] iii. In one example, if N is greater than or equal to M, the subgroup size is set to M.

[0396] iv. In one example, M and Q are set equal to 5 and 1 respectively.

[0397] (i) Alternatively, M and / or Q may be predefined, signaled, or derived from decoded information.

[0398] b. In one example, the subgroup size is decided according to the available number of neighboring spatial and temporal merging candidates denoted as N.

[0399] i. In one example, if N is less than M and greater than Q, the subgroup size is set to N;

[0400] ii. In one example, if N is less than or equal to Q, no reordering is performed;

[0401] iii. In one example, if N is greater than or equal to M, the subgroup size is set to M.

[0402] iv. In one example, M and Q are set equal to 5 and 1 respectively.

[0403] 7. The template shape can be adaptive.

[0404] a. In one example, the template may only include neighboring samples left for the current block.

[0405] b. In one example, the template may only include neighboring samples above the current block.

[0406] c. In one example, the template shape is selected according to the CU shape.

[0407] d. In one example, the width of the left template is selected according to the CU height.

[0408] i. For example, if H<=M, the left template size is w1xH; otherwise, the left template size is w2xH.

[0409] e. In one example, M, w1, and w2 are set equal to 8, 1, and 2, respectively.

[0410] f. In one example, the height of the upper template is selected according to the CU width.

[0411] i. For example, if W<=N, the size of the upper template is Wxh1; otherwise, the size of the upper template is Wxh2.

[0412] (i) In one example, N, h1, and h2 are set equal to 8, 1, and 2, respectively.

[0413] g. In one example, the width of the left template is selected according to the CU width.

[0414] i. For example, if W<=N, the size of the left template is w1xH; otherwise, the size of the left template is w2xH.

[0415] (i) In one example, N, w1, and w2 are set equal to 8, 1, and 2, respectively.

[0416] h. In one example, the height of the upper template is selected according to the CU height.

[0417] i. For example, if H<=M, the size of the upper template is Wxh1; otherwise, the size of the upper template is Wxh2.

[0418] (i) In one example, M, h1, and h2 are set equal to 8, 1, and 2, respectively.

[0419] i. In one example, samples of the template and reference samples of the template samples may be subsampled or downsampled before being used to calculate the cost.

[0420] i. Whether and / or how subsampling is performed may depend on the CU size.

[0421] ii. In one example, no subsampling is performed on the short edges of the CU.

[0422] 8. In the above examples, the merge candidate is one of the candidates included in the final merge candidate list (eg, after pruning).

[0423] a. Alternatively, the merge candidate is one derived from a given spatial or temporal block or HMVP list, or derived in other ways, even though it may not be included in the final merge candidate list.

[0424] 9. Templates can include samples of specific color components.

[0425] a. In one example, the template includes samples of the luma component only.

[0426] 10. Whether to apply adaptive merge candidate list reordering may depend on a message transmitted by signal in VPS / SPS / PPS / sequence header / picture header / slice header / CTU / CU. It may also be based on the region of the signal. For example, the picture is divided into groups of CTU / CU evenly or unevenly, and a flag is encoded and decoded for each group to indicate whether to apply merge candidate list reordering.

[0427] 2.11 Adaptive Motion Candidate List

[0428] 1. The motion candidates in the motion candidate list of the block may be adaptively rearranged according to one or some criteria to derive a reordered motion candidate list, and the block is encoded / decoded according to the reordered motion candidate list.

[0429] a. The motion candidates in the motion candidate list of blocks that are not in the regular merge candidate list may be adaptively rearranged according to one or some criteria to derive a reordered motion candidate list.

[0430] b. In one example, whether and / or how to reorder motion candidates may depend on the codec mode (e.g., affine merge, affine AMVP, regular merge, regular AMVP, GPM, TPM, MMVD, TM merge, CIIP, GMVD, affine MMVD).

[0431] c. In one example, whether and / or how to reorder motion candidates may depend on the category of the motion candidates (eg, spatial, temporal, STMVP, HMVP, pairwise, SbTMVP, constructive affine, inherited affine).

[0432] d. In one example, the motion candidate list may be an AMVP candidate list.

[0433] e. In one example, the motion candidate list may be a merge candidate list.

[0434] f. In one example, the motion candidate list may be an affine merge candidate list.

[0435] g. In one example, the motion candidate list may be a sub-block based merge candidate list.

[0436] h. In one example, the motion candidate list may be a GPM merge candidate list.

[0437] i. In one example, the motion candidate list may be a TPM merge candidate list.

[0438] j. In one example, the motion candidate list may be a TM merge candidate list.

[0439] k. In one example, the motion candidate list may be a candidate list of an MMVD codec block.

[0440] 1. In one example, the motion candidate list may be a candidate list of a DMVR codec block.

[0441] 2. How to adaptively rearrange the motion candidates in the motion candidate list may depend on decoded information, for example, the category of the motion candidate, the category of the motion candidate list, and the encoding and decoding tool.

[0442] a. In one example, for different motion candidate lists, different criteria may be used to re-rank the motion candidate lists.

[0443] i. In one example, the criteria may include how to select a template.

[0444] ii. In one example, the criteria may include how to calculate the template cost.

[0445] iii. In one example, the criteria may include how many candidates in the candidate list and / or how many subgroups need to be reordered.

[0446] b. In one example, motion candidates in a motion candidate list are first adaptively rearranged to construct a fully rearranged candidate list or a partially rearranged candidate list, and then at least one motion candidate indicated by at least one index is retrieved from the rearranged candidate list to derive final motion information to be used by the current block.

[0447] c. In one example, the motion candidates before refinement are first adaptively rearranged (e.g., using TM for TM-encoded blocks; adding MVD for MMVD-encoded blocks) to construct a fully rearranged candidate list or a partially rearranged candidate list. Then, at least one motion candidate indicated by at least one index is retrieved from the rearranged candidate list, and refinement (e.g., using TM for TM-encoded blocks; adding MVD for MMVD-encoded blocks) is applied to the retrieved motion candidate to derive final motion information of the current block.

[0448] d. In one example, refinement is applied to at least one motion candidate in the motion candidate list (e.g., using TM for TM codec blocks; adding MVD for MMVD codec blocks), and then they are adaptively rearranged to construct a fully rearranged candidate list or a partially rearranged candidate list, and then at least one motion candidate indicated by at least one index is retrieved from the rearranged candidate list to derive final motion information without any further refinement for the current block.

[0449] 3. In one example, new MERGE / AMVP motion candidates may be generated based on the candidate re-ranking.

[0450] i. For example, the candidate L0 motion and L1 motion may be reordered separately.

[0451] ii. For example, a new bi-prediction merging candidate may be constructed by combining one from the reordered L0 motion and another from the reordered L1 motion.

[0452] iii. For example, new uni-directional prediction merging candidates can be generated by reordering L0 or L1 motion.

[0453] 2.12 Adaptive Motion Candidate List

[0454] For sub-block motion prediction, if the sub-block size is Wsub*Hsub, the height of the upper template is Ht, and the width of the left template is Wt, then the upper template can be regarded as a composition of several sub-templates with a size of Wsub*Ht, and the left template can be regarded as a composition of several sub-templates with a size of Wt*Hsub. After deriving the reference samples of each sub-template in a similar manner as described above, the reference samples of the template are derived. Two examples are shown in FIG. Fig.24 and Fig.25 shown. Fig.24 An example diagram 2400 for displaying a template of a block having sub-block motion using motion information of a sub-block of a current block and a reference sample of the template is illustrated. Fig.25 An example diagram 2500 for showing a template of a block having sub-block motion using motion information of each sub-template and a reference sample of the template is illustrated.

[0455] It should be noted that the terms mentioned below are not limited to specific terms defined in existing standards. Any changes in codecs also apply. For example, the term "GPM" is used to represent any codec that derives two sets of motion information and uses the derived information and the segmentation pattern to obtain the final prediction. For example, TPM is also considered a GPM.

[0456] Note that the proposed method can be applied to the merge candidate list construction process of inter-frame codec blocks (e.g., translational motion), affine codec blocks, or IBC codec blocks; or other motion candidate list construction processes (e.g., normal AMVP list, affine AMVP list, and IBC AMVP list).

[0457] W and H are the width and height of the current block (eg, luma block).

[0458] 1. In one example, if the coding mode is TM merging, some or all TM merging candidates may be reordered.

[0459] a. In one example, if the coding mode is TM merging, part or all of the original TM merging candidates may be reordered before the TM refinement process.

[0460] b. Alternatively, if the coding mode is TM merging, some or all of the refined TM merging candidates may be reordered after the TM refinement process.

[0461] c. Alternatively, if the coding mode is TM merging, the TM merging candidates may not be reordered.

[0462] d. Alternatively, the reordering method may be different for TM merge mode and other merge modes.

[0463] 2. In one example, if the coding mode is a sub-block based merge mode, candidate merges based on part or all of the sub-blocks may be reordered.

[0464] a. Alternatively, the reordering method may be different for the sub-block based merge mode and other merge modes.

[0465] b. In one example, a template can be divided into sub-templates. Each sub-template can have individual motion information.

[0466] i. In one example, a cost for a reordering candidate may be derived based on the cost of each sub-template. For example, the cost for a reordering candidate may be calculated as the sum of the costs of all sub-templates. For example, the cost of a sub-template may be calculated as SAD, SATD, SSD, or any other distortion measure between a sub-template and its corresponding reference sub-template.

[0467] c. In one example, to derive reference samples of the sub-template, the motion information of the sub-blocks in the first row and first column of the current block may be used.

[0468] i. In one example, the motion information of a sub-template can be derived (eg, copied) from its neighboring sub-block in the current block. Fig.24 shown.

[0469] d. In one example, in order to derive a reference sample of a sub-template, the motion information of the sub-template may be derived without referring to the motion information of the sub-block in the current block. Fig.25 shown.

[0470] i. In one example, the motion information of each sub-template is calculated according to the affine model of the current block.

[0471] (i) In one example, the motion vector of the center sample of each sub-block containing the sub-template calculated according to the affine model of the current block is regarded as the motion vector of the sub-template.

[0472] (ii) In one example, the motion vector of the center sample of each sub-template calculated according to the affine model of the current block is regarded as the motion vector of the sub-template.

[0473] (iii) For the 4-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as:

[0474]

[0475] (iv) For the 6-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as:

[0476]

[0477] (v) For (iii) and (iv), the coordinates of the upper left corner, upper right corner, and lower left corner of the current block are (0, 0), (W, 0), and (0, H), and the motion vectors of the upper left corner, upper right corner, and lower left corner of the current block are (mv 0x ,mv 0y )、(mv 1x ,mv 1y ) and (mv 2x ,mv 2y ).

[0478] (vi) In one example, the coordinates (x, y) in the above equation may be set equal to the position in the template or the position of the sub-template. For example, the coordinates (x, y) may be set equal to the center position of the sub-template.

[0479] e. In one example, this scheme can be applied to affine merge candidates.

[0480] f. In one example, this scheme can be applied to affine AMVP candidates.

[0481] g. In one example, this scheme can be applied to SbTMVP merge candidates.

[0482] h. In one example, this scheme can be applied to GPM merge candidates.

[0483] i. In one example, the scheme can be applied to TPM merge candidates.

[0484] g. In one example, this scheme can be applied to TM to refine merge candidates.

[0485] k. In one example, this solution can be applied to DMVR to refine merge candidates.

[0486] 1. In one example, this solution can be applied to MULTI_PASS_DMVR to refine merge candidates.

[0487] 3. In one example, if the codec mode is MMVD, the merge candidates used to derive the base merge candidate may be reordered.

[0488] a. In one example, a reordering process may be applied to the merge candidates before they are refined by a signaled or derived MVD.

[0489] b. For example, the reordering method may be different for MMVD mode and other merge modes.

[0490] 4. In one example, if the coding mode is MMVD, the merge candidates after MMVD refinement may be reordered.

[0491] a. In one example, a reordering process may be applied to the merge candidates after they are refined by a signaled or derived MVD.

[0492] b. For example, the reordering method may be different for MMVD mode and other merge modes.

[0493] 5. In one example, if the coding mode is affine MMVD, the merge candidates from which the base merge candidate is derived may be reordered.

[0494] a. In one example, a reordering process may be applied to affine merge candidates before they are refined by a signaled or derived MVD.

[0495] b. For example, the reordering method can be different for the affine MMVD mode and other merge modes.

[0496] 6. In one example, if the coding mode is affine MMVD, the merge candidates after affine MMVD refinement may be reordered.

[0497] a. In one example, a re-ranking process may be applied to affine merge candidates after they are refined by a signaled or derived MVD.

[0498] b. For example, the reordering method can be different for the affine MMVD mode and other merge modes.

[0499] 7. In one example, if the codec mode is GMVD, the merge candidates used to derive the base merge candidate may be reordered.

[0500] a. In one example, a reordering process may be applied to the merge candidates before they are refined by a signaled or derived MVD.

[0501] b. For example, the reordering method may be different for GMVD mode and other merge modes.

[0502] 8. In one example, if the coding mode is GMVD, the merge candidates after GMVD refinement may be reordered.

[0503] a. In one example, a reordering process may be applied to the merge candidates after they are refined by a signaled or derived MVD.

[0504] b. For example, the reordering method may be different for GMVD mode and other merge modes.

[0505] 9. In one example, if the codec mode is GPM, the merge candidates may be reordered.

[0506] a. In one example, before the merge candidates are used to derive the GPM candidate list for each partition (also referred to as the unidirectional prediction candidate list of the GPM), a reordering process may be applied to the original merge candidates.

[0507] b. In one example, if the coding mode is GPM, the merge candidates in the unidirectional prediction candidate list may be reordered.

[0508] c. In one example, a GPM unidirectional prediction candidate list may be constructed based on re-ranking.

[0509] i. In one example, a candidate with bi-directional prediction (also referred to as a bi-directional prediction candidate) may be separated into two uni-directional prediction candidates.

[0510] (i) If the number of original merging candidates is M, at most 2M unidirectional prediction candidates can be separated from them.

[0511] ii. In one example, unidirectional prediction candidates separated from bidirectional prediction candidates may be put into an initial unidirectional prediction candidate list.

[0512] iii. In one example, the candidates in the initial unidirectional prediction candidate list may be reordered using the template matching cost.

[0513] iv. In one example, the first N unidirectional prediction candidates with smaller template matching costs can be used as the final GPM unidirectional prediction candidates. For example, N is equal to M.

[0514] d. In one example, after deriving the GPM unidirectional prediction candidate list, a combined bidirectional prediction list of partition 0 and partition 1 is constructed, and then the bidirectional prediction list is reordered.

[0515] i. In one example, if the number of GPM unidirectional prediction candidates is M, the number of combined bidirectional prediction candidates is M*(M-1).

[0516] e. Alternatively, the reordering method may be different for GPM mode and other merge modes.

[0517] 2.13 Adaptive Motion Candidate List

[0518] It should be noted that the terms mentioned below are not limited to specific terms defined in existing standards. Any changes in codecs also apply. For example, the term "GPM" is used to represent any codec that derives two sets of motion information and uses the derived information and the partitioning pattern to obtain the final prediction. For example, TPM is also considered a GPM.

[0519] Note that the proposed method can be applied to the merge candidate list construction process of inter-frame codec blocks (e.g., translational motion), affine codec blocks, or TM codec blocks; or other motion candidate list construction processes (e.g., normal AMVP list, affine AMVP list, and IBC AMVP list).

[0520] W and H are the width and height of the current block (eg, luma block).

[0521] 1. The reference sample (RT) of the template or sub-template used for bidirectional prediction is obtained by referencing the reference sample (RT) of the template or sub-template in the reference list0. 0 ) and the reference sample (RT 1 ) are equally weighted averaged. An example is as follows:

[0522] RT(x,y)=(RT 0 (x,y)+RT 1 (x,y)+1)>>1

[0523] 2. The reference sample (RT) of the template or sub-template used for bidirectional prediction is obtained by comparing the reference sample (RT) of the template or sub-template in the reference list0. 0 ) and the reference sample (RT 1 ) is obtained by taking the weighted average.

[0524] a. An example is as follows:

[0525] RT(x,y)=((2 N -w)*RT 0 (x,y)+w*RT 1 (x,y)+2 N-1 )>>N, for example, N=3.

[0526] b. The weights can be determined by the BCW index, or they can be derived on the fly or predefined or determined by the weights used in weighted prediction.

[0527] c. In one example, the weight of a reference template such as (8-w) in reference list 0 and the weight of a reference template such as (w) in reference list 1 may be determined by the BCW index of the merge candidate.

[0528] i. In one example, the BCW index is equal to 0 and w is set to -2.

[0529] ii. In one example, the BCW index is equal to 1 and w is set to 3.

[0530] iii. In one example, the BCW index is equal to 2 and w is set to 4.

[0531] iv. In one example, BCW index equals 3 and w is set to 5.

[0532] v. In one example, BCW index equals 4 and w is set to 10.

[0533] 3. Reference samples for deriving templates using the LIC method are proposed.

[0534] a. In one example, the LIC parameters of the left template and the upper template are the same as the LIC parameters of the current block.

[0535] b. In one example, the LIC parameters of the left template are derived as the LIC parameters of the current block, which uses its original motion vector plus the motion vector offset (-Wt, 0) as the motion vector of the current block.

[0536] c. In one example, the LIC parameters of the upper template are derived as the LIC parameters of the current block, which uses its original motion vector plus the motion vector offset (0, -Ht) as the motion vector of the current block.

[0537] d. Alternatively or additionally, if the local illumination compensation (LIC) flag of the merge candidate is true, the above items may be applied

[0538] 4. A reference sample for deriving a template or sub-template using the OBMC method is proposed. In the following discussion, "template" may refer to a template or a sub-template.

[0539] a. In one example, in order to derive the reference samples of the upper template, the motion information of the sub-block in the first row of the current block and its adjacent neighboring sub-blocks above it is used. The reference samples of all sub-templates constitute the reference samples of the upper template. Fig.26 Shown in. Fig.26 Samples for displaying a block with OBMC and reference samples for the samples are shown.

[0540] b. In one example, to derive the reference samples of the left template, the motion information of the sub-block in the first column of the current block and its adjacent sub-blocks to the left is used. And the reference samples of all sub-templates constitute the reference samples of the left template. An example is Fig.26 shown.

[0541] c. In one example, the sub-block size is 4×4.

[0542] d. The reference sample of the sub-template based on the motion vector of the neighboring sub-block is represented as P N, where N represents the index of the adjacent upper and left sub-blocks, and the reference sample of the sub-template based on the motion vector of the sub-block of the current block is represented by P C For P generated based on the motion vectors of vertically (horizontally) adjacent sub-blocks N , P N The samples in the same row (column) of P are added to P with the same weighting factor. C .

[0543] i. The reference sample (P) of the sub-template can be derived as P = W N *P N +W C *P C

[0544] ii. In one example, if the height of the upper template or the width of the left template is greater than or equal to 4, the weight factors {1 / 4, 1 / 8, 1 / 16, 1 / 32} are used for P N The {first, second, third, fourth} rows (columns) of P, and the weight factors {3 / 4, 7 / 8, 15 / 16, 31 / 32} are used for P C The {first, second, third, fourth} rows (columns) of .

[0545] iii. In one example, if the height of the upper template or the width of the left template is greater than or equal to 2, the weighting factor {1 / 4, 1 / 8} is used for P N The {first, second} rows (columns) of P, weighting factors {3 / 4,7 / 8} are used for P C The {first, second} row (column) of .

[0546] iv. In one example, if the height of the template above or the width of the template to the left is greater than or equal to 1, the weighting factor {1 / 4} is used for P N The first row (column) of , the weighting factor {3 / 4} is used for P C The first row (column) of .

[0547] e. If merge candidates are allocated with OBMC enabled, the above items may apply.

[0548] 5. In one example, if the merged candidate uses multi-hypothesis prediction, the reference sample of the template can be derived using the multi-hypothesis prediction method.

[0549] 6. A template may include samples of (multiple) specific color components.

[0550] a. In one example, the template includes samples of the luma component only.

[0551] b. Alternatively, the template only includes samples of any component, such as Cb / Cr / R / G / B.

[0552] 7. Whether and / or how to re-order motion candidates may depend on the category of the motion candidates.

[0553] a. In one example, only neighboring spatial motion candidates and temporal motion candidates may be reordered.

[0554] b. In one example, only neighboring spatial motion candidates, STMVP, and neighboring temporal motion candidates may be reordered.

[0555] c. In one example, only neighboring spatial motion candidates, STMVP, neighboring temporal motion candidates, and non-neighboring spatial motion candidates may be reordered.

[0556] d. In one example, only neighboring spatial motion candidates, STMVP, neighboring temporal motion candidates, non-neighboring spatial motion candidates, and HMVP may be reordered.

[0557] e. In one example, only neighboring spatial motion candidates, STMVP, neighboring temporal motion candidates, non-neighboring spatial motion candidates, HMVP, and pairwise average motion candidates may be re-ordered.

[0558] f. In one example, only neighboring spatial motion candidates, neighboring temporal motion candidates, HMVP and pairwise average motion candidates may be reordered.

[0559] g. In one example, only adjacent spatial motion candidates, adjacent temporal motion candidates, and

[0560] HMVP can be reordered.

[0561] h. In one example, only neighboring spatial motion candidates may be reordered.

[0562] i. In one example, motion candidates based on unidirectional prediction sub-blocks are not re-ordered.

[0563] j. In one example, SbTMVP candidates are not re-ordered.

[0564] k. In one example, inherited affine motion candidates are not reordered.

[0565] 1. In one example, reconstructed affine motion candidates are not reordered.

[0566] m. In one example, zero-padded affine motion candidates are not reordered.

[0567] n. In one example, only the top N motion candidates may be reordered.

[0568] i. In one example, N is set equal to 5.

[0569] 8. In one example, the motion candidates may be divided into multiple subgroups. Whether and / or how to reorder the motion candidates may depend on the subgroups of the motion candidates.

[0570] a. In one example, only the first subgroup can be reordered.

[0571] b. In one example, the last subgroup may not be reordered.

[0572] c. In one example, the last subgroup may not be reordered. However, if the last subgroup is also the first subgroup, it can be reordered.

[0573] d. Different subgroups can be reordered individually.

[0574] e. Two candidates in different subgroups cannot be compared and / or re-ranked.

[0575] f. If the first subgroup is before the second subgroup, then after reordering, the first candidate in the first subgroup needs to be placed before the second candidate in the second subgroup.

[0576] 9. In one example, motion candidates not included in the re-ordering process may be processed in a specified manner.

[0577] a. In one example, for candidates that are not reordered, they will be arranged in the merge candidate list according to the initial order.

[0578] b. In one example, candidates that are not reordered may be placed after candidates to be reordered.

[0579] c. In one example, candidates that are not reordered may be placed before candidates to be reordered.

[0580] 10. Whether to apply adaptive merge candidate list reordering may depend on a message transmitted by signal in VPS / SPS / PPS / sequence header / picture header / slice header / CTU / CU. It may also be a signaling-based area. For example, the picture is divided into groups of CTU / CU uniformly or unevenly, and a flag is encoded for each group to indicate whether to apply merge candidate list reordering.

[0581] 2.14 Cost functions used in codec data refinement in image / video coding

[0582] The term "block" may refer to a codec tree block (CTB), a codec tree unit (CTU), a coding block (CB), a CU, a PU, a TU, a PB, a TB, or a video processing unit including multiple samples / pixels, and a block may be rectangular or non-rectangular.

[0583] In the present disclosure, the phrase "motion candidate" may refer to a merged motion candidate in a regular / extended merge list indicated by a merge candidate index, or an AMVP motion candidate in a regular / extended AMVP list indicated by an AMVP candidate index, or an AMVP motion candidate, or a merged motion candidate.

[0584] In this disclosure, a motion candidate is referred to as "refined" if its motion information is modified based on information signaled from an encoder or derived at a decoder. For example, a motion vector may be refined by DMVR, FRUC, TM Merge, TMAMVP, TM GPM, TM CIIP, TM Affine, MMVD, GMVD, Affine MMVD, BDOF, etc.

[0585] In the present disclosure, the phrase "codec data refinement" may refer to a refinement process to derive or refine a signaled / decoded / derived prediction mode, prediction direction, or signaled / decoded / derived motion information, prediction, and / or reconstructed samples for a block. In one example, the refinement process may include motion candidate reordering.

[0586] In the following discussion, a "template-based" block may refer to a block that uses a template matching-based method to derive or refine the coding information during the encoding / decoding process, such as template matching-based motion derivation, template matching-based motion list reconstruction, LIC, symbol prediction, template matching-based block vector (e.g., used in IBC mode) derivation, DIMD, template matching-based non-inter-frame (e.g., intra-frame) prediction, etc. The template-based coding method can be combined with any other coding tools, such as MMVD, CIIP, GPM, FRUC, Affine, BDOF, DMVR, OBMC, etc. In another example, a "template-based" block may also refer to a block that uses (adjacent or non-adjacent) neighboring reconstructed samples based on a certain rule to derive or refine its decoding information, such as the DIMD method in 2.27 and the TIMD method in 2.29.

[0587] In the following discussion, a "bilateral codec-based" block may refer to a block that uses a bilateral matching-based method to derive or refine coding information during the encoding / decoding process, such as bilateral matching-based motion derivation, bilateral matching-based motion list reconstruction, etc. The bilateral codec-based method can be combined with any other codec tools, such as MMVD, CIIP, GPM, FRUC, Affine, DMVR, etc.

[0588] W and H are the width and height of the current block (e.g., luminance block). W*H is the size of the current block (e.g., luminance block)

[0589] In the following discussion, Shift(x,s) is defined as

[0590] Shift(x,s)=(x+offset)>>s, where the offset (offset) is an integer, such as offset=0 or offset=1<<(s-1) or offset=(1<<(s-1))-1.

[0591] In another example, offset depends on x. For example, offset = (x<0?(1<<(s-1)):((1<<(s-1)-1).

[0592] 1. In addition to error measurement, it is proposed to add conventional items to the cost calculation process.

[0593] a) In one example, the cost is defined as: E+W*RI, where E represents the output of the error function and W is the weight applied to the regular term represented by RI.

[0594] i. In one example, to process template-based codec blocks / bilateral-based codec blocks, the cost function is set to: E+W*RI, where E can be SAD / MRSAD / SATD or others, RI is the estimated bit of the true motion vector / motion vector difference, and W is the weight, which can depend on QP / temporal layer, etc., for example.

[0595] ii. Alternatively, the cost can be defined as: w0*E+W1*RI, where E represents the output of the error function, W1 is the weight applied to the regular term represented by RI, and w0 is the weight applied to the output of the error function.

[0596] (i) Alternatively, in addition, W1 may be set to 0.

[0597] b) In one example, the regular term may be multiplied by a weighting rate.

[0598] i. In one example, weights are derived on the fly.

[0599] ii. In one example, the weight is set to the lambda used in the full RDO process

[0600] iii In one example, the weights are set to the square root of the lambda used in the full RDO process.

[0601] c) In one example, the cost is calculated as E + Shift(W * RI, s), where s and W are integers.

[0602] i. Alternatively, the cost is calculated as Shift((E << s) + W * RI, s), where s and W are integers.

[0603] 2. An error function different from SAD / MR - SAD (Sum of Absolute Differences with Mean Removed) is proposed to process template - based encoding / decoding blocks / bilateral - based encoding / decoding blocks.

[0604] a) In one example, the error function can be

[0605] i. SATD

[0606] ii. MR - SATD

[0607] iii. Gradient information

[0608] iv. SSE / SSD

[0609] v. MR - SSE / MR - SSD

[0610] vi Weighted SAD / Weighted MR - SAD

[0611] vii. Weighted SATD / Weighted MR - SATD

[0612] viii. Weighted SSD / Weighted MR - SSD

[0613] ix. Weighted SSE / Weighted MR - SSE

[0614] b) Alternatively, in addition, a method for adaptively selecting an error function among different cost functions such as the above - mentioned error functions and SAD / MR - SAD is proposed.

[0615] i. The selection can be determined instantaneously.

[0616] 3. When using an error function based on MR - X (e.g., X is SATD, SAD, SSE) (such as MR - SAD / MR - SATD, etc.), the following can be further applied:

[0617] a) In one example, the mean can be calculated considering all samples in the block to be compared.

[0618] b) In one example, the mean can be calculated using some of the samples in the block to be compared.

[0619] c) In one example, the mean and the X function can depend on the same samples in the block.

[0620] i. In one example, the mean and X-function may be calculated using all samples in a block.

[0621] ii. In one example, a portion of the samples in a block may be used to calculate the mean and the X-function.

[0622] d) In one example, the mean and the X-function may depend on at least one different sample in the block.

[0623] In one example, the mean may be calculated using all samples, while the X function may depend on a subset of samples in a block.

[0624] ii In one example, the mean may be calculated using a portion of the samples, while the X function may depend on all samples in the block.

[0625] 4. The template / bilateral matching cost can be calculated by applying the cost factor to the error cost function.

[0626] a) In one example, it is proposed to bias motion candidates towards the front during the template / bilateral matching based re-ranking process.

[0627] i. In one example, the motion candidate in the i-th position is assigned a smaller cost factor than the cost factor of the motion candidate in the (i+1)-th position.

[0628] ii. In one example, motion candidates in the i-th group (eg, involving M motion candidates) are assigned a smaller cost factor than the cost factor of motion candidates in the (i+1)-th group (eg, involving N motion candidates).

[0629] (i) In one example, M may be equal to N. For example, M=N=2.

[0630] (ii) In one example, M may not be equal to N. For example, M=2, N=3.

[0631] b) In one example, it is proposed that the refinement process based on template / bilateral matching tends to be closer to the original

[0632] Search MV for MV.

[0633] i. In one example, each search region is assigned a cost factor, which may be determined by the distance between each search MV in the search region and the start MV (eg, in integer pixel precision MV increments).

[0634] ii. In one example, each search region is assigned a cost factor, which may be determined by the distance between a center search MV and a start MV in the search region (eg, in integer pixel precision MV increments).

[0635] iii. In one example, each search MV is assigned a cost factor, which may be determined by the distance between each search MV and the start MV (eg, in integer pixel precision MV increments).

[0636] 5. The above method can be applied to any codec data refinement process, for example, for blocks based on template codecs, for blocks based on bilateral codecs (such as DMVR in VVC).

[0637] 6. For different template matching refinement methods, the template matching cost measurement may be different.

[0638] a. In one example, the template matching refinement method may be a motion candidate re-ranking based on template matching.

[0639] b. In one example, the template matching refinement method may be a motion derivation based on template matching.

[0640] i. In one example, the refinement method may be TM AMVP, TM Merge, and / or FRUC.

[0641] c. In one example, the template matching refinement method may be template matching based motion refinement.

[0642] ii. In one example, the refinement method may be TM GPM, TM CIIP and / or TM Affine.

[0643] d. In one example, the template matching refinement method may be based on template matching block vector derivation.

[0644] e. In one example, the template matching refinement method may be intra-mode derivation based on template matching.

[0645] iii. In one example, the thinning method may be DIMD and / or TIMD.

[0646] f. In one example, the template matching cost measure may be calculated based on the sum of absolute differences (SAD) between the current template and the reference template.

[0647] g. In one example, the template matching cost measure may be calculated based on the mean removed SAD between the current template and the reference template.

[0648] h. In one example, SAD and mean removed SAD (MR-SAD) can be selectively utilized according to the size of the current block.

[0649] i. In one example, mean removal SAD is used for blocks of size greater than M, and SAD is used for blocks of size less than or equal to M.

[0650] (i) In one example, M is 64.

[0651] i. In one example, SAD and mean removed SAD (MR-SAD) can be selectively utilized according to the LIC flag of the current block.

[0652] i. In one example, if the LIC flag of the current block is false, the template matching cost measure may be SAD.

[0653] ii. In one example, if the LIC flag of the current block is true, the template matching cost measure may be MR-SAD.

[0654] j. In one example, the template matching cost measure may be calculated based on the sum of absolute transformed differences (SATD) between the current template and the reference template.

[0655] k. In one example, the template matching cost measure may be calculated based on the mean-removed SATD between the current template and the reference template.

[0656] 1. In one example, SATD and mean-removed SATD (MR-SATD) can be selectively utilized according to the size of the current block.

[0657] i. In one example, mean removal SATD is used for blocks of size greater than M, and SATD is used for blocks of size less than or equal to M.

[0658] (i) In one example, M is 64.

[0659] m. In one example, SATD and mean removal SATD (MR-SATD) can be selectively utilized according to the LIC flag of the current block.

[0660] i. In one example, if the LIC flag of the current block is false, the template matching cost measure can be SATD.

[0661] ii. In one example, if the LIC flag of the current block is true, the template matching cost measure can be MR-SATD.

[0662] n. In one example, the template matching cost measure may be calculated based on the sum of squared differences (SSD) between the current template and the reference template.

[0663] o. In one example, the template matching cost measure may be calculated based on the mean removed SSD between the current template and the reference template.

[0664] p. In one example, SSD and mean removed SSD (MR-SSD) can be selectively utilized according to the size of the current block.

[0665] i. In one example, mean removal SSD is used for blocks of size greater than M, and SSD is used for blocks of size less than or equal to M.

[0666] (i) In one example, M is 64.

[0667] q. In one example, the template matching cost measurement can be weighted SAD / weighted MR-SAD / selective weighted MR-SAD and SAD / weighted SATD / weighted MR-SATD / selective weighted MR-SATD and SATD / weighted SSD / weighted MR-SSD / selective weighted MR-SSD and SSD between the current template and the reference template.

[0668] i. In one example, weighting means applying different weights to each sample based on its row and column index in the template block when calculating the distortion between the current template and the reference template.

[0669] ii. In one example, weighting means applying different weights to each sample based on its position in the template block when calculating the distortion between the current template and the reference template.

[0670] iii. In one example, weighting means applying different weights to each sample according to its distance to the current block when calculating the distortion between the current template and the reference template.

[0671] r. In one example, the template matching cost may be calculated as follows: tplCost = w1*mvDis-tanceCost+w2*distortionCost.

[0672] i. In one example, distortionCost may be the weighted SAD / weighted MR-SAD / weighted SATD / weighted MR-SATD / weighted SSD / weighted MR-SSD / SAD / MR-SAD / SATD / MR-SATD / SSD / MR-SSD between the current template and the reference template.

[0673] ii. In one example, mvDistanceCost may be the sum of the absolute mv differences between the search point and the start point in the horizontal direction and the vertical direction.

[0674] iii. In one example, w1 and w2 may be predetermined, signaled, or derived from decoded information.

[0675] (i) In one example, w1 is a weighting factor set to 4 and w2 is a weighting factor set to 1

[0676] s. In addition to the SAD calculated in (f), the cost may also consider the continuity between the reference template and the reconstructed samples adjacent to or non-adjacent to the current template (Boundary_SAD). For example, consider the reconstructed samples adjacent to the left and / or above the current sample.

[0677] i. In one example, the cost can be calculated based on SAD and Boundary_SAD

[0678] (i) In one example, the cost may be calculated as (SAD + w*Bound-ary_SAD). w may be predetermined, signaled, or derived from decoded information.

[0679] 7. For different bilateral matching refinement methods, the bilateral matching cost measurement can be different.

[0680] a) In one example, the bilateral matching refinement method may be a motion candidate re-ranking based on bilateral matching.

[0681] b) In one example, the bilateral matching refinement method may be motion derivation based on bilateral matching.

[0682] i. In one example, the refinement method may be BM merging and / or FRUC.

[0683] c) In one example, the bilateral matching refinement method may be motion refinement based on bilateral matching.

[0684] i. In one example, the refinement method may be BM GPM, BM CIIP and / or BM Affine.

[0685] d) In one example, the bilateral matching refinement method may be a block vector derivation based on bilateral matching.

[0686] e) In one example, the bilateral matching refinement method may be an intra-mode derivation based on bilateral matching.

[0687] f) In one example, the bilateral matching cost measure may be calculated based on the sum of absolute differences (SAD) between two reference blocks / sub-blocks.

[0688] g) In one example, the bilateral matching cost measure may be calculated based on the mean removed SAD between two reference blocks / sub-blocks.

[0689] h) In one example, SAD and mean removed SAD (MR-SAD) can be selectively utilized according to the size of the current block / sub-block.

[0690] In one example, mean-removed SAD is used for blocks / sub-blocks with a size greater than M, and SAD is used for blocks / sub-blocks with a size less than or equal to M.

[0691] (i) In one example, M is 64.

[0692] i) In one example, SAD and mean removed SAD (MR-SAD) can be selectively utilized according to the LIC flag of the current block.

[0693] i. In one example, if the LIC flag of the current block is false, the bilateral matching cost measure can be SAD.

[0694] ii. In one example, if the LIC flag of the current block is true, the bilateral matching cost measure may be MR-SAD.

[0695] j) In one example, the bilateral matching cost measure may be calculated based on the sum of absolute transformed differences (SATD) between two reference blocks / sub-blocks.

[0696] k) In one example, the bilateral matching cost measure may be calculated based on the mean-removed SATD between two reference blocks / sub-blocks.

[0697] l) In one example, SATD and mean removed SATD (MR-SATD) can be selectively utilized according to the size of the current block / sub-block.

[0698] i. In one example, mean removal SATD is used for blocks / sub-blocks with size greater than M, and SATD is used for blocks / sub-blocks with size less than or equal to M.

[0699] (i) In one example, M is 64.

[0700] m) In one example, SATD and mean removed SATD (MR-SATD) can be selectively utilized according to the LIC flag of the current block.

[0701] (i) In one example, if the LIC flag of the current block is false, the bilateral matching cost measure can be SATD.

[0702] (ii) In one example, if the LIC flag of the current block is true, the bilateral matching cost measure can be MR-SATD.

[0703] n) In one example, the bilateral matching cost measure may be calculated based on the sum of squared differences (SSD) between two reference blocks / sub-blocks.

[0704] o) In one example, the bilateral matching cost measure may be calculated based on the mean removed SSD between two reference blocks / sub-blocks.

[0705] p) In one example, SSD and mean removed SSD (MR-SSD) can be selectively utilized according to the size of the current block / sub-block.

[0706] In one example, mean removal SSD is used for blocks / sub-blocks with a size greater than M, and SSD is used for blocks / sub-blocks with a size less than or equal to M.

[0707] (i) In one example, M is 64.

[0708] q) In one example, SSD and mean removed SSD (MR-SSD) can be selectively utilized according to the LIC flag of the current block.

[0709] In one example, if the LIC flag of the current block is false, the bilateral matching cost measure can be SSD.

[0710] ii In one example, if the LIC flag of the current block is true, the bilateral matching cost measure can be MR-SSD.

[0711] r) In one example, the bilateral matching cost metric can be weighted SAD / weighted MR-SAD / selective weighted MR-SAD and SAD / weighted SATD / weighted MR-SATD / selective weighted MR-SATD and SATD / weighted SSD / weighted MR-SSD / selective weighted MR-SSD and SSD between two reference blocks / sub-blocks.

[0712] i. In one example, weighting means applying different weights to each sample based on its row and column index in the reference block / sub-block when calculating the distortion between two reference blocks / sub-blocks.

[0713] ii. In one example, weighting means that when calculating the distortion between two reference blocks / sub-blocks, different weights are applied to each sample based on its position in the reference block / sub-block.

[0714] iii. In one example, weighting means that when calculating the distortion between two reference blocks / sub-blocks, different weights are applied to each sample according to its distance to the center position of the reference block / sub-block.

[0715] s) In one example, if MR-SAD / MR-SATD / MR-SSD is used for bilateral matching cost measurement, LIC may not be used when deriving reference blocks / sub-blocks.

[0716] t) In one example, the bilateral matching cost can be calculated as follows: bilCost = w1*mvDis-tanceCost+w2*distortionCost.

[0717] In one example, distortionCost may be the weighted SAD / weighted MR-SAD / weighted SATD / weighted MR-SATD / weighted SSD / weighted MR-SSD / SAD / MR-SAD / SATD / MR-SATD / SSD / MR-SSD between two reference blocks / subblocks.

[0718] ii. In one example, mvDistanceCost may be the sum of the absolute mv differences between the search point and the start point in the horizontal and vertical directions.

[0719] iii. In one example, w1 and w2 may be predetermined, or signaled, or derived from decoded information.

[0720] (i) In one example, w1 is a weighting factor set to 4, and w2 is a weighting factor set to 1.

[0721] 8. The bilateral matching cost or template matching cost can be calculated based on the prediction / reference samples modified by the function.

[0722] a) In one example, the prediction / reference samples may be filtered before being used to calculate the bilateral matching cost or the template matching cost.

[0723] b) In one example, the prediction / reference sample S can be modified to a*S+b before being used to calculate the bilateral matching cost or the template matching cost.

[0724] c) In one example, the modification may depend on the codec mode of the block, such as whether the block is LIC coded or BCW coded.

[0725] 2.15 Application of Multiple Cost Functions in Codec Data Refinement in Image / Video Codecs

[0726] The term "block" may refer to a codec tree block (CTB), a codec tree unit (CTU), a codec block (CB), a CU, a PU, a TU, a PB, a TB, or a video processing unit including a plurality of samples / pixels. A block may be rectangular or non-rectangular.

[0727] In the present disclosure, the phrase "motion candidate" may refer to a merged motion candidate in a regular / extended merge list indicated by a merge candidate index, or an AMVP motion candidate in a regular / extended AMVP list indicated by an AMVP candidate index, or an AMVP motion candidate, or a merged motion candidate.

[0728] In the present disclosure, a motion candidate is referred to as being "refined" if its motion information is modified according to information signaled from an encoder or derived at a decoder. For example, a motion vector may be refined by DMVR, FRUC, TM Merge, TM AMVP, TM GPM, TM CIIP, TM Affine, MMVD, GMVD, Affine MMVD, BDOF, etc.

[0729] In the present disclosure, the phrase "codec data refinement" may refer to a refinement process for deriving or refining a signaled / decoded / derived prediction mode, prediction direction, or signaled / decoded / derived motion information, prediction and / or reconstructed samples for a block. In one example, the refinement process may include motion candidate reordering.

[0730] In the following discussion, a "template-based codec" block may refer to a block that uses a template matching-based method to derive or refine codec information during the codec / decoding process, such as template matching-based motion derivation, template matching-based motion list reconstruction, LIC, symbol prediction, template matching-based block vector derivation (e.g., used in IBC mode), DIMD, template matching-based non-inter (e.g., intra) prediction, etc. The template matching-based method may be combined with any other codec tools, such as MMVD, CIIP, GMP, FRUC, Affine, BDOF, DMVR, OBMC, etc. In another example, a "template-based codec" block may also refer to a block that uses (adjacent or non-adjacent) neighboring reconstructed samples based on a certain rule to derive or refine its decoding information, such as the DIMD method in 2.27 and the TIMD method in 2.29.

[0731] In the following discussion, a "bilateral codec-based" block may refer to a block that uses a bilateral matching-based method to derive or refine codec information during the encoding / decoding process, such as bilateral matching-based motion derivation, bilateral matching-based motion list reconstruction, etc. The bilateral codec-based method can be combined with any other codec tools, such as MMVD, CIIP, GPM, FRUC, Affine, DMVR, etc.

[0732] W and H are the width and height of the current block (eg, luma block). W*H is the size of the current block (eg, luma block).

[0733] 1. The cost definition may rely on the output of multiple error functions (eg, distortion measurement methods) regarding the error / difference between two samples / blocks to be evaluated in a codec data refinement process of the current block.

[0734] a) In one example, the error function may be:

[0735] i.SAD

[0736] ii.SATD

[0737] iii.MR-SAD

[0738] iv.MR-SATD

[0739] v. Gradient information

[0740] vi.SSE / SSD

[0741] vii.MR-SSE / MR-SSD

[0742] viii. Weighted SAD / Weighted MR-SAD

[0743] ix. Weighted SATD / Weighted MR-SATD

[0744] x. Weighted SSD / Weighted MR-SSD

[0745] xi. Weighted SSE / Weighted MR-SSE

[0746] b) In one example, the error function may be performed at a block level or a sub-block level.

[0747] i. Alternatively, furthermore, the error function may be different for the two sub-blocks.

[0748] ii. Alternatively, furthermore, the final output of the estimated error for the block may be based on the outputs of the sub-blocks, for example the sum of the outputs of the error function applied to each sub-block.

[0749] 2. When the cost definition depends on the output of multiple functions, the following items may further apply:

[0750] a) In one example, the cost function may rely on a linear weighted sum of multiple error functions.

[0751] b) In one example, the cost function may rely on a non-linear weighted sum of multiple error functions.

[0752] c) In one example, the cost function may further depend on the estimated bits for the side information.

[0753] d) In one example, the cost function may be defined as:

[0754]

[0755] Where R represents the estimated bits for the side information, W i and Ei denote the weight applied to the i-th error function and the output of the i-th error function, respectively.

[0756] 3. Multiple refinement processes may be applied to one block, wherein at least more than two different cost functions are applied to at least two refinement processes.

[0757] a) In one example, a first refinement process may be invoked using a first cost function. Based on the output of the first refinement process, a second cost function is further applied to a second refinement process.

[0758] b) The above method can be applied to blocks based on template coding.

[0759] 4. Whether to use multiple refinement processes and / or how to select one or more error functions and / or how to define the cost function and / or which samples to involve in the error function may depend on the decoding information of the current block and / or the decoding information of neighboring (adjacent or non-adjacent) blocks of the current block.

[0760] a) In one example, how to select one or more error functions and / or how to define the cost function may depend on the codec tools applied to the current block and / or neighboring blocks of the current block.

[0761] i. In one example, the codec is LIC.

[0762] (i) In one example, SSD and mean removed SSD (MR-SSD) can be selectively utilized according to the LIC flag of the current block.

[0763] a) In one example, if the LIC flag of the current block is false, the template matching cost metric may be SSD.

[0764] b) In one example, if the LIC flag of the current block is true, the template matching cost metric may be MR-SSD.

[0765] (ii) In one example, if MR-SAD / MR-SATD / MR-SSD is used for the template matching cost metric, the linear function used in the LIC process may not be used when deriving the reference template.

[0766] (iii) In one example, if MR-SAD / MR-SATD / MR-SSD is used for the bilateral matching cost metric, the linear function used in the LIC process may not be used when deriving the reference block.

[0767] b) In one example, it may depend on block size (eg, W, H), temporal layer, low latency check flag, etc.

[0768] c) In one example, it may depend on whether the motion information of the current block is similar / identical to the motion information of the neighboring blocks.

[0769] d) In one example, it may depend on a reference picture list and / or reference picture information.

[0770] i. In one example, for list X, a first error function (eg, SAD / SSE) may be used, and for list Y (Y=1-X), a second error function (eg, MR-SAD / MR-SSE) may be used.

[0771] ii. Alternatively, furthermore, the final cost may be based on the cost of each reference picture list.

[0772] 5. The above method can be applied to any codec data refinement process, for example, for blocks based on template codecs, for blocks based on bilateral codecs (for example, DMVR in VVC).

[0773] 2.16 Samples used in codec data refinement for image / video codecs

[0774] The term "block" may refer to a codec tree block (CTB), a codec tree unit (CTU), a codec block (CB), a CU, a PU, a TU, a PB, a TB, or a video processing unit including a plurality of samples / pixels. A block may be rectangular or non-rectangular.

[0775] In the present disclosure, the phrase "motion candidate" may refer to a merged motion candidate in a regular / extended merge list indicated by a merge candidate index, or an AMVP motion candidate in a regular / extended AMVP list indicated by an AMVP candidate index, or an AMVP motion candidate, or a merged motion candidate.

[0776] In the present disclosure, a motion candidate is referred to as being "refined" if its motion information is modified according to information signaled from an encoder or derived at a decoder. For example, a motion vector may be refined by DMVR, FRUC, TM Merge, TM AMVP, TM GPM, TM CIIP, TM Affine, MMVD, GMVD, Affine MMVD, BDOF, etc.

[0777] In the present disclosure, the phrase "codec data refinement" may refer to a refinement process for deriving or refining a signaled / decoded / derived prediction mode, prediction direction, or signaled / decoded / derived motion information, prediction and / or reconstructed samples for a block. In one example, the refinement process may include motion candidate reordering.

[0778] In the following discussion, a "template-based codec" block may refer to a block that uses a template matching-based method to derive or refine codec information during the codec / decoding process, such as template matching-based motion derivation, template matching-based motion list reconstruction, LIC, symbol prediction, template matching-based block vector derivation (e.g., used in IBC mode), DIMD, template matching-based non-inter (e.g., intra) prediction, etc. The template matching-based method may be combined with any other codec tools, such as MMVD, CIIP, GMP, FRUC, Affine, BDOF, DMVR, OBMC, etc. In another example, a "template-based codec" block may also refer to a block that uses (adjacent or non-adjacent) neighboring reconstructed samples based on a certain rule to derive or refine its decoding information, such as the DIMD method in 2.27 and the TIMD method in 2.29.

[0779] In the following discussion, a "bilateral codec-based" block may refer to a block that uses a bilateral matching-based method to derive or refine codec information during the encoding / decoding process, such as bilateral matching-based motion derivation, bilateral matching-based motion list reconstruction, etc. The bilateral codec-based method can be combined with any other codec tools, such as MMVD, CIIP, GPM, FRUC, Affine, DMVR, etc.

[0780] W and H are the width and height of the current block (eg, luma block). W*H is the size of the current block (eg, luma block).

[0781] 1. The error / cost evaluation during codec data refinement may depend on reference samples corresponding to the current block (eg, the reference block used in bilateral matching) and reference samples corresponding to the template of the current block.

[0782] a) Alternatively, it may rely on both the reference samples corresponding to the current block and the samples in the template of the current block.

[0783] b) In one example, the template may be neighboring samples (adjacent or non-adjacent) of the current block.

[0784] 2. Multiple refinement processes can be applied to one block, and different templates are applied to at least two refinement processes.

[0785] a) In one example, a first refinement process may be invoked using a first template. Based on the output of the first refinement process, a second template is further applied to a second refinement process.

[0786] b) In one example, the first template may contain more samples than the second template.

[0787] c) In one example, the first and second templates may contain at least one different sample.

[0788] d) In one example, the first refinement process and the second refinement process may use different cost / error functions.

[0789] 3. Whether to use multiple refinement processes, and / or how to select one or more error functions and / or how to define the cost function and / or which samples are involved in the error function may depend on the coding information of the current block and / or neighboring (adjacent or non-adjacent) blocks.

[0790] a) In one example, how to select one or more error functions and / or how to define the cost function may depend on the codec tools applied to the current block and / or neighboring blocks.

[0791] i. In one example, the codec is LIC.

[0792] (i) In one example, SSD and mean removed SSD (MR-SSD) may be selectively used according to the LIC flag of the current block.

[0793] a) In one example, if the LIC flag of the current block is false, the template matching cost metric may be SSD.

[0794] b) In one example, if the LIC flag of the current block is true, the template matching cost metric may be MR-SSD.

[0795] (ii) In one example, if MR-SAD / MR-SATD / MR-SSD is used for the template matching cost measurement, the linear function used in the LIC process may not be used when deriving the reference template.

[0796] (iii) In one example, if MR-SAD / MR-SATD / MR-SSD is used for the bilateral matching cost measurement, the linear function used in the LIC process may not be used when deriving the reference block.

[0797] b) In one example, it may depend on block size (eg, W, H), temporal layer, low latency check flag, etc.

[0798] c) In one example, it may depend on whether the motion information of the current block and the neighboring blocks are similar / identical.

[0799] d) In one example, it may depend on a reference picture list and / or reference picture information.

[0800] i. In one example, for list X, a first error function (eg, SAD / SSE) may be used, and for list Y (Y=1-X), a second error function (eg, MR-SAD / MR-SSE) may be used.

[0801] ii. Alternatively, the final cost may be based on the cost of each reference picture list.

[0802] 4. In one example, LIC may be enabled for reference list X, while LIC may be disabled for reference list Y.

[0803] a) In one example, the final prediction of the current block may be a weighted average of the LIC prediction from reference list X and the conventional prediction from reference list Y.

[0804] 5. The above method can be applied to any codec data refinement process, for example, for template-based codec blocks, for bilateral-based codec blocks (e.g., DMVR in VVC).

[0805] 2.17 Adaptive Motion Candidate List

[0806] It should be noted that the terms mentioned below are not limited to specific terms defined in existing standards. Any changes in codecs also apply. For example, the term "GPM" is used to represent any codec that derives two sets of motion information and uses the derived information and the partitioning pattern to obtain the final prediction. For example, TPM is also considered a GPM.

[0807] Note that the proposed method can be applied to the merge candidate list construction process of inter-frame codec blocks (e.g., translational motion), affine codec blocks, TM codec blocks, or IBC codec blocks; or other motion candidate list construction processes (e.g., normal AMVP list, affine AMVP list, and IBC AMVP list).

[0808] Cost functions other than template matching cost are also applicable for motion candidate re-ranking.

[0809] W and H are the width and height of the current block (eg, luma block).

[0810] 1. The template / bilateral matching cost C can be calculated as f(C) before it is used to compare with another template matching cost.

[0811] a. In one example, f(C) = w*C, where w represents the cost factor.

[0812] b. In one example, f(C) = w*C + u.

[0813] c. In one example, f(C)=Shift((w*C),s).

[0814] d. In one example, w and / or u and / or s are integers.

[0815] e. In one example, before comparing the first template matching cost for the first motion candidate with the second template matching cost for the second motion candidate, the first template matching cost for the first motion candidate may be multiplied by a cost factor.

[0816] f. In one example, the cost factor proposed for a motion candidate may depend on the position of the candidate before re-ranking.

[0817] i. In one example, a motion candidate at the i-th position may be assigned a cost factor that is smaller than a cost factor of a motion candidate at the j-th position, where j>i, eg, j=i+1.

[0818] (i) In one example, the cost factor of the motion candidate at the i-th position is 4, and the cost factor of the motion candidate at the j-th position is 5.

[0819] (ii) In one example, the cost factor of the motion candidate at the i-th position is 1, and the cost factor of the motion candidate at the j-th position is 5.

[0820] ii. In one example, a motion candidate at the i-th position may be assigned a cost factor greater than a cost factor of a motion candidate at the j-th position, where j>i, eg, j=i+1.

[0821] iii. In one example, motion candidates in the pth group (eg, including M motion candidates) may be assigned a smaller cost factor than a cost factor of motion candidates in the qth group (eg, including N motion candidates), where q>p, eg, q=p+1.

[0822] (i) Alternatively, motion candidates in the pth group (eg, including M motion candidates) may be assigned a larger cost factor than motion candidates in the qth group (eg, including N motion candidates), where q>p, eg, q=p+1.

[0823] (ii) In one example, M may be equal to N. For example, M=N=2.

[0824] (iii) In one example, M may not be equal to N. For example, M=2, N=3.

[0825] (iv) In one example, the cost factor of the motion candidates of the pth group is 4, and the cost factor of the motion candidates of the qth group is 5.

[0826] (v) In one example, the cost factor of the motion candidates of the pth group is 1, and the cost factor of the motion candidates of the qth group is 5.

[0827] iv. In one example, the cost factor may not be applied to sub-block motion candidates.

[0828] v. In one example, the cost factor may not be applied to affine motion candidates.

[0829] vi. In one example, the cost factor may not be applied to the SbTMVP motion candidates.

[0830] g. In one example, the cost factor of a motion candidate in a group / position may be adaptive.

[0831] i. In one example, the cost factor of a motion candidate in a group / position may depend on the codec mode of the neighboring codec blocks.

[0832] (i) In one example, the cost factor of a SbTMVP merge candidate may depend on the number of neighboring affine codec blocks.

[0833] (ii) In one example, the adjacent codec block may include five spatially adjacent blocks (eg Figure 4 ) and / or temporal neighboring blocks (as shown Figure 7 at least one of the above (shown).

[0834] (iii) In one example, when spatially adjacent affine codec blocks (e.g. Figure 4 ) is 0, the cost factor of the SbTMVP merge candidate can be 0.2; when the spatially adjacent affine codec blocks (such as Figure 4 When the number of spatially adjacent affine codec blocks (such as Figure 4 When the number of merged candidates (as shown) is 2, the cost factor of the SbTMVP merge candidate may be 0.8. Otherwise, the cost factor of the SbTMVP merge candidate may be 1 (which means remains unchanged).

[0835] (iv) In one example, when spatially adjacent affine codec blocks (e.g. Figure 4 ) is 0, the cost factor of the SbTMVP merge candidate can be 0.2; when the spatially adjacent affine codec blocks (such as Figure 4 When the number of spatially adjacent affine codec blocks (such as Figure 4 When the number of SbTMVP merge candidates is greater than or equal to 2, the cost factor of the SbTMVP merge candidate can be 0.8.

[0836] (v) In one example, when spatially adjacent affine codec blocks (e.g. Figure 4 ) is 0, the cost factor of the SbTMVP merge candidate can be 2; when the spatially adjacent affine codec blocks (such as Figure 4 ) is 1, the cost factor of the SbTMVP merge candidate can be 5; when the spatially adjacent affine codec blocks (such as Figure 4 When the number of merge candidates (as shown) is 2, the cost factor of the SbTMVP merge candidate may be 8. Otherwise, the cost factor of the SbTMVP merge candidate may be 10 (wherein the cost factor of the affine merge candidate is 10).

[0837] (vi) In one example, when spatially adjacent affine codec blocks (e.g. Figure 4 When the number of spatially adjacent affine codec blocks (such as Figure 4 When the number of spatially adjacent affine codec blocks (such as Figure 4 When the number of merging candidates (as shown) is greater than or equal to 2, the cost factor of the SbTMVP merge candidate can be 8 (where the cost factor of the affine merge candidate is 10).

[0838] 2. For different codec modes, the subgroup size can be different.

[0839] a. The coding mode may include normal / sub-block / TM merge mode.

[0840] i. For normal merge mode, the subgroup size may be K1 (eg, K1=5).

[0841] ii. For the sub-block merging mode, the sub-group size may be K2 (eg, K2=3).

[0842] iii. For TM merging mode, the subgroup size may be K3 (eg, K3=3).

[0843] b. For sub-block merging mode, the sub-group size may be greater than or equal to the maximum number of sub-block merging candidates defined in the sps / picture / slice header (which means the whole list is re-ordered together).

[0844] c. For TM merge mode, the subgroup size can be greater than or equal to the maximum number of TM merge candidates defined in the sps / picture / slice header (which means the whole list is reordered together).

[0845] d. The subgroup size of a codec mode may depend on the maximum number of motion candidates in the codec mode.

[0846] e. The subgroup size of the sub-block merging mode can be adaptive according to the number of adjacent affine codec blocks.

[0847] i. In one example, the adjacent codec block may include five spatially adjacent blocks (eg Figure 4 ) and / or temporal neighboring blocks (as shown Figure 7 at least one of the above (shown).

[0848] ii. In one example, when spatially adjacent affine codec blocks (such as Figure 4 ) is 0 or 1, the subgroup size of the subblock merging mode can be 3; when the number of spatially adjacent affine codec blocks (such as Figure 4 When the number of sub-blocks (as shown) is greater than 1, the sub-group size of the sub-block merging mode can be 5.

[0849] 3. For different codec modes, the template size can be different.

[0850] a. The coding mode may include normal / sub-block / TM merge mode.

[0851] i. For normal merge mode, the template size may be K1 (eg, K1=1).

[0852] ii. For the sub-block merging mode, the template size may be K2 (eg, K2=1, 2, or 4).

[0853] iii. For TM merge mode, the template size may be K3 (eg, K3=1).

[0854] 4. Whether and / or how to reorder motion candidates may depend on the coding modes of neighboring coded blocks.

[0855] a. In one example, the adjacent coded blocks may include five spatially adjacent blocks (eg Figure 4 ) and / or temporal neighboring blocks (as shown Figure 7 at least one of the above (shown).

[0856] b. When the regular merge mode (such as Figure 4 When the number of spatially adjacent coded blocks under (as shown) is greater than or equal to K (eg, K=1), the conventional merge candidates may be reordered.

[0857] c. When the sub-block merging mode (such as Figure 4 When the number of spatially adjacent coded blocks under (as shown in FIG. 1 ) is greater than or equal to K (eg, K=1), the sub-block merging candidates may be reordered.

[0858] d. When the affine merge mode (such as Figure 4When the number of spatially adjacent coded blocks under (as shown) is greater than or equal to K (eg, K=1), the affine merging candidates may be reordered.

[0859] e. When the affine merge mode (such as Figure 4 When the number of spatially adjacent coded blocks under (as shown) is greater than or equal to K (eg, K=1, 2, or 3), the SbTMVP merge candidates may be reordered.

[0860] f. When TM merge mode (such as Figure 4 When the number of spatially adjacent coded blocks under (as shown in FIG. 1 ) is greater than or equal to K (eg, K=1), the TM merge candidates may be reordered.

[0861] 5. The HMVP motion candidates in the HMVP table can be reordered based on template / bilateral matching, etc.

[0862] a. In one example, HMVP motion candidates are assigned costs, and the HMVP candidates are adaptively reordered in descending order of the costs of the HMVP candidates.

[0863] i. In one example, the cost of the HMVP candidate may be a template matching cost.

[0864] b. In one example, HMVP motion candidates may be reordered before encoding and decoding the block.

[0865] i. In one example, HMVP motion candidates may be reordered before encoding an inter-coded block.

[0866] c. In one example, the HMVP motion candidates may be reordered in different ways according to the coding information of the current block and / or the neighboring blocks.

[0867] General Statement

[0868] 6. Whether and / or how to apply the above-disclosed method can be signaled at the sequence level / picture group level / picture level / slice level / tile group level, for example in the sequence header / picture header / SPS / VPS / DPS / DCI / PPS / APS / slice header / tile group header.

[0869] 7. Whether and / or how to apply the above disclosed methods can be signaled in PB / TB / CB / PU / TU / CU / VPDU / CTU / CTU row / slice / tile / sub-picture / other kinds of regions containing more than one sample or pixel.

[0870] 8. Whether and / or how to apply the method disclosed above may depend on the coded information, such as codec mode, block size, color format, single / dual tree partitioning, color component, slice / picture type.

[0871] 2.18 Adaptive GPM Candidate List

[0872] The term "block" may refer to a codec tree block (CTB), a codec tree unit (CTU), a codec block (CB), a CU, a PU, a TU, a PB, a TB, or a video processing unit including a plurality of samples / pixels. A block may be rectangular or non-rectangular.

[0873] It should be noted that the terms mentioned below are not limited to specific terms defined in existing standards. Any changes in codecs also apply. For example, the term "GPM" is used to represent any codec that derives two or more sets of motion information and uses the derived information and partitioning patterns / weighted masks to obtain the final prediction. For example, TPM is also considered a GPM.

[0874] Note that the proposed method can be applied to the merge candidate list construction process of inter-frame codec blocks (e.g., translational motion), affine codec blocks, TM codec blocks, GPM codec blocks, or IBC codec blocks; or other motion candidate list construction processes (e.g., normal AMVP list, affine AMVP list, and IBC AMVP list).

[0875] Cost functions other than template matching cost are also applicable for motion candidate re-ranking.

[0876] In the following, a template is a set of adjacent reconstruction / prediction samples that are adjacent or non-adjacent to the current block. The reference samples of the template (i.e., the reference template) are the mapping of the template in the reference picture according to the motion information of the current block. "Template above" indicates a template constructed by a set of adjacent reconstruction / prediction samples that are adjacent or non-adjacent to the current block and its reference template. "Left template" indicates a template constructed by a set of adjacent reconstruction / prediction samples that are adjacent or non-adjacent to the current block and its reference template. "Top left template" includes the top template and the left template.

[0877] Below, in one example, a GPM candidate list in which GPM candidates are directly derived from a conventional merge list (before or without motion refinement based on template matching) is called OGPMList; a GPM candidate list in which GPM candidates are refined by a first refinement method such as template matching using an upper template is called AGPMList; a GPM candidate list in which GPM candidates are refined by a second refinement method such as template matching using a left template is called LGPMList; and a GPM candidate list in which GPM candidates are refined by a third refinement method such as template matching using left and upper templates is called LAGPMList.

[0878] W and H are the width and height of the current block (eg, luma block).

[0879] 1. It is proposed that for a GPM codec block, the decoded candidate index may correspond to a candidate with a different candidate index in the candidate list for the GPM codec block.

[0880] a. Alternatively, furthermore, the candidate list constructed for the GPM codec block may be reordered before use, and the coded index corresponds to the reordered candidate list.

[0881] b. Alternatively, furthermore, for the first type of GPM codec blocks, the candidate list may be reordered, while for the second type of GPM codec blocks, the candidate list may not be reordered.

[0882] i. In one example, the first type is a template-based GPM codec block.

[0883] ii. In one example, the second type is a GPM codec block based on MMVD (eg, GMVD)

[0884] c. In addition, for the first type of GPM codec blocks, the candidate list can be reordered using the first rule, and for the second type of GPM codec blocks, the candidate list can be reordered using the second rule.

[0885] d. The reordering method for GPM codec blocks may be the same as the reordering method for non-GPM codec blocks.

[0886] i. The reordering method of GPM codec blocks may be different from the reordering method of non-GPM codec blocks.

[0887] 2. It is proposed that for a GPM codec block, the decoded candidate index may correspond to a candidate from a refined candidate list for the GPM codec block.

[0888] a. Alternatively, furthermore, the candidate list constructed for the GPM codec block may be refined before use, and the coded index corresponds to the refined candidate list.

[0889] b. Alternatively, furthermore, for the first type of GPM codec blocks, the candidate list may be refined, while for the second type of GPM codec blocks, the candidate list may not be refined.

[0890] i. In one example, the first type is a template-based GPM codec block.

[0891] ii. In one example, the second type is a GPM codec block based on MMVD (e.g., GMVD)

[0892] c. Alternatively, furthermore, for the first type of GPM codec blocks, the candidate list may be refined using a first rule, and for the second type of GPM codec blocks, the candidate list may be refined using a second rule.

[0893] d. The refinement method for GPM codec blocks may be the same as the refinement method for non-GPM codec blocks

[0894] i. The refinement method for GPM codec blocks may be different from the refinement method for non-GPM codec blocks

[0895] 3. In one example, the GPM candidates may be divided into subgroups. Whether and / or how to reorder the GPM candidates may depend on the subgroups of the GPM candidates.

[0896] a. In one example, the first subset may be reordered.

[0897] b. In one example, the last subgroup may not be reordered.

[0898] c. In one example, the last subgroup may not be reordered. However, if the last subgroup is also the first subgroup, it may be reordered.

[0899] d. Different subgroups can be reordered individually.

[0900] e. Two candidates in different subgroups cannot be compared and / or re-ordered.

[0901] f. If the first subgroup precedes the second subgroup, then after reordering, the first candidate in the first subgroup must be placed before the second candidate in the second subgroup.

[0902] 4. In one example, GPM candidates that are not included in the re-ordering process may be processed in a specified manner.

[0903] a. In one example, for the candidates that are not reordered, they are arranged in the merge candidate list according to the original order.

[0904] b. In one example, the candidates that are not reordered may be placed after the candidates that are to be reordered.

[0905] c. In one example, candidates that are not reordered may be placed before candidates that are to be reordered.

[0906] 5. The GPM candidate list to be re-ranked can be found at:

[0907] Case 1: A first candidate list prepared for two GPM partitions, which is used to derive a separate GPM candidate list for each GPM partition.

[0908] Case 2: Second GPM candidate list for each GPM partition. Typically, the second GPM candidates are derived from the first candidate list.

[0909] a. In one example, in case 1, the reordering method may be the same as the reordering method used for a conventional merge candidate list.

[0910] b. In one example, in case 1, if the corresponding candidate is bidirectionally predicted, the template matching method in the reordering method can be performed in a bidirectionally predicted manner.

[0911] c. In one example, in case 2, the template matching method in the reordering method cannot be performed in a bidirectional predictive manner.

[0912] d. In one example, in case 2, the reordering method can be the same for all GPM partitions.

[0913] e. In one example, in case 2, the reordering method may be different for different GPM partitions.

[0914] 6. In the above example, the GPM codec block can be a GPM codec block in merge mode or a GPM codec block in AMVP mode.

[0915] a. Alternatively, in addition, the above-mentioned merge candidates may be replaced by AMVP candidates.

[0916] General Statement

[0917] 7. Whether and / or how to apply the above-disclosed method can be signaled at the sequence level / picture group level / picture level / slice level / tile group level, for example in the sequence header / picture header / SPS / VPS / DPS / DCI / PPS / APS / slice header / tile group header.

[0918] 8. Whether and / or how to apply the above disclosed methods to signal multiple samples or pixels at a region containing a PB / TB / CB / PU / TU / CU / VPDU / CTU / CTU row / slice / tile / sub-picture / other kind.

[0919] 9. Whether and / or how to apply the above disclosed method may depend on the coded information, such as codec mode, block size, color format, single / dual tree partitioning, color component, slice / picture type.

[0920] 2.19 Adaptive GPM Candidate List

[0921] The term "block" may refer to a codec tree block (CTB), a codec tree unit (CTU), a codec block (CB), a CU, a PU, a TU, a PB, a TB, or a video processing unit including a plurality of samples / pixels. A block may be rectangular or non-rectangular.

[0922] It should be noted that the terms mentioned below are not limited to specific terms defined in existing standards. Any changes in codecs also apply. For example, the term "GPM" is used to represent any codec that derives two or more sets of motion information and uses the derived information and partitioning patterns / weighted masks to obtain the final prediction. For example, TPM is also considered a GPM.

[0923] Note that the proposed method can be applied to the merge candidate list construction process of inter-frame codec blocks (e.g., translational motion), affine codec blocks, TM codec blocks, GPM codec blocks, or IBC codec blocks; or other motion candidate list construction processes (e.g., normal AMVP list, affine AMVP list, and IBC AMVP list).

[0924] Cost functions other than template matching cost are also applicable for motion candidate re-ranking.

[0925] In the following, a template is a set of adjacent reconstruction / prediction samples that are adjacent or non-adjacent to the current block. The reference samples of the template (i.e., the reference template) are the mapping of the template in the reference picture according to the motion information of the current block. "Template above" indicates a template constructed by a set of adjacent reconstruction / prediction samples that are adjacent or non-adjacent to the current block and its reference template. "Left template" indicates a template constructed by a set of adjacent reconstruction / prediction samples that are adjacent or non-adjacent to the current block and its reference template. "Top left template" includes the top template and the left template.

[0926] Below, in one example, a GPM candidate list in which GPM candidates are directly derived from a conventional merge list (before or without motion refinement based on template matching) is called OGPMList; a GPM candidate list in which GPM candidates are refined by a first refinement method such as template matching using an upper template is called AGPMList; a GPM candidate list in which GPM candidates are refined by a second refinement method such as template matching using a left template is called LGPMList; and a GPM candidate list in which GPM candidates are refined by a third refinement method such as template matching using left and upper templates is called LAGPMList.

[0927] Regarding the GPM candidate types in the original GPM candidate list, the GPM candidates derived in the first step of the GPM candidate list construction process in Section 2.29 are called GPM parity-based candidates; the GPM candidates derived in the second step of the GPM candidate list construction process in Section 2.29 are called GPM anti-parity-based candidates; and the GPM candidates derived in the third step of the GPM candidate list construction process in Section 2.29 are called GPM-filled candidates.

[0928] W and H are the width and height of the current block (eg, luma block).

[0929] 1. In one example, if the codec mode is GPM, the merge candidates may be reordered.

[0930] a. In one example, merge candidates in OGPMList may be reordered.

[0931] i. In one example, at least two merge candidates in OGPMList may be reordered.

[0932] ii. In one example, at least one type of template may be used for OGPMList reordering.

[0933] iii. Alternatively, merge candidates in the OGPMList may not be reordered.

[0934] iv. In one example, the first type of template may include only neighboring samples to the left of the current block.

[0935] v. In one example, the second type of template may include only neighboring samples above the current block.

[0936] vi. In one example, the third type of template may include neighboring samples to the left and above the current block.

[0937] vii. The reordering process can be called after the parsing process but before the MV reconstruction process.

[0938] b. In one example, the merge candidates in AGPMList may be reordered.

[0939] i. In one example, at least two merge candidates in the AGPMList may be reordered.

[0940] ii. In one example, at least one type of template may be used for AGPMList reordering.

[0941] iii. In one example, the first type of template may include only neighboring samples above the current block.

[0942] iv. In one example, the second type of template may include neighboring samples to the left and above the current block.

[0943] c. In one example, the merge candidates in LGPMList may be reordered.

[0944] i. In one example, at least two merge candidates in LGPMList may be reordered.

[0945] ii. In one example, at least one type of template can be used for LGPMList reordering.

[0946] iii. In one example, the first type of template may include only neighboring samples to the left of the current block.

[0947] iv. In one example, the second type of template may include neighboring samples to the left and above the current block.

[0948] d. In one example, the merge candidates in LAGPMList may be reordered.

[0949] i. In one example, at least two merge candidates in the LAGPMList may be reordered.

[0950] ii. In one example, at least one type of template may be used for LAGPMList reordering.

[0951] iii. In one example, the first type of template may include only neighboring samples to the left of the current block.

[0952] iv. In one example, the second type of template may include only neighboring samples above the current block.

[0953] v. In one example, the third type of template may include neighboring samples to the left and above the current block.

[0954] e. In one example, whether and / or how to reorder merge candidates in the GPM list may depend on codec information.

[0955] i. In one example, whether to reorder merge candidates in the GPM list may depend on whether template matching based motion refinement is applied to a GPM partition or to two GPM partitions (ie, a GPM codec CU).

[0956] (i) For example, if the motion of one GPM partition or two GPM partitions (ie, GPM codec CUs) is not refined based on template matching (eg, the template matching flag is equal to false), the corresponding GPM lists may not be reordered.

[0957] a) For example, if the GPM partition is coded using the merge candidates in OGPMList (eg, no motion refinement is applied), the merge candidates in OGPMList may not be reordered.

[0958] (ii) For example, if the motion of one GPM partition or two GPM partitions (ie, GPM codec CUs) is refined based on template matching (eg, template matching flag equals true), the corresponding GPM lists may be reordered.

[0959] a) For example, if the GPM partition is encoded using the merge candidates in the AGPMList (eg, a template matching motion refinement method using an upper template is applied), the merge candidates in the AGPMList may be reordered.

[0960] b) For example, if the GPM partition is encoded and decoded using the merge candidates in the LGPMList (eg, a template matching motion refinement method using a left template is applied), the merge candidates in the LGPMList may be reordered.

[0961] c) For example, if the GPM partition is encoded and decoded using the merge candidates in the LAGPMList (eg, a template matching motion refinement method using left and above templates is applied), the merge candidates in the LAGPMList may be reordered.

[0962] ii. In one example, how to reorder the merge candidates in the GPM list may depend on the GPM partition information (eg, partition mode, partition angle, partition distance, etc.).

[0963] (i) For example, when the current GPM partition is divided by the first partition angle (or partition mode, or partition distance, etc.), the upper template can be used for merging candidate reordering.

[0964] (ii) For example, in the case where the current GPM partition is split by a second partition angle (or partition mode, or partition distance, etc.), the left template can be used for merge candidate reordering.

[0965] (iii) For example, in the case where the current GPM partition is divided by a third partition angle (or partition mode, or partition distance, etc.), the left and upper templates can be used for merge candidate reordering.

[0966] (iv) For example, the type of template may be specified corresponding to the first / second / third division angle (or division mode, or division distance, etc.).

[0967] (v) For example, at least one lookup table (i.e., mapping table) is used to map which specified division angle (or division method, or division distance, etc.) corresponds to which type of template (e.g., upper template, left template, or upper left template).

[0968] f. In one example, merge candidates in OGPMList may not be reordered, while merge candidates in AGPMList and / or LGPMList and / or LAGPMList may be reordered.

[0969] 2. The merge candidates can be adaptively reordered in the final GPM candidate list according to one or some criteria.

[0970] a. In one example, the GPM candidate list can be

[0971] i.OGPM List

[0972] ii.AGPM List

[0973] iii.LGPM List

[0974] iv.LAGPM List

[0975] b.GPM candidates can be divided into several subgroups.

[0976] i. For example, the number of GPM candidates in a sub-group (eg, X=3 or 5 or any other integer value) may be predefined.

[0977] c. In one example, part or all of the current GPM candidate list construction process is first called, and then the candidates in the GPM list are reordered.

[0978] i. Alternatively, the candidates in the first subgroup may be reordered and they should be added before those in the second subgroup, where the first subgroup is added before the second subgroup.

[0979] ii. The construction process may include pruning methods.

[0980] d. In one example, the merge candidates may be adaptively rearranged before being retrieved.

[0981] i. In one example, the process of adaptively arranging merge candidates may be processed before obtaining merge candidates to be used in the motion compensation process.

[0982] e. The criterion can be based on template matching cost.

[0983] i. In one example, the cost function between the current template and the reference template can be

[0984] (i) SAD / MR-SAD

[0985] (ii) SATD / MR-SATD

[0986] (iii) SSD / MR-SSD

[0987] (iv) SSE / MR-SSE

[0988] (v) Weighted SAD / Weighted MR-SAD

[0989] (vi) Weighted SATD / Weighted MR-SATD

[0990] (vii) Weighted SSD / Weighted MR-SSD

[0991] (viii) Weighted SSE / Weighted MR-SSE

[0992] (ix) Gradient information

[0993] 3. When deriving two motions for two geometric partitions, the process can be

[0994] a. In one example, if TM is not applied to a partition, the motion may be derived from the signaled merge index from the OGPMList / reordered OGPMList.

[0995] b. In one example, if TM is applied to one partition, the motion can be derived based on the signaled merge index from AGPMList / reorderedAGPMList or LGPMList / reorderedLGPMLIst or LAGPMList / reorderedLAGPMList, depending on the partition angle and partition index.

[0996] i. In one example, if the partition angle is X (eg, 0), for the first partition, AGPMList / reordered AGPMList will be used; for the second partition, LAGPMList / reordered LAGPMLIst will be used.

[0997] c. In one example, if TM is applied to one partition, the motion can be derived from the signaled merge index from the AGPMList / reordered AGPMList.

[0998] d. In one example, if TM is applied to one partition, the motion can be derived from the signaled merge index from LGPMList / reordered LGPMLIst.

[0999] e. In one example, if TM is applied to a partition, the motion can be derived from the signaled merge index from LAGPMList / reordered LAGPMLIst

[1000] 4. Whether and / or how to re-rank GPM candidates may depend on the category of the GPM candidates.

[1001] a. In one example, only GPM parity-based candidates can be reordered.

[1002] b. In one example, only GPM parity-based and GPM anti-parity-based candidates are reordered.

[1003] c. In one example, candidates that populate the GPM may not be reordered.

[1004] d. In one example, two candidates in different GPM lists cannot be compared and / or re-ordered.

[1005] e. In one example, only the top N GPM candidates may be reordered.

[1006] i. In one example, N is set equal to 5.

[1007] 5. In the above example, the GPM codec block can be a GPM codec block in merge mode or a GPM codec block in AMVP mode.

[1008] a. Alternatively, furthermore, the above merge candidates may be replaced by AMVP candidates.

[1009] General Statement

[1010] 6. Whether and / or how to apply the above-disclosed method can be signaled at the sequence level / picture group level / picture level / slice level / tile group level, for example in the sequence header / picture header / SPS / VPS / DPS / DCI / PPS / APS / slice header / tile group header.

[1011] 7. Whether and / or how to apply the above disclosed methods to signal multiple samples or pixels at a region containing PB / TB / CB / PU / TU / CU / VPDU / CTU / CTU row / slice / tile / sub-picture / other types.

[1012] 8. Whether and / or how to apply the above disclosed method may depend on the coded information, such as codec mode, block size, color format, single / dual tree partitioning, color component, slice / picture type.

[1013] 2.20 Hash-based motion estimation for picture content encoding and decoding

[1014] The VTM reference software uses hash-based motion estimation to handle the sometimes large and irregular motion in the picture content. For each reference picture, a hash table corresponding to 4×4 to 64×64 block sizes is generated using a bottom-up approach as follows:

[1015] - For each 2×2 block, a block hash value is calculated directly from the original sample values ​​(for 4:2:0 chroma format, luma samples are used; for 4:4:4 chroma format, luma sample values ​​are used). A cyclic redundancy check (CRC) value is used as the hash value.

[1016] - For 4×4, 8×8, 16×16, 32×32, and 64×64 blocks, the hash value of the current block is the CRC value calculated based on the CRC values ​​of its four sub-blocks.

[1017] In order to be able to efficiently search for matching blocks, an inverted index structure is used, in which a hash value is used as an index into the table, and the table entry contains all blocks with the same hash value as the corresponding table index. The blocks corresponding to a given table index are stored as a linked list. Two CRC values ​​are calculated for each block, one 16-bit hash value and the other 24-bit hash value. The two hash values ​​are calculated in a similar way, but using different CRC truncation polynomials. The first 16-bit CRC value is used as an inverted index. The second 24-bit hash value is stored with the block to resolve hash conflicts in the event that multiple matching blocks are found. In order to reduce the length of the hash table, the hash values ​​of all "simple" blocks (defined as blocks with only one sample value in each row or column) are excluded from the hash table.

[1018] In motion estimation, if the current block is a square block (except for 128×128 blocks), its hash value is calculated. Then, the encoder queries the corresponding hash table. If a hash match is found, the matched block is used as a reference. If the current block is a rectangular block of size N×M (and without loss of generality assume that M>N), it will be divided into several non-overlapping square sub-blocks of size N×N. Fig. 27 An example diagram 2700 is shown, which shows motion estimation for a rectangular block with hash values ​​for square sub-blocks. The encoder will find the first non-simple square sub-block and calculate its hash value. The encoder looks up the hash value of this N×N square sub-block in the hash table corresponding to the N×N block size. One or more matching reference blocks are considered as reference block candidates. For each matching reference block candidate, the encoder will continue to check the remaining square sub-blocks (i.e. Fig. 27 The white area after the first non-simple square sub-block depicted in the figure) is equal to the hash value of the square sub-blocks adjacent to the reference block candidate. If the hash values ​​of all square sub-blocks match, the reference block candidate will be considered a valid reference block.

[1019] For inter codecs, a hash-based motion search is performed before testing all codec modes. In addition, the encoder will reuse the MV of the hash mode as a starting point candidate in the normal motion estimation process. If a hash-based motion vector exists, which indicates that the block most likely contains picture content, fractional motion estimation is skipped.

[1020] To speed up the encoder, codec modes other than the skip and merge parts of ETM_MERGE_SKIP, ETM_AFFINE, and ETM_MERGE_GPM modes and finer-grained block splitting are skipped if all of the following conditions are met:

[1021] - Current block size is 64×64, 128×64 or 64×128.

[1022] - The same reference block was found in the reference image.

[1023] - The QP of the reference picture is not greater than the QP of the current picture.

[1024] 2.21 Luminance Mapping with Chroma Scaling (LMCS)

[1025] In VVC, a codec tool called "2.21 Luma mapping with chroma scaling (LMCS)" is added as a new processing block before the loop filter. LMCS has two main components: 1) In-loop mapping of luma components based on an adaptive piecewise linear model; 2) For chroma components, luma-dependent chroma residual scaling is applied. Fig.28 An example luma mapping and chroma scaling architecture 2800 is shown. Fig.28 shows the LMCS architecture from the decoder perspective. Fig.28 The light blue shaded blocks in indicate where processing is applied in the mapped domain; these include inverse quantization, inverse transform, luma intra prediction, and adding the luma prediction to the luma residual. Fig.28 The unshaded blocks in indicate where processing is applied in the original (i.e., unmapped) domain; these include loop filtering (e.g., deblocking, ALF, and SAO), motion compensated prediction, chroma intra prediction, adding the chroma prediction to the chroma residual, and storing the decoded image as a reference image. Fig.28 The light yellow shaded blocks in are the new LMCS functional blocks, including the forward and reverse mapping of the luma signal and the luma-dependent chroma scaling process. Like most other tools in VVC, LMCS can be enabled / disabled at the sequence level using the SPS flag.

[1026] 2.21.1 Brightness Mapping Using Piecewise Linear Model

[1027] The in-loop mapping of the luma component adjusts the dynamic range of the input signal by redistributing the codewords within the dynamic range to improve compression efficiency. Luma mapping uses a forward mapping function FwdMap and a corresponding inverse mapping function InvMap. The FwdMap function is signaled using a piecewise linear model with 16 equal segments. The InvMap function does not need to be signaled, but is derived from the FwdMap function.

[1028] The luma mapping model is signaled in the adaptive parameter set (APS) syntax structure by setting aps_params_type to 1 (LMCS_APS). A maximum of 4 LMCS APS can be used in the encoded video sequence. Only 1 LMCS APS can be used for a picture. The luma mapping model is signaled using a piecewise linear model. The piecewise linear model divides the dynamic range of the input signal into 16 equal segments, and for each segment, its linear mapping parameters are expressed using the number of codewords assigned to the segment. Take 10-bit input as an example. By default, each of the 16 segments is assigned 64 codewords. The signaled number of codewords is used to calculate the scaling factor and adjust the mapping function for the segment accordingly. At the slice level, the LMCS enable flag is signaled to indicate if Fig.28 Whether the LMCS process shown is applied to the current slice. If LMCS is enabled for the current slice, the aps_id signal is sent in the slice header to identify the APS carrying the brightness mapping parameters.

[1029] Each i-th part of the FwdMap piecewise linear model, i=0…15 is defined by two input pivot points InputPivot[] and two output (mapped) pivot points MappedPivot[].

[1030] InputPivot[] and MappedPivot[] are calculated as follows (assuming 10-bit video):

[1031] 1) Organization CW = 64

[1032] 2) For i = 0:16, InputPivot[i] = i * OrgCW

[1033] 3) For i = 0:16, MappedPivot[i] is calculated as follows:

[1034] MappedPivot[0] = 0;

[1035] for(i=0;i<16;i++)

[1036] MappedPivot[i+1]=MappedPivot[i]+SignalledCW[i]

[1037] where SignalledCW[i] is the number of codewords signaled for the ith segment.

[1038] like Fig.28 As shown, for the inter-frame coding block, motion compensation prediction is performed in the mapping domain. In other words, the motion compensation prediction block Y is calculated based on the reference signal in the DPB.pred Finally, the FwdMap function is applied to map the brightness prediction block in the original domain to the mapping domain Y′ pred =FwdMap(Y pred ). For intra-frame coded blocks, the FwdMap function is not applied because intra-frame prediction is performed in the mapping domain. r Finally, the InvMap function is applied to convert the reconstructed brightness value in the mapping domain back to the reconstructed brightness value in the original domain. The InvMap function is applicable to intra-frame coded luminance blocks and inter-frame coded luminance blocks.

[1039] The brightness mapping process (forward and / or inverse mapping) can be implemented using a lookup table (LUT) or using on-the-fly calculations. If LUTs are used, then the FwdMapLUT and InvMapLUT can be pre-calculated and pre-stored for use at the tile group level, and the forward and inverse mapping can be simply implemented as FwdMap(Y pred )=FwdMapLUT[Y pred ] and InvMap(Y r )=InvMapLUT[Y r ]. Alternatively, immediate calculation can be used. Take the forward mapping function FwdMap as an example. To find out the part to which the brightness sample belongs, the sample value is shifted right by 6 bits (equivalent to 16 equal parts). Then, the linear model parameters of the part are retrieved and applied immediately to calculate the mapped brightness value. Let i be the part index, a1, a2 are InputPivot[i] and InputPivot[i+1] respectively, and b1, b2 are MappedPivot[i] and MappedPivot[i+1] respectively. The FwdMap function is evaluated as follows:

[1040] FwdMap(Y pred )=((b2-b1) / (a2-a1))*(Y pred -a1)+b1

[1041] The InvMap function can be calculated on the fly in a similar way. Typically, the fragments in the mapped domain are of unequal size, so the most straightforward inverse mapping process will require comparisons in order to find out which fragment the current sample value belongs to. This comparison increases the complexity of the decoder. Therefore, VVC imposes a bitstream constraint on the value of the output pivot point MappedPivot[i], as shown below. Assume that the range of the mapped domain (for 10-bit video, the range is [0,1023]) is divided into 32 equal fragments. If MappedPivot[i] is not a multiple of 32, then MappedPivot[i+1] and MappedPivot[i] cannot belong to the same fragment among the 32 fragments of the same size, that is, MappedPivot[i+1]>>(BitDepth Y -5) must not be equal to MappedPivot[i]>>(BitDepth Y Due to this bitstream constraint, the InvMap function can also be performed using a simple right shift of 5 bits (corresponding to 32 equally sized blocks) in order to find out which fragment a sample value belongs to.

[1042] 2.21.2 Luma-Dependent Chroma Residual Scaling

[1043] Chroma residual scaling is intended to compensate for the interaction between luma signals and their corresponding chroma signals. Whether chroma residual scaling is enabled is also signaled on the slice level. If luma mapping is enabled, an additional flag is signaled to indicate whether luma-dependent chroma residual scaling is enabled. When luma mapping is not used, luma-dependent chroma residual scaling is disabled. Additionally, luma-dependent chroma residual scaling is always disabled for chroma blocks with area less than or equal to 4.

[1044] The chroma residual scaling depends on the average of the neighboring luma samples reconstructed on the top and / or left of the current VPDU. If the current CU is Inter 128x128, Inter 128x64, and Inter 64x128, the chroma residual scaling factor derived for the CU associated with the first VPDU is used for all chroma transform blocks in that CU. Denote avgYr as the average of the neighboring luma samples reconstructed (see Fig.28 ). ScaleInv The value of is calculated as follows:

[1045] 1) Find the index Y of the piecewise linear model to which avgYr belongs according to the InvMap function Idx .

[1046] 2) C ScaleInv =cScaleInv[Y Idx], where cScaleInv[] is a 16-segment LUT pre-computed based on the value of SignalledCW[i] and the offset value signaled in APS for the chroma residual scaling process.

[1047] Unlike brightness mapping performed based on samples, C ScaleInv is a constant value for the entire chrominance block. ScaleInv , chroma residual scaling is applied as follows:

[1048] Encoder side: C ResScale =C Res *C Scale =C Res / C ScaleInv

[1049] Decoder side: C Res =C ResScale / C Scale =C ResScale *C ScaleInv

[1050] 2.21.3 Encoder-side LMCS parameter estimation

[1051] A non-normative reference implementation is provided in the VTM encoder to estimate the LMCS model parameters. Since the VTM anchor handles SDR, HDR PQ, and HDR HLG differently, the reference algorithm in VTM13 is designed differently for SDR, HDR PQ, and HDR HLG sequences. For SDR and HDR HLG sequences, the encoder algorithm is based on local luminance variance and optimized for the PSNR metric. For HDR PQ sequences, the encoder algorithm is based on luminance values ​​and optimized for the wPSNR (weighted PSNR) metric.

[1052] 2.22 Merge Mode with MVD (MMVD)

[1053] In addition to the merge mode that uses implicitly derived motion information directly for prediction sample generation of the current CU, merge mode with motion vector difference (MMVD) is introduced in VVC. The MMVD flag is signaled immediately after the regular merge flag is sent to specify whether the MMVD mode is used for the CU.

[1054] In MMVD, after a merge candidate is selected, it is further refined by signaled MVD information. Further information includes a merge candidate flag, an index indicating the magnitude of motion, and an index indicating the direction of motion. In MMVD mode, one of the first two candidates in the merge list is selected as the MV basis. The MMVD candidate flag is signaled to specify which one to use between the first and second merge candidates.

[1055] The distance index specifies the motion magnitude information and indicates a predefined offset from the starting point. Fig.29 An example of an MMVD search point is shown. Fig.29 As shown in Table 2, the offset is added to the horizontal component or the vertical component of the starting MV.

[1056] Table 2. Relationship between distance index and predefined offsets

[1057]

[1058] The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate four directions, as shown in Table 3. Note that the meaning of the MVD symbol may vary depending on the information of the starting MV. When the starting MV is a unidirectional prediction MV or a bidirectional prediction MV, where both lists point to the same side of the current picture (i.e., both referenced POCs are greater than the POC of the current picture or both are less than the POC of the current picture), the symbol in Table 3 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectional prediction MV, where the two MVs point to different sides of the current picture (i.e., one referenced POC is greater than the POC of the current picture and the other referenced POC is less than the POC of the current picture), and the difference in POC in list 0 is greater than or equal to that in Table 1, the symbol in Table 3 specifies the sign of the MV offset added to the list0 MV component of the starting MV, and the sign of list1 MV has the opposite value. Otherwise, if the POC difference in list 1 is greater than that in list 0, the symbol in Table 3 specifies the sign of the MV offset added to the list1 MV component of the starting MV, and the sign of list0 MV has the opposite value.

[1059] The MVD is scaled based on the difference in POC in each direction. If the POC difference in both lists is the same, no scaling is required. Otherwise, if the POC difference in list 0 is greater than the POC difference in list 1, the MVD of list 1 is scaled by defining the POC difference of L0 as td and the POC difference of L1 as tb, as Figure 6 If the POC difference of L1 is greater than L0, the MVD of list 0 is scaled in the same way. If the starting MV is unidirectionally predicted, the MVD will be added to the available MVs.

[1060] MMVD is also known as Ultimate Motion Vector Expression (UMVE).

[1061] Table 3. Signs of MV offsets specified by direction index

[1062] Direction Index 00 01 10 11 x-axis + – N / A N / A y-axis N / A N / A + –

[1063] 2.23 Triangulation for Inter-frame Prediction

[1064] In VVC, triangulation partition mode (TPM) is supported for inter prediction. Triangulation partition mode is only applicable to CUs of 8×8 or larger. Triangulation partition mode is signaled using a CU-level flag as a merge mode. Other merge modes include normal merge mode, MMVD mode, CIIP mode, and sub-block merge mode.

[1065] When this mode is used, the CU is evenly split into two triangular partitions using diagonal or anti-diagonal splits ( Fig.30 ).exist Fig.30 In the illustrated inter prediction based on triangle partitioning, CU 3010 is partitioned using diagonal partitioning when triangleDir is set equal to 0, and CU 3020 is partitioned using anti-diagonal partitioning when triangleDir is set equal to 1. Each triangle partition in the CU uses its own motion for inter prediction; only unidirectional prediction is allowed for each partition, i.e., one motion vector and one reference index per partition. Unidirectional prediction motion constraints are applied to ensure that only two motion compensated predictions are required for each CU, as with conventional bidirectional prediction. The unidirectional prediction motion for each partition is derived using the process described in Section 2.23.1.

[1066] If the triangulation mode is used for the current CU, a flag indicating the direction of the triangulation (diagonal or anti-diagonal) and two merge indices (one for each partition) are further signaled. The number of maximum TPM candidate sizes is explicitly signaled at the slice level, and the syntax for specifying the TMP merge indices is binarized. After predicting each triangulation, a hybrid process with adaptive weights is used to adjust the sample values ​​along the diagonal or anti-diagonal edges. This is the prediction signal for the entire CU, and the transform and quantization process will be applied to the entire CU as in other prediction modes. Finally, the motion field of the CU predicted using the triangulation mode is stored as in 2.23.3.

[1067] The triangulation mode is not used in conjunction with SBT, i.e., when the signaled triangulation mode is equal to 1, cu_SBT_flag is inferred to be 0 if not signaled.

[1068] 2.23.1 One-way prediction candidate list construction

[1069] The unidirectional prediction candidate list is directly derived from the merge candidate list constructed according to the extended merge prediction process in Section 2.1. Denote n as the index of the unidirectional prediction motion in the triangular unidirectional prediction candidate list. The LX motion vector of the nth extended merge candidate, with X equal to the parity of n, is used as the nth unidirectional prediction motion vector of the triangular partitioning pattern. Fig.31 Figure 3100 shows unidirectional prediction MV selection for triangulation mode. These motion vectors are Fig.31 If there is no corresponding LX motion vector of the n-th extended merge candidate, the L(1-x) motion vector of the same candidate is used as the unidirectional prediction motion vector of the triangulation mode.

[1070] 2.23.2 Blending along triangulation edges

[1071] After each triangulation is predicted using its own motion, blending is applied to the two prediction signals to derive samples around diagonal or anti-diagonal edges. Fig.32 Graph 3200 showing weights used in the blending process. The following weights are used in the blending process:

[1072] {7 / 8, 6 / 8, 5 / 8, 4 / 8, 3 / 8, 2 / 8, 1 / 8} for luminance, {6 / 8, 4 / 8 and 2 / 8} for chrominance, e.g.

[1073] Fig.32 shown.

[1074] 2.23.3 Sports Field Storage

[1075] Figures 33A-33C Three example MV storage areas are shown with triangleDir equal to 0, where Fig.33A A 32×16 block 3310 is shown, Fig.33B A 16×32 block 3320 is shown, and Fig.33C A 32×32 block 3330 is shown. The motion vector of the triangular prediction unit ( Figures 33A-33C Mv1 and Mv2 in CU are stored in 4×4 grids. For each 4×4 grid, a unidirectional prediction or bidirectional prediction motion vector is stored according to the position of the 4×4 grid in the CU. Figures 33A-33C As shown, the unidirectional prediction motion vector Mv1 or Mv2 is stored for a 4×4 grid located in a non-weighted area (i.e., not located at a diagonal edge). On the other hand, a bidirectional prediction motion vector is stored for a 4×4 grid located in a weighted area. The bidirectional prediction motion vector is derived from Mv1 and Mv2 according to the following rules:

[1076] 1) If Mv1 and Mv2 are from different reference picture lists (one from L0 and the other from L1), then Mv1 and Mv2 are simply combined to form a portable prediction motion vector.

[1077] 2) Otherwise, if Mv1 and Mv2 are from the same list, only the unidirectional predicted motion Mv2 is stored.

[1078] 2.23.4 Motion vector storage process in triangle merging mode (alternative description)

[1079] The variables numSbX and numSbY, which specify the number of 4x4 blocks in the horizontal and vertical directions in the current codec block, are set equal to numSbX=cbWidth>>2 and numSbY=cbHeight>>2.

[1080] Where cbWidth and cbHeight specify the width and height of the current codec block in luma samples.

[1081] The variable minSb is set equal to min(numSbX, numSbY)-1.

[1082] The variable cbRatio is derived as follows:

[1083] cbRatio=(cbWidth>cbHeight)? (cbWidth / cbHeight):(cbHeight / cbWidth)

[1084] For each 4x4 sub-block at sub-block index (xSbIdx, ySbIdx), where xSbIdx = 0..numSbX-1, and ySbIdx = 0..numSbY-1, the following rules apply:

[1085] - The variables xIdx and yIdx are derived as follows:

[1086] -xIdx=(cbWidth>cbHeight)? (xSbIdx / cbRatio):xSbIdx

[1087] -yIdx=(cbWidth>cbHeight)? ySbIdx:(ySbIdx / cbRatio)

[1088] The variable sType is derived as follows:

[1089] - If triangleDir is equal to 0, the following applies:

[1090] sType=(xIdx==yIdx)? 2:((xIdx>yIdx)?0:1)

[1091] - Otherwise (triangleDir is equal to 1), the following rules apply:

[1092] sType=(xIdx+yIdx==minSb)? 2:((xIdx+yIdx <minSb)?0:1)

[1093] Where triangleDir specifies the division direction.

[1094] like Figures 33A-33C As shown, sType equals 0 corresponds to the P1 area; sType equals 1 corresponds to the P2 area; sType equals 2 corresponds to the weighted area.

[1095] The motion information of the P1 region is represented as (Mv1, refIdx1); the motion information of the P2 region is represented as (Mv2, refIdx2).

[1096] - Depending on the value of sType, the following assignments are made:

[1097] - If sType is equal to 0, the motion information of the 4×4 sub-block is (Mv1, refIdx1).

[1098] - If sType is equal to 1, the motion information of the 4×4 sub-block is (Mv2, refIdx2).

[1099] - Otherwise (sType is equal to 2), the following rules apply:

[1100] 1) If Mv1 and Mv2 come from different reference picture lists (one from L0 and the other from L1), then Mv1 and Mv2 are simply combined to form a bi-directional prediction motion vector.

[1101] 2) Otherwise, if Mv1 and Mv2 are from the same list, only the unidirectional predicted motion Mv2 is stored.

[1102] 2.24 Geometric Partitioning Model (GPM)

[1103] In VVC, geometric partitioning mode is supported for inter prediction. Use CU level flag as a merge mode to signal geometric partitioning mode. Other merge modes include normal merge mode, MMVD mode, CIIP mode and sub-block merge mode. For each possible CU size w×h=2 m ×2 n (m,n∈{3…6}), the geometric partitioning mode supports a total of 64 partitions, excluding 8x64 and 64x8.

[1104] Fig.34Figure 3400 shows an example of a GPM segmentation grouped by the same angle. When this mode is used, the CU is divided into two parts by a geometrically positioned straight line ( Fig.34 ). The position of the partition line is mathematically derived from the angle and offset parameters of the specific partition. Each part of the geometric partition in the CU uses its own motion for inter-frame prediction; each partition only allows unidirectional prediction, that is, each part has a motion vector and a reference index. Unidirectional prediction motion constraints are applied to ensure that, as with traditional bidirectional prediction, only two motion compensated predictions are required per CU. The unidirectional prediction motion for each partition is derived using the process described in Section 2.24.1.

[1105] If a geometric partitioning mode is used for the current CU, a geometric partitioning index indicating the partitioning mode (angle and offset) of the geometric partitioning and two merge indices (one for each partition) are further transmitted through a signal. The number of maximum GPM candidate sizes is explicitly transmitted through a signal in the SPS, and the syntax binarization for the GPM merge indices is specified. After predicting each part of the geometric partitioning, the sample values ​​along the edges of the geometric partitioning are adjusted using the hybrid process with adaptive weights in 2.24.2. This is the prediction signal for the entire CU, and the transformation and quantization processes will be applied to the entire CU as in other prediction modes. Finally, the motion field of the CU predicted using the geometric partitioning mode is stored as shown in Section 2.24.3.

[1106] 2.24.1 One-way prediction candidate list construction

[1107] The unidirectional prediction candidate list is derived directly from the merge candidate list constructed according to the extended merge prediction process in 3.4.1. Denote n as the index of the unidirectional prediction motion in the geometric unidirectional prediction candidate list. The LX motion vector of the nth extended merge candidate, with X equal to the parity of n, is used as the nth unidirectional prediction motion vector of the geometric partitioning mode. Fig.35 Figure 3500 shows unidirectional MV selection for geometric partitioning mode. These motion vectors are Fig.35 If the corresponding LX motion vector of the n-th extended merge candidate does not exist, the L(1-X) motion vector of the same candidate is used as the unidirectional prediction motion vector of the geometric partitioning mode.

[1108] 2.24.2 Blending Along Geometric Partition Edges

[1109] After predicting each part of the geometric partition using its own motion, blending is applied to the two prediction signals to derive samples around the geometric partition edges. The blending weight for each position of the CU is derived based on the distance between the individual position and the partition edge.

[1110] The distance from a location to the partition edge is derived as:

[1111]

[1112]

[1113]

[1114] where i,j are the angle and offset indices of the geometric partition, which depend on the geometric partition index transmitted by the signal. x,j and ρ y,j The sign of depends on the angle index i.

[1115] The weight of each part of the geometric partition is derived as follows:

[1116] wIdxL(x,y)=partIdx? 32+d(x,y):32-d(x,y) (2-5)

[1117]

[1118] w 1 (x,y)=1-w 0 (x,y) (2-7)

[1119] partIdx depends on the angle index i. Fig.36 shows the warping weight w using the geometric partitioning pattern 0 3600. The weight w 0 An example of Fig.36 Shown in.

[1120] 2.24.3 Motion Field Storage for Geometry Partitioning Mode

[1121] Mv1 from the first part of the geometric partitioning, Mv2 from the second part of the geometric partitioning, and the combination Mv of Mv1 and Mv2 are stored in the motion field of the geometric partitioning mode coded CU.

[1122] The type of motion vector stored for each individual position in the motion field is determined as:

[1123] sType=abs(motionIdx)<32?2:(motionIdx≤0?(1-pariIdx):partIdx)(2-8)

[1124] Where motionIdx is equal to d(4x+2,4y+2), which is recalculated according to equation (2-1). partIdx depends on the angle index i.

[1125] If sType is equal to 0 or 1, then Mv1 or Mv2 is stored in the corresponding motion field, otherwise, if sTType is equal to 2, the combined Mv from Mv1 and Mv2 is stored. The combined Mv is generated using the following process:

[1126] 1) If Mv1 and Mv2 are from different reference picture lists (one from L0 and the other from L1), then Mv1 and Mv2 are simply combined to form a bidirectional prediction motion vector.

[1127] 2) Otherwise, if Mv1 and Mv2 are from the same list, only the unidirectional predicted motion Mv2 is stored.

[1128] 2.25 Combined Inter and Intra Prediction (CIIP)

[1129] In VVC, when encoding a CU in merge mode, if the CU contains at least 64 luma samples (i.e., the CU width multiplied by the CU height is equal to or greater than 64), and if both the CU width and the CU height are less than 128 luma samples, an additional flag is signaled to indicate whether the combined inter / intra prediction (CIIP) mode is applied to the current CU. As its name suggests, CIIP prediction combines the inter prediction signal with the intra prediction signal. The inter prediction signal P in CIIP mode inter is derived using the same inter prediction process applied to the normal merge mode; and the intra prediction signal P is derived after the normal intra prediction process for the planar mode intra Then, the intra-frame and inter-frame prediction signals are combined using weighted averaging, where the weight values ​​are calculated depending on the coding mode of the top and left neighboring blocks as follows (e.g. Fig.37 As shown in Figure 3700):

[1130] - If the top neighbor is available and intra-coded, set isIntraTop to 1, otherwise set isIntra top to 0;

[1131] - If the left neighbor is available and intra-frame decoding is performed, isIntraLeft is set to 1, otherwise isIntralLeft is set to 0;

[1132] -If (isIntraLeft+isIntraTop) is equal to 2, then wt is set to 3;

[1133] - Otherwise, if (isIntraLeft + isIntraTop) is equal to 1, then wt is set to 2;

[1134] - Otherwise, set wt to 1.

[1135] The CIIP forecast format is as follows:

[1136] P CIIP =((4-wt)*P inter +wt*P intra +2)>>2

[1137] 2.26 Decoder-side Intra Mode Derivation (DIMD)

[1138] When DIMD is applied, two intra modes are derived from the reconstructed neighboring samples and these two predictors are combined with a planar mode predictor with weights derived from the gradients.

[1139] The derived intra modes are included in the master list of intra most probable modes (MPMs), so the DIMD process is performed before constructing the MPM list. The master derived intra modes of a DIMD block are stored with the block and used for MPM list construction of neighboring blocks.

[1140] 2.27 IBC Movement Candidates

[1141] The term "block" may refer to a codec tree block (CTB), a codec tree unit (CTU), a codec block (CB), a CU, a PU, a TU, a PB, a TB, or a video processing unit including a plurality of samples / pixels. A block may be rectangular or non-rectangular.

[1142] For IBC codec blocks, a block vector (BV) is used to indicate the displacement from the current block to a reference block that has been reconstructed in the current picture.

[1143] W and H are the width and height of the current block (eg, luma block).

[1144] The non-adjacent spatial candidates of the current codec block are the adjacent spatial candidates of the virtual block in the i-th search round (e.g. Fig. 9 As shown). The calculation formulas for the width and height of the virtual block of the i-th search round are: newWidth = i × 2 × gridX + W, newHeight = i × 2 × gridY + H. Obviously, if the search round i is 0, the virtual block is the current block.

[1145] Below, the BV predictor is also a BV candidate. Skip mode is also a merge mode.

[1146] BV candidates can be divided into several groups according to certain criteria. Each group is called a subgroup. For example, we can take the adjacent spatial and temporal BV candidates as the first subgroup and the remaining BV candidates as the second subgroup; in another example, we can also take the first N (N ≥ 2) BV candidates as the first subgroup, the following M (M ≥ 2) BV candidates as the second subgroup, and the remaining BV candidates as the third subgroup.

[1147] About the use of BV candidates

[1148] 1. If a BV candidate (e.g., a BV search point or a BV predictor) is invalid, it is not allowed to be used in the encoding / decoding process of a block.

[1149] a. In one example, only when a BV candidate is valid can it be used in the encoding / decoding process of a block.

[1150] i. For example, only when a BV candidate is valid, it can be used for BV search or BV prediction.

[1151] b. In one example, whether to use a BV candidate in the encoding / decoding process of a block may depend on a validity check of the BV candidate.

[1152] i. In one example, before inserting a new BV candidate into the BV candidate list, a validity check of the BV candidate needs to be performed.

[1153] c. Only when a BV candidate is valid can it be inserted into the IBC candidate list.

[1154] i. In one example, the IBC candidate list may be an IBC merge candidate list.

[1155] ii. In one example, the IBC candidate list may be an IBC AMVP candidate list.

[1156] iii. In one example, the IBC candidate list may be an IBC template matching candidate list.

[1157] iv. In one example, the IBC candidate list may be an intra template matching candidate list.

[1158] d. Only when a BV candidate is valid, it can be used for hash-based search of IBC.

[1159] e. Only when a BV candidate is valid, it can be used for block matching based local search of IBC.

[1160] f. Only when a BV candidate is valid, it can be used for intra template matching.

[1161] g. Alternatively, in addition, the above-mentioned BV candidates may be those from specific neighboring blocks (eg, adjacent or non-adjacent) or the HMVP table, or some virtual candidates generated from these BV candidates.

[1162] h. Alternatively, in addition, the above BV candidates may exclude some default candidates (eg, a default zero vector).

[1163] i. Alternatively, further, when a BV candidate is marked as invalid, a virtual candidate derived from the invalid BV candidate may alternatively be used.

[1164] i. In one example, a virtual candidate may be derived by adding an offset to an invalid BV candidate.

[1165] ii. In one example, virtual candidates may be derived by applying a pruning function to invalid BV candidates.

[1166] Verification Checks for BV Candidates

[1167] 2. In one example, a BV candidate may be determined to be valid when it satisfies one of the following conditions or a combination of at least one of the following conditions.

[1168] a. The corresponding reference block has been reconstructed in the current image.

[1169] b. The corresponding reference block is located in the same CTU row as the current block.

[1170] c. The corresponding reference block is located in the same tile / sub-picture as the current block.

[1171] d. The corresponding reference block is located in the same stripe as the current block.

[1172] e. The BV candidate satisfies the block vector constraints (such as those described in 2.4.2 and 2.4.3)

[1173] f. The BV candidate satisfies the IBC virtual buffer conditions (such as described in 2.4.4).

[1174] 3. In one example, when a BV candidate violates one of the conditions in item 2 or a combination of at least one of the conditions, it may be determined to be invalid.

[1175] About BV Candidate List

[1176] 4. BV candidates can be derived / obtained from non-neighboring blocks.

[1177] b. In one example, the distance between the non-adjacent spatial candidate and the current codec block can be based on the width and height of the current codec block (e.g. Fig. 9 or Fig.10 , gridX=W, gridY=H).

[1178] i. Alternatively, the distance between the non-adjacent spatial candidates and the current codec block may be a multiple of a constant value.

[1179] (i) For example, the multiplication factor depends on the search round index (eg, the multiplication factor for the i-th search round is i) and gridX=M, gridY=N (M and N are constant values).

[1180] c. In one example, non-adjacent spatial candidates may be inserted after the TMVP candidate.

[1181] i. Alternatively, non-adjacent spatial candidates may be inserted after the adjacent spatial candidates and before the TMVP candidate.

[1182] 5. BV candidates can come from spatial neighboring blocks, temporal neighboring blocks, HMVP, pairwise and / or STMVP candidates. Figures 38A-38B Example candidate positions are shown for spatial candidates and temporal candidates.

[1183] a. In one example, a spatial candidate may consist of adjacent and / or non-adjacent spatial candidates.

[1184] i. In one example, the neighboring spatial candidates may consist of left and / or above and / or above right and / or below left and / or above left spatial candidates (examples of candidate positions for spatial candidates are in Fig.38A 3800)

[1185] b. In one example, for the TMVP candidate, the position of the time domain candidate is selected between candidates C0 and C1, such as Fig.38B As depicted in FIG. 3810 of FIG. If the CU at position C0 is not available, is intra-coded, is outside the current row of the CTU, or its BV is invalid for the current block, position C1 is used. Otherwise, position C0 is used to derive the TMVP candidate.

[1186] i. Alternatively, for TMVP candidates, both candidates C0 and C1 can be used, such as Fig.38B shown.

[1187] (i) For example, the order is C0->C1.

[1188] (ii) Alternatively, the order is C1->C0.

[1189] c. In one example, for paired candidates, paired average candidates are generated by averaging predefined candidate pairs in an existing BV candidate list, and the predefined pairs are defined as {(0,1), (0,2), (1,2), (0,3), (1,3), (2,3)}, where the numbers represent the BV candidate indexes in the BV candidate list.

[1190] i. In one example, the number of pairwise candidates is P. P is an integer from 0 to 6.

[1191] ii. In one example, the pairwise candidates may be inserted after the HMVP.

[1192] d. In one example, for the STMVP candidate, it is generated by averaging the predefined E spatial BV candidate and the predefined G temporal BV candidate.

[1193] i. In one example, E is less than or equal to the number of spatial candidates (F) inserted into the current BV candidate list before the STMVP.

[1194] ii. In one example, the predefined E-spatial BV candidates may be the first E spatial BV candidates among the F-spatial candidates inserted into the current BV candidate list before the STMVP.

[1195] (i) Alternatively, the predefined E spatial BV candidates may be the selected E spatial BV candidates among the F spatial candidates inserted into the current BV candidate list before the STMVP.

[1196] iii. In one example, E is 2 and G is 1.

[1197] iv. In one example, the STMVP may be inserted before the top left spatial BV candidate.

[1198] v. In one example, STMVP can be inserted after the pairwise candidates.

[1199] e. In one example, the BV candidate insertion order is adjacent space->HMVP->pairwise.

[1200] f. In one example, the BV candidate insertion order is adjacent space->time domain->HMVP->paired.

[1201] g. In one example, the BV candidate insertion order is adjacent space->time domain->non-adjacent space->HMVP->paired.

[1202] h. In one example, the BV candidate insertion order is adjacent space->non-adjacent space->HMVP->paired.

[1203] i. In one example, the BV candidate insertion order is adjacent space (STMVP is inserted before the upper left spatial BV candidate) -> temporal domain -> non-adjacent space -> HMVP -> paired.

[1204] 6. The BV candidate list can also consist of pruned BV candidates.

[1205] a. In one example, if a BV candidate is invalid in the sense of item 3, it can be converted to a valid BV according to the given rules and then inserted into the BV candidate list.

[1206] i. In one example, a BV candidate may be converted to the closest valid BV candidate.

[1207] ii. In one example, a BV candidate may be converted to the closest valid BV candidate from a predefined set of BV candidates.

[1208] b. In one example, if a non-zero BV candidate is invalid, it can be pruned to the closest valid BV, which is then inserted into the BV candidate list.

[1209] c. In one example, if a non-zero BV candidate is invalid, it can be pruned to the closest valid BV from the predefined BV candidate set and then inserted into the BV candidate list.

[1210] i. In one example, the predefined BV candidate set may consist of (-m*W, 0), (0, -n*H), (-m*H, 0), (0, -n*W). m and n are positive values.

[1211] d. In one example, the pruned BV candidates may be inserted after the unpruned BV candidates.

[1212] 7. The BV candidate list can be used as the IBC merge / AMVP candidate list.

[1213] a. Alternatively, the BV candidate list can be used to derive the IBC merge / AMVP candidate list.

[1214] i. In one example, for the IBC merge mode, the first R entries of the BV candidate list will be used to construct the IBC merge candidate list; for the IBC AMVP mode, the first S entries of the BV candidate list will be used to construct the IBC AMVP candidate list.

[1215] (i) In one example, R is 6 and S is 2.

[1216] 8. In one example, sub-block based temporal block vector prediction (SbTBVP) may be supported as a BV candidate or BV prediction mode.

[1217] a. Similar to SbTMVP, SbTBVP uses the BV motion field in the co-located picture to improve the block vector prediction and IBC merge mode of the CU in the current picture. For SbTBVP, the co-located picture used by TMVP is used.

[1218] b. In one example, SbTBVP applies motion translation before obtaining temporal BV information from the co-located picture, where the motion translation is obtained from the motion vector of one of the spatial neighboring blocks of the current CU (for example, Fig.39shown). Fig.39 Diagram 3900 shows the derivation of a sub-CU bv motion field from a corresponding co-located sub-CU by applying motion shifts from spatial neighbors.

[1219] i. In one example, if A1 has a motion vector that uses the co-located picture as its reference picture, then that motion vector is selected as the motion offset to be applied. If no such motion is identified, then the motion shift is set to (0,0).

[1220] ii. In one example, other spatial candidate locations (eg, A0, B0, B1, B2) may be used to derive motion shifts.

[1221] (i) In one example, the checking order may be A1->B1->B0->A0->B2.

[1222] (ii) In one example, the checking order may be B1->A1->B0->A0->B2.

[1223] (iii) In one example, the checking order may be A0->A1->B0->B1->B2.

[1224] c. In one example, after deriving the motion shift, for each sub-CU, the BV information of its corresponding block (the minimum motion grid covering the center sample) in the same-position picture is used to derive the BV information of the sub-CU ( Fig.39 Example in (assuming motion shift is set to block motion of A1).

[1225] Regarding the re-ranking of the BV candidate list

[1226] 9. An initial BV candidate list may be derived first, and then a reordering / refining process may be performed, and the reordered / refined list may be used in the encoding / decoding process of the block.

[1227] 10. The BV candidates can be adaptively rearranged in the final BV candidate list according to one or some criteria.

[1228] a. In one example, firstly, part or all of the process of building the current BV candidate list is called, and then the candidates in the list are reordered.

[1229] i. Alternatively, the candidates in the first subgroup may be reordered and they should be added before those in the second subgroup, where the first subgroup is added before the second subgroup.

[1230] b. In one example, the criterion may be based on template matching cost.

[1231] i. In one example, the cost function between the current template and the reference template can be

[1232] (i) SAD / MR-SAD

[1233] (ii) SATD / MR-SATD

[1234] (iii) SSD / MR-SSD

[1235] (iv) SSE / MR-SSE

[1236] (v) Weighted SAD / Weighted MR-SAD

[1237] (vi) Weighted SATD / Weighted MR-SATD

[1238] (vii) Weighted SSD / Weighted MR-SSD

[1239] (viii) Weighted SSE / Weighted MR-SSE

[1240] (ix) Gradient information

[1241] ii. In one example, if LMCS is enabled, the current template and the reference template may consist of samples in the mapped domain.

[1242] (i) The current template and the reference template may consist of samples in the original domain.

[1243] iii. In one example, the BV candidates in each subgroup may be re-sorted in ascending order according to the cost value based on template matching.

[1244] iv. In one example, if only the upper template is available for the current block, the template matching reordering can only use the upper template.

[1245] v. In one example, if only the left template is available for the current block, the template matching reordering can only use the left template.

[1246] vi. In one example, if both the above template and the left template are available for the current block, the template matching reordering may use the left template, the above template, or both the above template and the left template.

[1247] vii. In one example, if the reference template is outside the current picture, the corresponding BV candidates can still be re-ranked.

[1248] (i) In one example, if the reference template is outside the current picture, it can be filled from the closest sample within the current picture.

[1249] viii. In one example, the reference template should have been reconstructed within the current picture.

[1250] (i) In one example, if the reference template is not reconstructed or is outside the current picture, the corresponding BV candidates may not be reordered.

[1251] c. In one example, whether and / or how to re-rank BV candidates may depend on the category of the BV candidates.

[1252] d. In one example, the BV candidates to be re-ranked may be BV candidates in the final BV candidate list.

[1253] i. Alternatively, the BV candidate to be re-ordered may be part / all of the adjacent spatial BV candidates, even though it may not be included in the final BV candidate list.

[1254] ii. Alternatively, the BV candidate to be re-ordered may be part / all of the non-adjacent spatial BV candidates, even though it may not be included in the final BV candidate list.

[1255] iii. Alternatively, the re-ordered BV candidate may be part / all of the HMVP BV candidate, even though it may not be included in the final BV candidate list.

[1256] iv. Alternatively, the BV candidate to be re-ranked may be part / all of the pairwise average BV candidate, even though it may not be included in the final BV candidate list.

[1257] v. Alternatively, the BV candidate to be re-ranked may be part / all of the STMVP BV candidate, even though it may not be included in the final BV candidate list.

[1258] 11. In one example, the cost disclosed in item 10 may be derived for a first BV candidate, which may or may not be placed in the BV candidate list.

[1259] a. In one example, whether to put the first BV candidate into the BV candidate list may depend on the cost derived for the first BV candidate.

[1260] b. In one example, whether to place a first BV candidate in the BV candidate list may depend on a comparison between a first cost derived for the first BV candidate and a second cost derived for a second BV candidate, which may or may not be placed in the BV candidate list.

[1261] 12. In one example, for intra-frame TMP, the L-shaped template can be replaced by excluding the left and upper templates (such as Fig.40 example shown). Fig.40Diagram 4000 shows intra-frame template matching.

[1262] a. In one example, if only the above template is available for the current block, the intra TMP can only use the above template.

[1263] b. In one example, if only the left template is available for the current block, the intra TMP can only use the left template.

[1264] c. In one example, if both the above template and the left template are available for the current block, the intra TMP may use the left template, the above template, or both the above template and the left template.

[1265] 3. Question

[1266] The current design of the IBC model can be further improved.

[1267] More IBC-based modes can be supported to improve the encoding and decoding efficiency of the IBC mode.

[1268] 4. Detailed description

[1269] The following detailed description should be considered as an example to explain the general concept. These embodiments should not be interpreted narrowly. In addition, these embodiments can be combined in any way.

[1270] Furthermore, these embodiments may be combined in any manner. In the present disclosure, intra block copy (IBC) may not be limited to the current IBC technology, but may be interpreted as a technology in which a reference (or prediction) block is obtained using samples in the current slice / tile / sub-picture / picture / other video unit (e.g., CU, PU, ​​TU, CTU, CTU row) outside of a conventional intra prediction method.

[1271] The term "block" may refer to a codec tree block (CTB), a codec tree unit (CTU), a codec block (CB), a CU, a PU, a TU, a PB, a TB, or a video processing unit including a plurality of samples / pixels. A block may be rectangular or non-rectangular.

[1272] For IBC codec blocks, a block vector (BV) is used to indicate the displacement from the current block to a reference block that has been reconstructed in the current picture.

[1273] W and H are the width and height of the current block (eg, luma block).

[1274] It should be noted that the following terms are not limited to the specific terms defined in existing standards. Any variants of codecs are also applicable. For example, the term "GPM" is used to represent any codec that derives two or more sets of motion information and uses the derived motion information and segmentation mode / weighted mask to obtain the final prediction. For example, TPM can also be regarded as GPM.

[1275] Below, Mv1 and Mv2 are the motion vectors from the first and second parts of the triangulation or geometric partitioning.

[1276] 1. In one example, IBC merge mode with block vector difference (MBVD) may be used.

[1277] a. In MBVD, BV can be derived based on IBC merge candidates, which can be further refined by signaled BVD information.

[1278] b. In one example, the BVD information may include one or more IBC merge candidate indexes, one or more indications (eg, multiple indices) for specifying (one or more) motion magnitudes, and one or more indications (eg, multiple indices) for indicating motion directions.

[1279] i. In MBVD mode, at least one is selected from the candidates in the IBC merge list as the BV basis. At least one MBVD candidate index is signaled to specify which one or more of the IBC merge candidates are used.

[1280] (i) In one example, the MBVD candidate index is signaled to specify which of the top N IBC merge candidates to use.

[1281] a) In one example, N is set to 2.

[1282] b) In one example, the candidate index may be binarized into a truncated code with a maximum value equal to N-1.

[1283] (ii) In one example, the IBC merge candidates may be reordered before use. ii In one example, the distance index specifies motion magnitude information and indicates a predefined offset from a starting point.

[1284] (i) An offset may be added to the horizontal or vertical component of the starting MV.

[1285] (ii) An offset may be added to both the horizontal and vertical components of the starting MV.

[1286] (iii) In one example, the distance set may be {1 pixel, 2 pixels, 4 pixels, 8 pixels, 16 pixels, 32 pixels}.

[1287] (iv) In one example, the distance set may be {1 pixel, 2 pixels, 4 pixels, 8 pixels, 16 pixels, 32 pixels, 64 pixels, 128 pixels}.

[1288] (v) In one example, the distance set may be {1 pixel, 2 pixels, 3 pixels, 4 pixels, 6 pixels, 8 pixels, 16 pixels}.

[1289] (vi) In one example, the distance set may be {1 pixel, 2 pixels, 3 pixels, 4 pixels, 6 pixels, 8 pixels, 16 pixels, 32 pixels, 64 pixels}.

[1290] (vii) In one example, the relationship between the distance index and the predefined offset is specified in Table 4.

[1291] (viii) In one example, the relationship between the distance index and the predefined offset is specified in Table 5.

[1292] (ix) In one example, the relationship between the distance index and the predefined offset is specified in Table 6.

[1293] (x) In one example, the relationship between the distance index and the predefined offset is specified in Table 7.

[1294] (xi) In one example, the relationship of the distance index and the predefined offset may be signaled from the encoder to the decoder at a sequence / picture / slice / CTU / CU level.

[1295] (xii) The index can be binarized using unary encoding and decoding, truncated unary encoding and decoding, exponential Golomb code, truncated exponential Golomb code, fixed length code or any other binarization method.

[1296] iii In one example, the direction index indicates the direction of the BVD relative to the starting point. The direction index may indicate M BVD directions.

[1297] (i) In one example, M is set to 4.

[1298] (a) In one example, 4 horizontal / vertical directions can be used.

[1299] (b) In one example, 4 diagonal directions may be used.

[1300] (c) In one example, the relationship between the direction index and the predefined direction is specified in Table 8. For direction index 0, (offset, 0) is BVD; for direction index 1, (-offset, 0) is BVD; for direction index 2, (0, offset) is BVD; for direction index 3, (0, -offset) is BVD.

[1301] (d) In one example, the relationship between the direction index and the predefined direction is specified in Table 9. For direction index 0, (offset, offset) is BVD; for direction index 1, (offset, -offset) is BVD; for direction index 2, (-offset, offset) is BVD; for direction index 3, (-offset, -offset) is BVD.

[1302] (ii) In one example, M is set to 8.

[1303] a) In one example, 4 horizontal / vertical directions plus 4 diagonal directions may be used.

[1304] b) In one example, the relationship between the direction index and the predefined directions is specified in Table 10.

[1305] (iii) The index can be binarized using unary codec, truncated unary codec, exponential Golomb code, truncated exponential Golomb code, fixed length code or any other binarization method.

[1306] iv. In one example, the distance and / or number of directions of the MBVD of a block may be determined by:

[1307] (i) Resolution of the image.

[1308] (ii) Configuration of the encoding and decoding process.

[1309] (iii) BVD of the neighboring blocks of this block.

[1310] a) In one example, the neighboring blocks above and to the left can be used (e.g. Fig.41 Depicted, it shows a diagram 4100 of the adjacent spatial neighboring blocks used.

[1311] b) In one example, adjacent spatial neighboring blocks including left and / or above and / or above right and / or below left and / or above left spatial neighboring blocks may be used ( Fig.41 An example is shown).

[1312] (iv) Alternatively, the number of distances and / or directions of MBVD may be sent from the encoder to the decoder at the sequence / picture / slice / CTU / CU level.

[1313] c. In one example, candidates / directions / amplitudes in the MBVD that may produce a BV outside the valid range of the BV may be excluded from the set of candidates / directions / amplitudes to be selected or signaled.

[1314] Alternatively, the BV generated in the MBVD may be restricted to a valid range.

[1315] ii Alternatively, the BV generated in the MBVD must be within the valid range of the conforming bitstream.

[1316] Table 4 – Relationship between distance index and predefined offsets

[1317]

[1318] Table 5 – Relationship between distance index and predefined offsets

[1319]

[1320] Table 6 – Relationship between distance index and predefined offsets

[1321]

[1322] Table 7 – Relationship between distance index and predefined offsets

[1323]

[1324] Table 8 – Sign of MV offset specified by direction index

[1325] Directions to IDX 00 01 10 11 x-axis + - N / A N / A y-axis N / A N / A + -

[1326] Table 9 – Sign of MV offset specified by direction index

[1327] Directions to IDX 00 01 10 11 x-axis + + - - y-axis + - + -

[1328] Table 10 – Sign of BV offset specified by direction index

[1329] Directions to IDX 000 001 010 011 100 101 110 111 x-axis + - N / A N / A + + - - y-axis N / A N / A + - + - + -

[1330] 2. In one example, a new CIIP prediction mode (referred to as CIIP_N) combines at least one IBC prediction signal and at least one prediction signal generated by a second prediction method.

[1331] a. The second prediction method can be intra-frame prediction or inter-frame prediction.

[1332] b. The second prediction signal and the IBC prediction signal may be combined by weighted averaging.

[1333] i. The form of CIIP_N prediction is

[1334] P CIIP_N =((2 N -wt)*P IBC +wt*P sec +offset)>>N

[1335] ii. In one example, offset is an integer, such as 2 N >>1.

[1336] iii. In one example, N=2.

[1337] iv. In one example, the weight values ​​may be predefined.

[1338] (i) In one example, wt is set to 2.

[1339] v. In one example, the weight value for each sample may be position-dependent.

[1340] (i) For example, for some positions wt = 2 N .

[1341] (ii) For example, for some positions, wt=0.

[1342] vi In one example, the weight values ​​can be signaled from the encoder to the decoder.

[1343] c. IBC prediction signal P in CIIP_N mode IBC It can be derived using the same IBC prediction process applied to the regular IBC merge mode.

[1344] d. Second prediction signal P sec It can be derived according to the conventional intra prediction process, and the combined mode is named CIIP_N1.

[1345] i. In one example, the intra prediction mode may be planar mode.

[1346] ii. In one example, the intra prediction mode may be an intra prediction mode implicitly derived by the DIMD method.

[1347] iii. In one example, the intra prediction mode may be an intra prediction mode implicitly derived by the TIMD method.

[1348] iv. In one example, the weight value may be calculated according to the coding mode of the neighboring blocks.

[1349] (i) In one example, the weight value may be calculated based on the encoding and decoding modes of the top and left neighboring blocks (e.g. Fig.42 As shown, Figure 4200 ) which shows the top and left neighboring blocks used in the CIIP_N1 and CIIP_N2 weight derivation is as follows:

[1350] ● If the top neighbor is available and is intra-coded, set isIntraTop to 1, otherwise set isIntraTop to 0;

[1351] ● If the left neighbor is available and is intra-coded, set isIntraLeft to 1, otherwise set isIntraLeft to 0;

[1352] ●If (isIntraLeft+isIntraTop) is equal to 2, set wt to 3;

[1353] ● Otherwise, if (isIntraLeft + isIntraTop) is equal to 1, set wt to 2;

[1354] ●Otherwise, set wt to 1.

[1355] e. Second prediction signal P sec It can be derived using the same inter prediction process applied to the regular merge mode, and the combined mode is named CIIP_N2.

[1356] i. In one example, the weight value may be calculated according to the coding mode of the neighboring blocks.

[1357] (ii) In one example, the top and left neighboring blocks (such as Fig.42 The weight value is calculated based on the encoding and decoding mode shown in the figure, as shown below:

[1358] ● If the top neighbor is available and is encoded and decoded by IBC, set isIBCTop to 1, otherwise set isIBCTop to 0;

[1359] ●If the left neighbor is available and is encoded and decoded by IBC, set isIBCLeft to 1, otherwise set isIBCLeft to 0;

[1360] ●If (isIBCLeft+isIBCTop) is equal to 2, set wt to 1;

[1361] ● Otherwise, if (isIBCLeft + isIBCTop) is equal to 1, set wt to 2;

[1362] Otherwise, set wt to 3.

[1363] f. In CIIP_N mode, select one of the candidates in the IBC merge list for IBC prediction. An IBC candidate index may be signaled to specify which of the IBC merge candidates to use.

[1364] i. In one example, an IBC candidate index is signaled to specify which of the top N IBC merge candidates to use.

[1365] (i) In one example, N is set to 4.

[1366] (ii) In one example, N is set to the valid number of IBC merge candidates in the IBC merge list.

[1367] (iii) In one example, N may be signaled from the encoder to the decoder.

[1368] (iv) In one example, N is set to the number of complete RDs merged for IBC.

[1369] (v) In one example, N is set to the number of complete RDs merged for IBC plus an integer.

[1370] (vi) In one example, IBC merge candidates may be reordered before use.

[1371] ii. In one example, an IBC candidate index is signaled to specify which one to use among the top N IBC merge candidates in ascending order of SATD cost value.

[1372] (i) In one example, N is set to 4.

[1373] (ii) In one example, N is set to the valid number of IBC merge candidates in the IBC merge list.

[1374] (iii) In one example, N may be signaled from the encoder to the decoder.

[1375] (iv) In one example, N is set to the number of full RDs merged for IBC.

[1376] (v) In one example, N is set to the number of full RDs used for IBC merging plus an integer.

[1377] (vi) In one example, IBC merge candidates may be reordered before calculating SATD cost.

[1378] g. In CIIP_N2 mode, select one of the candidates in the regular merge list for inter prediction. A merge candidate index may be signaled to specify which of the regular merge candidates to use.

[1379] i. In one example, a merge candidate index is signaled to specify which of the top N regular merge candidates to use.

[1380] (i) In one example, N is set to 4.

[1381] (ii) In one example, N is set to the effective number of regular merge candidates in the regular merge list.

[1382] (iii) In one example, N may be signaled from the encoder to the decoder.

[1383] (iv) In one example, N is set to the number of full RDs for frame merging.

[1384] (v) In one example, N is set to the number of full RDs for frame merging plus an integer.

[1385] (vi) In one example, regular merge candidates may be reordered before use.

[1386] ii. In one example, a merge candidate index is signaled to specify which of the top N regular merge candidates to use in ascending order of SATD cost value.

[1387] (i) In one example, N is set to 4.

[1388] (ii) In one example, N is set to the effective number of regular merge candidates in the regular merge list.

[1389] (iii) In one example, N may be signaled from the encoder to the decoder.

[1390] (iv) In one example, N is set to the number of full RDs for inter-frame merging.

[1391] (v) In one example, N is set to the number of full RDs for inter-frame merging plus an integer.

[1392] (vi) In one example, regular merge candidates may be reordered before calculating the SATD cost.

[1393] h. In one example, whether and / or how to use CIIP_N mode may depend on codec information such as block size / QP / neighboring block mode, etc.

[1394] i. In one example, when a block is encoded and decoded in IBC merge mode, if the block contains at least P luma samples (ie, the block width multiplied by the block height is equal to or greater than P), an additional flag is signaled to indicate whether CIIP_N mode is applied to the current block.

[1395] (i) In one example, P is set to 64.

[1396] ii. In one example, when a block is encoded and decoded in IBC merge mode, if both the block width and the block height are less than Q samples, an additional flag is signaled to indicate whether CIIP_N mode is applied to the current block.

[1397] (i) In one example, Q is set to 128.

[1398] (ii) In one example, Q is set to 64.

[1399] iii. In one example, the above two conditions can be used together.

[1400] i. In one example, whether and / or how to use CIIP_N2 mode may depend on codec information, such as block size / QP / neighboring block mode, etc.

[1401] i. In one example, when a block is encoded and decoded in normal merge mode, if the block contains at least P luma samples (ie, the block width multiplied by the block height is equal to or greater than P), an additional flag is signaled to indicate whether CIIP_N2 mode is applied to the current block.

[1402] (i) In one example, P is set to 64.

[1403] ii. In one example, when a block is encoded and decoded in normal merge mode, if both the block width and the block height are less than Q luma samples, an additional flag is signaled to indicate whether CIIP_N2 mode is applied to the current block.

[1404] (i) In one example, Q is set to 128.

[1405] (ii) In one example, Q is set to 64.

[1406] iii. In one example, the above two conditions can be used together.

[1407] 3. In one example, a triangulation mode may be supported for IBC prediction (referred to as TPM_IBC).

[1408] a. When using this mode, use diagonal split or anti-diagonal split to evenly split the block into two triangle-shaped partitions (for example, Fig.43 An example of triangulation based on IBC prediction is shown). Fig.43 , block 4310 is partitioned using diagonal partitioning, and block 4320 is partitioned using anti-diagonal partitioning.

[1409] b. Each triangulation in the block is IBC predicted using its own motion.

[1410] c. The uni-directional prediction motion of each partition is derived from the uni-directional prediction IBC candidate list.

[1411] i. In one example, the unidirectional prediction IBC candidate list is directly derived from part or all of the IBC merge candidate list.

[1412] ii. In one example, the unidirectional prediction IBC candidate list can be reordered before use. d. After predicting each triangulation, a blending process with adaptive weights can be used to adjust sample values ​​along diagonal or anti-diagonal edges.

[1413] i. In one example, the weight can be adaptively determined according to the distance between the sample and the segmentation line.

[1414] (i) Section 2.23.2 shows an example.

[1415] ii. Alternatively, sample values ​​along a diagonal or anti-diagonal edge may not be adjusted using the blending process. Instead, samples along a diagonal or anti-diagonal edge can only be predicted by one of the two predictions.

[1416] e. For TPM_IBC's motion field storage, if sType is equal to 0 or 1, store Mv1 or Mv2 in the corresponding motion field, otherwise if sType is equal to 2, store Mv2.

[1417] i. In one example, the sType calculation is the same as that for the inter-frame TPM.

[1418] f. For TPM_IBC signaling, the following applies:

[1419] i. Use CU level flag to signal TPM_IBC mode as an IBC merge mode.

[1420] ii. If TPM_IBC mode is used for the current CU, a flag indicating the direction of the triangulation (diagonal or anti-diagonal) is further signaled, as well as two merge indices (one for each partition).

[1421] iii. If the TPM_IBC mode is used for the current CU, a triangulation partitioning index indicating a partitioning mode of triangulation partitioning and two merge indexes (one index for each partition) are further signaled.

[1422] 4. In one example, a geometric partitioning mode for IBC prediction (called GPM_IBC) may be supported.

[1423] a. When using this mode, the block is split into two parts by a geometrically positioned straight line (e.g. Fig.34 ). The location of the dividing line is derived mathematically from the angle and offset parameters of the specific division.

[1424] b. Each part of the geometric partition in the block is predicted by IBC using its own motion.

[1425] c. The unidirectional prediction motion for each partition is derived from the unidirectional prediction IBC candidate list.

[1426] i. In one example, the unidirectional prediction IBC candidate list is directly derived from part or all of the IBC merge candidate list.

[1427] ii. In one example, the unidirectional prediction IBC candidate list can be reordered before use. d. After predicting each part of a geometric partition, a blending process with adaptive weights can be used to adjust sample values ​​along the edges of the geometric partition.

[1428] i. In one example, the weight can be adaptively determined according to the distance between the sample and the segmentation line.

[1429] (i) Section 2.24.2 shows an example.

[1430] ii. Alternatively, sample values ​​along the edges of geometric partitions may not be adjusted using blending. Instead, samples along the edges of geometric partitions can only be predicted by one of the two predictions. e. For the field storage of GPM_IBC, if sType is equal to 0 or 1, Mv1 or Mv2 is stored in the corresponding motion field, otherwise if sType is equal to 2, Mv2 is stored.

[1431] i. In one example, the sType calculation is the same as that for inter-frame GPM.

[1432] f. For GPM_IBC signaling, the following applies:

[1433] i. Use CU level flag to signal GPM_IBC mode as an IBC merge mode.

[1434] ii. If GPM_IBC mode is used for the current CU, a geometric partition index indicating the partition mode (angle and offset) of the geometric partition and two merge indices (one for each partition) are further signaled.

[1435] 5. In one example, TM_AMVP for IBC is supported (referred to as TM_AMVP_IBC).

[1436] a. In TM_AMVP_IBC mode, K IBC MVP candidates are determined based on template matching to select the candidates that achieve the top K minimum differences between the current block template and the reference block template from the IBC AMVP list.

[1437] i. The selected starting point candidate set consists of K IBC MVP candidates.

[1438] b. TM can be performed for MV refinement only on the selected starting point candidate set.

[1439] i.TM refines the starting point candidates within the search range starting from full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode).

[1440] (i) In one example, this is refined to a [–8, +8] pixel search range using an iterative diamond search. M rounds of searching are performed until the center search point has Fig.44A Minimum matching cost for the diamond search pattern shown. Fig.44A An example 4400 of a search pattern is shown.

[1441] a) In one example, M is MAX_UINT.

[1442] b) In one example, M is 375.

[1443] ii The selected starting point candidates can be further refined by using a cross search with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode).

[1444] (i) In one example, a round of searching is used in a cross-search mode, such as Fig.44B shown. Fig.44B An example 4410 of a search pattern is shown.

[1445] c In one example, TM_AMVP_IBC may generate K refined IBC AMVP candidates, and one of them may be selected and the selection may be signaled from the encoder to the decoder.

[1446] In one example, K=1, and no selection information is signaled.

[1447] ii In one example, if at least one refined IBC AMVP candidate obtained by template matching is available, they are used as TM_AMVP_IBC candidates. Otherwise, the first K existing unrefined IBC AMVP candidates are used.

[1448] iii. In one example, the selected refined IBC AMVP candidate is used as a starting point for a local search of the block matching based IBC mode.

[1449] d. Alternatively, the TM_IBC derived BV is used as the starting point for the IBC mode block matching based local search.

[1450] e. For example, when IBC AMVR is enabled, the refined IBC AMVP candidates in MVD precision can be reused at another MVD precision.

[1451] i. In one example, refined IBC AMVP candidates at full-pixel MVD precision can be reused at 4-pixel MVD precision.

[1452] 6. In one example, TM_merge for IBC is supported (referred to as TM_merge_IBC).

[1453] a. In TM_merge_IBC mode, K IBC merge candidates are determined based on template matching to select candidates that achieve the top K minimum differences between the current block template and the reference block template from the IBC merge list.

[1454] i. The selected starting point candidate set consists of K IBC merge candidates.

[1455] b. TM can be performed for MV refinement only on the selected starting point candidate set.

[1456] i.TM refines the starting point candidates within the search range starting from full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode).

[1457] (i) In one example, this is refined to a [–8, +8] pixel search range using an iterative diamond search. M rounds of searching are performed until the center search point has Fig.44A

[1458] Minimum matching cost for the diamond search pattern shown.

[1459] a) In one example, M is MAX_UINT.

[1460] b) In one example, M is 375.

[1461] ii. The selected starting point candidates can be further refined by using cross search with full-pixel MVD precision (or 4 pixels for 4-pixel AMVR mode).

[1462] (i) In one example, a round of searching is used in a cross-search mode, such as Fig.44B shown.

[1463] c. In one example, TM_merge_IBC may generate K refined IBC merge candidates, and one of them may be selected and the selection may be signaled from the encoder to the decoder.

[1464] In one example, K=1, and no selection information is signaled.

[1465] ii In one example, if at least one refined IBC merge candidate obtained by template matching is available, they are used as TM_merge_IBC candidates. Otherwise, M_merge_IBC is invalid.

[1466] iii. In one example, the best TM-refined IBC merge candidate is selected by a criterion.

[1467] (i) In one example, the criterion is an RD decision.

[1468] d. Alternatively, perform TM for MV for each IBC merge candidate. Then select the best TM-refined IBC merge candidate according to the criteria.

[1469] i. In one example, the criterion is an RD decision.

[1470] 7. It is proposed to use a combination of intra block copy (IBC) and intra prediction (CIBCIP) to derive the prediction / reconstruction of a video unit obtained by fusing the IBC prediction signal and the intra prediction signal.

[1471] a. In one example, P(x, y) = w ip1 *IP 1 (x,y)+w ip2 *IP 2 (x,y)+…+w ipn *IP n (x,y)+w ibc1 *IBC 1 (x,y)+w ibc2 *IBC 2 (x,y)+…+w ibcm *IBC m (x,y), where P(x,y) is the generated prediction value, IP kis the prediction generated by the k-th intra prediction, IBCj is the prediction generated by the j-th IBC, and w ipk and w ibcj is the corresponding weight value.

[1472] b. In one example, the intra prediction may be generated by angular intra prediction, DC, planar, cross component prediction (CCLM), multi-model CCLM, left CCLM, top CCLM, etc.

[1473] i. Intra prediction mode can be encoded using MPM or TIMD or DIMD or any other method to signal the intra prediction mode.

[1474] ii. A specific set of intra prediction modes may be allowed to be used in CIBCIP.

[1475] c. In one example, IBC merge mode may be used.

[1476] i. In one example, one or more BV candidates in the IBC merge list may be allowed to be used in the CIBCIP.

[1477] ii. In one example, one or more BV offsets may be used for CIBCIP.

[1478] 1) In one example, a BV offset may be added to the BV candidate before the BV candidate is used to obtain an IBC prediction.

[1479] 2) In one example, the BV offset may be signaled or derived.

[1480] iii. In one example, the IBC merge mode may be at least one of a conventional IBC merge mode and an IBC-MBVD merge mode and an IBC-TM merge mode.

[1481] d. In one example, the IBC AMVP mode may be used.

[1482] i. In one example, one or more BV predictors in the IBC AMVP list may be allowed to be used in the CIBCIP.

[1483] ii. In one example, the block vector difference (BVD) used in CIBCIP may be signaled in the same manner as in IBC mode.

[1484] 1) Alternatively, the BVD may not be signaled but predefined.

[1485] iii. In one example, the BVD may be derived using codec information.

[1486] e. In one example, a merge index (mergeIdx) indicating a BV candidate in the IBC merge list and / or a BVP index (bvpIdx) indicating a BV predictor in the IBC AMVP list for obtaining an IBC prediction signal may be signaled.

[1487] 1) In one example, the binarization or signaling method of the merge index or BVP index may be the same as that in the IBC mode.

[1488] 2) Alternatively, a merge index or a BVP index may be predefined, such as mergeldx=0 or mergeldx=1; bvpldx=0 or bvpldx=1.

[1489] 3) Alternatively, the merge index or BVP index may be derived using codec information.

[1490] 4) Alternatively, template matching can be used (e.g., with minimum template matching cost)

[1491] Export the merged index or BVP index.

[1492] f. In one example, the construction of the IBC merge list or IBC AMVP list used in CIBCIP mode may be the same as or different from that used in IBC mode.

[1493] g. In one example, the number of BV candidates (N) that can be used in the IBC merge (or AMVP) list for CIBCIP is less than or equal to the number of BV candidates (M) that can be used in the IBC merge (or AMVP) list for IBC. N is an integer greater than 0 and less than or equal to M.

[1494] i. In one example, N=1, or N=2, or N=3, or N=4, or N=5, or N=6.

[1495] ii. In one example, the top N BV candidates of the IBC merge (or AMVP) list may be used for CIBCIP.

[1496] h. In one example, template matching can be used to derive / refine the BV used to obtain the IBC prediction signal.

[1497] i. In one example, the BV offset may be derived using template matching, which is added to the BV candidate in the IBC merge list.

[1498] ii. In one example, the BVD may be derived using a template matching based approach.

[1499] iii. In one example, a template matching based approach may be used to derive the BVD symbols.

[1500] iv. In one example, the intra prediction mode or intra prediction method used to obtain the intra prediction signal may be used for template matching to derive / refine the BV.

[1501] i. In one example, the BV list may be reordered before being used in CIBCIP.

[1502] i. In one example, template matching or bilateral matching cost can be used for re-ranking.

[1503] ii. In one example, template matching or bilateral matching may be used during the construction of the BV list for CIBCIP.

[1504] iii. In one example, the BV list may refer to the IBC Consolidated List or the IBC AMVP List.

[1505] iv. In one example, the reordering method for the BV list of CIBCIP may be the same as that of IBC.

[1506] v. Alternatively, the reordering method for the BV list of the CIBCIP may be different from the reordering method of the IBC.

[1507] 1) In one example, the number of BV candidates (N) in the reordered BV list for CIBCIP 1 ) may be less than or equal to the number of reordered BV candidates for IBC mode (M 1 ).

[1508] a) In one example, when IBC merge mode is used for CIBCIP, N 1 =1 or 2 or 3 or 4.

[1509] b) In one example, when IBC AMVP is used for CIBCIP, N 1 =1 or 2 or 3.

[1510] j. In one example, intra prediction may refer to a conventional intra prediction method (e.g., intra prediction using 35 intra prediction modes in HEVC or 67 intra prediction modes in VVC), or other intra prediction methods that obtain a prediction block from samples in the current slice / sub-block / sub-picture / picture / other video unit (e.g., CU, PU, ​​TU, CTU, CTU row) excluding IBC.

[1511] i. In one example, one or more predefined intra prediction modes may be used to obtain an intra prediction signal.

[1512] 1) In one example, the predefined intra prediction mode may refer to a planar mode, a DC mode, a horizontal mode, and a vertical mode.

[1513] ii. In one example, one or more of the Most Probable Modes (MPMs) may be used to obtain an intra prediction signal.

[1514] iii. In one example, an intra prediction mode may be used to obtain an intra prediction signal, the intra prediction mode being derived using a block vector used to obtain an IBC prediction signal.

[1515] iv. In one example, the intra prediction signal may be obtained using an intra prediction mode derived using a template-based method such as TIMD.

[1516] v. In one example, an intra prediction signal may be obtained using an intra prediction mode, which is derived using neighboring samples or gradients of neighboring samples (such as DIMD).

[1517] vi. In one example, an intra prediction signal may be obtained using ISP.

[1518] vii. In one example, MIP may be used to obtain an intra prediction signal.

[1519] viii. In one example, MRL may be used to obtain an intra prediction signal.

[1520] ix. In one example, the intra prediction signal may be intra template matching prediction (IntraTMP).

[1521] 8. In one example, weight parameters for fusing the IBC prediction signal and the intra prediction signal may be signaled or derived.

[1522] a. In one example, weight parameters may be signaled.

[1523] i. In one example, a set of weight parameters is constructed and an index indicating the weight parameter is signaled.

[1524] b. In one example, the weight parameters may be derived using codec information.

[1525] i. In one example, the codec information may refer to the codec mode of the neighboring unit.

[1526] 1) In one example, the weight parameter may depend on whether one or more neighboring units are coded in intra prediction or IBC mode.

[1527] ii. In one example, the codec information may refer to an intra prediction mode used to obtain an intra prediction signal.

[1528] iii. In one example, the codec information may refer to the block size or block dimension of the current video unit and / or the neighboring video units.

[1529] iv. In one example, the weight parameters may be derived using a template matching method (eg, with minimum template matching cost).

[1530] c. In one example, the weight parameters may be predefined.

[1531] 9. In one example, the reference area of ​​CIBCIP may be smaller than or equal to the reference area of ​​IBC.

[1532] a. In one example, the reference area of ​​CIBCIP may depend on the coding information of intra prediction.

[1533] i. In one example, the reference region of CIBCIP may depend on the intra prediction mode.

[1534] b. Alternatively, the reference region of CIBCIP may be different from that of IBC.

[1535] 10. Whether and / or how to apply CIBCIP mode for a video unit may depend on codec information, which may refer to:

[1536] a. Whether to allow IBC or intra prediction method

[1537] b. Block size and / or block size

[1538] c. Block Depth

[1539] d. Strip / picture type and / or partition tree type (single or dual tree or local dual tree)

[1540] e. Time domain layer identification

[1541] f. Block location

[1542] g. Color component

[1543] Combined signaling of intra block copy and intra prediction

[1544] 11. Indications for CIBCIP mode can be derived on the fly.

[1545] 12. The indication of CIBCIP mode may be signaled conditionally, where the condition may include:

[1546] a. Whether to allow IBC or intra prediction method

[1547] b. Block size and / or block size

[1548] c. Block Depth

[1549] d. Strip / picture type and / or partition tree type (single or dual tree or local dual tree)

[1550] e. Time domain layer identification

[1551] f. Block location

[1552] g. Color component

[1553] 13. One or more syntax elements may be used to signal whether the current block is coded in CIBCIP mode.

[1554] a. In one example, the syntax element may be binarized using a fixed length codec, or a truncated unary codec, or a unary codec, or an EG codec, or a coded flag.

[1555] b. In one example, the syntax element may be bypass coded or context coded.

[1556] i. The context may depend on coded information, such as block dimensions and / or block size, and / or slice / picture type, and / or information of neighboring blocks (adjacent or non-adjacent), and / or information of other codec tools used for the current block, and / or information of the temporal layer.

[1557] c. In one example, when the current video unit is coded by IBC, an indication of CIBCIP mode may be signaled.

[1558] d. In one example, a syntax element may be signaled before or after the indication of IBC-TM mode or IBC-MBVD mode.

[1559] i. In one example, whether and / or how the syntax elements are signaled may depend on whether IBC mode or IBC-TM mode or IBC-MBVD mode is enabled for the video unit.

[1560] e. In one example, one or more syntax elements may be signaled in a sequence header / picture header / SPS / VPS / DPS / DCI / PPS / APS / slice header / tile group header.

[1561] f. In one example, syntax elements may be encoded in a predictive manner.

[1562] g. For example, the syntax elements of the current block can be predicted from the syntax elements of neighboring blocks.

[1563] 14. In one example, for the above points, RR-IBC or symmetric IBC method can be used in CIBCIP.

[1564] 15. In one example, in response to the above points, the RR-IBC or symmetric IBC method can be disabled in CIBCIP.

[1565] 16. In one example, the flip type of the IBC prediction portion may be set to NO_FLIP (eg, 0).

[1566] General Statement

[1567] 17. In the above examples, a video unit may refer to a color component / sub-picture / slice / tile / codec tree unit (CTU) / CTU row / CTU group / codec unit (CU) / prediction unit (PU) / transform unit (TU) / codec tree block (CTB) / codec block (CB) / prediction block (PB) / transform block (TB) / block / sub-block of a block / sub-region in a block / any other region of color containing more than one sample or pixel.

[1568] 18. Whether and / or how to apply the method disclosed above may be signaled at sequence level / picture group level / picture level / slice level / tile group level, for example in sequence header / picture header / SPS / VPS / DPS / DCI / PPS / PPS / slice header / tile group header.

[1569] 19. Whether and / or how the above approach is applied may depend on the following information:

[1570] a. Messages signaled in DPS / SPS / VPS / PPS / APS / Picture Header / Slice Header / Tile Group Header / Largest Codec Unit (LCU) / Codec Unit (CU) / LCU Line / LCU Group / TU / PU Block / Video Codec Unit

[1571] b. Location of CU / PU / TU / block / video codec unit

[1572] c. The block size of the current block and / or its neighboring blocks

[1573] d. Block shape of the current block and / or its neighboring blocks

[1574] e. The mode in which the block is encoded or decoded, such as IBC or non-IBC inter-frame mode or non-IBC sub-block mode

[1575] f. Indication of color format (e.g. 4:2:0, 4:4:4)

[1576] g. Codec tree structure

[1577] h. Strip / tile group type and / or picture type

[1578] i. Color components (e.g., may be applied only to chroma components or luma components)

[1579] j. Time domain layer ID

[1580] k. Standard grade / level / tier

[1581] Embodiments of the present disclosure relate to IBC mode extension. As used herein, the term "block" may refer to a codec tree block (CTB), a codec tree unit (CTU), a codec block (CB), a codec unit (CU), a prediction unit (PU), a transform unit (TU), a prediction block (PB), a transform block (TB), or a video processing unit including multiple samples or pixels. A block may be rectangular or non-rectangular.

[1582] For intra block copy (IBC) codec blocks, a block vector (BV) may be used to indicate the displacement from the current block and a reference block, which is reconstructed within the current picture.

[1583] W and H are the width and height of the current block (eg, luma block).

[1584] It should be noted that the following terms are not limited to the specific terms defined in existing standards. Any variants of codecs are also applicable. For example, the term "GPM" is used to represent any codec that derives two or more sets of motion information and uses the derived motion information and segmentation mode / weight mask to obtain the final prediction. For example, TPM can also be regarded as GPM.

[1585] Below, Mv1 and Mv2 are the motion vectors from the first and second parts of the triangulation or geometric partitioning.

[1586] As used herein, the terms "video unit" or "codec unit" or "block" used may refer to one or more of a color component, a sub-picture, a slice, a tile, a decoding tree unit (CTU), a CTU row, a CTU group, a codec unit (CU), a prediction unit (PU), a transform unit (TU), a codec tree block (CTB), a codec block (CB), a prediction block (PB), a transform block (TB), a block, a sub-block of a block, a sub-region in a block, and a region containing more than one sample or pixel.

[1587] In the present disclosure, with respect to “block encoded and decoded with mode N”, the term “mode N” may refer to a prediction mode (such as MODE_INTRA, MODE_INTER, MODE_PLT, MODE_IBC, etc.), or a coding and decoding technology (such as AMVP, Merge, SMVD, BDOF, PROF, DMVR, AMVR, TM, Affine, CIIP, GPM, MMVD, BCW, HMVP, SbTMVP, etc.).

[1588] It is important to note that the terms mentioned below are not limited to specific terms defined in existing standards. Any changes in codec tools also apply.

[1589] Fig.45 A flow chart of a method 4500 for video processing according to some embodiments of the present disclosure is shown. The method 4500 may be implemented during conversion between bitstreams of blocks.

[1590] like Fig.45 As shown, at box 4510, during the conversion between a target block of a video and a bitstream of the target block, it is determined that an intra block copy (IBC) and combined intra prediction (CIBCIP) mode is applied to the target block.

[1591] At block 4520, an IBC prediction signal and an intra prediction signal are obtained based on the CIBCIP mode.

[1592] At block 4530, a prediction or reconstruction of the target block is derived by combining the IBC prediction signal and the intra prediction signal. In one example, P(x, y) = w ip1 *IP 1 (x,y)+w ip2 *IP 2 (x,y)+…+w ipn *IP n (x,y)+w ibc1 *IBC 1 (x,y)+w ibc2 *IBC 2 (x,y)+…+w ibcm *IBC m (x,y), where P(x,y) is the generated prediction value, IP k is the prediction generated by the k-th intra prediction, IBC j is the prediction generated by the jth IBC, and w ipk and w ibcj is the corresponding weight value.

[1593] At block 4540, a conversion is performed based on the prediction or reconstruction of the target block. In some embodiments, the conversion may include encoding the target block into a bitstream. Alternatively, the conversion may include decoding the target block from the bitstream. Compared to existing solutions, embodiments of the present disclosure may advantageously improve codec efficiency, codec performance, and flexibility.

[1594] In some embodiments, the intra prediction is generated by at least one of the following: angular intra prediction, direct currency (DC), plane, cross component prediction (CCLM), multi-model CCLM, left CCLM or the above CCLM. In some embodiments, the intra prediction mode used in CIBCIP mode is encoded using at least one of the following: most probable mode (MPM), template-based intra mode derivation (TIMD), decoder-side intra mode derivation (DIMD) or another method of signaling intra prediction mode. In some embodiments, a set of intra prediction modes are allowed to be used in CIBCIP mode.

[1595] In some embodiments, the IBC merge mode is used in the CIBCIP mode. In one example, at least one block vector (BV) candidate in the IBC merge list is allowed to be used in the CIBCIP mode. In one example, at least one BV offset is used in the CIBCIP mode. In some embodiments, at least one BV offset is added to the BV candidate before the BV candidate is used to obtain the IBC prediction. In some embodiments, at least one BV offset is indicated. Alternatively, at least one BV offset can be derived.

[1596] In some embodiments, the IBC merge mode includes at least one of the following: a conventional IBC merge mode, an IBC block vector difference (MBVD) merge mode, or an IBC template matching (TM) merge mode. In some embodiments, the IBC Advanced Motion Vector Prediction (AMVP) mode may be used in the CIBCIP mode. In some embodiments, at least one BV predictor in the IBC AMVP list is allowed to be used in the CIBCIP mode.

[1597] In some embodiments, the block vector difference (BVD) used in CIBCIP mode is indicated in the same manner as in IBC mode. In some embodiments, the BVD is predefined. Alternatively, the BVD is derived using the codec information of the target block.

[1598] In some embodiments, at least one of the following items used to obtain the IBC prediction signal is indicated: a merge index indicating a BV candidate in an IBC merge list, or a BVP index indicating a BV predictor in an IBC AMVP list. In one example, a merge index indicating a BV candidate in an IBC merge list (merge Idx) and / or a BVP index indicating a BV predictor in an IBC AMVP list used to obtain the IBC prediction signal (bvp Idx) may be signaled.

[1599] In some embodiments, the binarization or signaling method of at least one of the merge index or the BVP index is the same as that in the IBC mode. Alternatively, at least one of the merge index or the BVP index is predefined. For example, the merge index is set to 0 or 1. Alternatively, the BVP index is set to 0 or 1. For example, mergeIdx=0 or mergeIdx=1, bvpIdx=0 or bvpIdx=1.

[1600] In some embodiments, at least one of the merge index or the BVP index is derived using the codec information of the target block. In some embodiments, at least one of the merge index or the BVP index is derived using template matching. For example, at least one of the merge index or the BVP index is derived using a minimum template matching cost.

[1601] In some embodiments, the construction of the IBC merge list used in CIBCIP mode is the same as the construction used in IBC mode. Alternatively, the construction of the IBC merge list used in CIBCIP mode is different from the construction used in IBC mode.

[1602] In some embodiments, the construction of the IBC merge list used in CIBCIP mode is the same as the construction used in IBC mode. Alternatively, the construction of the IBC merge list used in CIBCIP mode is different from the construction used in IBC mode.

[1603] In some embodiments, the first number of BV candidates in the first IBC merge list for CIBCIP mode is less than or equal to the second number of BV candidates in the second IBC merge list for IBC mode. In this case, the first number and the second number are integers greater than 0, and the first number is less than or equal to the second number. In some embodiments, the first number of BV candidates in the first IBC AMVP list for CIBCIP mode is less than or equal to the second number of BV candidates in the second IBC AMVP list for IBC mode. In this case, the first number and the second number are integers greater than 0, and the first number is less than or equal to the second number. In one example, the number (N) of BV candidates in the IBC merge (or AMVP) list that can be used for CIBCIP is less than or equal to the number (M) of BV candidates in the IBC merge (or AMVP) list that can be used for IBC. N is an integer greater than 0 and less than or equal to M.

[1604] In some embodiments, the first number is one of: 1, 2, 3, 4, 5, or 6. In one example, N=1, or N=2, or N=3, or N=4, or N=5, or N=6.

[1605] In some embodiments, the first N BV candidates of the IBC merge list or the IBC AMVP list are used for CIBCIP mode, and N represents a first number. In one example, the first N BV candidates of the IBC merge (or AMVP) list may be used for CIBCIP.

[1606] In some embodiments, template matching is used to derive or refine the BV used to obtain the IBC prediction signal. In some embodiments, the BV offset is derived using template matching, and the BV offset is added to the BV candidate in the IBC merge list. In some embodiments, the BVD is derived using a template matching-based method. In some embodiments, the BVD symbol is derived using a template matching-based method. In some embodiments, the intra-frame prediction mode or intra-frame prediction method used to obtain the intra-frame prediction signal is used for template matching to derive or refine the BV.

[1607] In some embodiments, the BV list is reordered before the BV list is used in CIBCIP mode. In one example, template matching or bilateral matching costs are used for reordering of the BV list. In one example, template matching or bilateral matching is used during construction of the BV list for CIBCIP mode. In one example, the BV list includes at least one of the following: an IBC merge list or an IBC AMVP list.

[1608] In some embodiments, the reordering method of the BV list for CIBCIP mode is the same as the reordering method for IBC mode. Alternatively, the reordering method of the BC list for CIBCIP mode is different from the reordering method for IBC mode.

[1609] In some embodiments, the third number of BV candidates in the reordered BV list for CIBCIP mode is less than or equal to the fourth number of BV candidates for IBC mode. For example, if IBC merge mode is used for CIBCIP mode, the third number is one of the following: 1, 2, 3, or 4. Alternatively, if IBC AMVP is used for CIBCIP mode, the third number is one of the following: 1, 2, or 3. In one example, the number of BV candidates in the reordered BV list for CIBCIP (N 1 ) can be less than or equal to the number of BV candidates for reordering in IBC mode (M 1 In one example, when the IBC merge mode is used for CIB-CIP, N 1 = 1 or 2 or 3 or 4. In one example, when IBC AMVP is used for CIBCIP, N 1 =1 or 2 or 3.

[1610] In some embodiments, intra prediction refers to one of the following: a conventional intra prediction method, or another intra prediction method other than IBC, obtaining a prediction block with samples in at least one of the following: a current slice, a current tile, a current sub-picture, a current picture, or another video unit. In one example, intra prediction may refer to a conventional intra prediction method (e.g., intra prediction using 35 intra prediction modes in HEVC or 67 intra prediction modes in VVC), or an intra prediction method other than IBC, obtaining a prediction block with samples in at least one of the following: a current slice, a current tile, a current sub-picture, a current picture, or another video unit (e.g., a CU, a PU, a TU, a CTU, a CTU row).

[1611] In some embodiments, at least one predefined intra prediction mode may be used to obtain the intra prediction signal. In one example, the at least one predefined intra prediction mode may include at least one of the following: planar mode, DC mode, horizontal mode, or vertical mode.

[1612] In some embodiments, the intra prediction signal is obtained by at least one MPM.In some embodiments, the intra prediction signal is obtained using an intra prediction mode, the intra prediction mode being derived using a block vector used to obtain the IBC prediction signal.

[1613] In some embodiments, the intra prediction signal is obtained using an intra prediction mode derived using a template-based method, such as TIMD. In some embodiments, the intra prediction signal is obtained using an intra prediction mode derived using neighboring samples or gradients of neighboring samples, such as DIMD.

[1614] In some embodiments, the intra prediction signal may be obtained using at least one of: intra sub-partitioning (ISP), matrix weighted intra prediction (MIP), or multiple reference lines (MRL). In some embodiments, the intra prediction signal is intra template matching prediction (Intra TMP).

[1615] In some embodiments, the IBC prediction signal and the intra prediction signal may be combined based on a set of weight parameters. For example, the set of weight parameters may be indicated. In some embodiments, a set of weight parameters is constructed, and an index indicating a set of weight parameters may be indicated. Alternatively, a set of weight parameters may be derived.

[1616] In some embodiments, a set of weight parameters is derived using coding information. For example, the coding solution information may include a coding method of at least one neighboring unit associated with the target block.

[1617] In some embodiments, a set of weight parameters may depend on whether at least one neighboring unit is coded in intra prediction or IBC mode. For example, the codec information may include an intra prediction mode for obtaining an intra prediction signal. In some embodiments, the codec information may include a block size or block dimension of at least one of the target block or the neighboring video units associated with the target block.

[1618] In some embodiments, a set of weight parameters is derived using a template matching method. In one example, a set of weight parameters can be derived using a template matching method (e.g., with a minimum template matching cost). In some embodiments, a set of weight parameters is predefined.

[1619] In some embodiments, the first reference area of ​​the CIB CIP mode is less than or equal to the second reference area of ​​the IBC. In one example, the first reference area of ​​the CIB CIP mode depends on the codec information of the intra prediction. In one example, the first reference area of ​​the CIB CIP mode depends on the intra prediction mode of the intra prediction. Alternatively, the first reference area of ​​the CIB-CIP mode is different from the second reference area of ​​the IBC.

[1620] In some embodiments, an indication of whether and / or how to derive a prediction or reconstruction of a target block by combining an IBC prediction signal and an intra prediction signal is indicated at one of the following: sequence level, picture level group, picture level, slice level, or tile group level.

[1621] In some embodiments, an indication of whether and / or how to derive a prediction or reconstruction of a target block by combining an IBC prediction signal and an intra prediction signal is indicated in one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header.

[1622] In some embodiments, whether and / or how to derive the prediction or reconstruction of the target block by combining the IBC prediction signal and the intra-frame prediction signal can be determined based on the following information: a message indicated in at least one of a DPS, an SPS, a VPS, a PPS, a picture header, a slice header, a tile group header, a maximum codec unit (LCU), a codec unit (CU), an LCU row, a group of LCUs, a TU, a PU block, and a video codec unit; the position of at least one of a CU, a PU, a TU, a block or a video codec unit, a block size of a current block, a block size of a neighboring block of the current block, a block shape of a current block, a block shape of a neighboring block of the current block, a mode in which the block is coded, an indication of a color format, a codec tree structure, a slice group type, a tile group type, a picture type, a color component, a temporal layer identifier, a standard profile, a standard level or a standard layer.

[1623] In some embodiments, a non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by a video processing device. The method includes: determining that an intra-block copy (IBC) and intra-prediction combination (CIBCIP) mode is applied to a target block; obtaining an IBC prediction signal and an intra-prediction signal based on the CIBCIP mode; deriving a prediction or reconstruction of the target block by combining the IBC prediction signal and the intra-prediction signal; and generating a bitstream of the target block based on the prediction or reconstruction of the target block.

[1624] In some embodiments, a method for storing a bitstream of a video includes: determining that an intra block copy (IBC) combined with intra prediction (CIBCIP) mode is applied to a target block; obtaining an IBC prediction signal and an intra prediction signal based on the CIBCIP mode; deriving a prediction or reconstruction of the target block by combining the IBC prediction signal and the intra prediction signal; generating a bitstream of the target block based on the prediction or reconstruction of the target block; and storing the bitstream in a non-transitory computer-readable recording medium.

[1625] In some embodiments, the residual rotation (RR)-IBC or symmetric IBC method can be used in CIBCIP mode. Alternatively, the RR-IBC or symmetric IBC mode can be disabled in CIBCIP mode. In some embodiments, the flip type of the IBC prediction part is set to NO_FLIP, for example, zero.

[1626] In some embodiments, the target block may refer to one of the following: a color component, a sub-picture, a slice, a tile, a codec tree unit (CTU), a CTU row, a group of CTUs, a codec unit (CU), a prediction unit (PU), a transform unit (TU), a codec tree block (CTB), a codec block (CB), a prediction block (PB), a transform block (TB), a block, a sub-block of a block, a sub-region within a block, or a region including more than one samples or pixels.

[1627] Fig.46 A flow chart of a method 4600 for video processing according to some embodiments of the present disclosure is shown. The method 4600 may be implemented during conversion between bitstreams of blocks.

[1628] like Fig.46 As shown, at block 4610, during the conversion process between the target block of the video and the bitstream of the target block, at least one of the following is determined: whether to apply an intra-block copy (IBC) and intra-prediction combination (CIBCIP) mode to the target block, or a process of applying the CIBCIP mode to the target block based on codec information. In one example, whether and / or how to apply the CIBCIP mode of the video unit may depend on the codec information.

[1629] At block 4620, a conversion is performed based on the determination. In some embodiments, the conversion may include encoding the target block into a bitstream. Alternatively, the conversion may include decoding the target block from the bitstream. Compared to existing solutions, embodiments of the present disclosure may advantageously improve codec efficiency, codec performance, and flexibility.

[1630] In some embodiments, the coding information may include at least one of the following: whether the IBC method is allowed, whether the intra-frame prediction method is allowed, block size, block size, block depth, slice type, picture type, partition tree type, temporal layer identifier, block position or color component.

[1631] In some embodiments, the indication of whether and / or how to apply the CIBCIP mode for a target block may be indicated at: a sequence level, a picture level group, a picture level, a slice level, or a tile group level.

[1632] In some embodiments, an indication of whether and / or how the CIBCIP mode is applied for a target block may be indicated in one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header.

[1633] In some embodiments, whether and / or how to apply the CIBCIP mode for a target block can be determined based on the following information: a message indicated in at least one of the following: a message indicated in at least one of a DPS, an SPS, a VPS, a PPS, a picture header, a slice header, a tile group header, a maximum codec unit (LCU), a codec unit (CU), an LCU row, a group of LCUs, a TU, a PU block, a video coding unit; the position of at least one of a CU, a PU, a TU, a block or a video codec unit, a block size of a current block, a block size of a neighboring block of the current block, a block shape of a current block, a block shape of a neighboring block of the current block, a mode in which the block is encoded, an indication of a color format, a codec tree structure, a slice group type, a tile group type, a picture type, a color component, a temporal layer identifier, a standard profile, a standard level or a standard layer.

[1634] In some embodiments, a non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by a video processing device. The method may include: determining at least one of the following: whether to apply an intra-block copy (IBC) and intra-prediction combination (CIBCIP) mode to a target block, or a process of applying the CIBCIP mode to the target block based on codec information; and generating a bitstream of the target block based on the determination.

[1635] In some embodiments, a method for storing a bitstream of a video includes: determining at least one of the following: whether to apply an intra block copy (IBC) and intra prediction combination (CIBCIP) mode to a target block, or a process of applying the CIBCIP mode to the target block based on codec information; generating a bitstream of the target block based on the determination; and storing the bitstream in a non-transitory computer-readable recording medium.

[1636] In some embodiments, the residual rotation (RR)-IBC or symmetric IBC method is used in CIBCIP mode. Alternatively, the RR-IBC or symmetric IBC mode is disabled in CIBCIP mode. In some embodiments, the flip type of the IBC prediction part is set to NO_FLIP, for example, zero.

[1637] In some embodiments, the target block may refer to one of the following: a color component, a sub-picture, a slice, a tile, a codec tree unit (CTU), a CTU row, a group of CTUs, a codec unit (CU), a prediction unit (PU)...

Claims

1. A video processing method, comprising: During conversion between a target block of a video and a bitstream of the target block, determining that a combined intra block copy (IBC) and intra prediction (CIBCIP) mode is applied to the target block; Acquire an IBC prediction signal and an intra-frame prediction signal based on the CIBCIP mode; deriving a prediction or reconstruction of the target block by combining the IBC prediction signal and the intra prediction signal; and The converting is performed based on the prediction or the reconstruction of the target block.

2. The method of claim 1, wherein the prediction is determined by: P(x,y)=w ip1 *IP1(x,y)+w ip2 *IP2(x,y)+…+w ipn *IP n (x,y)+w ibc1 *IBC1(x,y)+w ibc2 *IBC2(x,y)+…+w ibcm *IBC m (x,y), Where P(x,y) represents the prediction, IP k represents the prediction signal generated by the k-th intra-frame prediction, IBC j is the prediction signal generated by the jth IBC, w ipk and w ibcj Represent the corresponding weight values, n represents the total number of prediction signals generated by the intra-frame prediction, m represents the total number of prediction signals generated by the IBC prediction, k is an integer and is in the range of 1 to n, j is an integer and is in the range of 1 to m, and m, n, x, and y are integers.

3. The method of claim 1 , wherein the intra prediction is generated by at least one of: Angular intra prediction, Direct current (DC), flat, Cross-component prediction (CCLM), Multi-model CCLM, Left CCLM, or CCLM above.

4. The method of claim 1 , wherein the intra prediction mode used in the CIBCIP mode is encoded and decoded using at least one of: Most Probable Model (MPM), Template-based intra mode derivation (TIMD), Decoder-side intra mode derivation (DIMD), or Another method to signal intra prediction mode.

5. The method of claim 1, wherein a set of intra prediction modes are allowed to be used in the CIBCIP mode. The method according to claim 1 , wherein an IBC merge mode is used in the CIBCIP mode.

7. The method of claim 6, wherein at least one block vector (BV) candidate in an IBC merge list is allowed to be used in the CIBCIP mode.

8. The method of claim 6, wherein at least one BV offset is used in the CIBCIP mode.

9. The method of claim 8, wherein the at least one BV offset is added to a BV candidate before the BV candidate is used to obtain the IBC prediction.

10. The method of claim 8, wherein the at least one BV offset is indicated, or Wherein the at least one BV offset is derived.

11. The method of claim 6, wherein the IBC merge mode comprises at least one of the following: Conventional IBC merge mode, IBC - Block Vector Difference (MBVD) merging mode, or IBC - Template Matching (TM) Merge Mode.

12. The method of claim 1, wherein an IBC Advanced Motion Vector Prediction (AMVP) mode is used in the CIBCIP mode.

13. The method of claim 12, wherein at least one BV predictor in an IBC AMVP list is allowed to be used in the CIBCIP mode.

14. The method of claim 12, wherein a block vector difference (BVD) used in the CIBCIP mode is indicated in the same manner as in the IBC mode.

15. The method of claim 12, wherein the BVD is predefined, or The BVD is derived using codec information for the target block.

16. The method of claim 1, wherein at least one of the following items used to obtain the IBC prediction signal is indicated: The merge index indicating the BV candidate in the IBC merge list, or Indicates the BVP index of the BV predictor in the IBC AMVP list. 17 . The method of claim 16 , wherein a binarization or signaling method of at least one of the merge index or the BVP index is the same as that in an IBC mode.

18. The method of claim 16, wherein at least one of the merge index or the BVP index is predefined.

19. The method according to claim 18, wherein the merge index is set to 0 or 1, or The BVP index is set to 0 or 1.

20. The method of claim 16, wherein at least one of the merge index or the BVP index is derived using codec information of the target block, or Wherein at least one of the merge index or the BVP index is derived using template matching.

21. The method of claim 20, wherein the at least one of the merge index or the BVP index is derived with a minimum template matching cost.

22. The method of claim 1, wherein the construction of the IBC merge list used in the CIBCIP mode is the same as the construction used in the IBC mode, or The construction of the IBC merge list used in the CIBCIP mode is different from the construction used in the IBC mode.

23. The method of claim 1, wherein the construction of the IBC AMVP list used in the CIBCIP mode is the same as that used in the IBC mode, or The construction of the IBC AMVP list used in the CIBCIP mode is different from the construction used in the IBC mode.

24. The method of claim 1, wherein a first number of BV candidates in a first IBC merge list for the CIBCIP mode is less than or equal to a second number of BV candidates in a second IBC merge list for the IBC mode, and The first number and the second number are integers greater than 0, and the first number is less than or equal to the second number.

25. The method of claim 1, wherein a first number of BV candidates in a first IBC AMVP list for the CIBCIP mode is less than or equal to a second number of BV candidates in a second IBC AMVP list for the IBC mode, and The first number and the second number are integers greater than 0, and the first number is less than or equal to the second number.

26. The method of claim 24 or 25, wherein the first number is one of: 1, 2, 3, 4, 5 or 6.

27. The method according to claim 24 or 25, wherein the top N BV candidates of the IBC merge list or the IBC AMVP list are used for the CIBCIP mode, and Wherein N represents the first number.

28. The method of claim 1, wherein template matching is used to derive or refine the BV used to obtain the IBC prediction signal.

29. The method of claim 28, wherein a BV offset is derived using the template matching, and the BV offset is added to a BV candidate in an IBC merge list.

30. The method of claim 28, wherein the BVD is derived using a template matching based approach.

31. The method of claim 28, wherein the BVD symbols are derived using a template matching based approach.

32. The method of claim 28, wherein an intra prediction mode or an intra prediction method for obtaining the intra prediction signal is used in the template matching to derive or refine the BV.

33. The method of claim 1, further comprising: The BV list is reordered before it is used in the CIBCIP mode.

34. The method of claim 33, wherein template matching or bilateral matching cost is used for the reordering of the BV list.

35. The method of claim 33, wherein during construction of the BV list for the CIBCIP mode, template matching or bilateral matching is used.

36. The method of claim 33, wherein the BV list includes at least one of the following: IBC Consolidated List, or IBC AMVP list.

37. The method of claim 33, wherein a reordering method for the BV list for the CIBCIP mode is the same as a reordering method for the IBC mode, or The reordering method for the BV list for the CIBCIP mode is different from the reordering method for the IBC mode.

38. The method of claim 37, wherein a third number of BV candidates in the BV list used for the reordering for the CIBCIP mode is less than or equal to a fourth number of BV candidates used for the reordering for the IBC mode.

39. The method of claim 38, wherein if an IBC merge mode is used for the CIBCIP mode, the third number is one of: 1, 2, 3 or 4.

40. The method of claim 38, wherein if IBC AMVP is used for the CIBCIP mode, the third number is one of: 1, 2, or 3.

41. The method of claim 1, wherein the intra prediction is one of: Conventional intra prediction methods, or Another intra prediction method other than IBC obtains a prediction block having samples in at least one of the following: Current strip, Current tile, Current sub-image, The current image, or Another video unit.

42. The method of claim 41, wherein the intra prediction signal is obtained using at least one predefined intra prediction mode.

43. The method of claim 42, wherein the at least one predefined intra prediction mode comprises at least one of: Plane mode, DC mode, Horizontal mode, or Vertical mode.

44. The method according to claim 41, wherein the intra prediction signal is obtained by at least one MPM.

45. The method of claim 41, wherein the intra prediction signal is obtained using an intra prediction mode, the intra prediction mode being derived using a block vector used to obtain the IBC prediction signal.

46. ​​The method of claim 41, wherein the intra prediction signal is obtained using an intra prediction mode, the intra prediction mode being derived using a template based approach.

47. The method of claim 41, wherein the intra prediction signal is obtained using an intra prediction mode, the intra prediction mode being derived using neighboring samples or gradients of the neighboring samples.

48. The method of claim 41, wherein the intra prediction signal is obtained using at least one of: Intra-frame sub-partitioning (ISP), Matrix Weighted Intra Prediction (MIP), or Multiple Reference Line (MRL).

49. The method of claim 41, wherein the intra prediction signal is intra template matching prediction (IntraTMP).

50. The method of claim 1, wherein the IBC prediction signal and the intra prediction signal are combined based on a set of weight parameters.

51. The method of claim 50, wherein the set of weight parameters is indicated.

52. The method of claim 51, wherein the set of weight parameters is constructed and an index indicating the set of weight parameters is indicated.

53. The method of claim 50, wherein the set of weight parameters is derived.

54. The method of claim 53, wherein the set of weight parameters is derived using codec information.

55. The method of claim 54, wherein the codec information comprises a codec method of at least one neighboring unit associated with the target block.

56. The method of claim 55, wherein the set of weight parameters depends on whether the at least one neighboring unit is coded by intra prediction or by IBC mode.

57. The method of claim 54, wherein the codec information includes an intra prediction mode used to obtain the intra prediction signal.

58. The method of claim 54, wherein the codec information comprises a block size or block dimension of at least one of the target block or a neighboring video unit associated with the target block.

59. The method of claim 53, wherein the set of weight parameters is derived using a template matching method.

60. The method of claim 50, wherein the set of weight parameters is predefined.

61. The method of claim 1, wherein the first reference area of ​​the CIBCIP mode is less than or equal to the second reference area of ​​the IBC.

62. The method of claim 61, wherein the first reference area of ​​the CIBCIP mode depends on codec information of the intra prediction.

63. The method of claim 61, wherein the first reference region of a CIBCIP mode depends on an intra prediction mode of the intra prediction.

64. The method of claim 1, wherein the first reference region of the CIBCIP pattern is different from the second reference region of the IBC.

65. The method according to any one of claims 1-64, wherein the indication of whether and / or how the prediction or the reconstruction of the target block is derived by combining the IBC prediction signal and the intra prediction signal is indicated in one of the following: Sequence level, Picture level group, Picture level, Stripe level, or Tile group level.

66. The method according to any one of claims 1-64, wherein the indication of whether and / or how the prediction or the reconstruction of the target block is derived by combining the IBC prediction signal and the intra prediction signal is indicated in one of the following: Sequence header, Picture header, Sequence Parameter Set (SPS), Video Parameter Set (VPS), Dependent Parameter Set (DPS), Decoding Capability Information (DCI), Picture Parameter Set (PPS), Adaptive Parameter Set (APS), Strip header, or Tile group header.

67. The method of any one of claims 1-64, further comprising: It is determined whether and / or how to derive the prediction or the reconstruction of the target block by combining the IBC prediction signal and the intra prediction signal based on the following information: A message indicated in at least one of a DPS, an SPS, a VPS, a PPS, an APS, a picture header, a slice header, a tile group header, a largest codec unit (LCU), a codec unit (CU), an LCU row, a group of LCUs, a TU, a PU block, a video codec unit; a position of at least one of the CU, the PU, the TU, the block, or the video codec unit, the block size of the current block, The block sizes of the neighboring blocks of the current block, the block shape of the current block, The block shapes of the neighboring blocks of the current block, The mode in which the block is encoded or decoded, Instructions for color format, Codec tree structure, Stripe group type, Tile group type, Image type, Color components, Time domain layer identification, Standard grade, the level of the standard, or The standard layer.

68. A method of video processing, comprising: During conversion between a target block of a video and a bitstream of the target block, determining at least one of the following: whether to apply a combined intra block copy (IBC) and intra prediction (CIBCIP) mode to the target block, or a process of applying the CIBCIP mode to the target block based on codec information; as well as The converting is performed based on the determination.

69. The method according to claim 68, wherein the codec information comprises at least one of the following: Whether the IBC method is allowed, Whether the intra prediction method is allowed, Block size, Block size, Block depth, Strip type, Image type, Partition tree type, Time domain layer identification, block location, or Color components.

70. The method according to any one of claims 68-69, wherein the indication of whether and / or how to apply the CIBCIP mode for the target block is indicated at: Sequence level, Picture level group, Picture level, Stripe level, or Tile group level.

71. A method according to any one of claims 68 to 69, wherein the indication of whether and / or how to apply the CIBCIP mode for the target block is indicated by one of: Sequence header, Picture header, Sequence Parameter Set (SPS), Video Parameter Set (VPS), Dependent Parameter Set (DPS), Decoding Capability Information (DCI), Picture Parameter Set (PPS), Adaptive Parameter Set (APS), Strip header, or Tile group header.

72. The method of any one of claims 68 to 69, further comprising: Determine whether and / or how to apply the CIBCIP mode to the target block based on the following information: A message indicated in at least one of a DPS, an SPS, a VPS, a PPS, an APS, a picture header, a slice header, a tile group header, a largest codec unit (LCU), a codec unit (CU), an LCU row, a group of LCUs, a TU, a PU block, a video codec unit; a position of at least one of the CU, the PU, the TU, the block, or the video codec unit, the block size of the current block, The block sizes of the neighboring blocks of the current block, the block shape of the current block, The block shapes of the neighboring blocks of the current block, The mode in which the block is encoded or decoded, Instructions for color format, Codec tree structure, Stripe group type, Tile group type, Image type, Color components, Time domain layer identification, Standard grade, the level of the standard, or The standard layer.

73. A method of video processing, comprising: During conversion between a target block of video and a bitstream of the target block, determining based on an indication associated with a combined intra block copy (IBC) and intra prediction (CIBCIP) mode whether the CIBCIP mode is applied to the target block; as well as The converting is performed based on the determination.

74. The method of claim 73, wherein the indication associated with the CIBCIP mode is derived dynamically.

75. The method of claim 73, wherein the indication associated with the CIBCIP mode is indicated based on a condition.

76. The method of claim 75, wherein the condition comprises at least one of: Whether IBC or intra prediction method is allowed, Block size, Block size, Block depth, Strip type, Image type, Partition tree type, Time domain layer identification, block location, or Color components.

77. The method of claim 73, wherein whether the target block is encoded or decoded using the CIBCIP mode is indicated using at least one syntax element.

78. The method of claim 77, wherein the at least one syntax element is binarized by at least one of: Fixed length codec, Truncated unary codec, Unary codec, or EG codec, or The at least one syntax element is encoded as a flag.

79. The method of claim 77, wherein the at least one syntax element is bypass coded or context coded.

80. The method of claim 79, wherein the context depends on encoded information.

81. The method of claim 80, wherein the encoded information comprises at least one of: Block size, Block size, Strip type, Image type, Information about neighboring blocks, information about other codecs used for the target block, or Time domain layer information.

82. The method of claim 77, wherein if the target block is IBC coded, the indication associated with the CIBCIP mode is indicated.

83. The method of claim 77, wherein the at least one syntax element is indicated before an indication of an IBC-TM mode or an indication of an IBC-MBCD mode, or The at least one syntax element is indicated after the indication of the IBC-TM mode or the indication of the IBC-MBCD mode.

84. The method of claim 77, wherein whether and / or how to indicate the at least one syntax element depends on at least one of the following: Whether IBC mode is enabled for the target block, Whether IBC-TM mode is enabled for the target block, or Whether IBC-MBCD mode is enabled for the target block.

85. The method of claim 77, wherein the at least one syntax element is indicated at: Sequence header, Picture header, Sequence Parameter Set (SPS), Video Parameter Set (VPS), Dependent Parameter Set (DPS), Decoding Capability Information (DCI), Picture Parameter Set (PPS), Adaptive Parameter Set (APS), Strip header, or Tile group header.

86. The method of claim 77, wherein the at least one syntax element is encoded in a predictive manner.

87. The method of claim 77, wherein the at least one syntax element is predicted by syntax elements of neighboring blocks.

88. A method according to any one of claims 73 to 87, wherein an indication of whether and / or how to determine that the CIBCIP mode is applied to the target block is indicated at: Sequence level, Picture level group, Picture level, Stripe level, or Tile group level.

89. A method according to any one of claims 73 to 87, wherein the indication of whether and / or how to determine that the CIBCIP mode is applied to the target block is indicated in one of: Sequence header, Picture header, Sequence Parameter Set (SPS), Video Parameter Set (VPS), Dependent Parameter Set (DPS), Decoding Capability Information (DCI), Picture Parameter Set (PPS), Adaptive Parameter Set (APS), Strip header, or Tile group header.

90. The method of any one of claims 73-87, further comprising: Determine whether and / or how to determine that the CIBCIP mode is applied to the target block based on the following information: A message indicated in at least one of a DPS, an SPS, a VPS, a PPS, an APS, a picture header, a slice header, a tile group header, a largest codec unit (LCU), a codec unit (CU), an LCU row, a group of LCUs, a TU, a PU block, a video codec unit; a position of at least one of the CU, the PU, the TU, the block, or the video codec unit, the block size of the current block, The block sizes of the neighboring blocks of the current block, the block shape of the current block, The block shapes of the neighboring blocks of the current block, The mode in which the block is encoded or decoded, Instructions for color format, Codec tree structure, Stripe group type, Tile group type, Image type, Color components, Time domain layer identification, Standard grade, the level of the standard, or The standard layer.

91. The method according to any one of claims 1-90, wherein Residual Rotation (RR)-IBC or Symmetric IBC method is used in the CIBCIP mode, or The RR-IBC or the symmetric IBC mode is disabled in the CIBCIP mode.

92. The method of any one of claims 1-90, wherein the flip type of the IBC prediction part is set to NO_FLIP.

93. The method according to any one of claims 1-90, wherein the target block refers to one of the following: Color components, Sub-picture, Strips, Tile, Codec Tree Unit (CTU), CTU line, A group of CTUs, Codec Unit (CU), Prediction Unit (PU), Transformation Unit (TU), Codec Tree Block (CTB), Codec Block (CB), Prediction Block (PB), Transform Block (TB), piece, Sub-blocks of a block, A sub-region within a block, or An area that contains more than one sample or pixel.

94. The method of any one of claims 1 to 93, wherein the converting comprises encoding the target block into the bitstream.

95. The method of any one of claims 1 to 93, wherein the converting comprises decoding the target block from the bitstream.

96. A device for processing video data, comprising a processor and a non-volatile memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform a method according to any one of claims 1-95.

97. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method of any one of claims 1-95.

98. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing device, wherein the method comprises: determining that a combined intra block copy (IBC) and intra prediction (CIBCIP) mode is applied to a target block of the video; Acquire an IBC prediction signal and an intra-frame prediction signal based on the CIBCIP mode; deriving a prediction or reconstruction of the target block by combining the IBC prediction signal and the intra prediction signal; and Based on the prediction or the reconstruction of the target block, a bitstream of the target block is generated.

99. A method for storing a bitstream of a video, comprising: determining that a combined intra block copy (IBC) and intra prediction (CIBCIP) mode is applied to a target block of the video; Acquire an IBC prediction signal and an intra-frame prediction signal based on the CIBCIP mode; deriving a prediction or reconstruction of the target block by combining the IBC prediction signal and the intra prediction signal; as well as Based on the prediction or the reconstruction of the target block, a bitstream of the target block is generated. The bit stream is stored in a non-transitory computer-readable recording medium.

100. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing device, wherein the method comprises: A process of determining whether to apply an intra block copy (IBC) and combined intra prediction (CIBCIP) mode to a target block of the video, or applying the CIBCIP mode to the target block based on codec information; as well as A bitstream of the target block is generated based on the determination.

101. A method for storing a bitstream of a video, comprising: A process of determining whether to apply an intra block copy (IBC) and combined intra prediction (CIBCIP) mode to a target block of the video, or applying the CIBCIP mode to the target block based on codec information; generating a bitstream of the target block based on the determining; as well as The bit stream is stored in a non-transitory computer-readable recording medium.

102. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing device, wherein the method comprises: determining, based on an indication associated with a combined intra block copy (IBC) and intra prediction (CIBCIP) mode, whether the CIBCIP mode is applied to a target block of the video; as well as A bitstream of the target block is generated based on the determination.

103. A method for storing a bitstream of a video, comprising: determining, based on an indication associated with a combined intra block copy (IBC) and intra prediction (CIBCIP) mode, whether the CIBCIP mode is applied to a target block of the video; generating a bitstream of the target block based on the determining; as well as The bit stream is stored in a non-transitory computer-readable recording medium.

Images