Method and device for video processing and medium

Abstract

Embodiments of the present disclosure provide a video processing solution. A method for video processing is presented. The method includes applying a combination of Intra Block Copying (IBC) and Intra Prediction (CIBIP) to a video unit of a video for a conversion between the video unit and a bitstream of the video unit; deriving a prediction of the video unit by combining the IBC prediction signal and the intra prediction signal; and performing the conversion based on the prediction of the video unit.

Description

Technical Field

[0001] Embodiments of the present disclosure relate generally to video processing techniques, and more particularly, to combined intra block copying and intra prediction. Background Art

[0002] Nowadays, digital video capabilities are being applied to all aspects of people's lives. Various video compression technologies, such as MPEG-2, MPEG-4, ITU-TH.263, ITU-TH.264 / MPEG-4 Part 10 Advanced Video Codec (AVC), ITU-TH.265 High Efficiency Video Codec (HEVC) standard, and Versatile Video Codec (VVC) standard have been proposed for video encoding / decoding. However, the encoding and decoding efficiency of video encoding and decoding technologies is generally expected to be further improved. Summary of the invention

[0003] The embodiments of the present disclosure provide a video processing solution.

[0004] In a first aspect, a method for video processing is proposed. The method comprises: for conversion between a video unit of a video and a bitstream of the video unit, applying a combination of intra-frame block copy (IBC) and intra-frame prediction (CIBIP) to the video unit; deriving a prediction of the video unit by combining an IBC prediction signal and an intra-frame prediction signal; and performing the conversion based on the prediction of the video unit. In this way, coding efficiency can be improved.

[0005] In a second aspect, another method of video processing is proposed. The method comprises: combining a plurality of reference lines for conversion between a video unit of a video and a bitstream of the video unit; deriving an intra-frame prediction of the video unit based on the combined plurality of reference lines; and performing the conversion based on the intra-frame prediction. In this way, the coding efficiency can be improved.

[0006] In a third aspect, a video processing device is provided. The device includes a processor and a non-volatile memory having instructions thereon. The instructions, when executed by the processor, cause the processor to perform the method according to the first aspect or the second aspect of the present disclosure.

[0007] In a fourth aspect, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores instructions, wherein the instructions enable a processor to execute the method according to the first aspect or the second aspect of the present disclosure.

[0008] In a fifth aspect, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video, the bitstream of the video being generated by a method performed by a video processing device. The method comprises: applying a combination of intra-block copy (IBC) and intra-frame prediction (CIBCIP) to a video unit of the video; deriving a prediction of the video unit by combining an IBC prediction signal and an intra-frame prediction signal; and generating the bitstream based on the prediction of the video unit.

[0009] In a sixth aspect, a method for storing a bitstream of a video is provided. The method comprises: applying a combination of intra-block copy (IBC) and intra-frame prediction (CIBCIP) to a video unit of the video; deriving a prediction of the video unit by combining an IBC prediction signal and an intra-frame prediction signal; generating the bitstream based on the prediction of the video unit; and storing the bitstream in a non-transitory computer-readable recording medium.

[0010] In a seventh aspect, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video, the bitstream of the video being generated by a method performed by a video processing device. The method comprises: combining a plurality of reference lines; deriving an intra-frame prediction of the video unit based on the combined plurality of reference lines; and generating the bitstream based on the prediction of the video unit.

[0011] In an eighth aspect, a method for storing a bitstream of a video is provided. The method comprises: combining a plurality of reference lines; deriving an intra-frame prediction of the video unit based on the combined plurality of reference lines; generating the bitstream based on the prediction of the video unit; and storing the bitstream in a non-transitory computer-readable recording medium.

[0012] This summary is provided to introduce some concepts in a simplified form, which are further described in the following detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other purposes, features and advantages of the embodiments of the present disclosure will become more apparent through the following detailed description and with reference to the accompanying drawings. In the embodiments of the present disclosure, the same reference numerals generally refer to the same components.

[0014] Figure 1 illustrates a block diagram of an example video encoding and decoding system according to some embodiments of the present disclosure;

[0015] Figure 2illustrates a block diagram illustrating a first example video encoder according to some embodiments of the present disclosure;

[0016] Figure 3 illustrates a block diagram of an example video decoder according to some embodiments of the present disclosure;

[0017] Figure 4 illustrates an example of an encoder block diagram;

[0018] Figure 5 67 intra prediction modes are shown;

[0019] Figure 6 Reference samples for wide-angle intra prediction are shown;

[0020] Figure 7 It shows the problem of discontinuity when the direction exceeds 45°;

[0021] Figure 8 The MMVD search point is shown;

[0022] Fig. 9 This is an explanation of the symmetric MVD mode;

[0023] Fig.10 shows the extended CU area used in BDOF;

[0024] Fig.11 An affine motion model based on control points is shown;

[0025] Fig.12 The affine MVF of each sub-block is shown;

[0026] Fig.13 The location of the inherited affine motion predictor is shown;

[0027] Fig.14 Control point motion vector inheritance is shown;

[0028] Fig.15 The positions of candidate positions for the constructed affine merge pattern are shown;

[0029] Fig.16 is a schematic diagram of the use of motion vectors for the proposed combination method;

[0030] Fig.17 Sub-block MV VSB and pixel Δv(i,j) are shown;

[0031] Fig.18A shows the spatially adjacent blocks used by ATVMP;

[0032] Fig.18BIt is shown that the sub-CU region is derived by applying motion shifts from spatially neighboring blocks and scaling the motion information from the corresponding co-located sub-CU.

[0033] Fig.19 Positional illumination compensation is shown;

[0034] Fig. 20 No subsampling for the short edges is shown;

[0035] Fig.21 Decoding side motion vector refinement is shown.

[0036] Fig. 22 A diamond-shaped area in the search area is shown;

[0037] Fig.23 The positions of spatial merging candidates are shown;

[0038] Fig.24 shows candidate pairs considered for redundancy check of spatial merging candidates;

[0039] Fig.25 is an illustration of motion vector scaling of temporal merging candidates;

[0040] Fig.26 Candidate positions for temporal merging candidates C0 and C1 are shown;

[0041] Fig. 27 The VVC spatial neighboring blocks of the current block are shown;

[0042] Fig.28 is a diagram of the virtual blocks in the i-th round of search;

[0043] Fig.29 An example of GPM segmentation grouped at the same angle is shown;

[0044] Fig.30 Unidirectional prediction MV selection for geometric partitioning mode is shown;

[0045] Fig.31 An exemplary generation of warping weights w_0 using a geometric partitioning pattern is shown;

[0046] Fig.32 The spatial neighboring blocks used to derive spatial merging candidates are shown;

[0047] Fig.33 shows the template matching performed on the search area around the initial MV;

[0048] Fig.34 is a diagram of the sub-blocks of the OBMC application;

[0049] Fig.35The SBT position, type and transformation type are shown;

[0050] Fig.36 Neighboring samples used to calculate SAD are shown;

[0051] Fig.37 Neighboring samples used to calculate SAD for sub-CU level motion information are shown;

[0052] Fig.38 The sorting process is shown;

[0053] Fig.39 shows the reordering process in the encoder;

[0054] Fig.40 shows the reordering process in the decoder;

[0055] Fig.41 is a graphic representation of the extended reference area;

[0056] Fig.42 shows the IBC reference area depending on the current CU position;

[0057] Fig.43 An example of symmetry in a picture of screen content is shown;

[0058] FIG44( a ) is an illustration of BV adjustment for horizontal flipping;

[0059] Fig. 44(b) is an illustration of BV adjustment for vertical flipping;

[0060] Fig.45 The intra-frame template matching search area used is shown;

[0061] Fig.46 Examples of different numbers of samples in different reference lines for fusion are shown;

[0062] Fig.47 Examples of different numbers of samples in different reference lines used for fusion are shown, and samples surrounded by square boxes are discarded and not used for fusion of the reference lines;

[0063] Fig.48 Examples of different numbers of samples in different reference lines used for fusion are shown, and the samples represented by the empty circles in the reference Ln are filled and used for fusion of the reference lines;

[0064] Fig.49 A flowchart of a method for video processing according to an embodiment of the present disclosure is shown;

[0065] Fig.50 A flowchart of a method for video processing according to an embodiment of the present disclosure is shown;

[0066] Fig.51 Illustrated is a block diagram of a computing device in which various embodiments of the present disclosure may be implemented.

[0067] Throughout the drawings, the same or similar reference numbers generally refer to the same or similar elements. DETAILED DESCRIPTION

[0068] The principle of the present invention is now described in conjunction with some embodiments. It should be understood that these embodiments are described only for illustrative purposes to help those skilled in the art understand and implement the present invention, and do not constitute any limitation to the scope of the present invention. The disclosure described herein can be implemented in various ways except for the way described below.

[0069] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0070] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but it is not necessary that every embodiment include the particular feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an example, it is considered within the knowledge of those skilled in the art to affect embodiments related to such features, structures, or characteristics, for example, whether or not explicitly described.

[0071] It should be understood that although the terms "first" and "second" etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0072] The terms used herein are used only for the purpose of describing specific embodiments and are not intended to limit the example embodiments. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "including", "comprising", and / or "having", when used herein, specify the presence of stated features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof. .

[0073] Example Environment

[0074] Figure 11 is a block diagram illustrating an example video codec system 100 that may utilize the techniques of the present disclosure. As shown, the video codec system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a video encoding device, and the destination device 120 may also be referred to as a video decoding device. In operation, the source device 110 may be configured to generate encoded video data, and the destination device 120 may be configured to decode the encoded video data generated by the source device 110. The source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0075] Video source 112 may include a source such as a video capture device. Examples of a video capture device include, but are not limited to, an interface that receives video data from a video content provider, a computer graphics system for generating video data, and / or combinations thereof.

[0076] The video data may include one or more pictures. The video encoder 114 encodes the video data from the video source 112 to generate a bitstream. The bitstream may include a bit sequence that forms a coded representation of the video data. The bitstream may include coded pictures and associated data. The coded pictures are coded representations of pictures. The associated data may include sequence parameter sets, picture parameter sets, and other grammatical structures. The I / O interface 116 may include a modulator / demodulator and / or a transmitter. The coded video data may be directly transmitted to the destination device 120 via the network 130A via the I / O interface 116. The coded video data may also be stored on a storage medium / server 130B for access by the destination device 120.

[0077] The destination device 120 may include an I / O interface 126, a video decoder 124, and a display device 122. The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain encoded video data from the source device 110 or the storage medium / server 130B. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to the user. The display device 122 may be integrated with the destination device 120, or may be outside the destination device 120, which is configured to be connected to an external display device interface.

[0078] The video encoder 114 and the video decoder 124 may operate according to a video compression standard, such as the High Efficiency Video Codec (HEVC) standard, the Versatile Video Codec (VVC) standard, and other existing and / or further standards.

[0079] Figure 2is a block diagram showing an example of a video encoder 200 according to some embodiments of the present disclosure, which may be Figure 1 An example of a video encoder 114 in the system 100 is shown.

[0080] Video encoder 200 may be configured to implement any or all of the techniques of this disclosure. Figure 2 In the example of , video encoder 200 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video encoder 200. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0081] In some embodiments, the video encoder 200 may include a partitioning unit 201, a prediction unit 202, a residual generation unit 207, a transformation unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transformation unit 211, a reconstruction unit 212, a buffer 213 and an entropy coding and decoding unit 214, and the prediction unit 202 may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205 and an intra-frame prediction unit 206.

[0082] In other examples, the video encoder 200 may include more, fewer, or different functional components. In one example, the prediction unit 202 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode, where at least one reference picture is a picture where the current video block is located.

[0083] Furthermore, although some components (such as the motion estimation unit 204 and the motion compensation unit 205) may be integrated, for the purpose of explanation, these components are described in detail below. Figure 2 are represented separately in the example.

[0084] The partitioning unit 201 may partition a picture into one or more video blocks. The video encoder 200 and the video decoder 300 may support various video block sizes.

[0085] The mode selection unit 203 may select one of a plurality of codec modes (intra-frame coding or inter-frame coding), for example, based on the error result, and provide the generated intra-frame codec block or inter-frame codec block to the residual generation unit 207 to generate residual block data, and to the reconstruction unit 212 to reconstruct the codec block for use as a reference picture. In some examples, the mode selection unit 203 may select a combination of intra-frame and inter-frame prediction (CIIP) modes, where the prediction is based on an inter-frame prediction signal and an intra-frame prediction signal. In the case of inter-frame prediction, the mode selection unit 203 may also select a resolution for the motion vector (e.g., sub-pixel precision or integer pixel precision) for the block.

[0086] To perform inter-frame prediction on the current video block, the motion estimation unit 204 may generate motion information for the current video block by comparing the current video block with one or more reference frames from the buffer 213. The motion compensation unit 205 may determine a predicted video block for the current video block based on the motion information and decoded samples of pictures from the buffer 213 other than the picture associated with the current video block.

[0087] The motion estimation unit 204 and the motion compensation unit 205 may perform different operations on the current video block, for example, depending on whether the current video block is in an I slice, a P slice, or a B slice. As used herein, an "I slice" may refer to a portion of a picture consisting of macroblocks, all of which are based on macroblocks within the same picture. Furthermore, as used herein, in some aspects, a "P slice" and a "B slice" may refer to a portion of a picture consisting of macroblocks that are independent of macroblocks in the same picture.

[0088] In some examples, the motion estimation unit 204 may perform unidirectional prediction on the current video block, and the motion estimation unit 204 may search the reference pictures of list 0 or list 1 to find the reference video block for the current video block. The motion estimation unit 204 may then generate a reference index and a motion vector, the reference index indicating the reference picture in list 0 or list 1 containing the reference video block, and the motion vector indicating the spatial displacement between the current video block and the reference video block. The motion estimation unit 204 may output the reference index, the prediction direction indicator, and the motion vector as the motion information of the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0089] Alternatively, in other examples, the motion estimation unit 204 may perform bidirectional prediction on the current video block. The motion estimation unit 204 may search the reference pictures in list 0 for a reference video block for the current video block, and may also search the reference pictures in list 1 for another reference video block for the current video block. The motion estimation unit 204 may then generate a plurality of reference indexes and a plurality of motion vectors, the plurality of reference indexes indicating a plurality of reference pictures in list 0 and list 1 containing a plurality of reference video blocks, and the plurality of motion vectors indicating a plurality of spatial displacements between the plurality of reference video blocks and the current video block. The motion estimation unit 204 may output the plurality of reference indexes and the plurality of motion vectors of the current video block as motion information of the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the plurality of reference video blocks indicated by the motion information of the current video block.

[0090] In some examples, motion estimation unit 204 may output a complete set of motion information for use in a decoding process by a decoder. Alternatively, in some embodiments, motion estimation unit 204 may signal motion information of a current video block with reference to motion information of another video block. For example, motion estimation unit 204 may determine that motion information of a current video block is sufficiently similar to motion information of a neighboring video block.

[0091] In one example, motion estimation unit 204 may indicate to video decoder 300 a value in a syntax structure associated with the current video block that indicates that the current video block has the same motion information as another video block.

[0092] In another example, the motion estimation unit 204 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 300 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0093] As discussed above, the video encoder 200 may signal motion vectors in a predictive manner.Two examples of prediction signaling techniques that may be implemented by the video encoder 200 include Advanced Motion Vector Prediction (AMVP) and merge mode signaling.

[0094] The intra prediction unit 206 may perform intra prediction on the current video block. When the intra prediction unit 206 performs intra prediction on the current video block, the intra prediction unit 206 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a prediction video block and various syntax elements.

[0095] The residual generation unit 207 may generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) the prediction video block(s) of the current video block from the current video block. The residual data of the current video block may include residual video blocks corresponding to different sample portions of samples in the current video block.

[0096] In other examples, such as in skip mode, there may be no residual data for the current video block, and the residual generation unit 207 may not perform a subtraction operation.

[0097] Transform processing unit 208 may generate one or more transform coefficient video blocks for a current video block by applying one or more transforms to the residual video block associated with the current video block.

[0098] After transform processing unit 208 generates a transform coefficient video block associated with the current video block, quantization unit 209 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0099] The inverse quantization unit 210 and the inverse transform unit 211 may apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct a residual video block from the transform coefficient video block. The reconstruction unit 212 may add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by the prediction unit 202 to generate a reconstructed video block associated with the current video block for storage in the buffer 213.

[0100] After reconstruction unit 212 reconstructs the video block, a loop filtering operation may be performed to reduce video blocking artifacts in the video block.

[0101] The entropy codec unit 214 may receive data from other functional components of the video encoder 200. When the entropy codec unit 214 receives data, the entropy codec unit 214 may perform one or more entropy encoding operations to generate entropy-coded data and output a bitstream including the entropy-coded data.

[0102] Figure 3 is a block diagram showing an example of a video decoder 300 according to some embodiments of the present disclosure, which may be Figure 1 An example of a video decoder 124 in the system 100 is shown.

[0103] Video decoder 300 may be configured to perform any or all of the techniques of this disclosure. Figure 3 In the example of , video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video decoder 300. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0104] exist Figure 3 In the example of , the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, and a reconstruction unit 306 and a buffer 307. In some examples, the video decoder 300 can perform a decoding process that is generally opposite to the encoding process described with respect to the video encoder 200.

[0105] The entropy decoding unit 301 can retrieve the encoded bitstream. The encoded bitstream may include entropy encoded video data (e.g., encoded video data blocks). The entropy decoding unit 301 can decode the entropy encoded video data, and the motion compensation unit 302 can determine motion information from the entropy decoded video data, the motion information including motion vectors, motion vector precision, reference picture list indexes, and other motion information. The motion compensation unit 302 can determine the information, for example, by performing AMVP and merge mode. AMVP is used, including deriving several most likely candidates based on data and reference pictures of adjacent PBs. The motion information typically includes horizontal and vertical motion vector displacement values, one or two reference picture indexes, and in the case of prediction areas in B strips, also includes an identification of which reference picture list is associated with each index. As used herein, in some aspects, "merge mode" may refer to deriving motion information from blocks that are adjacent in space or time.

[0106] The motion compensation unit 302 may generate motion compensated blocks, possibly performing interpolation based on interpolation filters.Identifiers for the interpolation filters used with sub-pixel precision may be included in the syntax elements.

[0107] The motion compensation unit 302 may calculate interpolated values ​​for sub-integer pixels of a reference block using interpolation filters used by the video encoder 200 during encoding of the video block. The motion compensation unit 302 may determine the interpolation filters used by the video encoder 200 based on received syntax information, and the motion compensation unit 302 may use the interpolation filters to generate a prediction block.

[0108] The motion compensation unit 302 may use at least part of the syntax information to determine the size of blocks used to encode (multiple) frames and / or (multiple) slices of the encoded video sequence, partition information describing how each macroblock of a picture of the encoded video sequence is partitioned, a mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-frame coding block, and other information for decoding the encoded video sequence. As used herein, in some aspects, a "slice" may refer to a data structure that can be decoded independently of other slices of the same picture in terms of entropy coding, signal prediction, and residual signal reconstruction. A slice may be an entire picture, or it may be a region of a picture.

[0109] The intra prediction unit 303 may use, for example, an intra prediction mode received in the bitstream to form a prediction block from spatially neighboring blocks. The inverse quantization unit 304 inverse quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 305 applies an inverse transform.

[0110] The reconstruction unit 306 may obtain the decoded block, for example, by adding the residual block to the corresponding prediction block generated by the motion compensation unit 302 or the intra prediction unit 303. If necessary, a deblocking filter may also be applied to filter the decoded block to remove blocking artifacts. The decoded video block is then stored in a buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction, and the buffer 307 also generates the decoded video for presentation on a display device.

[0111] Some example embodiments of the present disclosure will be described in detail below. It should be noted that the section titles used in this document are for ease of understanding, and the embodiments disclosed in the section are not limited to that section. In addition, although some embodiments are described with reference to general video codecs or other specific video codecs, the disclosed technology is also applicable to other video coding and decoding technologies. In addition, although some embodiments describe the video encoding steps in detail, it should be understood that the corresponding decoding steps of canceling the encoding will be implemented by the decoder. In addition, the term video processing includes video coding or compression, video decoding or decompression, and video transcoding, in which video pixels are represented from one compression format to another compression format or at different compression bit rates.

[0112] Brief Overview

[0113] The present disclosure relates to video coding techniques. Specifically, it relates to combined intra-block copying, in which a reference (or prediction) block is obtained using samples in a current picture and intra-frame prediction, and other coding tools in image / video coding. It can be applied to existing video coding standards, such as HEVC, or the Universal Video Codec (VVC). It may also be applicable to future video coding standards or video codecs.

[0114] introduce

[0115] Video codec standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T developed H.261 and H.263, ISO / IEC developed MPEG-1 and MPEG-4 Visual, and the two organizations jointly developed H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Codec (AVC) and H.265 / HEVC standards. Since H.262, video codec standards have been based on a hybrid video codec structure that uses temporal prediction plus transform codecs. To explore future video codec technologies beyond HEVC, VCEG and MPEG jointly founded the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and put them into reference software called the Joint Exploration Model (JEM). In April 2018, the Joint Video Experts Group (JVET) between VCEG (Q6 / 16) and ISO / IEC JTC1 SC29 / WG11 (MPEG) was created to work on the VVC standard, which aims for a 50% bitrate reduction compared to HEVC. ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 5) are investigating the potential need for standardization of future video codec technologies with compression capabilities significantly exceeding the current VVC standard. Such future standardization action may take the form of (one or more) further extensions to VVC or an entirely new standard. The groups are working together to conduct this exploratory activity, called the Joint Video Exploration Team (JVET), to evaluate compression technology designs proposed by experts in the field. New codec features and codec methods have been implemented in the Enhanced Compression Model (ECM) software and are being explored by the Joint Video Exploration Team (JVET) of ITU-T VCEG and ISO / IEC MPEG as potential enhanced video codec technologies that aim to exceed the capabilities of VVC.

[0116] 2.1. Encoding and decoding process of typical video codecs

[0117] Figure 4 An example of an encoder block diagram for VVC is shown, which contains three loop filter blocks: Deblocking Filter (DF), Sample Adaptive Offset (SAO), and ALF. Unlike DF, which uses predefined filters, SAO and ALF use the original samples of the current picture to reduce the mean square error between the original and reconstructed samples by adding an offset and applying a finite impulse response (FIR) filter, respectively, using the coded side information that transmits the offset and filter coefficients through signaling. ALF is located at the last processing stage of each picture and can be seen as a tool that tries to capture and repair artifacts created by the previous stages.

[0118] 2.2. Intra-mode codec with 67 intra-prediction modes

[0119] To capture arbitrary edge directions present in natural videos, the number of directional intra modes is expanded from 33 used in HEVC to 65, as shown in Figure 5 As shown, planar and DC modes remain unchanged. These more dense directional intra prediction modes apply to intra prediction for all block sizes and for both luma and chroma.

[0120] In HEVC, each intra-coded block has a square shape, and the length of each side of each square is a power of 2. Therefore, no division operation is required to generate the intra predictor using DC mode. In VVC, blocks can have a rectangular shape, which in general requires the use of a division operation for each block. To avoid division operations for DC prediction, only the longer sides are used to calculate the average for non-square blocks.

[0121] Wide-angle intra prediction

[0122] Although 67 modes are defined in VVC, the exact prediction direction for a given intra prediction mode index further depends on the block shape. Conventional angular intra prediction directions are defined as 45 degrees to -135 degrees clockwise. In VVC, for non-square blocks, several conventional angular intra prediction modes are adaptively replaced with wide-angle intra prediction modes. The replaced mode is transmitted via a signal using the original mode index, which is remapped to the index of the wide-angle mode after parsing. The total number of intra prediction modes remains unchanged, i.e. 67, and the intra mode encoding and decoding method remains unchanged.

[0123] To support these prediction directions, a top reference of length 2W+1 and a left reference of length 2H+1 are provided as Figure 6 is defined as shown.

[0124] The number of alternative modes in the wide-angle direction mode depends on the aspect ratio of the block. The alternative intra-frame prediction modes are shown in Table 2-1.

[0125] Table 2-1 - Intra-frame prediction modes replaced by wide angle mode

[0126]

[0127] like Figure 7 As shown, in the case of wide-angle intra prediction, two vertically adjacent prediction samples can use two non-adjacent reference samples. Therefore, a low-pass reference sample filter and side smoothing are applied to wide-angle prediction to reduce the increased gap Δp αnegative impact. If the wide-angle mode represents a non-fractional offset. There are 8 modes in the wide-angle mode that meet this condition, and the 8 modes are [-14, -12, -10, -6, 72, 76, 78, 80]. When a block is predicted by these modes, the samples in the reference buffer are directly copied without applying any interpolation. With this modification, the number of samples that need to be smoothed is reduced. In addition, it aligns the design of non-fractional modes in the traditional prediction mode and the wide-angle mode.

[0128] In VVC, 4:2:2 and 4:4:4 as well as 4:2:0 chroma formats are supported. The chroma derivation mode (DM) derivation table for the 4:2:2 chroma format was originally ported from HEVC, which expanded the number of entries from 35 to 67 to be consistent with the expansion of intra prediction modes. Since the HEVC specification does not support prediction angles below -135 degrees and above 45 degrees, the luma intra prediction modes ranging from 2 to 5 are mapped to 2. Therefore, the chroma DM derivation table for the 4:2:2: chroma format is updated by replacing the values ​​of the entries of the mapping table to more accurately convert the prediction angles of the chroma blocks.

[0129] 2.3. Inter-frame prediction

[0130] For each inter-predicted CU, motion parameters consisting of motion vector, reference picture index and reference picture list usage index, as well as additional information required by the new codec features of VVC for sample generation for inter-prediction. Motion parameters can be signaled explicitly or implicitly. When a CU is coded and decoded in skip mode, the CU is associated with one PU and has no significant residual coefficients, no coded motion vector increments or reference picture indexes. Merge mode is specified, whereby the motion parameters of the current CU are obtained from neighboring CUs, which include spatial candidates and temporal candidates, as well as additional arrangements introduced in VVC. Merge mode can be applied to any inter-predicted CU, not just skip mode. An alternative to merge mode is explicit transmission of motion parameters, where motion vectors, corresponding reference picture indexes for each reference picture list and reference picture list usage flags, as well as other required information, are explicitly signaled for each CU.

[0131] 2.4. Intra-block copy (IBC)

[0132] Intra-block copying (IBC) is a tool adopted in the HEVC extension on SCC. It is well known that it significantly improves the coding efficiency of screen content materials. Since the IBC mode is implemented in a block-level coding mode, block matching (BM) is performed at the encoder to find the best block vector (or motion vector) for each CU. Here, the block vector is used to indicate the displacement from the current block to the reference block, which has been reconstructed in the current picture. The luminance block vector of the IBC-coded CU is integer precision. The chrominance block vector is also rounded to integer precision. When combined with AMVR, the IBC mode can switch between 1-pixel l and 4-pixel motion vector precision. The IBC-coded CU is regarded as a third prediction mode in addition to the intra or inter prediction mode. The IBC mode is applicable to CUs with width and height less than or equal to 64 luminance samples.

[0133] At the encoder side, hash-based motion estimation is performed for IBC. The encoder performs RD checks on blocks with width or height no greater than 16 luma samples. For non-merge mode, a block vector search is first performed using a hash-based search. If the hash search does not return a valid candidate, a local search based on block matching is performed.

[0134] In hash-based search, the hash key match (32-bit CRC) between the current block and the reference block is extended to all allowed blocks. The hash key calculation for each position in the current picture is based on a 4×4 sub-block. For a current block of larger size, the hash key is determined to match the hash key of the reference block when all hash keys of all 4×4 sub-blocks match the hash key of the corresponding reference position. If the hash keys of multiple reference blocks are found to match the hash key of the current block, the block vector cost of each matching reference is calculated and the one with the smallest cost is selected. In block matching search, the search range is set to cover previous and current CTUs.

[0135] At the CU level, the IBC mode is signaled using a flag. The IBC mode can be signaled as IBC AMVP mode or IBC skip / merge mode as shown below:

[0136] – IBC skip / merge mode: The merge candidate index is used to indicate which block vector from the list of neighboring candidate IBC codec blocks is used to predict the current block. The merge list consists of spatial candidates, HMVP candidates and pairwise candidates.

[0137] – IBC AMVP mode: Block vector differences are encoded in the same way as motion vector differences. The block vector prediction method uses two candidates as predictors, one from the left neighbor and one from the upper neighbor (if IBC is encoded). When either neighbor is not available, the default block vector will be used as the predictor. A flag is signaled to indicate the block vector predictor index.

[0138] 2.5. IBC Movement Candidates

[0139] The term "block" may refer to a codec tree block (CTB), a codec tree unit (CTU), a codec block (CB), a CU, a PU, a TU, a PB, a TB, or a video processing unit including a plurality of samples / pixels. A block may be rectangular or non-rectangular.

[0140] For IBC codec blocks, a block vector (BV) is used to indicate the displacement from the current block to a reference block that has been reconstructed inside the current picture.

[0141] W and H are the width and height of the current block (eg, luma block).

[0142] The non-adjacent spatial candidates of the current codec block are the adjacent spatial candidates of the virtual block in the i-th search round (e.g. Fig. 9 As shown). The width and height of the virtual block of the i-th search round are calculated by the following formulas: newWidth = i × 2 × gridX + W, newHeight = i × 2 × gridY + H. Obviously, if the search round i is 0, the virtual block is the current block.

[0143] In the following, the BV predictor is also a BV candidate. The skip mode is also a merge mode. The BV candidates can be divided into several groups according to some criteria. Each group is called a subgroup. For example, we can take the adjacent spatial and temporal BV candidates as the first subgroup and the remaining BV candidates as the second subgroup; in another example, we can also take the first N (N≥2) BV candidates as the first subgroup, the next M (M≥2) BV candidates as the second subgroup, and the remaining BV candidates as the third subgroup.

[0144] 2.6. Merged mode with MVD (MMVD)

[0145] In addition to the merge mode that uses implicitly derived motion information directly for prediction sample generation of the current CU, a merge mode with motion vector difference (MMVD) is introduced in VVC. The MMVD flag is signaled immediately after the regular merge flag is sent to specify whether the MMVD mode is used for the CU.

[0146] In MMVD, after a merge candidate is selected, it is further refined by MVD information transmitted by signal. Further information includes a merge candidate flag, an index specifying the magnitude of motion, and an index indicating the direction of motion. In MMVD mode, one of the first two candidates in the merge list is selected to be used as the MV basis. The MMVD candidate flag is transmitted by signal to specify which one between the first and second merge candidates is used.

[0147] The distance index specifies the motion magnitude information and indicates a predefined offset from the starting point. Figure 8 As shown, the offset is added to the horizontal component or the vertical component of the starting MV. The relationship between the distance index and the predefined offset is specified in Table 2-2.

[0148] Table 2-2 – Relationship between distance index and predefined offsets

[0149]

[0150] The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate four directions, as shown in Table 2-3. Note that the meaning of the MVD symbol may vary depending on the information of the starting MV. When the starting MV is a non-predicted MV or a bidirectionally predicted MV, where both lists point to the same side of the current picture (i.e., both referenced POCs are greater than the POC of the current picture, or both are less than the POC of the current picture), the symbol in Table 2-3 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectionally predicted MV, where the two MVs point to different sides of the current picture (i.e., the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture), and the difference in POC in list0 is greater than the difference in POC in list1, the symbol in Table 2-3 specifies the sign of the MV offset added to the MV component of the starting MV of column list0, and the sign for list1 MV has an opposite value. Otherwise, if the difference of POC in list1 is greater than list0, the sign in Table 2-3 specifies the sign of the MV offset added to the list1 MV component of the starting MV and the sign for list0 MV has the opposite value.

[0151] The MVD is scaled according to the difference in POC in each direction. If the POC difference in both lists is the same, no scaling is required. Otherwise, if the POC difference in list0 is greater than the POC difference in list1, the MVD for list1 is scaled by defining the POC difference of L0 as td and the POC difference of L1 as tb, as Fig.26 If the POC difference of L1 is greater than L0, the MVD for list 0 is scaled in the same way. If the starting MV is unidirectionally predicted, the MVD is added to the available MVs.

[0152] Table 2-3 – MV offset symbols specified by direction index

[0153] Direction Index 00 01 10 11 x-axis + - N / A N / A y-axis N / A N / A + -

[0154] 2.7. Symmetric MVD encoding and decoding

[0155] In VVC, in addition to normal uni-prediction and bi-prediction mode MVD signaling, symmetric MVD mode for bi-prediction MVD signaling is applied. In symmetric MVD mode, the motion information including reference picture indexes of both list-0 and list-1 and MVD of list-1 is not signaled but derived.

[0156] The decoding process of the symmetric MVD mode is as follows:

[0157] At the stripe level, the variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1 are derived as follows:

[0158] If mvd_l1_zero_flag is 1, BiDirPredFlag is set equal to 0.

[0159] Otherwise, if the nearest reference picture in list-0 and the nearest reference picture in list-1 form a preceding pair of reference pictures or a preceding pair of reference pictures, BiDirPredFlag is set to 1, and both the list-0 reference picture and the list-1 reference picture are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0.

[0160] At CU level, if the CU is bi-predictively coded and BiDirPredFlag is equal to 1, a symmetry mode flag indicating whether the symmetry mode is used is explicitly signaled.

[0161] When the symmetric mode flag is true, only mvp_l0_flag, mvp_l1_flag and MVD0 are explicitly signaled. The reference index for list-0 and list-1 are set equal to the reference picture pair, respectively. MVD1 is set equal to (-MVD0). The final motion vector is shown below.

[0162]

[0163] In the encoder, symmetric MVD motion estimation starts with initial MV evaluation. A set of initial MV candidates includes MVs obtained from unidirectional prediction search, MVs obtained from bidirectional prediction search, and MVs from the AMVP list. The one with the lowest rate-distortion cost is selected as the initial MV for symmetric MVD motion search.

[0164] 2.8. Bidirectional Optical Flow (BDOF)

[0165] The Bidirectional Optical Flow (BDOF) tool is included in VVC. BDOF, formerly known as BIO, is included in JEM. Compared to the JEM version, BDOF in VVC is a simpler version that requires much less computation, especially in terms of the number of multiplications and the size of the multipliers.

[0166] BDOF is used to refine the bidirectional prediction signal of a CU at the 4×4 sub-block level. BDOF is applied to a CU if all of the following conditions are met:

[0167] The CU is encoded using "true" bi-prediction mode, i.e., one of the two reference pictures precedes the current picture in display order, and the other of the two reference pictures follows the current picture in display order;

[0168] The distances from the two reference images to the current image (i.e., POC difference) are the same;

[0169] Both reference images are short-term reference images;

[0170] The CU is not encoded or decoded using affine mode or SbTMVP merge mode;

[0171] CU has more than 64 luminance samples;

[0172] CU height and CU width are both greater than or equal to 8 luma samples;

[0173] The BCW weight index indicates equal weights;

[0174] WP is not enabled in the current CU;

[0175] CIIP mode is not used for the current CU.

[0176] BDOF is applied only to the luma component. As its name suggests, BDOF mode is based on the concept of optical flow, which assumes that the motion of objects is smooth. For each 4×4 sub-block, motion refinement (v x ,v y ) is calculated by minimizing the difference between L0 prediction samples and L1 prediction samples. Then motion refinement is used to adjust the bi-prediction sample values ​​in the 4x4 sub-block. The following steps are applied in the BDOF process.

[0177] First, by directly calculating the difference between two adjacent samples, the horizontal and vertical gradients of the two prediction signals, and is calculated, that is,

[0178]

[0179] Where I( k)(i,j) is the sample value at the prediction signal coordinate (i,j) in list k, k=0,1, and shift1 is calculated based on the luma bit depth bitDepth as shift1=max(6, bitDepth-6).

[0180] Then, the autocorrelations and cross-correlations of the gradients S1, S2, S3, S5, and S6 are calculated as follows:

[0181] S1=∑ (i,j)∈Ω Abs(ψ x (i,j)),S3=∑ (i,j)∈Ω θ(i,j)·Sign(ψ x (i,j))

[0182]

[0183] S5=∑ (i，j)∈Ω Abs(ψ y (i,j)), S6=∑ (i,j)∈Ω θ(i,j)·Sign(ψ y (i,j))

[0184] (2-3)

[0185] in,

[0186]

[0187] θ(i, j)=(I (1) (i, j)>>n b )-(I (0) (i, j)>>n b ) (2-4)

[0188] where Ω is a 6×6 window around a 4×4 sub-block, and n a and n b The values ​​are set to min(1, bitDepth-11) and min(4, bitDepth-8) respectively.

[0189] Then the cross-correlation and autocorrelation terms are used to refine the motion (v x ,v y ) is exported using:

[0190]

[0191] in th′ BIO =2 max(5,BD-7) . is the round-down (floor) function, and

[0192] Based on the motion refinement and gradients, the following adjustments are computed for each sample in the 4×4 sub-block:

[0193]

[0194] Finally, the BDOF samples of the CU are calculated by adjusting the bidirectional prediction samples as shown below:

[0195] pred BDOF (x, y) = (I (0) (x,y)+I (1) (x, y) + b(x, y) + o offset )>>shift (2-7)

[0196] These values ​​are chosen so that the multipliers in the BDOF process do not exceed 15 bits and the maximum bit width of the intermediate parameters in the BDOF process remains within 32 bits.

[0197] In order to derive the gradient value, some prediction samples I in the list k (k = 0, 1) outside the current CU boundary are (k) (i,j) needs to be generated. Fig.10 As shown, BDOF in VVC uses an extended row / column around the boundary of the CU. In order to control the computational complexity of generating prediction samples outside the boundary, the prediction samples in the extended area (white positions) are generated by directly taking the reference samples at nearby integer positions (using the floor() operation on the coordinates) without interpolation, and a conventional 8-head motion compensated interpolation filter is used to generate prediction samples within the CU (gray positions). These extended sample values ​​are only used for gradient calculations. For the rest of the steps in the BDOF process, if any sample values ​​and gradient values ​​outside the CU boundary are needed, these sample values ​​and gradient values ​​are filled (i.e. repeated) from their nearest neighbors.

[0198] When the width and / or height of a CU is greater than 16 luma samples, it will be divided into sub-blocks with width and / or height equal to 16 luma samples, and the sub-block boundaries are regarded as CU boundaries in the BDOF process. The maximum unit size of the BDOF process is limited to 16x16. For each sub-block, the BDOF process can be skipped. When the SAD between the initial L0 prediction sample and the L1 prediction sample is less than a threshold, the BDOF process is not applied to the sub-block. The threshold is set equal to (8*W*(H>>1), where W represents the sub-block width and H represents the sub-block height. In order to avoid the additional complexity of SAD calculation, the SAD between the initial L0 prediction sample and the L1 prediction sample calculated in the DVMR process is reused here.

[0199] If BCW is enabled for the current block, that is, the BCW weight index indicates unequal weights, bidirectional optical flow is disabled. Similarly, if WP is enabled for the current block, that is, luma_weight_lx_flag of any of the two reference pictures is 1, BDOF is also disabled; BDOF is also disabled when the CU is encoded and decoded using symmetric MVD mode or CIIP mode.

[0200] 2.9. Combined Inter- and Intra-frame Prediction (CIIP)

[0201] 2.10. Affine Motion Compensated Prediction

[0202] In HEVC, motion compensated prediction (MCP) only applies the translational motion model. In the real world, there are many kinds of motion, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, block-based affine transformation motion compensated prediction is applied. Fig.11 As shown, the affine motion field of a block is described by motion information of two control points (4 parameters) or three control point motion vectors (6 parameters).

[0203] For the 4-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as:

[0204]

[0205] For the 6-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as:

[0206]

[0207] Where (mv0x,mv0y) is the motion vector of the upper left control point, (mv1x,mv1y) is the motion vector of the upper right control point, and (mv2x,mv2y) is the motion vector of the lower left control point.

[0208] To simplify motion compensated prediction, block-based affine transform prediction is applied. To derive the motion vector for each 4×4 luminance subblock, Fig.12 As shown, the motion vector of the center sample of each subblock is calculated according to the above equation and rounded to 1 / 16 fractional precision. Then the motion compensated interpolation filter is applied to generate a prediction of each subblock with a derived motion vector. The subblock size of the chroma component is also set to 4×4. The MV of a 4×4 chroma subblock is calculated as the average of the MVs of the four corresponding 4×4 luminance subblocks.

[0209] Like translational motion inter prediction, there are two affine motion inter prediction modes: affine merge mode and affine AMVP mode.

[0210] 2.10.1. Affine Merge Prediction

[0211] AF MERGE mode can be applied to CUs with width and height greater than or equal to 8. In this mode, the CPMV of the current CU is generated based on the motion information of spatially neighboring CUs. There can be up to five CPMVP candidates, and the index is transmitted by signal to indicate the CPMVP used for the current CU. The following three types of CPVM candidates are used to form the affine merge candidate list:

[0212] - Inherited affine merge candidates inferred from the CPMV of neighboring CUs

[0213] -Affine merge candidate CPMVP constructed using the translation MV of the neighboring CU

[0214] -Zero MV

[0215] In VVC, there are at most two inherited affine candidates, which are derived from the affine motion models of neighboring blocks, one from the left neighboring CU and one from the upper neighboring CU. Fig.13 As shown. For the left predictor, the scanning order is A0->A1, and for the upper predictor, the scanning order is B0->B1->B2. Only the first inheritance candidate from each side is selected. Pruning check between two inheritance candidates is not performed. When a neighboring affine CU is identified, its control point motion vector is used to derive the CPMVP candidate in the affine merge list of the current CU. As shown, if the neighboring lower left block A is encoded and decoded in affine mode, the motion vectors v2, v3 and v4 of the upper left, upper right and lower left corners of the CU containing block A are obtained. When block A is encoded and decoded with a 4-parameter affine model, two CPMVs of the current CU are calculated based on v2, v3 and v4. When block A is encoded and decoded with a 6-parameter affine model, three CPMVs of the current CU are calculated based on v2, v3 and v4.

[0216] Constructing affine candidates means that candidates are constructed by combining the neighboring translation motion information of each control point. Fig.15 The designated spatial neighbors and designated temporal neighbors shown are derived. CPMVk (k = 1, 2, 3, 4) represents the kth control point. For CPMV1, check B2->B3->A2 block, and the MV of the first available block is used. For CPMV2, check B1->B0 block, and for CPMV3, check A1->A0 block. For TMVP, if available, TMVP is used as CPMV4.

[0217] After the MVs of the four control points are obtained, an affine merge candidate is constructed based on the motion information. The following combinations of control point MVs are used to construct in order:

[0218] {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}

[0219] A combination of 3 CPMVs constructs a 6-parameter affine merge candidate, and a combination of 2 CPMVs constructs a 4-parameter affine merge candidate. To avoid the motion scaling process, the relevant combination of control point MVs is discarded if the reference indices of the control points are different.

[0220] After the inherited affine merge candidates and the constructed affine merge candidates are checked, if the list is still not full, a zero MV is inserted at the end of the list.

[0221] 2.10.2 Affine AMVP Prediction

[0222] Affine AMVP mode can be applied to CUs with width and height both greater than or equal to 16. A CU-level affine flag is signaled in the bitstream to indicate whether affine AMVP mode is used, and another flag is signaled to indicate whether it is 4-parameter affine or 6-parameter affine. In this mode, the difference between the CPMVP of the current CU and their predictor CPMVP is signaled in the bitstream. The affine AVMP candidate list size is 2, and it is generated in order using the following four types of CPVM candidates:

[0223] - Inherited affine AMVP candidates inferred from the CPMV of neighboring CUs

[0224] -Affine AMVP candidate CPMVP constructed using the translation MV of the neighboring CU

[0225] -Translated MV from neighboring CU

[0226] -Zero MV

[0227] The order in which inherited affine AMVP candidates are checked is the same as the order in which inherited affine merge candidates are checked. The only difference is that for AVMP candidates, only affine CUs with the same reference picture as in the current block are considered. No pruning process is applied when inserting the inherited affine motion predictor into the candidate list. The constructed AMVP candidates are Fig.13The specified spatial neighbors shown are derived. The same check order as in the affine merge candidate construction is used. In addition, the reference picture index of the neighboring blocks is also checked. The first block in the check order is used, which is inter-coded and has the same reference picture as in the current CU. There is only one. When the current CU is coded with a 4-parameter affine mode and both mv0 and mv1 are available, they are added as a candidate in the affine AMVP list. When the current CU is coded with a 6-parameter affine mode and all three CPMVs are available, they are added as a candidate in the affine AMVP list. Otherwise, the constructed AMVP candidate is set to unavailable.

[0228] If the affine AMVP list candidate is still less than 2 after the inherited affine AMVP candidate and the constructed AMVP candidate are checked, mv0, mv1 and mv2 are added as translation MVs in sequence to predict all control point MVs of the current CU when available. Finally, if the affine AMVP list is still not full, the zero MV is used to fill the affine AMVP list.

[0229] 2.10.3. Affine motion information storage

[0230] In VVC, the CPMV of the affine CU is stored in a separate buffer. The stored CPMV is only used to generate the inherited CPMV in the affine merge mode and the inherited CPMV in the affine AMVP mode for the most recently coded CU. The sub-block MV derived from the CPMV is used for motion compensation, merging of translation MVs / MV derivation of AMVP lists, and deblocking.

[0231] To avoid picture line buffering for additional CPMV, the affine motion data inheritance from the CU of the upper CTU is handled differently from the inheritance from the normal neighboring CU. If the candidate CU for affine motion data inheritance is in the upper CTU row, the lower left and lower right sub-block MVs in the line buffer are used for affine MVP derivation instead of CPMV. In this way, CPMV is only stored in the local buffer. If the candidate CU is 6-parameter affine coded, the affine model is downgraded to a 4-parameter model. Fig.16 As shown, along the top CTU boundary, the bottom left and bottom right sub-block motion vectors of the CU are used for affine inheritance of the CU in the bottom CTU.

[0232] 2.10.4. Prediction refinement using optical flow for affine modality

[0233] Compared with pixel-based motion compensation, sub-block-based affine motion compensation can save memory access bandwidth and reduce computational complexity, but at the expense of prediction accuracy. In order to achieve finer motion compensation granularity, prediction refinement using optical flow (PROF) is used to refine sub-block-based affine motion compensation predictions without increasing the memory access bandwidth of motion compensation. In VVC, after sub-block-based affine motion compensation is performed, the brightness prediction samples are refined by adding the differences derived from the optical flow equation. PROF is described as the following four steps:

[0234] Step 1) Sub-block based affine motion compensation is performed to generate sub-block predictions.

[0235] Step 2) Using a 3-tap filter [-1, 0, 1], the spatial gradient g of the sub-block prediction x (i, j) and g y (i, j) is calculated at each sample position. The gradient calculation is exactly the same as the gradient calculation in BDOF.

[0236] g x (i,j)=(I(i+1,j)>>shift1)-(I(i-1,j)>>shift1) (2-10)

[0237] g y (i,j)=(I(i,j+1)>>shift1)-(I(i,j-1)>>shift1) (2-11)

[0238] shift1 is used to control the precision of the gradient. The sub-block (i.e. 4x4) prediction is extended by one sample on each side of the gradient calculation. To avoid extra memory bandwidth and extra interpolation calculations, these extended samples on the extended boundary are copied from the nearest integer pixel position in the reference picture.

[0239] Step 3) The brightness prediction refinement is calculated by following the optical flow equation.

[0240] ΔI(i, j) = g x (i, j)*Δv x (i, j)+g y (i, j)*Δv y (i, j) (2-12)

[0241] where Δv(i, j) is the difference between the sample MV calculated for the sample position (i, j), denoted as v(i, j), and the sub-block MV of the sub-block to which the sample (i, j) belongs, as Fig.17 As shown in the figure, Δv(i, j) is quantized in units of 1 / 32 brightness sample accuracy.

[0242] Since the affine model parameters and the sample positions relative to the subblock center do not change from subblock to subblock, Δv(i, j) can be calculated for the first subblock and reused for other subblocks in the same CU. Let dx(i, j) and dy(i, j) be the distance from the sample position (i, j) to the subblock center (xS B , yS B )’s horizontal and vertical offsets, Δv(x, y) can be derived by the following equations:

[0243]

[0244]

[0245] To maintain accuracy, the sub-block (x SB ,y SB ) is calculated as ((W SB -1) / 2,(H SB -1) / 2), where W SB and H SB are the width and height of the sub-block respectively.

[0246] For the 4-parameter affine model,

[0247]

[0248] For the 6-parameter affine model,

[0249]

[0250] Where (v 0x ,v 0y ), (v 1x ,v 1y ), (v 2x ,v 2y ) are the control point motion vectors of the upper left, upper right and lower left, and w and h are the width and height of the CU.

[0251] Step 4) Finally, the luma prediction refinement ΔI(i,j) is added to the sub-block prediction I(i,j) and the final prediction I' is generated as follows.

[0252] I'(i,j)=I(ij)+ΔI(i,j) (2-17)

[0253] PROF is not applicable to affine codec CUs in two cases: 1) all control point MVs are the same, which indicates that the CU has only translational motion; 2) the affine motion parameters are larger than the specified limit, because the sub-block based affine MC is downgraded to CU based MC to avoid large memory access bandwidth requirements.

[0254] A fast codec method is applied to reduce the codec complexity of affine motion estimation using PROF. PROF is not applied to the affine motion estimation stage in the following two cases: a) If the CU is not a root block and the parent block of the CU does not select the affine mode as its best mode, PROF is not applied because the possibility that the current CU selects the affine mode as the best mode is low; b) If the magnitudes of the four affine parameters (C, D, E, F) are all less than a predefined threshold and the current picture is not a low-latency picture, PROF is not applied. In this way, affine motion estimation using PROF can be accelerated.

[0255] 2.11. Sub-block based temporal motion vector prediction (SbTMVP) VVC supports the sub-block based temporal motion vector prediction (SbTMVP) method. Similar to the temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses the motion field in the co-located picture to improve the motion vector prediction and merge mode for the CU in the current picture. The same co-located picture used by TMVP is also used for SbTVMP. SbTMVP differs from TMVP in the following two main aspects:

[0256] TMVP predicts motion at the CU level, but SbTMVP predicts motion at the sub-CU level;

[0257] TMVP obtains the temporal motion vector from the co-located block in the co-located picture (the co-located block is the bottom-right or center block relative to the current CU), while SbTMVP applies motion shifting before obtaining the temporal motion information from the co-located picture, where the motion shifting is obtained from the motion vector of one of the spatial neighboring blocks from the current CU.

[0258] SbTVMP process Fig.18A and Fig.18B As shown. SbTMVP predicts the motion vector of the sub-CU in the current CU in two steps. In the first step, Fig.18A The spatial neighbors A1 in are checked. If A1 has a motion vector that uses the co-located picture as its reference picture, then that motion vector is selected as the motion shift to be applied. If no such motion is identified, the motion offset is set to (0,0).

[0259] In the second step, the motion shift identified in step 1 is applied (i.e., added to the coordinates of the current block) to Fig.18B The co-located picture 1822 shown obtains sub-CU level motion information (motion vector and reference index). Fig.18BThe example in assumes that the motion shift is set to the motion of block A1. Then, for each sub-CU, the motion information of the corresponding block in the co-located picture (the minimum motion grid covering the center sample) is used to derive the motion information for the sub-CU. After the motion information of the co-located sub-CU is identified, it is converted into a motion vector and reference index of the current sub-CU in a manner similar to the TMVP process of HEVC, where temporal motion scaling is applied to align the reference picture of the temporal motion vector with the reference picture of the current CU.

[0260] In VVC, a sub-block based merge list containing a combination of both SbTVMP candidates and affine merge candidates is used for signaling of sub-block based merge mode. SbTVMP mode is enabled / disabled by a sequence parameter set (SPS) flag. If SbTMVP mode is enabled, the SbTMVP predictor is added as the first entry in the list of sub-block based merge candidates and is followed by the affine merge candidate. The size of the sub-block based merge list is signaled in the SPS, and the maximum allowed size of the sub-block based merge list is 5 in VVC. The sub-CU size used in SbTMVP is fixed to 8x8, and like the affine merge mode, SbTMVP mode is only applicable to CUs with width and height both greater than or equal to 8.

[0261] The encoding logic of the additional SbTMVP merge candidate is the same as that for other merge candidates, ie, for each CU in a P slice or a B slice, an additional RD check is performed to decide whether to use the SbTMVP candidate.

[0262] 2.12. Adaptive Motion Vector Resolution (AMVR)

[0263] In HEVC, when use_integer_mv_flag in the slice header is equal to 0, the motion vector difference (MVD) (between the motion vector of the CU and the predicted motion vector) is transmitted by signal in units of quarter luma samples. In VVC, the CU-level adaptive motion vector resolution (AMVR) scheme is introduced. AMVR allows the MVD of a CU to be encoded and decoded with different precisions. Depending on the mode for the current CU (normal AMVP mode or affine AVMP mode), the MVD of the current CU can be adaptively selected as follows:

[0264] - Normal AMVP mode: quarter-brightness-sample, half-brightness-sample, whole-brightness-sample or quad-brightness-sample.

[0265] - Affine AMVP modes: quarter-luminance-sample, whole-luminance-sample, or 1 / 16-luminance-sample.

[0266] The CU-level MVD resolution indication is conditionally signaled if the current CU has at least one non-zero MVD component.If all MVD components (ie, both horizontal and vertical MVD for reference list L0 and reference list L1) are zero, a quarter-luminance-sample MVD resolution is inferred.

[0267] For a CU with at least one non-zero MVD component, a first flag is transmitted by a signal to indicate whether quarter-luminance-sample MVD precision is used for the CU. If the first flag is 0, no further signaling is required, and quarter-luminance-sample MVD precision is used for the current CU. Otherwise, a second flag is transmitted by a signal to indicate that half-luminance-sample or other MVD precision (whole or four-luminance samples) is used for a normal AMVP CU. In the case of half-luminance-samples, a 6-tap interpolation filter is used for half-luminance sample positions instead of the default 8-tap interpolation filter. Otherwise, a third flag is transmitted by a signal to indicate whether whole-luminance-sample MVD precision or four-luminance-sample MVD precision is used for a normal AMVP CU. In the case of an affine AMVP CU, a second flag is used to indicate whether whole-luminance-sample MVD precision or 1 / 16 luminance-sample MVD precision is used. To ensure that the reconstructed MV has the expected precision (quarter-luma-sample, half-luma-sample, whole-luma-sample, or quad-luma-sample), the motion vector predictor for the CU is rounded to the same precision as the MVD before the MVD is added. The motion vector predictor is rounded towards zero (i.e., a negative motion vector predictor is rounded towards positive infinity, and a positive motion vector predictor is rounded towards negative infinity).

[0268] The encoder uses RD check to determine the motion vector resolution for the current CU. In order to avoid always performing four CU-level RD checks for each MVD resolution, in VTM11, only the RD check of MVD precision other than a quarter-brightness-sample is conditionally called. For the normal AVMP mode, the RD cost of a quarter-brightness-sample MVD precision and a whole-brightness-sample MV precision is calculated first. Then, the RD cost of the whole-brightness-sample MVD precision is compared with the RD cost of the quarter-brightness-sample MVD precision to determine whether it is necessary to further check the RD cost of the four-brightness-sample MVD precision. When the RD cost for the quarter-brightness-sample MVD precision is much less than the whole-brightness-sample MVD precision, the RD check of the four-brightness-sample MVD precision is skipped. Then, if the RD cost of the whole-brightness-sample MVD precision is significantly greater than the best RD cost of the previously tested MVD precision, the check of the half-brightness-sample MVD precision is skipped. For the affine AMVP mode, if the affine inter mode is not selected after checking the rate-distortion cost of the affine merge / skip mode, merge / skip mode, quarter-luma-sample MVD accuracy normal AMVP mode, and quarter-luma-sample MVD accuracy affine AMVP mode, the 1 / 16 luma-sample MV accuracy and 1-pixel MV accuracy affine inter modes are not checked. In addition, the affine parameters obtained in the quarter-luma-sample MV accuracy affine inter mode are used as the starting search points for the 1 / 16 luma-sample and quarter-luma-sample MV accuracy affine inter modes.

[0269] 2.13. Bidirectional Prediction with CU-Level Weights (BCW)

[0270] In HEVC, a bidirectional prediction signal is generated by averaging two prediction signals obtained from two different reference pictures and / or using two different motion vectors. In VVC, the bidirectional prediction mode is extended beyond simple averaging to allow weighted averaging of the two prediction signals.

[0271] P bi-pred =((8-w)*P0+w*P1+4)>>3 (2-18) Five weights are allowed in weighted average bidirectional prediction, w∈{-2,3,4,5,10}. For each bidirectional prediction CU, the weight w is determined in one of two ways: 1) For non-merged CUs, the weight index is signaled after the motion vector difference; 2) For merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is only applied to CUs with 256 or more luma samples (i.e., the CU width multiplied by the CU height is greater than or equal to 256). For low-latency pictures, all 5 weights will be used. For non-low-latency pictures, only 3 weights (w∈{3,4,5}) are used.

[0272] At the encoder, a fast search algorithm is applied to find the weight index without significantly increasing the encoder complexity. These algorithms are summarized below. The VTM software can be referenced for further details. When combined with AMVR, if the current picture is a low-latency picture, the unequal weights of 1-pixel and 4-pixel motion vector accuracy are only conditionally checked.

[0273] When combined with affine, affine ME will be performed for unequal weights if and only if the affine mode is selected as the current best mode.

[0274] Unequal weights are only conditionally checked when the two reference pictures in bidirectional prediction are the same.

[0275] Unequal weights are not searched when certain conditions are met, which depend on the POC distance between the current picture and its reference pictures, the codec QP, and the temporal level.

[0276] The BCW weight index is encoded and decoded using a context encoded binary followed by a bypass encoded binary. The first context encoded binary indicates whether equal weights are used; and if unequal weights are used, an additional binary is signaled using the bypass codec to indicate which unequal weights are used. Weighted prediction (WP) is a codec tool supported by the H.264 / AVC and HEVC standards for efficient encoding and decoding of video content in the presence of fading. Support for WP has also been added to the VVC standard. WP allows weighting parameters (weights and offsets) to be signaled for each reference picture list L0 and each reference picture in list L1. Then, during motion compensation, the weights and offsets of the corresponding reference pictures are applied. WP and BCW are designed for different types of video content. To avoid interaction between WP and BCW (which would complicate VVC decoder design), if the CU uses WP, the BCW weight index is not signaled and w is inferred to be 4 (i.e., equal weights are applied). For merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. This applies to both normal merge mode and inherited affine merge mode. For constructed affine merge mode, affine motion information is constructed based on motion information of up to 3 blocks. The BCW index of a CU using constructed affine merge mode is simply set equal to the BCW index of the first control point MV.

[0277] In VVC, CIIP and BCW cannot be applied jointly to a CU. When a CU is encoded and decoded using CIIP mode, the BCW index of the current CU is set to 2, for example, equal weight.

[0278] 2.14. Local Illumination Compensation (LIC)

[0279] Local illumination compensation (LIC) is a codec tool that addresses the problem of local illumination variations between a current picture and its temporal reference picture. LIC is based on a linear model where a scaling factor and an offset are applied to a reference sample to obtain a predicted sample for the current block. Specifically, LIC can be mathematically modeled by the following equation:

[0280] P(x,y)=α·P r (x+v x ,y+v y )+β

[0281] Among them, P(x,y) is the prediction signal of the current block at the coordinate (x,y); P r (x+v x ,y+v y ) is the motion vector (v x ,v y) points to the reference block; and α and β are the corresponding scaling factors and offsets applied to the reference block. Fig.19 The LIC process is shown. Fig.19 In , when applying LIC to a block, the minimum mean square error (LMSE) method is used to minimize the neighboring samples of the current block (i.e. Fig.19 The template T in the temporal reference picture) and its corresponding reference sample in the temporal reference picture (i.e., Fig.19 In addition, in order to reduce the computational complexity, both the template samples and the reference template samples are subsampled (adaptive subsampling) to derive the LIC parameters, that is, only Fig.19 The shadow samples in are used to derive α and β.

[0282] To improve the codec performance, the short edge subsampling is not performed, such as Fig. 20 shown.

[0283] 2.15. Decoder-side Motion Vector Refinement (DMVR)

[0284] To improve the accuracy of the merge mode MV, decoder-side motion vector refinement based on bilateral matching (BM) is applied in VVC. In the bidirectional prediction operation, the refined MV is searched around the initial MV in the reference picture list L0 and the reference picture list L1. The BM method calculates the distortion between two candidate blocks in the reference picture list L0 and the list L1. Fig.21 As shown, based on each MV candidate (eg, MV0' and MV1') around the initial MV, the SAD between two blocks is calculated. The MV candidate with the lowest SAD becomes the refined MV and is used to generate a bidirectional prediction signal.

[0285] - In VVC, the application of DMVR is limited and only applies to CUs with the following modes and capabilities:

[0286] - CU level merge mode with bi-predictive MV.

[0287] -Relative to the current picture, one reference picture is in the past and the other reference picture is in the future.

[0288] - The distances from the two reference pictures to the current picture (ie, the POC difference) are the same.

[0289] - Both reference images are short-term reference images.

[0290] -CU has more than 64 luma samples.

[0291] - CU height and CU width are both greater than or equal to 8 luma samples.

[0292] -BCW weight index indicates equal weight.

[0293] -Currently the block is not WP enabled.

[0294] -CIIP mode is not used for the current block.

[0295] The refined MV derived by the DMVR process is used to generate inter-frame prediction samples and is also used for temporal motion vector prediction of future picture encoding and decoding. The original MV is used in the deblocking process and is also used for spatial motion vector prediction of future CU encoding and decoding.

[0296] Additional features of DMVR are mentioned in the following sub-clauses.

[0297] 2.15.1. Search solution

[0298] In DVMR, the search point is around the initial MV, and the MV offset obeys the MV difference mirror rule. In other words, any point examined by DMVR represented by the candidate MV pair (MV0, MV1) follows the following two equations:

[0299] MV0′=MV0+MV_offset (2-19)

[0300] MV1′=MV1-MV_offset (2-20)

[0301] Wherein, MV_offset represents the refinement offset between the initial MV and the refinement MV in one of the reference pictures. The refinement search range is two integer luma samples starting from the initial MV. The search includes an integer sample offset search phase and a fractional sample refinement phase.

[0302] The integer sample offset search uses a 25-point full search. First, the SAD of the initial MV pair is calculated. If the SAD of the initial MV pair is less than the threshold, the integer sampling stage of DMVR is terminated. Otherwise, the SAD of the remaining 24 points is calculated and checked in raster scan order. The point with the smallest SAD is selected as the output of the integer sample offset search stage. In order to reduce the impact of DMVR refinement uncertainty, it is proposed to support the original MV in the DMVR process. The SAD between the reference blocks referenced by the initial MV candidates is reduced by 1 / 4 of the SAD value.

[0303] The integer sample search is followed by fractional sample refinement. To save computational complexity, fractional sample refinement is derived using the parametric error surface equations instead of doing an additional search using SAD comparisons. Fractional sample refinement is conditionally called based on the output of the integer sample search stage. When the integer sample search stage ends with a center with minimum SAD in either the first or second iteration search, fractional sample refinement is further applied.

[0304] In the sub-pixel offset estimation based on the parameter error surface, the cost of the center position and the costs of the four neighboring positions from the center are used to fit a two-dimensional parabolic error surface equation of the following form

[0305] E(x, y) = A(x min ) 2 +B(yy min ) 2 +C (2-21)

[0306] Where (x min ,y min ) corresponds to the fractional position with the minimum cost, and C corresponds to the minimum cost value. By solving the above equation using the cost values ​​of the five search points, (x min ,y min ) is calculated as:

[0307] x min =(E(-1,0)-E(1,0)) / (2(E(-1,0)+E(1,0)-2E(0,0))) (2-22)

[0308] y min =(E(0,-1)-E(0,1)) / (2((E(0,-1)+E(0,1)-2E(0,0))) (2-23).

[0309] x min and min The value of is automatically clamped between -8 and 8, since all cost values ​​are positive, with a minimum value of E(0,0). This corresponds to a half-pixel offset with 1 / 16-pixel MV precision in VVC. The calculated fraction (x min ,y min ) is added to the integer distance refinement MV to obtain a sub-pixel accurate refinement delta MV.

[0310] 2.15.2. Bilinear interpolation and sample padding

[0311] In VVC, the resolution of MV is 1 / 16 luma sample. Samples at fractional positions are interpolated using an 8-tap interpolation filter. In DMVR, the search points are around the initial fractional pixel MV with integer sample offsets, so the samples at these fractional positions need to be interpolated for the DMVR search process. In order to reduce the computational complexity, a bilinear interpolation filter is used to generate fractional samples for the search process in DMVR. Another important effect is that by using a bilinear filter, DVMR does not access more reference samples than the normal motion compensation process within the 2-sample search range. After the refined MV is obtained through the DMVR search process, an ordinary 8-tap interpolation filter is applied to generate the final prediction. In order not to access more reference samples for the normal MC process, samples will be filled from those available samples, which are not needed for the interpolation process based on the original MV, but are needed for the interpolation process based on the refined MV.

[0312] 2.15.3. Maximum DMVR processing unit

[0313] When the width and / or height of a CU is greater than 16 luma samples, it will be further split into sub-blocks with width and / or height equal to 16 luma samples. The maximum unit size for the DMVR search process is limited to 16x16.

[0314] 2.16. Multi-channel decoder-side motion vector refinement

[0315] In this contribution, multi-pass decoder-side motion vector refinement is applied instead of DMVR. In the first pass, bilateral matching (BM) is applied to one codec block. In the second pass, BM is applied to each 16x16 sub-block within the codec block. In the third pass, MVs in each 8x8 sub-block are refined by applying bidirectional optical flow (BDOF). The refined MVs are stored for spatial and temporal motion vector prediction.

[0316] 2.16.1. First Pass – Block-based Bilateral Matching MV Refinement

[0317] In the first pass, refined MVs are derived by applying BM to the codec blocks. Similar to decoder-side motion vector refinement (DMVR), refined MVs are searched around two initial MVs (MV0 and MV1) in reference picture lists L0 and L1. Refined MVs (MV0_pass1 and MV1_pass1) are derived around the initiating MVs based on the minimum bilateral matching cost between the two reference blocks in L0 and L1.

[0318] BM performs a local search to derive integer sample precision intDeltaMV and half-pel sample precision halfDeltaMV. The local search applies a 3×3 square search pattern, looping in the horizontal search range [–sHor, sHor] and the vertical search range [–sVer, sVer], where the values ​​of sHor and sVer are determined by the block scale, and the maximum values ​​of sHor and sVer are 8.

[0319] The bilateral matching cost is calculated as: bilCost = mvDistanceCost + sadCost. When the block size cbW*cbH is greater than 64, the MRSAD cost function is applied to remove the DC effect of the distortion between the reference blocks. When the bilCost of the center point of the 3×3 search pattern has the minimum cost, the intDeltaMV or halfDeltaMV local search terminates. Otherwise, the current minimum cost search point becomes the new center point of the 3×3 search pattern and continues to search for the minimum cost until it reaches the end of the search range.

[0320] The existing fractional sample refinement is further applied to derive the final deltaMV. Then, the refined MV after the first pass is derived as:

[0321] MV0_pass1=MV0+deltaMV,

[0322] MV1_pass1=MV1–deltaMV.

[0323] 2.16.2. Second pass – Subblock-based bilateral matching MV refinement In the second pass, refined MVs are derived by applying BM to 16×16 grid subblocks. For each subblock, the refined MVs are searched around the two MVs (MV0_pass1 and MV1_pass1) obtained in the first pass in the reference picture lists L0 and L1. The refined MVs (MV0_pass2(sbIdx2) and MV1_pass2(sbIdx2)) are derived based on the minimum bilateral matching cost between the two reference subblocks in L0 and L1.

[0324] For each subblock, BM performs a full search to derive integer sample precision intDeltaMV. The full search has a search range of [–sHor, sHor] in the horizontal direction and a search range of [–sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block scale and the maximum values ​​of sHor and sVert are 8.

[0325] The bilateral matching cost is calculated by applying the cost factor to the SATD cost between two reference sub-blocks, such as: bilCost = satdCost * costFactor. The search area (2*sHor+1)*(2*sVer+1) is divided into 5 diamond search areas, such as Fig. 22 As shown above. Each search area is assigned a costFactor determined by the distance (intDeltaMV) between each search point and the starting MV, and each diamond area is processed in order starting from the center of the search area. In each area, the search points are processed in raster scan order, starting from the upper left corner of the area and going to the lower right corner. When the minimum bilCost in the current search area is less than or equal to the threshold of sbW*sbH, the integer pixel (int-pel) full search is terminated, otherwise, the int-pel full search continues to the next search area until all search points are checked.

[0326] BM performs a local search to derive the half-sample accuracy halfDeltaMv. The search pattern and cost function are the same as defined in 2.9.1.

[0327] The existing VVC DMVR fractional sample refinement is further applied to derive the final deltaMV (sbIdx2). Then, the refined MV of the second channel is derived as:

[0328] MV0_pass2(sbIdx2)=MV0_pass1+deltaMV(sbIdx2),

[0329] MV1_pass2(sbIdx2)=MV1_pass1-deltaMV(sbIdx2).

[0330] 2.16.3. Third channel – sub-block based bidirectional optical flow MV refinement

[0331] In the third pass, a refined MV is derived by applying BDOF to an 8×8 grid sub-block. For each 8×8 sub-block, BDOF refinement is applied to derive scaled Vx and Vy without clipping from the refined MV of the parent sub-block in the second pass. The derived bioMv(Vx, Vy) is rounded to 1 / 16 sample precision and clipped between -32 and 32.

[0332] The refined MVs of the third pass (MV0_pass3(sbIdx3) and MV1_pass3(sbIdx3)) are derived as:

[0333] MV0_pass3(sbIdx3)=MV0_pass2(sbIdx2)+bioMv,

[0334] MV1_pass3(sbIdx3)=MV0_pass2(sbIdx2)–BioMv.

[0335] 2.17. Sample-based BDOF

[0336] In sample-based BDOF, instead of deriving motion refinement (Vx, Vy) on a block basis, it is performed per sample.

[0337] The codec block is divided into 8×8 sub-blocks. For each sub-block, whether to apply BDOF is determined by checking the SAD between the two reference sub-blocks against a threshold. If it is decided to apply BDOF to a sub-block, for each sample in the sub-block, Vx and Vy are derived using a sliding 5×5 window and applying the existing BDOF process for each sliding window. The derived motion refinement (Vx, Vy) is applied to adjust the bi-predicted sample value of the window center sample.

[0338] 2.18. Extended Merge Prediction

[0339] In VVC, the merge candidate list is constructed by including the following five types of candidates in order:

[0340] (1) Spatial MVP from spatially neighboring CUs.

[0341] (2) Temporal MVP from the same CU.

[0342] (3) History-based MVP from FIFO table.

[0343] (4) Paired average MVP.

[0344] (5) Zero MV.

[0345] The size of the merge list is signaled in the sequence parameter set header, and the maximum allowed size of the merge list is 6. For each CU code in merge mode, the index of the best merge candidate is encoded and decoded using truncated unary binarization (TU). The first bin of the merge index is encoded and decoded using context, while bypass encoding and decoding is used for other bins.

[0346] The derivation process of merge candidates for each category is provided in this section. As in HEVC, VVC also supports parallel derivation of merge candidate lists for all CUs within a certain size region.

[0347] 2.18.1. Spatial candidate export

[0348] The derivation of spatial merge candidates in VVC is the same as that in HEVC, except that the positions of the first two merge candidates are swapped. Among the candidates located at the described positions, at most four merge candidates are selected. The derivation order is B0, A0, B1, A1 and B2. Position B2 is only considered when one or more CUs at positions B0, A0, B1 and A1 are not available (for example, because it belongs to another slice or tile) or is intra-coded. After the candidate at position A1 is added, a redundancy check is performed on the addition of the remaining candidates, which ensures that candidates with the same motion information are excluded from the list, thereby improving the coding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only the Fig.24 The pairs linked with arrows in , and candidates are added to the list only if the corresponding candidates for redundancy check do not have the same motion information.

[0349] 2.18.2. Time candidate export

[0350] In this step, only one candidate is added to the list. In particular, in the derivation of this temporal merge candidate, the scaled motion vector is derived based on the co-located CU belonging to the co-located reference picture. The reference picture list to be used for deriving the co-located CU is explicitly signaled in the slice header. Fig.25 As shown by the dashed line in , the scaled motion vector for the temporal merge candidate is obtained, which is scaled from the motion vector of the co-located CU using the POC distances, tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal merge candidate is set equal to zero.

[0351] like Fig.26 As shown, the position of the temporal candidate is selected between candidates C0 and C1. If the CU at position C0 is not available and is intra-coded, or the CU at position C0 is outside the current row of the CTU, position C1 is used. Otherwise, position C0 is used in the derivation of the temporal merge candidate.

[0352] 2.18.3 History-based Merge Candidate Export

[0353] History-based MVP (HMVP) merge candidates are added to the merge list after spatial MVP and TMVP. In this method, the motion information of the previous codec block is stored in a table and used as the MVP of the current CU. A table with multiple HMVP candidates is maintained during the encoding / decoding process. The table is reset (cleared) when a new CTU row is encountered. Whenever there is a non-sub-block inter-coded CU, the associated motion information is added as a new HMVP candidate to the last entry of the table.

[0354] The HMVP table size S is set to 6, which indicates that up to 6 history-based MVP (HMVP) candidates can be added to the table. When a new motion candidate is inserted into the table, a constrained first-in-first-out (FIFO) rule is used, where a redundancy check is first applied to find if there is an identical HMVP in the table. If found, the identical HMVP is removed from the table, and then all HMVP candidates are moved forward.

[0355] HMVP candidates can be used in the merge candidate list construction process. The nearest HMVP candidates in the table are checked in order and inserted into the candidate list after the TMVP candidate. Redundancy check is applied to HMVP candidates to spatial or temporal merge candidates.

[0356] To reduce the number of redundant checking operations, the following simplifications are introduced:

[0357] The number of HMPV candidates for merge list generation is set to (N<=4)?M:(8-N), where N indicates the number of existing candidates in the merge list and M indicates the number of available HMVP candidates in the table.

[0358] Once the total number of available merge candidates reaches the maximum allowed merge candidates minus 1, the merge candidate list construction process from HMVP is terminated.

[0359] 2.18.4. Pairwise Average Merge Candidate Derivation

[0360] The pairwise average candidate is generated by averaging the predefined candidate pairs in the existing merge candidate list, and the predefined pairs are defined as {(0, 1), (0, 2), (1, 2), (0, 3), (1, 3), (2, 3)}, where the number represents the merge index of the merge candidate list. The average motion vector is calculated separately for each reference list. If two motion vectors are available in one list, they are averaged even if they point to different reference pictures; if only one motion vector is available, it is used directly; if no motion vector is available, this list is kept invalid.

[0361] When the merge list is not full after adding pairwise average merge candidates, zero MVPs will be inserted at the end until the maximum number of merge candidates is encountered.

[0362] 2.18.5. Merging Estimated Regions

[0363] The merge estimation region (MER) allows the merge candidate list to be derived independently for a CU in the same merge estimation region (MER). For generating the merge candidate list for the current CU, candidate blocks in the same MER as the current CU are not included. In addition, the update process for the history-based motion vector predictor candidate list is updated only when (xCb+cbWidth)>>Log2ParMrgLevel is greater than xCb>>Log2Par MrgLevel and (yCb+cbHeight)>>Log2parMrglevel is greater than (yCb>>Log2ParMrgLevel), and where (xCb, yCb) is the top left luma sample position of the current CU in the picture and (cbWidth, cbHeight) is the CU size. The MER size is selected at the encoder side and signaled in the sequence parameter set as log2_parallel_merge_level_minus2.

[0364] 2.19. New Merge Candidates

[0365] 2.19.1 Non-adjacent merge candidate export

[0366] In VVC, use Fig. 27 The five spatial neighboring blocks shown and one temporal neighbor are used to derive the merge candidate.

[0367] It is proposed to derive additional merge candidates from non-adjacent locations to the current block using the same pattern as VVC. To this end, for each search round i, a virtual block is generated based on the current block as follows:

[0368] First, the relative position of the virtual block to the current block is calculated as follows:

[0369] Offsetx=-i×gridX, Offsety=-i×gridY

[0370] Where Offsetx and Offsety represent the offset of the upper left corner of the virtual block relative to the upper left corner of the current block, and gridX and gridY are the width and height of the search grid.

[0371] Second, the width and height of the virtual block are calculated in the following way:

[0372] newWidth=i×2×gridX+currWidth newHeight=i×2×gridY+currHeight.

[0373] Where currWidth and currHeight are the width and height of the current block. newWidth and newHeight are the width and height of the new virtual block.

[0374] gridX and gridY are currently set to currWidth and currHeight respectively.

[0375] Fig.28 The relationship between the virtual block and the current block is shown.

[0376] After the virtual block is generated, blocks Ai, Bi, Ci, Di, and Ei can be regarded as VVC spatial neighbors of the virtual block, and their positions are obtained in the same mode as VVC. Obviously, if the search round i is 0, the virtual block is the current block. In this case, blocks Ai, Bi, Ci, Di, and Ei are the spatial neighbors used in the VVC merge mode.

[0377] When constructing the merge candidate list, pruning is performed to ensure the uniqueness of each element in the merge candidate list. The maximum search round is set to 1, which means that five non-adjacent spatial neighboring blocks are utilized.

[0378] Non-adjacent spatial merge candidates are inserted into the merge list after the temporal merge candidates in the order of B1->A1->C1->D1->E1.

[0379] 2.19.2.STMVP

[0380] It is proposed to use three spatial merging candidates and one temporal merging candidate to derive an average candidate as the STMVP candidate.

[0381] The STMVP is inserted before the top left spatial merge candidate.

[0382] The STMVP candidate is pruned along with all previous merge candidates in the merge list.

[0383] For spatial candidates, the first three candidates in the current merge candidate list are used.

[0384] For temporal candidates, the same positions as the VTM / HEVC co-location positions are used.

[0385] For spatial candidates, the first, second, and third candidates inserted into the current merge candidate list before STMVP are denoted as F, S, and T.

[0386] The temporal candidate having the same position as the VTM / HEVC co-location used in TMVP is denoted Col.

[0387] The motion vector of the STMVP candidate in prediction direction X (denoted as mvLX) is derived as follows:

[0388] 1) If the reference indices of the four merge candidates are all valid and equal to zero in the prediction direction X (X=0 or 1),

[0389] mvLX=(mvLX_F+mvLX_S+mvLX_T+mvLX_Col)>>2.

[0390] 2) If the reference indexes of three of the four merge candidates are valid and equal to zero in the prediction direction X (X=0 or 1),

[0391] mvLX=(mvLX_F×3+mvLX_S×3+mvLX_Col×2)>>3 or

[0392] mvLX=(mvLX_F×3+mvLX_T×3+mvLX_Col×2)>>3 or

[0393] mvLX=(mvLX_S×3+mvLX_T×3+mvLX_Col×2)>>3.

[0394] 3) If the reference indexes of two of the four merge candidates are valid and equal to zero in the prediction direction X (X=0 or 1),

[0395] mvLX=(mvLX_F+mvLX_Col)>>1 or

[0396] mvLX=(mvLX_S+mvLX_Col)>>1 or

[0397] mvLX=(mvLX_T+mvLX_Col)>>1.

[0398] NOTE: If time candidate is not available, STMVP mode is turned off.

[0399] 2.19.3. Merge List Size

[0400] If both non-adjacent merge candidates and STMVP merge candidates are considered, the size of the merge list is signaled in a set of sequence parameter headers, and the maximum allowed size of the merge list is 8.

[0401] 2.20. Geometric Partitioning Model (GPM)

[0402] In VVC, geometric partitioning mode is supported for inter prediction. The geometric partitioning mode is signaled using a CU level flag as a merge mode. Other merge modes include normal merge mode, MMVD mode, CIIP mode, and sub-block merge mode. For each possible CU size w×h=2 m ×2 n , where m,n∈{3…6} and excluding 8x64 and 64x8, the geometric partitioning mode supports a total of 64 partitions.

[0403] When this mode is used, the CU is split into two parts by a geometrically positioned straight line ( Fig.29 ). The position of the partition line is mathematically derived from the angle and offset parameters of the specific partition. Each part of the geometric partition in the CU uses its own motion for inter-frame prediction; each partition only allows unidirectional prediction, that is, each part has a motion vector and a reference index. Unidirectional prediction motion constraints are applied to ensure that, as with traditional bidirectional prediction, only two motion compensated predictions are required for each CU. The unidirectional prediction motion for each partition is derived using the process described in 2.20.1.

[0404] If geometric partitioning mode is used for the current CU, a geometric partitioning index indicating the partitioning mode (angle and offset) of the geometric partitioning and two merge indices (one for each partition) are further transmitted through the signal. The number of maximum GPM candidate sizes is explicitly transmitted through the signal in the SPS, and the syntax binarization for the GPM merge indices is specified. After predicting each part of the geometric partitioning, the sample values ​​along the edges of the geometric partitioning are adjusted using a hybrid process with adaptive weights as in 2.20.2. This is the prediction signal for the entire CU, and the transformation and quantization processes will be applied to the entire CU as in other prediction modes. Finally, the motion field of the CU predicted using the geometric partitioning mode is stored as in 2.20.3.

[0405] 2.20.1. One-way prediction candidate list construction

[0406] The unidirectional prediction candidate list is derived directly from the merge candidate list constructed according to the extended merge prediction process in 2.18. Denote n as the index of the unidirectional prediction motion in the geometric unidirectional prediction candidate list. The LX motion vector of the nth extended merge candidate is used as the nth unidirectional prediction motion vector of the geometric partitioning mode, with X equal to the parity of n. These motion vectors are Fig.30 If the corresponding LX motion vector of the n-th extended merge candidate does not exist, the L(1-X) motion vector of the same candidate is used instead as the unidirectional prediction motion vector of the geometric partitioning mode.

[0407] 2.20.2. Blending Along Geometric Partition Edges

[0408] After predicting each part of the geometric partition using its own motion, blending is applied to the two prediction signals to derive samples around the geometric partition edges. The blending weights for each position of the CU are derived based on the distance between the individual position and the partition edge.

[0409] The distance from position (x,y) to the partition edge is derived as:

[0410]

[0411] where i,j are the angle and offset indices of the geometric partition, which depend on the geometric partition index transmitted by the signal. x,j and ρ y,j The sign of depends on the angle index i.

[0412] The weight of each part of the geometric partition is derived as follows:

[0413] wIdxL(x,y)=partIdx? 32+d(x,y):32-d(x,y) (2-28),

[0414]

[0415] w1(x,y)=1-w0(x,y) (2-30).

[0416] partIdx depends on the angle index i. An example of weight w0 is Fig.31 is shown in the figure.

[0417] 2.20.3. Motion Domain Storage for Geometry Partitioning

[0418] Mv1 from the first part of the geometric partitioning, Mv2 from the second part of the geometric partitioning, and the combination Mv of Mv1 and Mv2 are stored in the motion domain of the CU encoded in the geometric partitioning mode.

[0419] The type of motion vector stored for each individual position in the motion field is determined as:

[0420] sType=abs(motionldx)<32?2:(motionldx≤0?(1-partldx):patldx) (2-31)

[0421] Where motionIdx is equal to d(4x+2,4y+2), which is recalculated according to equation (2-18). partIdx depends on the angle index i.

[0422] If sType is equal to 0 or 1, then Mv0 or Mv1 is stored in the corresponding motion field, otherwise, if sTType is equal to 2, the combined Mv from Mv0 and Mv2 is stored. The combined Mv is generated using the following process:

[0423] 1) If Mv1 and Mv2 are from different reference picture lists (one from L0 and the other from L1), then Mv1 and Mv2 are simply combined to form a bidirectional prediction motion vector.

[0424] Otherwise, if Mv1 and Mv2 are from the same list, only the unidirectional predicted motion Mv2 is stored.

[0425] 2.21. Multi-hypothesis prediction

[0426] In Multiple Hypothesis Prediction (MHP), up to two additional predictors are signaled on top of the Inter AMVP mode, Normal Merge mode, Affine Merge mode, and MMVD mode. The resulting overall prediction signal is iteratively accumulated with each additional prediction signal. n+1 =(1-α n+1 ) n +α n+1 h n+1

[0427] The weight factor α is specified according to the following Table 2-4:

[0428] Table 2-4 – Weighting factors for MHP

[0429] add_hyp_weight_idx α 0 1 / 4 1 -1 / 8

[0430] For inter-AMVP mode, MHP is applied only when unequal weights in BCW are selected in bi-prediction mode.

[0431] Additional assumptions may be merge mode or AMVP mode. In the case of merge mode, motion information is indicated by a merge index, and the merge candidate list is the same as in the geometry partition mode. In the case of AMVP mode, reference index, MVP index, and MVD are transmitted by signal.

[0432] 2.22. Non-adjacent spatial candidates

[0433] Non-adjacent spatial merge candidates are inserted after TMVP in the regular merge candidate list. The pattern of spatial merge candidates is as follows: Fig.32 As shown above, the distance between non-adjacent spatial candidates and the current codec block is based on the width and height of the current codec block.

[0434] 2.23. Template Matching (TM)

[0435] Template Matching (TM) is a decoder-side MV derivation method used to refine the motion information of the current CU by finding the closest match between a template in the current picture (i.e., the top and / or left neighboring blocks of the current CU) and a block in a reference picture (i.e., the same size as the template). Fig.33 It is shown that template matching is performed on the search area around the initial MV. Fig.33 As shown, in the [-8, +8] pixel search range, a better MV is searched around the initial motion of the current CU. This contribution adopts the template matching previously proposed in JVET-J0021 and makes two modifications: the search step size is determined based on the AMVR mode, and the TM can cascade the bilateral matching process in the merge mode.

[0436] In AMVP mode, MVP candidates are determined based on template matching error to select the one that achieves the minimum difference between the current block template and the reference block template, and then TM performs MV refinement only on this specific MVP candidate. TM refines this MVP candidate by using an iterative diamond search, starting with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode) within the [-8, +8] pixel search range. The AMVP candidate can be further refined by using a cross search with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode), and then using half-pixels and quarter-pixels in turn according to the AMVR mode specified in Table 2-5. This search process ensures that the MVP candidate still maintains the same MV accuracy as indicated by the AMVR mode after the TM process.

[0437] Table 2-5 – Search modes of AMVR and merge modes with AMVR

[0438]

[0439]

[0440] In merge mode, a similar search method is applied to the merge candidates indicated by the merge index. As shown in Table 2-5, TM can be performed all the way down to 1 / 8 pixel MVD accuracy, or skip accuracy beyond half-pixel MVD accuracy, depending on whether an alternative interpolation filter is used based on the merged motion information (used when AMVR is half-pixel mode). In addition, when TM mode is enabled, template matching can work as an independent process between block-based and sub-block-based bilateral matching (BM) methods or as an additional MV refinement process, depending on whether BM can be enabled based on its enabling condition check.

[0441] 2.24. Overlapped Block Motion Compensation (OBMC)

[0442] Overlapped block motion compensation (OBMC) has been previously used in H.263. In JEM, unlike H.263, OBMC can be turned on and off using CU-level syntax. When OBMC is used in JEM, OBMC is performed on all motion compensated (MC) block boundaries except the right and bottom boundaries of the CU. In addition, it also applies to luma and chroma components. In JEM, an MC block corresponds to a codec block. When a CU is encoded and decoded with sub-CU modes (including sub-CU merging, affine, and FRUC modes), each sub-block of the CU is an MC block. In order to handle CU boundaries in a unified manner, OBMC is performed on all MC block boundaries at the sub-block level, where the sub-block size is set equal to 4×4, as Fig.34 shown.

[0443] When OBMC is applied to the current sub-block, in addition to the current motion vector, the motion vectors of the four connected neighboring sub-blocks, if available and different from the current motion vector, are also used to derive the prediction block of the current sub-block. These multiple prediction blocks based on multiple motion vectors are combined to generate the final prediction signal of the current sub-block.

[0444] The prediction block based on the motion vector of the neighboring sub-block is denoted as PN, where N indicates the index of the neighboring upper, lower, left and right sub-blocks, and the prediction block based on the motion vector of the current sub-block is denoted as PC. When PN is based on the motion information of the neighboring sub-block containing the same motion information as the current sub-block, OBMC is not performed from PN. Otherwise, each sample of PN is added to the same sample in PC, that is, four rows / columns of PN are added to PC. Weighting factors {1 / 4, 1 / 8, 1 / 16, 1 / 32} are used for PN, and weighting factors {3 / 4, 7 / 8, 15 / 16, 31 / 32} are used for PC. Except for small MC blocks (i.e., when the height or width of the coding block is equal to 4, or when the CU is coded and decoded with sub-CU mode), only two rows / columns of PN are added to PC. In this case, weighting factors {1 / 4, 1 / 8} are used for PN, and weighting factors {3 / 4, 7 / 8} are used for PC. For a PN generated based on the motion vectors of vertically (horizontally) neighboring sub-blocks, samples in the same row (column) of the PN are added to the PC with the same weighting factor.

[0445] In JEM, for CUs with a size less than or equal to 256 luma samples, a CU-level flag is signaled to indicate whether OBMC is applied to the current CU. For CUs with a size greater than 256 luma samples or not encoded and decoded using AMVP mode, OBMC is applied by default. At the encoder, when OBMC is applied to a CU, its impact is taken into account in the motion estimation stage. The prediction signal formed by OBMC using the motion information of the top neighboring blocks and the left neighboring blocks is used to compensate the top and left boundaries of the original signal of the current CU, and then the normal motion estimation process is applied.

[0446] 2.25. Multiple Transform Selection (MTS) for Core Transformations

[0447] In addition to DCT-II already adopted in HEVC, the Multiple Transform Selection (MTS) scheme is also used for residual coding of inter-frame and intra-frame coded blocks. It uses multiple selected transforms in DCT8 / DST7. The newly introduced transform matrices are DST-VII and DCT-VIII. Table 2-6 shows the basic functions of the selected DST / DCT.

[0448] Table 2-6 – Transform basis functions of DCT-II / VIII and DSTVII for N-point input

[0449]

[0450] To maintain the orthogonality of the transform matrix, the quantization of the transform matrix is ​​more accurate than that in HEVC. To keep the intermediate values ​​of the transform coefficients within 16 bits, all coefficients are 10 bits after horizontal and vertical transforms.

[0451] To control the MTS scheme, separate enable flags are specified at the SPS level for intra and inter frames respectively. When MTS is enabled at SPS, a CU level flag signal is issued to indicate whether MTS is applied. Here, MTS is only applied to luma. When one of the following conditions applies, MTS signaling will be skipped.

[0452] - The position of the last significant coefficient of the luma TB is less than 1 (ie DC only).

[0453] -The last significant coefficient of the luminance TB is located within the MTS zeroing region.

[0454] If the MTS CU flag is equal to 0, DCT2 is applied in both directions. However, if the MTS CU flag is equal to 1, two other flags are additionally signaled to indicate the transform types in the horizontal and vertical directions, respectively. The transform and signaling mapping table is shown in Table 2-7. The transform selection for ISP and implicit MTS is unified by eliminating intra-frame mode and block shape dependencies. If the current block is in ISP mode or the current block is an intra-frame block and both intra-frame and inter-frame explicit MTS are turned on, only DST7 is used for horizontal and vertical transform cores. In terms of transform matrix accuracy, an 8-bit main transform core is used. Therefore, all transform cores used in HEVC remain the same, including 4-point DCT-2 and DST-7, 8-point, 16-point and 32-point DCT-2. In addition, other transform cores (including 64-point DCT-2, 4-point DCT-8, 8-point, 16-point, 32-point DST-7 and DCT-8) all use 8-bit main transform cores.

[0455] Table 2-7 – Transformation and signaling mapping table

[0456]

[0457] To reduce the complexity of large-size DST-7 and DCT-8, high-frequency transform coefficients are cleared to zero for DST-7 and DCT-8 blocks with size (width or height, or width and height) equal to 32. Only coefficients in the 16×16 low-frequency region are retained.

[0458] As in HEVC, the residual of a block can be coded using transform skip mode. To avoid syntax coding redundancy, the transform skip flag is not signaled when the CU level MTS_CU_flag is not equal to 0. Note that when LFNST or MIP is activated for the current CU, the implicit MTS transform is set to DCT2. In addition, when MTS is enabled for an inter-coded block, implicit MTS can still be enabled.

[0459] 2.26. Sub-Block Transform (SBT)

[0460] In VTM, sub-block transform is introduced for inter-predicted CU. In this transform mode, only a sub-part of the residual block is encoded and decoded for the CU. When cu_cbf of an inter-predicted CU is equal to 1, cu_sbt_flag can be signaled to indicate whether the entire residual block or a sub-part of the residual block is encoded and decoded. For the former case, the inter-MTS information is further parsed to determine the transform type of the CU. In the latter case, a part of the residual block is encoded and decoded by inferring the adaptive transform, while the other part of the residual block is cleared.

[0461] When SBT is used for an inter-coded CU, the SBT type and SBT position information are signaled in the bitstream. There are two SBT types and two SBT positions, such as Fig.35 As shown. For SBT-V (or SBT-H), the TU width (or height) can be equal to half of the CU width (or height) or 1 / 4 of the CU width (or height), resulting in 2:2 partitioning or 1:3 / 3:1 partitioning. 2:2 partitioning is similar to binary tree (BT) partitioning, while 1:3 / 3:1 partitioning is similar to asymmetric binary tree (ABT) partitioning. In ABT splitting, only small areas contain non-zero residuals. If one dimension of the CU is 8 (in units of luminance samples), 1:3 / 3:1 partitioning along that dimension is not allowed. A CU has a maximum of 8 SBT modes.

[0462] Position-dependent transform kernel selection is applied to luma transform blocks in SBT-V and SBT-H (chroma TBs always use DCT-2). Different kernel transforms are associated with the two positions, SBT-H and SBT-V. More specifically, the horizontal and vertical transforms for each SBT position are in Fig.35 For example, the horizontal and vertical transforms for SBT-V position 0 are DCT-8 and DST-7, respectively. When one side of the residual TU is larger than 32, the transforms for both dimensions are set to DCT-2. Therefore, the sub-block transforms jointly specify the TU tiling, cbf, and the horizontal and vertical core transform types for the residual block.

[0463] SBT is not applied to CUs encoded in inter-intra combination mode.

[0464] 2.27. Adaptive Merge Candidate Reordering Based on Template Matching

[0465] In order to improve the encoding and decoding efficiency, after building the merge candidate list, the order of each merge candidate is adjusted according to the template matching cost. The merge candidates are arranged in the list according to the template matching cost in ascending order. It operates in the form of subgroups.

[0466] The template matching cost is measured by the SAD (sum of absolute differences) between the neighboring samples of the current CU and its corresponding reference samples. If the merge candidate includes bidirectional prediction motion information, the corresponding reference sample is the average of the corresponding reference sample in reference list 0 and the corresponding reference sample in reference list 1, such as Fig.36 If the merge candidate includes sub-CU level motion information, the corresponding reference sample is composed of the neighboring samples of the corresponding reference sub-block, such as Fig.37 shown.

[0467] The sorting process is operated in the form of subgroups, such as Fig.38As shown. The first three merge candidates are sorted together. The next three merge candidates are sorted together. The template size (width of the left template or height of the upper template) is 1, and the subgroup size is 3.

[0468] 2.28. Adaptive Merge Candidate List

[0469] We can assume that the number of merge candidates is 8. We use the first 5 merge candidates as the first subgroup, and the following 3 merge candidates as the second subgroup (ie, the last subgroup).

[0470] For the encoder, after building the merge candidate list, some merge candidates are adaptively reordered in ascending order of merge candidate cost, e.g. Fig.39 More specifically, the template matching costs of the merge candidates in all subgroups except the last subgroup are calculated; then the merge candidates in its own subgroup except the last subgroup are reordered; finally, the final merge candidate list is obtained.

[0471] For the decoder, after constructing the merge candidate list, some / no merge candidates are adaptively reordered in ascending order of merge candidate cost, e.g. Fig.40 As shown. Fig.40 In , the subgroup where the selected (transmitted by signal) merging candidate belongs is called the selected subgroup.

[0472] More specifically, if the selected merge candidate is located in the last subgroup, the merge candidate list construction process is terminated after the selected merge candidate is derived, and no reordering is performed and the merge candidate list is not changed; otherwise, the execution process is as follows:

[0473] After all merge candidates in the selected subgroup are derived, the merge candidate list construction process is terminated; the template matching costs of the merge candidates in the selected subgroup are calculated; the merge candidates in the selected subgroup are reordered; and finally, a new merge candidate list is obtained.

[0474] For both encoder and decoder,

[0475] The template matching cost is derived as a function of T and RT, where T is the set of samples in the template and RT is the set of reference samples for the template.

[0476] When deriving the reference samples of the template of the merge candidate, the motion vector of the merge candidate is rounded to integer pixel precision.

[0477] The reference samples of the template (RT) for bidirectional prediction are derived by weighted averaging the reference samples of the template in reference list 0 (RT0) and the reference samples of the template in reference list 1 (RT1) as follows.

[0478] RT=((8-w)*RT0+w*RT1+4)>>3 (2-32)

[0479] The weight of the reference template in reference list 0 (8-w) and the weight of the reference template in reference list 1 (w) are determined by the BCW index of the merge candidate. BCW indexes equal to {0, 1, 2, 3, 4} correspond to w equal to {-2, 3, 4, 5, 10} respectively.

[0480] If the local illumination compensation (LIC) flag of the merge candidate is true, the reference samples of the template are derived using the LIC method.

[0481] The template matching cost is calculated based on the sum of absolute differences (SAD) between T and RT.

[0482] The template size is 1. This means that the width of the left template and / or the height of the top template is 1.

[0483] If the codec mode is MMVD, the merge candidates used to derive the base merge candidate are not reordered.

[0484] If the coding mode is GPM, the merge candidates used to derive the unidirectional prediction candidate list are not reordered.

[0485] 2.29. IBC with extended reference area

[0486] An IBC reference area design that does not increase the current memory area required by ECM-3 is proposed and the performance is tested.

[0487] Fig.41 The design is shown in FIG. In the figure, the blue square represents the current CTU, and the green square represents the CTU that can be used by IBC reference. Specifically, assuming that W represents the maximum horizontal CTU index and the current CTU index is (m, n), then for the codec unit in the current CTU, the CTUs with indices (0, n) ... (m, n) and (m-1, n) ... (W, n) define the reference area that can be used by IBC.

[0488] One reason for having such a design is that in the current ECM, the left, upper, and upper-left CTUs are being used and therefore need to be saved. To achieve this, all CTUs to the right of the upper CTU in the upper CTU row (for the CTU to be encoded in the current CTU row) and all CTUs to the left of the current CTU in the current CTU row (for the CTU to be encoded in the next CTU row) must be kept. This means that this design does not increase the buffer size required for the current ECM.

[0489] 2.30. IBC with Template Matching

[0490] It is proposed to use template matching in IBC also for both IBC merge mode and IBC AMVP mode.

[0491] Compared to the merge list used by the regular IBC merge mode, the IBC-TM merge list has been modified so that candidates are selected according to a pruning method that exploits the motion distance between candidates (as in the regular TM merge mode). The ending zero motion complement (which is meaningless with respect to intra coding) has been replaced by motion vectors pointing to the left (-W, 0), top (0, -H) and top-left (-W, -H) CUs, then the list is complemented with blocks from the left without pruning if necessary.

[0492] In IBC-TM merge mode, the selected candidates are refined with template matching method before RDO or decoding process.IBC-TM merge mode has been competed with regular IBC merge mode and TM merge flag is signaled.

[0493] In IBC-TM AMVP mode, up to 3 candidates are selected from the IBC merge list. Each of these 3 selected candidates is refined using a template matching method and ranked according to their resulting template matching cost. Only the first two are then typically considered in the motion estimation process.

[0494] Template matching refinement for both IBC-TM merge and AMVP modes is very simple, since IBC motion vectors are constrained to be integers and in Fig.42 . Therefore, in IBC-TM merge mode, all refinements are performed with integer precision, and in IBC-TM AMVP mode, they are performed with integer or 4-pixel precision. In both cases, the refined motion vectors in each refinement step must refer to the constraints of the reference region.

[0495] 2.31. Reconstruction-Reordering IBC (RR-IBC)

[0496] Screen content codec tools like intra-block copy (IBC) generate prediction blocks by directly copying previously coded reference areas in the same picture. Symmetry is often observed in video content, especially in text character areas and computer-generated graphics in screen content sequences, such as Fig.43 Therefore, a screen content-specific codec that takes symmetry into account will efficiently compress such video content.

[0497] A reconstruction-reordering IBC (RR-IBC) mode for screen content video coding and decoding is proposed. When this mode is applied, the samples in the reconstructed block are flipped according to the flip type of the current block. On the encoder side, the original block is flipped before motion search and residual calculation, while the prediction block is derived without flipping. On the decoder side, the reconstructed block is flipped back to restore the original block.

[0498] For RR-IBC codec blocks, two flipping methods are supported, horizontal flipping and vertical flipping. First, a syntax flag is signaled for the IBC AMVP codec block, indicating whether the reconstruction is flipped, and if flipped, another flag specifying the flip type is further signaled. For IBC merging, the flip type is inherited from the neighboring block without syntax signaling. Taking into account horizontal or vertical symmetry, the current block and the reference block are usually aligned horizontally or vertically. Therefore, when horizontal flipping is applied, the vertical component of BV is not signaled and inferred to be equal to 0. Similarly, when vertical flipping is applied, the horizontal component of BV is not signaled and inferred to be equal to 0.

[0499] In order to better utilize the symmetry characteristics, the flip-aware BV adjustment method is applied to refine the block vector candidates. For example, as shown in FIG. 44, (x nbr ,y nbr ) and (x cur ,y cur ) represent the coordinates of the center samples of the neighboring blocks and the current block, BV nbr and BV cur Represents the BV of the neighboring block and the current block respectively. Instead of inheriting the BV directly from the neighboring block, the motion shift is added to the BV by encoding the neighboring block with horizontal flipping. cur The horizontal component (expressed as BV nbr h ), that is, BV cur h =2(x nbr -x cur )+BV nbr h , to calculate BV cur Similarly, in the case where the adjacent block is coded with vertical flipping, by adding nbr The vertical component of the motion shift (denoted as BV nbr v ), that is, BV cur v =2(y nbr -y cur )+BV nbr v , to calculate BV cur The vertical component of .

[0500] 2.32. Intra-frame template matching

[0501] Intra Template Matching Prediction (Intra TMP) is a special intra prediction mode that copies the best prediction block (whose L-shaped template matches the current template) from the reconstructed part of the current frame. For a predefined search range, the encoder searches for the template that is most similar to the current template in the reconstructed part of the current frame and uses the corresponding block as the prediction block. The encoder then signals the use of this mode and performs the same prediction operation on the decoder side.

[0502] By combining the L-shaped causal neighboring blocks of the current block with the Fig.45 Generate a prediction signal by matching another block in a predefined search area:

[0503] R1: Current CTU

[0504] R2: Upper left CTU

[0505] R3: Upper CTU

[0506] R4: Left CTU

[0507] SAD is used as the cost function.

[0508] In each region, the decoder searches for the template with the minimum SAD relative to the current one and uses its corresponding block as the prediction block.

[0509] The dimensions of all regions (Search Range_w, SearchRange_h) are set to be proportional to the block dimensions (BlkW, BlkH) to have a fixed number of SAD comparisons for each pixel. That is: SearchRange_w = a*BlkW

[0510] SearchRange_h=a*BlkH

[0511] Where 'a' is a constant that controls the gain / complexity tradeoff. In practice, 'a' is equal to 5. The intra template matching tool is enabled for CUs whose size is less than or equal to 64 in width and height. This maximum CU size for intra template matching is configurable.

[0512] When DIMD is not used for the current CU, the intra template matching prediction mode is signaled at the CU level through a dedicated flag.

[0513] 3. Question

[0514] In the current design of IBC, the prediction of the current block is obtained using the samples in the current picture indicated by the block vector. The coding performance of IBC is very good for screen content videos with repeated content. However, for natural content videos, due to different characteristics, the codec gain of IBC is much lower than that of screen content videos.

[0515] 4. Detailed solution

[0516] The following embodiments should be considered as examples to explain the overall concept. These embodiments should not be interpreted in a narrow manner. In addition, these embodiments may be combined in any manner. In the present disclosure, intra-block copying (IBC) may not be limited to current IBC techniques, but may be interpreted as a technique in which a reference (or prediction) block is obtained using samples in the current strip / tile / sub-picture / picture / other video unit (e.g., CTU row), excluding conventional intra-frame prediction methods.

[0517] A reference line may refer to row and / or column reconstructed samples that are adjacent or non-adjacent to the current block, which are used to derive intra-frame prediction of the current video unit via an interpolation filter along a specific direction, and the specific direction is determined by an intra-frame prediction mode (e.g., conventional intra-frame prediction with an intra-frame prediction mode), or used to derive intra-frame prediction of the current video unit by weighting the reference samples of the reference line with a matrix or vector (e.g., MIP).

[0518] Combination of intra block copy and intra prediction

[0519] 1. It is proposed to use a combination of intra block copy (IBC) and intra prediction (CIBIP) to derive the prediction / reconstruction of a video unit obtained by fusing the IBC prediction signal and the intra prediction signal.

[0520] a. In one example,

[0521] P(x,y)=w ip1 *IP1(x,y)+w ip2 *IP2(x,y)+...+w ipn *IP n (x,y)+w ibc1 *IBC1(x,y)+w ibc2 *IBC2(x,y)+...+w ibcm *IBC m (x, y), where P(x, y) is the generated prediction value, IPk is the prediction generated by the k-th intra prediction, and IBC j is the prediction generated by the jth IBC, and wipk and wibc j is the corresponding weight value.

[0522] b. In one example, intra prediction may be generated by angular intra prediction, DC, planar, cross component prediction (CCLM), multi-model CCLM, left CCLM, top CCLM, etc.

[0523] i. Intra prediction mode can be coded using MPM or TIMD or DIMD or any other method to signal the intra prediction mode.

[0524] ii. A specific set of intra prediction modes may be allowed to be used in CIBIP.

[0525] c. In one example, IBC merge mode may be used.

[0526] i. In one example, one or more block vector (BV) candidates in the IBC merge list may be allowed to be used in the CIBIP.

[0527] ii. In one example, one or more BV offsets are used for CIBCIP.

[0528] 1). In one example, a BV offset may be added to a BV candidate before the BV candidate is used to obtain an IBC prediction.

[0529] 2) In one example, the BV offset may be signaled or derived.

[0530] iii. In one example, the IBC merge mode can be at least one of the following: conventional IBC

[0531] Merger mode, IBC-MBVD merge mode and IBC-TM merge mode.

[0532] d. In one example, IBC AMVP mode may be used.

[0533] i. In one example, one or more BV predictors in the IBC AMVP list may be allowed to be used in the CIBCIP.

[0534] ii. In one example, the block vector difference (BVD) used in CIBCIP may be signaled in the same manner as in IBC mode.

[0535] 1) Alternatively, the BVD may not be signaled but predefined.

[0536] iii. In one example, the BVD may be derived using codec information.

[0537] e. In one example, the following items used to obtain the IBC prediction signal may be signaled: a merge index (mergeldx) indicating a BV candidate in an IBC merge list and / or a BVP index (bvpldx) indicating a BV predictor in an IBC AMVP list.

[0538] 1) In one example, the binarization or signaling manner of the merge index or BVP index may be the same as in the IBC mode.

[0539] 2) Alternatively, the merge index or BVP index may be predefined, such as mergeldx=0 or mergeldx=1; bvpldx=0 or bvpldx=1.

[0540] 3) Alternatively, the merge index or BVP index can be derived using codec information.

[0541] 4) Alternatively, the merge index or BVP index may be derived using template matching (eg, with minimum template matching cost).

[0542] f. In one example, the construction of the IBC merge list or IBC AMVP list used in CIBIP mode may be the same as or different from that used in IBC mode.

[0543] g. In one example, the number (N) of BV candidates available for CIBCIP in the IBC merge (or AMVP) list is less than or equal to the number (M) of BV candidates available for IBC in the IBC merge (or AMVP) list. N is an integer greater than 0 and less than or equal to M.

[0544] i. In one example, N=1, or N=2, or N=3, or N=4, or N=5, or N=6.

[0545] ii. In one example, the top N BV candidates of the IBC merge (or AMVP) list may be used for CIBCIP.

[0546] h. In one example, template matching can be used to derive / refine the BV used to obtain the IBC prediction signal.

[0547] i. In one example, the BV offset can be derived using template matching and the BV offset is added to the BV candidate in the IBC merge list.

[0548] ii. In one example, the BVD can be derived using a template matching based approach.

[0549] iii. In one example, the BVD symbols can be derived using a template matching based approach.

[0550] iv. In one example, the intra prediction mode or intra prediction method used to obtain the intra prediction signal can be used in template matching to derive / refine the BV.

[0551] i. In one example, the BV list may be reordered before being used in CIBIP.

[0552] i. In one example, template matching or bilateral matching cost can be used for re-ranking.

[0553] ii. In one example, during the construction of the BV list for CIBCIP, template matching or bilateral matching is used.

[0554] iii. In one example, the BV List may refer to the IBC Consolidated List or the IBC AMVP

[0555] List.

[0556] iv. In one example, the reordering method of the BV list for CIBCIP can be the same as the reordering method for IBC mode.

[0557] v. Alternatively, the reordering method for the BV list of CIBIP can be the same as that for IBC

[0558] The reordering method of the modes is different.

[0559] 1) In one example, the number of BV candidates (N1) used for reordering for CIBCIP in the BV list may be less than or equal to the number of BV candidates (M1) used for reordering for IBC mode.

[0560] a) In one example, when the IBC merge mode is used for CIBCIP,

[0561] N1=1 or 2 or 3 or 4.

[0562] b) In one example, when IBC AMVP is used for CIBIP, N1

[0563] =1 or 2 or 3.

[0564] j. In one example, intra prediction may refer to a conventional intra prediction method (e.g., intra prediction using 35 intra prediction modes in HEVC or 67 intra prediction modes in VVC), or other intra prediction methods that utilize samples in the current slice / tile / sub-picture / picture / other video units (e.g., CU, PU, ​​TU, CTU, CTU row) excluding IBC to obtain a prediction block.

[0565] i. In one example, the intra prediction signal may be obtained using one or more predefined intra prediction modes.

[0566] 1) In one example, the predefined intra prediction mode may refer to planar mode,

[0567] DC mode, horizontal mode, vertical mode.

[0568] ii. In one example, the intra prediction signal may be obtained using one or more of the Most Probable Modes (MPMs).

[0569] iii. In one example, the intra prediction signal may be obtained using an intra prediction mode derived using a block vector used to obtain the IBC prediction signal.

[0570] iv. In one example, the intra prediction signal may be obtained using an intra prediction mode derived using a template-based method such as TIMD.

[0571] v. In one example, the intra prediction signal may be obtained using an intra prediction mode derived using neighboring samples or gradients of neighboring samples, such as DIMD.

[0572] vi. In one example, the intra prediction signal can be obtained using ISP.

[0573] vii. In one example, the intra prediction signal can be obtained using MIP.

[0574] viii. In one example, the intra prediction signal can be obtained using MRL.

[0575] ix. In one example, the intra prediction signal may be intra template matching prediction (IntraTMP).

[0576] 2. In one example, weight parameters for fusing the IBC prediction signal and the intra prediction signal may be signaled or derived.

[0577] a. In one example, weight parameters may be signaled.

[0578] i. In one example, a set of weight parameters may be constructed and an index indicating the weight parameters may be signaled.

[0579] b. In one example, the weight parameters can be derived using codec information.

[0580] i. In one example, the codec information may refer to the codec mode of the neighboring unit.

[0581] 1) In one example, the weight parameter may depend on whether one or more neighboring units are coded using intra prediction or IBC mode.

[0582] ii. In one example, the codec information may refer to an intra prediction mode used to obtain an intra prediction signal.

[0583] iii. In one example, the codec information may refer to the block size or block dimension of the current video unit and / or the neighboring video units.

[0584] iv. In one example, the weight parameters may be derived using a template matching method (eg, with minimum template matching cost).

[0585] c. In one example, the weight parameters may be predefined.

[0586] 3. In one example, the reference area of ​​CIBCIP may be smaller than or equal to the reference area of ​​IBC.

[0587] a. In one example, the reference area of ​​CIBCIP may depend on the coding information of intra prediction.

[0588] i. In one example, the reference area of ​​CIBCIP may depend on the intra prediction mode.

[0589] b. Alternatively, the reference region of CIBCIP may be different from that of IBC.

[0590] 4. Whether and / or how the CIBCIP mode is applied to a video unit may depend on codec information, which may refer to:

[0591] a. Whether IBC or intra prediction method is allowed,

[0592] b. block dimensions and / or block size,

[0593] c. Block depth,

[0594] d. Stripe / picture type and / or partition tree type (single or dual tree, or local dual tree),

[0595] e. Time layer identification,

[0596] f. Block location,

[0597] g. Color component.

[0598] Combined signaling of intra block copy and intra prediction

[0599] 5. Indications of CIBCIP patterns can be derived on the fly.

[0600] 6. The indication of CIBCIP mode may be signaled conditionally, where the condition may include:

[0601] a. Whether IBC or intra prediction method is allowed,

[0602] b. block dimensions and / or block size,

[0603] c. Block depth,

[0604] d. Stripe / picture type and / or partition tree type (single or dual tree, or local dual tree),

[0605] e. Time layer identification,

[0606] f. Block location,

[0607] g. Color component.

[0608] 7. Whether the current block is coded using CIBCIP mode may be signaled using one or more syntax elements.

[0609] a. In one example, the syntax element may be binarized using a fixed length codec or a truncated unary codec or a unary codec or an EG codec or a coded flag.

[0610] b. In one example, the syntax element may be bypass coded or context coded.

[0611] i. The context may depend on coded information, such as block dimension and / or block size, and / or slice / picture type, and / or information of neighboring blocks (adjacent or non-adjacent), and / or information of other codec tools used for the current block, and / or information of the temporal layer.

[0612] c. In one example, when the current video unit is coded by IBC, an indication of CIBCIP mode may be signaled.

[0613] d. In one example, the syntax element may be signaled before the indication of IBC-TM mode or IBC-MBVD mode, or the syntax element may be signaled after the indication of IBC-TM mode or IBC-MBVD mode.

[0614] i. In one example, whether to signal and / or how to signal a syntax element may depend on whether IBC mode or IBC-TM mode or IBC-MBVD mode is enabled for a video unit.

[0615] e. In one example, one or more syntax elements may be signaled at a sequence header / picture header / SPS / VPS / DPS / DCI / PPS / APS / slice header / tile group header.

[0616] f. In one example, syntax elements may be encoded and decoded in a predictive manner.

[0617] g. For example, the syntax elements of the current block can be predicted by the syntax elements of the neighboring blocks.

[0618] 8. In one example, for the above items, RR-IBC or symmetric IBC method can be used in CIBCIP.

[0619] 9. In one example, for the above entries, RR-IBC or symmetric IBC methods may be disabled in CIBCIP.

[0620] 10. In one example, the flip type of the IBC prediction portion may be set to NO_FLIP (eg, 0).

[0621] Intra prediction using fused reference lines

[0622] 11. It is proposed that more than one reference line be fused before being used to derive the intra prediction of a video unit.

[0623] a. In one example, the number of reference lines (N) and which reference lines are to be fused may be predefined, signaled in the bitstream, or derived at runtime, where N is an integer greater than 1.

[0624] i. In one example, N may be predefined, such as N=2 or N=3.

[0625] ii. In one example, N may be signaled at a sequence header / picture header / SPS / VPS / DPS / DCI / PPS / APS / slice header / tile group header.

[0626] iii. In one example, N may be determined based on codec information.

[0627] 1) In one example, the codec information may refer to a block size, a block dimension, or a block position, or a codec mode, or an intra prediction mode.

[0628] iv. In one example, which reference lines are used in the fusion can be indicated by the reference line index.

[0629] 1) The reference line index can be predefined, signaled in the bitstream, or derived at runtime.

[0630] 2) In one example, one of the reference line indexes may be predefined, and the other remaining N-1 reference line indexes may be signaled.

[0631] 3) In one example, one of the reference line indexes may be signaled, and the other remaining N-1 reference line indexes are predefined or derived at runtime.

[0632] b. In one example, N reference lines can be fused using weight parameters.

[0633] i. In one example, L = W1*L1+W2*L2+...+W N-1 *L N , where L i and W i denotes the i-th reference line used in fusion and the corresponding weight, and L denotes the fused reference line for intra prediction.

[0634] 1) In another example, L = (W'1*L1+W'2*L2+...+W' N-1 *L N )>>Shift1, where W'1+W'2+...+W' N-1 =2^Shoft1.

[0635] ii. In one example, the weight parameters may be predefined, or signaled in the bitstream, or derived at runtime.

[0636] iii. In one example, when the reference line L a Compared with reference line L b When you are closer to the current video unit, L a The corresponding weight parameter W a or W' a Can be equal to or greater than L b W b or W' b .

[0637] iv. In one example, when N=2, W1=3 / 4 and W2=1 / 4, or W1=5 / 8 and W2=3 / 8, or W1=1 / 2 and W2=1 / 2.

[0638] v. In one example, when N=3, W1=1 / 2 and W2=1 / 4 and W3=1 / 4, or W1=5 / 8 and W2=1 / 4 and W3=1 / 8.

[0639] c. In one example, the number of samples in one reference line can be the same as the number of samples in another reference line. The fusion of samples is expressed as follows: P(x, y) = W1*P(x1, y1)1+W2*P(x2, y2)2+...+W N-1 *P(x N-1 ,y N-1 ) N-1, where P(xi,yi)i represents a sample in the i-th reference line.

[0640] i. In one example, when fusing the above-mentioned parts of the reference lines, samples in different reference lines having the same horizontal position may be fused.

[0641] 1) In one example, x1=x2=...=x N-1 .

[0642] ii. In one example, when blending the left portion of a reference line, samples in different reference lines having the same vertical position may be fused.

[0643] 1) In one example, y1=y2=...=y N-1 .

[0644] d. In one example, the number of samples in one reference line may be different from the number of samples in another reference line. m and L n The number of samples in is denoted as S m and S n The example is shown as Fig.46 .

[0645] i. In one example, the number of samples in the fused reference line may be the same as the number of samples in the reference line with the least number of samples.

[0646] 1) In one example, in order to fuse the samples of the fused reference line, the reference line L m Use the reference line L n Many samples, among which L m The total number of samples in is greater than L n .

[0647] a) In one example, two or more samples may be taken at L m is used, and a sample can be n Used in.

[0648] 2) In one example, reference line L m S n The samples can be compared with the reference line L n

[0649] S n The examples are depicted as Fig.47 .

[0650] ii. In one example, the number of samples in the fused reference line may be the same as the number of samples in the reference line having the largest number of samples.

[0651] 1) In one example, when S n Less than S m When (S m -S n ) samples can use the reference line S n The samples in are filled or derived and used for fusion. The example is depicted as Fig.48 .

[0652] e. In one example, after the derivation of each reference line, fusion of the reference lines may be performed.

[0653] i. Alternatively, during the derivation of the reference lines, a fusion of the reference lines may be performed.

[0654] f. In one example, the derivation of reference samples used for fusion in a reference line may be the same as the derivation of reference samples not used for fusion of the reference line.

[0655] i. Alternatively, these derivations may be different.

[0656] 1) In one example, how to handle unavailable reference samples may be different.

[0657] g. In one example, after the fusion of reference lines, reference sample filtering may be performed.

[0658] i. Alternatively, prior to fusion of reference lines, reference sample filtering may be performed.

[0659] 1) In one example, reference sample filtering may be different for different reference lines.

[0660] 12. Whether and how to use the fused reference lines to derive intra prediction for the current video unit may depend on codec information.

[0661] a. In one example, the codec information may indicate one or more intra prediction methods.

[0662] i. In one example, the fused reference lines can be used for conventional intra prediction.

[0663] ii. In one example, the fused reference lines can be used in MRL / ISP / MIP / DIMD / TIMD.

[0664] iii. In one example, the fused reference lines can be used for conventional chroma intra prediction.

[0665] iv. In one example, the fused reference line can be used for LM of chromaticity and fusion of angle.

[0666] v. In one example, the fused reference lines can be used as an additional method or to replace the current intra prediction method.

[0667] b. In one example, the codec information may refer to a color component.

[0668] i. In one example, the fused reference lines can be used for intra prediction of the luma component.

[0669] ii. In one example, the fused reference lines can be used for intra prediction of chroma components.

[0670] c. In one example, the codec information may refer to an intra prediction mode.

[0671] i. In one example, when DC mode is used, fused reference lines may be used.

[0672] ii. In one example, when planar mode is used, fused reference lines may be used.

[0673] iii. In one example, when angular intra prediction mode is used, the fused reference lines may be used.

[0674] iv. In one example, when angular intra prediction mode with non-integer slope is used, a fused reference line may be used.

[0675] v. In one example, the fused reference line can be used for more than one intra prediction mode.

[0676] d. In one example, the encoding information may refer to the block size / dimension of the current block and / or neighboring blocks.

[0677] i. In one example, when the block size of the current block is greater than and equal to T1, the fused reference line may be used.

[0678] ii. In another example, when the block size of the current block is smaller than T2, the fused reference line may be used.

[0679] e. In one example, the codec information may refer to slice type and / or temporal layer and / or QP.

[0680] f. In one example, the fused reference line may not be allowed to be used for video units in different CTUs.

[0681] 13. In one example, how to fuse reference lines, and whether and how to use the fused reference lines to derive intra prediction for the current video unit can be signaled in the bitstream.

[0682] General Claims

[0683] 14. In the above examples, a video unit may refer to a color component / sub-image / slice / tile / codec tree unit (CTU) / CTU row / group of CTUs / codec unit (CU) / prediction unit (PU) / transform unit (TU) / codec tree block (CTB) / codec block (CB) / prediction block (PB) / transform block (TB) / block / sub-block of a block / sub-region within a block / any region containing more than one sample or pixel.

[0684] 15. Whether and / or how to apply the method disclosed above can be signaled at the sequence level / picture group level / picture level / slice level / tile group level, for example in the sequence header / picture header / SPS / VPS / DPS / DCI / PPS / PPS / slice header / tile group header.

[0685] 16. Whether and / or how the above approach is applied may depend on the following information:

[0686] a. Message signaled in DPS / SPS / VPS / PPS / APS / picture header / slice header / tile group header / largest codec unit (LCU) / codec unit (CU) / LCU row / LCU group / TU / PU block / video codec unit;

[0687] b. The location of CU / PU / TU / block / video codec unit;

[0688] c. The block dimensions of the current block and / or its neighboring blocks;

[0689] d. Block shapes of the current block and / or its neighboring blocks;

[0690] e. The coded mode of the block, such as IBC or non-IBC inter-frame mode or non-IBC sub-block mode;

[0691] f. Indication of color format (e.g. 4:2:0, 4:4:4);

[0692] g. Codec tree structure;

[0693] h. Strip / tile group type and / or picture type;

[0694] i. Color components (e.g., may apply only to chroma components or luma components);

[0695] j.Time layer ID;

[0696] k. Standard profiles / levels / tiers

[0697] As used herein, the term "video unit" or "video block" may be a sequence, a picture, a slice, a tile, a block, a sub-picture, a codec tree unit (CTU) / codec tree block (CTB), a CTU / CTB row, one or more codec units (CU) / codec blocks (CB), one or more CTU / CTBs, one or more virtual pipeline data units (VPDU), a sub-region within a picture / slice / tile / brick. The term "reference line" may refer to row and / or column reconstructed samples adjacent or non-adjacent to a current block, which are used to derive intra-frame prediction of the current video unit via an interpolation filter along a specific direction, and the specific direction is determined by an intra-frame prediction mode (e.g., conventional intra-frame prediction with an intra-frame prediction mode), or to derive intra-frame prediction of the current video unit by weighting the reference samples of the reference line with a matrix or vector (e.g., MIP).

[0698] Fig.49 A flow chart of a method 4900 for video processing according to an embodiment of the present invention is illustrated. The method 4900 is implemented during conversion between a video unit of a video and a bitstream of the video.

[0699] At block 4910, for conversion between a video unit of a video and a bitstream of the video unit, a combination of intra block copy (IBC) and intra prediction (CIBIP) is applied to the video unit.

[0700] At block 4920, a prediction for the video unit is derived by combining the IBC prediction signal and the intra prediction signal.

[0701] At block 4930, based on the prediction of the video unit, a conversion is performed. In some embodiments, the conversion may include encoding the video unit into a bitstream. Alternatively or additionally, the conversion may include decoding the video unit from the bitstream. In this way, codec efficiency and codec performance may be improved.

[0702] In some embodiments, the prediction of a video unit is obtained by: P(x, y) = w ip1 *IP1(x,y)+w ip2 *IP2(x,y)+w ipn *IP n (x,y)+w ibc1 *IBC1(x,y)+w ibc2 *IBC2(x,y)+...+w ibcm *IBC m (x, y), and where P(x, y) represents the prediction of the video unit, IP k represents the prediction generated by the k-th intra prediction, IBC j is the prediction generated by the jth IBC prediction, Wipk and W ibcj represents the corresponding weight value, and k and j are integers.

[0703] In some embodiments, the intra prediction signal is generated by at least one of the following: angular intra prediction mode, direct current (DC) mode, planar mode, cross component prediction (CCLM) mode, multi-model CCLM mode, left CCLM mode, or upper CCLM mode. In some embodiments, the intra prediction mode of the intra prediction signal is encoded and decoded using one of the following items to indicate the intra prediction mode: most probable mode (MPM), template-based intra mode derivation (TIMD), or decoder-side intra mode derivation (DIMD). In some embodiments, a set of intra prediction modes are allowed to be used in CIBCIP.

[0704] In some embodiments, the IBC merge mode is used in the CIBCIP. In some embodiments, one or more block vector (BV) candidates in the IBC merge list are allowed to be used in the CIBIP. In some embodiments, one or more BV offsets are used for the CIBCIP. In some embodiments, the one or more BV offsets are added to the BV candidate before the BV candidate is used to obtain the IBC prediction signal. In some embodiments, one or more BV offsets are indicated or derived. In some embodiments, the IBC merge mode includes at least one of the following: a conventional IBC merge mode, an IBC with block vector difference (IBC-MBVD) merge mode, or an IBC-template matching (TM) merge mode.

[0705] In some embodiments, the IBC Advanced Motion Vector Prediction (AMVP) mode is used in CIBCIP. In some embodiments, one or more BV predictors in the IBC AMVP list are allowed to be used in CIBCIP. In some embodiments, the block vector difference (BVD) used in CIBCIP is indicated in the same manner as the IBC mode. In some embodiments, the BVD used in CIBCIP is predefined. In some embodiments, the BVD used in CIBCIP is derived using the codec information of the video unit.

[0706] In some embodiments, at least one of the following items for obtaining an IBC prediction signal is indicated: a merge index indicating a BV candidate in an IBC merge list, or a BVP index indicating a BV predictor in an IBC AMVP list. In some embodiments, the binarization or signaling method of the merge index or BVP index is the same as in the IBC mode. In some embodiments, at least one of the merge index or BVP index is predefined. In some embodiments, the merge index is 0 or 1. Alternatively, the BVP index is 0 or 1. In some embodiments, at least one of the merge index or BVP index is derived using the codec information of the video unit. In some embodiments, at least one of the merge index or BVP index is derived using template matching.

[0707] In some embodiments, the construction of the IBC merge list or IBC AMVP list used in CIBIP is the same as the construction used in IBC mode. Alternatively, the construction of the IBC merge list or IBC AMVP list used in CIBIP is different from the construction used in IBC mode.

[0708] In some embodiments, the first number of BV candidates for CIBCIP in the IBC merge list is less than or equal to the number of BV candidates for IBC in the IBC merge list, and wherein the first number is an integer greater than 0 and less than or equal to the second number. Alternatively, the first number of BV candidates for CIBCIP in the AMVP list is less than or equal to the number of BV candidates for IBC in the AMVP list, and wherein the first number is an integer greater than 0 and less than or equal to the second number. In some embodiments, the first number is one of the following: 1, 2, 3, 4, 5, or 6. In some embodiments, the first N BV candidates of the IBC merge list are used for CIBCIP. Alternatively, the first N BV candidates of the AMVP list are used for CIBCIP, and wherein N is an integer.

[0709] In some embodiments, template matching is used to derive / refine the BV used to obtain the IBC prediction signal. In some embodiments, a BV offset is derived using template matching and the BV offset is added to the BV candidate in the IBC merge list.

[0710] In some embodiments, the block vector difference BVD is derived using a template matching based approach. In some embodiments, the sign of the BVD is derived using a template matching based approach.

[0711] In some embodiments, the intra prediction mode or intra prediction method used to obtain the intra prediction signal is used in template matching to derive / refine the BV. In some embodiments, the BV list is reordered before being used for CIBCIP.

[0712] In some embodiments, template matching or bilateral matching costs are used for reordering of the BV list. In some embodiments, template matching or bilateral matching is used during construction of the BV list for CIBCIP. In some embodiments, the BV list includes an IBC merge list or an IBC AMVP list.

[0713] In some embodiments, the reordering of the BV list for CIBCIP is the same as the reordering for IBC mode. Alternatively, the reordering of the BV list for CIBIP is different from the reordering for IBC mode.

[0714] In some embodiments, the number of BV candidates in the BV list for reordering for CIBCIP is less than or equal to the number of BV candidates for reordering for IBC mode. In some embodiments, if IBC merge mode is used for CIBCIP, the number of BV candidates is one of 1, 2, 3, or 4. Alternatively, if IBC AMVP is used for CIBCIP, the number of BV candidates is one of 1, 2, or 3.

[0715] In some embodiments, intra prediction includes conventional intra prediction or other intra prediction methods, wherein the other intra prediction methods utilize samples in the following items excluding IBC to obtain a prediction block: a current slice, a current tile, a current sub-picture, a current picture, or other video units.

[0716] In some embodiments, the intra prediction signal is obtained using one of the following: one or more predefined intra prediction modes, one or more of the MPMs, an intra prediction mode derived using a block vector used to obtain the IBC prediction signal, one or more intra prediction modes derived using a template-based approach, one or more intra prediction modes derived using neighboring samples or gradients of neighboring samples, intra sub-partitioning (ISP), matrix weighted intra prediction (MIP), multi-reference line intra prediction (MRL), or intra template matching prediction (Intra TMP).

[0717] In some embodiments, the one or more predefined intra prediction modes include at least one of: planar mode, DC mode, horizontal mode or vertical mode. In some embodiments, the intra prediction signal is intra template matching prediction (IntraTMP).

[0718] In some embodiments, a weight parameter for combining the IBC prediction signal and the intra prediction signal is indicated or derived. In some embodiments, a set of weight parameters is constructed and an index indicating the weight parameter is indicated. In some embodiments, the weight parameter is derived using codec information.

[0719] In some embodiments, the coding information comprises an intra prediction mode for obtaining the intra prediction signal. In some embodiments, the weight parameter depends on whether one or more neighboring units are coded using intra prediction or IBC mode.

[0720] In some embodiments, the codec information includes an intra-frame prediction mode for obtaining an intra-frame prediction signal. Alternatively or additionally, the codec information includes at least one of the following: a block size or block dimension of a video unit, or a block size or block dimension of an adjacent video unit. In some embodiments, the weight parameter is derived using a template matching method. In some embodiments, the weight parameter is predefined.

[0721] In some embodiments, the reference area of ​​CIBCIP is less than or equal to the reference area of ​​IBC. The reference area of ​​CIBCIP depends on the codec information of intra prediction. In some embodiments, the reference area of ​​CIBCIP depends on the intra prediction mode. In some embodiments, the reference area of ​​CIBCIP is different from the reference area of ​​IBC.

[0722] In some embodiments, whether CIBCIP is applied to a video unit and / or the manner in which CIBCIP is applied to a video unit depends on codec information, wherein the codec information includes at least one of: whether IBC or intra prediction is allowed, block dimension and / or block size, block depth, slice type, picture type, partition tree type, temporal layer identifier, block position, or color component.

[0723] In some embodiments, the indication of CIBCIP is dynamically derived. In some embodiments, the indication of CIBIP is indicated based on a condition. The indication of CIBCIP is based on a condition. The condition includes at least one of the following: whether IBC or intra prediction mode is allowed, block dimension and / or block size, block depth, slice type, picture type, partition tree type, temporal layer identification, block position, or color component.

[0724] In some embodiments, whether a video unit is coded using CIBCIP is indicated using one or more syntax elements. In some embodiments, the one or more syntax elements are binarized using one of the following: fixed length codec, truncated unary codec, unary codec, EG codec, or an encoded flag.

[0725] In some embodiments, one or more syntax elements are bypass coded or context coded. In some embodiments, the context depends on coded information. The coded information includes at least one of the following: block dimensions, block size, slice type, picture type, information about neighboring blocks, information about other codecs for the video unit, or information about a temporal layer.

[0726] In some embodiments, when the video unit is IBC coded, an indication of CIBCIP is indicated. In some embodiments, one or more syntax elements are indicated before an indication of IBC-TM mode or IBC-MBVD mode, or one or more syntax elements are indicated after an indication of IBC-TM mode or IBC-MBVD mode.

[0727] In some embodiments, whether to indicate one or more syntax elements and / or the manner in which one or more syntax elements are indicated depends on whether at least one of the following is enabled for the video unit: IBC mode, IBC-TM mode, IBC-MBVD mode.

[0728] In some embodiments, one or more syntax elements are indicated at one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header. In some embodiments, one or more syntax elements are encoded and decoded in a predictive manner. In some embodiments, one or more syntax elements of a video unit are predicted by one or more syntax elements of neighboring blocks.

[0729] In some embodiments, reconstruction reorder IBC (RR-IBC) or symmetric IBC method is used in CIBCIP. Alternatively, RR-IBC or symmetric IBC method is disabled in CIBCIP. In some embodiments, the flip type of the IBC prediction part is set to NO_FLIP.

[0730] In some embodiments, the video unit includes at least one of the following: a color component, a prediction block (PB), a transform block (TB), a codec block (CB), a prediction unit PU, a transform unit (TU), a codec tree block (CTB), a codec unit (CU), a codec tree unit (CTU), a CTU row, a group of CTUs, a slice, a tile, a sub-picture, a block, a sub-region within a block, or a region containing more than one sample or pixel.

[0731] In some embodiments, an indication of whether and / or how to derive a prediction for a video unit by combining an IBC prediction signal and an intra prediction signal is indicated at one of: a sequence level, a group of pictures level, a picture level, a slice level, or a tile group level.

[0732] In some embodiments, an indication of whether and / or how to derive a prediction for a video unit by combining an IBC prediction signal and an intra prediction signal is indicated in one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header.

[0733] In some embodiments, method 4900 also includes: determining whether and / or how to derive the prediction of the video unit by combining the IBC prediction signal and the intra-frame prediction signal based on at least one of the following: a message indicated in one of the following: DPS, SPS, VPS, PPS, APS, picture header, slice header, tile group header, maximum codec unit (LCU), codec unit (CU), LCU row, group of LCUs, TU, PU block, video codec unit, the position of one of the following: CU, PU, ​​TU, block, video codec unit, block dimensions of the current block and / or its neighboring blocks, block shape of the current block and / or its neighboring blocks, the coded mode of the video unit; an indication of color format, codec tree structure, slice type, tile group type, picture type, color component, temporal layer identifier, profile or level or level of the standard.

[0734] According to another embodiment of the present disclosure, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by a video processing device. The method includes: applying a combination of intra-frame block copy (IBC) and intra-frame prediction (CIBIP) to a video unit of the video; deriving a prediction of the video unit by combining an IBC prediction signal and an intra-frame prediction signal; and generating a bitstream based on the prediction of the video unit.

[0735] According to some other embodiments of the present disclosure, a method for storing a bitstream of a video is provided. The method includes: applying a combination of intra-block copy (IBC) and intra-frame prediction (CIBIP) to a video unit of the video; deriving a prediction of the video unit by combining an IBC prediction signal and an intra-frame prediction signal; generating a bitstream based on the prediction of the video unit; and storing the bitstream in a non-transitory computer-readable recording medium.

[0736] Fig.50 A flow chart of a method 5000 for video processing according to an embodiment of the present disclosure is illustrated. The method 5000 is implemented during conversion between a video unit of a video and a bit stream of the video.

[0737] At block 5010, a plurality of reference lines are combined for conversion between a video unit of a video and a bitstream of the video unit.

[0738] At block 5020, an intra prediction for the video unit is derived based on the combined multiple reference lines.

[0739] At block 5030, the conversion is performed based on intra-frame prediction. In some embodiments, the conversion may include encoding the video unit into a bitstream. Alternatively or additionally, the conversion may include decoding the video unit from the bitstream. In this way, the coding efficiency and coding performance can be improved.

[0740] In some embodiments, the number of reference lines and which reference lines are to be combined is one of: predefined, indicated in the bitstream, or dynamically derived.Alternatively, the number of reference lines is an integer number greater than one.

[0741] In some embodiments, the number of reference lines is predefined. Alternatively or additionally, the number of reference lines is 2 or 3.

[0742] In some embodiments, the number of reference lines is indicated in one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header.

[0743] In some embodiments, the number of reference lines is determined based on codec information. In some embodiments, the codec information includes one of the following: block size, block dimension, block position, codec mode, and intra-frame prediction mode.

[0744] In some embodiments, which reference lines are used in the combination is indicated by a reference line index. In some embodiments, the reference line index is predefined. Alternatively, the reference line index is indicated in the bitstream. Alternatively, the reference line index is dynamically derived.

[0745] In some embodiments, one of the reference line indices is indicated, or the other remaining reference line indices are indicated. In some embodiments, one of the reference line indices is indicated, and the other remaining reference line indices are predefined or dynamically derived.

[0746] In some embodiments, multiple reference lines are combined using weight parameters. In some embodiments, the combined multiple reference lines are obtained by the following formula: L = W1*L1+W2*L2+...+WN-1*LN, where Li and Wi represent the i-th reference line used in the combination and the corresponding weight, and L represents the combined multiple reference lines for intra-frame prediction. In some embodiments, L = (W'1*L1+W'2*L2+...+W'N-1*LN)>>Shift1, where W'1+W'2+...+W'N-1=2^Shoft1.

[0747] In some embodiments, the weight parameters are predefined. Alternatively, the weight parameters are indicated in the bitstream. Alternatively, the weight parameters are dynamically derived.

[0748] In some embodiments, when the reference line La is closer to the video unit than the reference line Lb, the corresponding weight parameter Wa or W'a of the reference line La is equal to or greater than Wb or W'b of the reference line Lb. In some embodiments, if N=2, W1=3 / 4 and W2=1 / 4, or W1=5 / 8 and W2=3 / 8, or W1=1 / 2 and W2=1 / 2. In some embodiments, if N=3, W1=1 / 2 and W2=1 / 4 and W3=1 / 4, or W1=5 / 8 and W2=1 / 4 and W3=1 / 8.

[0749] In some embodiments, the number of samples in one reference line is the same as the number of samples in another reference line. In some embodiments, the combination of samples is obtained as follows: P(x, y) = W1*P(x1, y1)1 + W2*P(x2, t2)2 + ... + WN-1*P, (xN-1, yN-1)N-1, where P(xi, yi)i represents a sample in the i-th reference line, and Wi represents the weight parameter corresponding to the i-th reference line.

[0750] In some embodiments, when combining the upper portion of the reference lines, samples in different reference lines having the same horizontal position are combined. In some embodiments, x1 = x2 = ... = xN-1.

[0751] In some embodiments, when combining the left portion of the reference lines, samples in different reference lines with the same vertical position are combined. In some embodiments, y1 = y2 = ... = yN-1.

[0752] In some embodiments, the number of samples in one reference line is different from the number of samples in another reference line. In some embodiments, the number of samples in the combined plurality of reference lines is the same as the number of samples in the reference line with the least number of samples.

[0753] In some embodiments, the number of samples used in reference line Lm is greater than the number of samples used in reference line Ln to combine the samples of the combined reference lines, wherein the total number of samples in reference line Lm is greater than the total number of samples in reference line Ln. In some embodiments, two or more samples are used in reference line Lm, and one sample is used in reference line Ln. In some embodiments, Sn samples in reference line Lm are used in combination with Sn samples in reference line Ln, where Sn is an integer and represents the number of samples in reference line Ln. In some embodiments, the number of samples in the combined multiple reference lines is the same as the number of samples of the reference line having the largest number of samples.

[0754] In some embodiments, if the number of samples in reference line Ln is less than the number of samples in reference line Lm, the difference between the number of samples in reference line Ln and the number of samples in reference line Lm is filled or derived using the samples in reference line Ln and used for combining. In some embodiments, combining of the multiple reference lines is performed after derivation of the multiple reference lines. Alternatively, combining of the multiple reference lines is performed during derivation of the multiple reference lines.

[0755] In some embodiments, the derivation of the reference samples used for combination in the plurality of reference lines is the same as the derivation of the reference samples not used for combination. Alternatively, the derivation of the reference samples used for combination in the plurality of reference lines is different from the derivation of the reference samples not used for combination.

[0756] In some embodiments, unavailable reference samples are handled differently in the derivation of reference samples used for combination and in the derivation of reference samples not used for combination among the plurality of reference lines.

[0757] In some embodiments, reference sample filtering is performed after combining the plurality of reference lines. Alternatively, reference sample filtering is performed before combining the plurality of reference lines. In some embodiments, reference sample filtering is different for different reference lines.

[0758] In some embodiments, whether and / or how to use the combined multiple reference lines to derive intra prediction for a video unit depends on the encoding and decoding information. In some embodiments, the encoding information includes one or more intra prediction methods.

[0759] In some embodiments, the combined multiple reference lines are used in at least one of: conventional intra prediction, intra sub-partitioning (ISP), matrix weighted intra prediction (MIP), multiple reference line intra prediction (MRL), template-based intra mode derivation (TIMD), or decoder-side intra mode derivation (DIMD), conventional chroma intra prediction, a combination of chroma LM and angle, or a method in addition to or used to replace the current intra prediction method.

[0760] In some embodiments, the codec information comprises a color component. In some embodiments, the combined plurality of reference lines are used in intra prediction of a luma component. Alternatively, the combined plurality of reference lines are used in intra prediction of a chroma component.

[0761] In some embodiments, the codec information includes an intra prediction mode. In some embodiments, when DC mode is used, a combined plurality of reference lines are used. Alternatively, when planar mode is used, a combined plurality of reference lines are used. Alternatively, when angular intra prediction mode is used, a combined plurality of reference lines are used. Alternatively, when angular intra prediction mode with a non-integer slope is used, a combined plurality of reference lines are used. Alternatively, a combined plurality of reference lines are used for more than one intra prediction mode.

[0762] In some embodiments, the encoding information includes at least one of the following: a block size of the video unit; a block dimension of the video unit, a block size of a neighboring block, or a block dimension of a neighboring block. In some embodiments, when the block size of the video unit is greater than or equal to a first threshold, the combined multiple reference lines are used. Alternatively, when the block size of the video unit block is less than a second threshold, the combined multiple reference lines are used.

[0763] In some embodiments, the encoding information includes at least one of: a slice type, a temporal layer, or a quantization parameter (QP). In some embodiments, the combined multiple reference lines are not allowed for video units in different CTUs.

[0764] In some embodiments, the manner in which the multiple reference lines are combined is indicated in the bitstream. Alternatively or additionally, whether to use the combined multiple reference lines to derive intra-frame predictions for the video unit and / or the manner in which the combined multiple reference lines are used to derive intra-frame predictions for the video unit is indicated in the bitstream.

[0765] In some embodiments, the video unit comprises at least one of: a color component, a prediction block (PB), a transform block (TB), a codec block (CB), a prediction unit (PU), a transform unit (TU), a codec tree block (CTB), a codec unit (CU), a codec tree unit (CTU), a CTU row, a group of CTUs, a slice, a tile, a sub-image, a block, a sub-region within a block, or a region containing more than one sample or pixel. In some embodiments, an indication of whether and / or how to derive intra-frame prediction for the video unit based on the combined multiple reference lines is indicated at one of the following: a sequence level, a picture group level, a picture level, a slice level, or a tile group level.

[0766] In some embodiments, an indication of whether and / or how to derive intra-prediction for a video unit based on the combined multiple reference lines is indicated in one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header.

[0767] In some embodiments, method 5000 also includes: determining whether and / or how to derive the intra-frame prediction of the video unit based on the combined multiple reference lines based on at least one of the following: a message indicated in one of the following: DPS, SPS, VPS, PPS, APS, picture header, slice header, tile group header, maximum codec unit (LCU), codec unit (CU), LCU row, group of LCUs, TU, PU block, video codec unit, the position of one of the following: CU, PU, ​​TU, block, video codec unit, block dimensions of the current block and / or its neighboring blocks, block shape of the current block and / or its neighboring blocks, the coded mode of the video unit; an indication of color format, coding tree structure, slice type, tile group type, picture type, color component, temporal layer identifier, profile or level or level of the standard.

[0768] According to another embodiment of the present disclosure, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method executed by a video processing device. The method includes: combining a plurality of reference lines; deriving an intra-frame prediction of a video unit based on the combined plurality of reference lines; and generating a bitstream based on the prediction of the video unit.

[0769] According to some other embodiments of the present disclosure, a method for storing a bitstream of a video is provided. The method includes: combining multiple reference lines; deriving intra-frame prediction of a video unit based on the combined multiple reference lines; generating a bitstream based on the prediction of the video unit; and storing the bitstream in a non-transitory computer-readable recording medium.

[0770] Embodiments of the present disclosure may be described according to the following clauses, features of which may be combined in any reasonable way.

[0771] Clause 1. A method of video processing, comprising: applying a combination of intra-block copy (IBC) and intra-frame prediction (CIBIP) to a video unit of a video for conversion between the video unit and a bitstream of the video unit; deriving a prediction of the video unit by combining an IBC prediction signal and an intra-frame prediction signal; and performing the conversion based on the prediction of the video unit.

[0772] Clause 2. The method of clause 1, wherein the prediction of the video unit is obtained by: P(x, y) = w ip1 *IP1(x,y)+w ip2 IP2(x,y)+...+w ipn *IP n (x,y)+w ibc1 *IBC1(x,y)+w ibc2 *IBC2(x,y)+...+w ibcm *IBC m (x, y), and wherein P(x, y) represents the prediction of the video unit, IP k represents the prediction generated by the k-th intra prediction, IBC j is the prediction generated by the jth IBC prediction, W ipk and W ibcj represents the corresponding weight value, and k and j are integers.

[0773] Clause 3. A method according to clause 1, wherein the intra prediction signal is generated by at least one of: an angular intra prediction mode, a direct current (DC) mode, a planar mode, a cross-component prediction (CCLM) mode, a multi-model CCLM mode, a left CCLM mode, or an upper CCLM mode.

[0774] Clause 4. A method according to clause 1, wherein the intra-frame prediction mode of the intra-frame prediction signal is encoded and decoded using one of the following items to indicate the intra-frame prediction mode: most probable mode (MPM), template-based intra-frame mode derivation (TIMD), or decoder-side intra-frame mode derivation (DIMD).

[0775] Clause 5. The method of clause 1, wherein a set of intra prediction modes are allowed for use in the CIBCIP.

[0776] Clause 6. The method of clause 1, wherein an IBC merge mode is used in the CIBCIP.

[0777] Clause 7. The method of clause 6, wherein one or more block vector (BV) candidates in an IBC merge list are allowed to be used in the CIBIP.

[0778] Clause 8. The method of clause 6, wherein one or more BV offsets are used for the CIBCIP.

[0779] Clause 9. The method of clause 8, wherein the one or more BV offsets are added to the BV candidate before the BV candidate is used to obtain the IBC prediction signal.

[0780] Clause 10. The method of clause 8, wherein the one or more BV offsets are indicated or derived.

[0781] Clause 11. The method of clause 6, wherein the IBC merge mode comprises at least one of: a regular IBC merge mode, an IBC with block vector difference (IBC-MBVD) merge mode, or an IBC-template matching (TM) merge mode.

[0782] Clause 12. The method of clause 1, wherein an IBC Advanced Motion Vector Prediction (AMVP) mode is used in the CIBCIP.

[0783] Clause 13. The method of clause 12, wherein one or more BV predictors in the IBC AMVP list are allowed to be used in the CIBCIP.

[0784] Clause 14. The method of clause 12, wherein a block vector difference (BVD) used in the CIBCIP is indicated in the same manner as in IBC mode.

[0785] Clause 15. The method of clause 12, wherein the BVD used in the CIBCIP is predefined.

[0786] Clause 16. The method of clause 12, wherein the BVD used in the CIBCIP is derived using codec information of the video unit.

[0787] Clause 17. The method of clause 1, wherein at least one of the following is indicated for obtaining the IBC prediction signal: a merge index indicating a BV candidate in an IBC merge list, or a BVP index indicating a BV predictor in an IBC AMVP list.

[0788] Clause 18. The method of clause 17, wherein the binarization or signaling manner of the merge index or the BVP index is the same as in the IBC mode.

[0789] Clause 19. The method of clause 17, wherein at least one of the merge index or the BVP index is predefined.

[0790] Clause 20. The method of clause 19, wherein the merge index is 0 or 1, or wherein the BVP index is 0 or 1.

[0791] Clause 21. The method of clause 17, wherein at least one of the merge index or the BVP index is derived using codec information of the video unit.

[0792] Clause 22. The method of clause 17, wherein at least one of the merge index or the BVP index is derived using template matching.

[0793] Clause 23. A method according to clause 1, wherein the construction of the IBC merge list or IBC AMVP list used in the CIBIP is the same as the construction used in the IBC mode, or wherein the construction of the IBC merge list or IBC AMVP list used in the CIBIP is different from the construction used in the IBC mode.

[0794] Clause 24. A method according to Clause 1, wherein a first number of BV candidates for the CIBCIP in the IBC merge list is less than or equal to the number of BV candidates for IBC in the IBC merge list, and wherein the first number is an integer greater than 0 and less than or equal to the second number, and / or wherein the first number of BV candidates for the CIBCIP in the AMVP list is less than or equal to the number of BV candidates for IBC in the AMVP list, and wherein the first number is an integer greater than 0 and less than or equal to the second number.

[0795] Clause 25. The method of clause 24, wherein the first number is one of: 1, 2, 3, 4, 5, or 6.

[0796] Clause 26. The method of clause 24, wherein the top N BV candidates of the IBC merge list are used for the CIBCIP, and / or wherein the top N BV candidates of the AMVP list are used for the CIBCIP, and wherein N is an integer.

[0797] Clause 27. The method of clause 1, wherein template matching is used to derive / refine the BV used to obtain the IBC prediction signal.

[0798] Clause 28. The method of clause 27, wherein a BV offset is derived using template matching, the BV offset being added to the BV candidate in the IBC merge list.

[0799] Clause 29. The method according to clause 27, wherein the block vector difference BVD is derived using a template matching based approach.

[0800] Clause 30. The method of clause 27, wherein the sign of the BVD is derived using a template matching based method.

[0801] Clause 31. The method of clause 27, wherein an intra prediction mode or an intra prediction method used to obtain the intra prediction signal is used in the template matching to derive / refine the BV.

[0802] Clause 32. The method of clause 1, wherein the BV list is reordered prior to use in the CIBCIP.

[0803] Clause 33. The method of clause 32, wherein template matching or bilateral matching cost is used for reordering the BV list.

[0804] Clause 34. The method of clause 32, wherein during construction of the BV list for the CIBCIP, template matching or bilateral matching is used.

[0805] Clause 35. The method of clause 32, wherein the BV list comprises an IBC consolidated list or an IBC AMVP list.

[0806] Clause 36. A method according to clause 32, wherein the reordering method of the BV list for the CIBCIP is the same as the reordering method for the IBC mode, or wherein the reordering method of the BV list for the CIBIP is different from the reordering method for the IBC mode.

[0807] Clause 37. The method of clause 36, wherein the number of BV candidates in the BV list for reordering for the CIBCIP is less than or equal to the number of BV candidates for reordering for IBC mode.

[0808] Clause 38. The method of clause 37, wherein if IBC merge mode is used for the CIBCIP, the number of BV candidates is one of 1, 2, 3, or 4, or wherein if IBC AMVP is used for the CIBCIP, the number of BV candidates is one of 1, 2, or 3.

[0809] Clause 39. A method according to Clause 1, wherein intra-frame prediction includes a conventional intra-frame prediction method or other intra-frame prediction methods, and the other intra-frame prediction methods use samples from the following items excluding IBC to obtain a prediction block: a current slice, a current tile, a current sub-picture, a current picture, or other video units.

[0810] Clause 40. A method according to clause 39, wherein the intra-frame prediction signal is obtained using one of the following items: one or more predefined intra-frame prediction modes, one or more of the MPMs, an intra-frame prediction mode derived using a block vector used to obtain the IBC prediction signal, one or more intra-frame prediction modes derived using a template-based approach, one or more intra-frame prediction modes derived using neighboring samples or gradients of neighboring samples, intra-frame sub-partitioning (ISP), matrix-weighted intra-frame prediction (MIP), multi-reference line intra-frame prediction (MRL), or intra-frame template matching prediction (IntraTMP).

[0811] Clause 41. The method of clause 40, wherein the one or more predefined intra-prediction modes include at least one of: a planar mode, a DC mode, a horizontal mode, or a vertical mode.

[0812] Clause 42. The method of clause 1, wherein weight parameters for combining the IBC prediction signal and the intra prediction signal are indicated or derived.

[0813] Clause 43. The method of clause 42, wherein a set of weight parameters is constructed and an index indicating the weight parameters is indicated.

[0814] Clause 44. The method of clause 42, wherein the weight parameter is derived using the codec information.

[0815] Clause 45. The method of clause 44, wherein the codec information comprises a codec mode of a neighboring unit.

[0816] Clause 46. The method of clause 45, wherein the weight parameter depends on whether one or more neighboring units are encoded using intra prediction or IBC mode.

[0817] Clause 47. The method of clause 44, wherein the codec information comprises an intra prediction mode used to obtain the intra prediction signal.

[0818] Clause 48. The method of clause 44, wherein the codec information comprises at least one of: a block size or block dimension of the video unit, or a block size or block dimension of a neighboring video unit.

[0819] Clause 49. The method of clause 42, wherein the weight parameters are derived using a template matching approach.

[0820] Clause 50. The method of clause 42, wherein the weight parameters are predefined.

[0821] Clause 51. The method of Clause 1, wherein the reference area of ​​the CIBCIP is less than or equal to the reference area of ​​the IBC.

[0822] Clause 52. The method of clause 51, wherein the reference region of the CIBCIP depends on codec information of intra prediction.

[0823] Clause 53. The method of clause 51, wherein the reference region of the CIBCIP depends on an intra prediction mode.

[0824] Clause 54. The method of Clause 51, wherein the reference region of the CIBCIP is different from the reference region of the IBC.

[0825] Clause 55. A method according to Clause 1, wherein whether the CIBCIP is applied to the video unit and / or the manner in which the CIBCIP is applied to the video unit depends on codec information, wherein the codec information includes at least one of the following: whether IBC or intra-frame prediction method is allowed, block dimension and / or block size, block depth, slice type, picture type, partition tree type, temporal layer identifier, block position, or color component.

[0826] Clause 56. The method of Clause 1, wherein the indication of the CIBCIP is derived dynamically.

[0827] Clause 57. A method according to clause 1, wherein the indication of the CIBCIP is based on conditions, wherein the conditions include at least one of the following: whether IBC or intra-frame prediction mode is allowed, block dimension and / or block size, block depth, slice type, picture type, partition tree type, temporal layer identifier, block position, or color component.

[0828] Clause 58. The method of clause 1, wherein whether the video unit is encoded and decoded using the CIBCIP is indicated using one or more syntax elements.

[0829] Clause 59. The method of clause 58, wherein the one or more syntax elements are binarized using one of: a fixed length codec, a truncated unary codec, a unary codec, an EG codec, or an encoded flag.

[0830] Clause 60. The method of clause 58, wherein the one or more syntax elements are bypass coded or context coded.

[0831] Clause 61. A method according to clause 60, wherein the context depends on encoded information, and wherein the encoded information includes at least one of the following: block dimension, block size, slice type, picture type, information of neighboring blocks, information of other encoding and decoding tools used for the video unit, or information of temporal layers.

[0832] Clause 62. The method of clause 58, wherein the indication of the CIBCIP is indicated when the video unit is encoded and decoded by IBC.

[0833] Clause 63. A method according to clause 58, wherein the one or more syntax elements are indicated before the indication of IBC-TM mode or IBC-MBVD mode, or the one or more syntax elements are indicated after the indication of IBC-TM mode or IBC-MBVD mode.

[0834] Clause 64. A method according to clause 63, wherein whether to indicate the one or more syntax elements and / or the manner of indicating the one or more syntax elements depends on whether at least one of the following is enabled for the video unit: IBC mode, IBC-TM mode, IBC-MBVD mode.

[0835] Clause 65. A method according to clause 58, wherein the one or more syntax elements are indicated at one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header.

[0836] Clause 66. The method of clause 58, wherein the one or more syntax elements are encoded in a predictive manner.

[0837] Clause 67. The method of clause 58, wherein the one or more syntax elements of the video unit are predicted by one or more syntax elements of neighboring blocks.

[0838] Clause 68. The method of any one of clauses 1-67, wherein reconstruction reordering IBC (RR-IBC) or symmetric IBC method is used in the CIBCIP, or wherein the RR-IBC or symmetric IBC method is disabled in the CIBCIP.

[0839] Clause 69. The method of clause 1, wherein the flip type of the IBC prediction portion is set to NO_FLIP.

[0840] Clause 70. A method according to any one of clauses 1-69, wherein the video unit comprises at least one of the following: a color component, a prediction block (PB), a transform block (TB), a codec block (CB), a prediction unit PU, a transform unit (TU), a codec tree block (CTB), a codec unit (CU), a codec tree unit (CTU), a CTU row, a group of CTUs, a slice, a tile, a sub-picture, a block, a sub-region within a block, or a region containing more than one sample or pixel.

[0841] Clause 71. A method according to any one of clauses 1-69, wherein an indication of whether and / or how to derive the prediction of the video unit by combining the IBC prediction signal and the intra-frame prediction signal is indicated at one of: a sequence level, a picture group level, a picture level, a slice level, or a tile group level.

[0842] Clause 72. A method according to any one of clauses 1-69, wherein an indication of whether and / or how to derive the prediction of the video unit by combining the IBC prediction signal and the intra-frame prediction signal is indicated in one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header.

[0843] Clause 73. The method according to any one of clauses 1-69 further includes: determining whether and / or how to derive the prediction of the video unit by combining the IBC prediction signal and the intra-frame prediction signal based on at least one of the following: a message indicated in one of the following: DPS, SPS, VPS, PPS, APS, picture header, slice header, tile group header, maximum codec unit (LCU), codec unit (CU), LCU row, group of LCUs, TU, PU block, video codec unit, the position of one of the following: CU, PU, ​​TU, block, video codec unit, block dimensions of the current block and / or its neighboring blocks, block shape of the current block and / or its neighboring blocks, the coded mode of the video unit; indication of color format, codec tree structure, slice type, tile group type, picture type, color component, temporal layer identifier, standard profile or level or hierarchy.

[0844] Item 74. A video processing method, comprising: combining a plurality of reference lines for conversion between a video unit of a video and a bitstream of the video unit; deriving an intra-frame prediction of the video unit based on the combined plurality of reference lines; and performing the conversion based on the intra-frame prediction.

[0845] Clause 75. A method according to clause 74, wherein the number of reference lines and which reference lines are to be combined are one of: predefined, indicated in the bitstream, or dynamically derived, wherein the number of reference lines is an integer number greater than 1.

[0846] Clause 76. The method of clause 75, wherein the number of reference lines is predefined and / or wherein the number of reference lines is 2 or 3.

[0847] Clause 77. A method according to clause 75, wherein the number of reference lines is indicated in one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header.

[0848] Clause 78. The method of clause 75, wherein the number of reference lines is determined based on codec information.

[0849] Clause 79. The method according to clause 78, wherein the coding and decoding information includes one of the following: block size, block dimension, block position, coding and decoding mode; intra-frame prediction mode.

[0850] Clause 80. The method of clause 75, wherein which reference lines to use in the combination are indicated by a reference line index.

[0851] Clause 81. The method of clause 80, wherein the reference line index is predefined, or wherein the reference line index is indicated in the bitstream, or wherein the reference line index is dynamically derived.

[0852] Clause 82. The method of clause 75, wherein one of the reference line indices is indicated, or the other remaining reference line indices are indicated.

[0853] Clause 83. The method of clause 75, wherein one of the reference line indices is indicated and the other remaining reference line indices are predefined or dynamically derived.

[0854] Clause 84. The method of clause 74, wherein the plurality of reference lines are combined using a weight parameter.

[0855] Clause 85. A method according to clause 84, wherein the combined multiple reference lines are obtained by the following formula: L = W1*L1+W2*L2+...+WN-1*LN, wherein Li and Wi represent the i-th reference line and the corresponding weight used in the combination, and L represents the combined multiple reference lines for intra-frame prediction.

[0856] Clause 86. The method of clause 85, wherein L=(W'1*L1+W'2*L2+...+W'N-1*LN)>>Shift1, wherein W'1+W'2+...+W'N-1=2^Shift1.

[0857] Clause 87. The method of clause 84, wherein the weight parameters are predefined, or wherein the weight parameters are indicated in the bitstream, or wherein the weight parameters are dynamically derived.

[0858] Clause 88. The method of clause 84, wherein when reference line La is closer to the video unit than reference line Lb, the corresponding weight parameter Wa or W'a of the reference line La is equal to or greater than Wb or W'b of the reference line Lb.

[0859] Clause 89. The method of clause 85 or 86, wherein if N=2, then W1=3 / 4 and W2=1 / 4, or W1=5 / 8 and W2=3 / 8, or W1=1 / 2 and W2=1 / 2.

[0860] Clause 90. The method of clause 85 or 86, wherein if N=3, then W1=1 / 2 and W2=1 / 4 and W3=1 / 4, or W1=5 / 8 and W2=1 / 4 and W3=1 / 8.

[0861] Clause 91. The method of clause 74, wherein the number of samples in one reference line is the same as the number of samples in another reference line.

[0862] Clause 92. A method according to Clause 91, wherein the combination of samples is obtained as follows: P(x, y) = W1*P(x1, y1)1+W2*P(x2, y2)2+...+WN-1*P, (xN-1, yN-1)N-1, wherein P(xi, yi)i represents a sample in the i-th reference line, and Wi represents the weight parameter corresponding to the i-th reference line.

[0863] Clause 93. The method of clause 92, wherein when combining the upper portion of the reference lines, samples in different reference lines having the same horizontal position are combined.

[0864] Clause 94. The method of clause 93, wherein x1=x2=...=xN-1.

[0865] Clause 95. The method of clause 92, wherein when combining the left portion of the reference lines, samples in different reference lines having the same vertical position are combined.

[0866] Clause 96. The method according to clause 95, wherein y1=y2=...=yN-1.

[0867] Clause 97. The method of clause 74, wherein the number of samples in one reference line is different than the number of samples in another reference line.

[0868] Clause 98. The method of clause 97, wherein the number of samples in the combined plurality of reference lines is the same as the number of samples in the reference line having the least number of samples.

[0869] Item 99. A method according to Item 98, wherein the number of samples used in reference line Lm is greater than the number of samples used in reference line Ln, so as to combine the samples of the combined reference lines, wherein the total number of samples in the reference line Lm is greater than the total number of samples in the reference line Ln.

[0870] Clause 100. The method of clause 99, wherein two or more samples are used in the reference line Lm and one sample is used in the reference line Ln.

[0871] Clause 101. The method of clause 98, wherein Sn samples in the reference line Lm are used in combination with Sn samples in the reference line Ln, where Sn is an integer and represents the number of samples in the reference line Ln.

[0872] Clause 102. The method of clause 97, wherein the number of samples in the combined plurality of reference lines is the same as the number of samples of the reference line having the largest number of samples.

[0873] Clause 103. A method according to clause 102, wherein if the number of samples in reference line Ln is less than the number of samples in reference line Lm, the difference between the number of samples in reference line Ln and the number of samples in reference line Lm is filled or derived using the samples in reference line Ln and is used for the combination.

[0874] Clause 104. The method of clause 74, wherein the combining of the plurality of reference lines is performed after derivation of the plurality of reference lines, or wherein the combining of the plurality of reference lines is performed during derivation of the plurality of reference lines.

[0875] Clause 105. A method according to clause 74, wherein the derivation of the reference samples used for the combination in the multiple reference lines is the same as the derivation of the reference samples not used for the combination, or wherein the derivation of the reference samples used for the combination in the multiple reference lines is different from the derivation of the reference samples not used for the combination.

[0876] Clause 106. The method of clause 105, wherein unavailable reference samples are handled differently in the derivation of reference samples used for the combination in the plurality of reference lines and in the derivation of reference samples not used for the combination.

[0877] Clause 107. The method of clause 74, wherein reference sample filtering is performed after the combining of the plurality of reference lines, or wherein reference sample filtering is performed before the combining of the plurality of reference lines.

[0878] Clause 108. The method of clause 107, wherein the reference sample filtering is different for different reference lines.

[0879] Clause 109. A method according to clause 74, wherein whether to use the combined multiple reference lines to derive the intra-frame prediction of the video unit and / or the manner in which the combined multiple reference lines are used to derive the intra-frame prediction of the video unit depends on codec information.

[0880] Clause 110. The method of clause 109, wherein the codec information comprises one or more intra prediction methods.

[0881] Clause 111. A method according to clause 110, wherein the combined multiple reference lines are used in at least one of: conventional intra-frame prediction, intra-frame sub-partitioning (ISP), matrix-weighted intra-frame prediction (MIP), multiple reference line intra-frame prediction (MRL), template-based intra-frame mode derivation (TIMD), or decoder-side intra-frame mode derivation (DIMD), conventional chroma intra-frame prediction, a combination of chroma LM and angle, or a method in addition to or used to replace the current intra-frame prediction method.

[0882] Clause 112. The method of clause 109, wherein the codec information includes color components.

[0883] Clause 113. The method of clause 112, wherein the combined plurality of reference lines are used in intra-prediction of luma components, or wherein the combined plurality of reference lines are used in intra-prediction of chroma components.

[0884] Clause 114. The method of clause 109, wherein the codec information comprises an intra prediction mode.

[0885] Clause 115. A method according to clause 114, wherein the combined multiple reference lines are used when DC mode is used, or wherein the combined multiple reference lines are used when planar mode is used; or wherein the combined multiple reference lines are used when angular intra-frame prediction mode is used, or wherein the combined multiple reference lines are used when angular intra-frame prediction mode with a non-integer slope is used; or wherein the combined multiple reference lines are used for more than one intra-frame prediction mode.

[0886] Clause 116. A method according to clause 109, wherein the encoding information includes at least one of the following: a block size of the video unit; a block dimension of the video unit, a block size of a neighboring block, or a block dimension of a neighboring block.

[0887] Clause 117. A method according to Clause 116, wherein the combined multiple reference lines are used when the block size of the video unit is greater than or equal to a first threshold, or wherein the combined multiple reference lines are used when the block size of the video unit block is less than a second threshold.

[0888] Clause 118. The method of clause 109, wherein the encoding information comprises at least one of: a slice type, a temporal layer, or a quantization parameter (QP).

[0889] Clause 119. The method of clause 109, wherein the combined plurality of reference lines are not allowed to be used for video units in different CTUs.

[0890] Clause 120. A method according to clause 74, wherein the manner in which the multiple reference lines are combined is indicated in the bitstream and / or wherein whether the combined multiple reference lines are used to derive the intra-frame prediction of the video unit and / or the manner in which the combined multiple reference lines are used to derive the intra-frame prediction of the video unit is indicated in the bitstream.

[0891] Clause 121. A method according to any of clauses 74-120, wherein the video unit includes at least one of the following: a color component, a prediction block (PB), a transform block (TB), a codec block (CB), a prediction unit (PU), a transform unit (TU), a codec tree block (CTB), a codec unit (CU), a codec tree unit (CTU), a CTU row, a group of CTUs, a slice, a block, a sub-image, a block, a sub-region within a block, or a region containing more than one sample or pixel.

[0892] Clause 122. A method according to any of clauses 74-120, wherein an indication of whether and / or how to derive the intra-frame prediction of the video unit based on the combined multiple reference lines is indicated at one of: sequence level, picture group level, picture level, slice level, or tile group level.

[0893] Clause 123. A method according to any one of clauses 74-120, wherein an indication of whether and / or how the intra-frame prediction of the video unit is derived based on the combined multiple reference lines is indicated in one of the following: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependency parameter set (DPS), decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header.

[0894] Clause 124. The method according to any one of clauses 74-120 further includes: determining whether and / or how to derive the intra-frame prediction of the video unit based on the combined multiple reference lines based on at least one of the following: a message indicated in one of the following: DPS, SPS, VPS, PPS, APS, picture header, slice header, tile group header, maximum codec unit (LCU), codec unit (CU), LCU row, group of LCUs, TU, PU block, video codec unit, the position of one of the following: CU, PU, ​​TU, block, video codec unit, block dimensions of the current block and / or its neighboring blocks, block shape of the current block and / or its neighboring blocks, the coded mode of the video unit; indication of color format, coding tree structure, band type, tile group type, picture type, color component, temporal layer identifier, profile or level or level of the standard.

[0895] Clause 125. The method of any of clauses 1-124, wherein the converting comprises encoding the video unit into the bitstream.

[0896] Clause 126. The method of any of clauses 1-124, wherein the converting comprises decoding the video unit from the bitstream.

[0897] Clause 127. A video processing device comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any of clauses 1-126.

[0898] Clause 128. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method of any of clauses 1-126.

[0899] Item 129. A non-transitory computer-readable storage medium storing a bitstream of a video, the bitstream of the video being generated by a method performed by a video processing device, the method comprising: applying a combination of intra-block copy (IBC) and intra-frame prediction (CIBCIP) to a video unit of the video; deriving a prediction of the video unit by combining the IBC prediction signal and the intra-frame prediction signal; and generating the bitstream based on the prediction of the video unit.

[0900] Item 130. A method for storing a bitstream of a video, comprising: applying a combination of intra block copy (IBC) and intra prediction (CIBCIP) to a video unit of the video; deriving a prediction of the video unit by combining the IBC prediction signal and the intra prediction signal; generating the bitstream based on the prediction of the video unit; and storing the bitstream in a non-transitory computer-readable recording medium.

[0901] Item 131. A non-transitory computer-readable storage medium storing a bitstream of a video, wherein the bitstream of the video is generated by a method executed by a video processing device, the method comprising: combining a plurality of reference lines; deriving an intra-frame prediction of the video unit based on the combined plurality of reference lines; and generating the bitstream based on the prediction of the video unit.

[0902] Item 132. A method for storing a bitstream of a video, comprising: combining a plurality of reference lines; deriving an intra-frame prediction of the video unit based on the combined plurality of reference lines; generating the bitstream based on the prediction of the video unit; and storing the bitstream in a non-transitory computer-readable recording medium.

[0903] Example Device

[0904] Fig.51 A block diagram of a computing device 5100 in which various embodiments of the present disclosure may be implemented is shown. The computing device 5100 may be implemented as or included in the source device 110 (or video encoder 114 or 200) or the destination device 120 (or video decoder 124 or 300).

[0905] It should be understood that Fig.51 The computing device 5100 shown is for illustrative purposes only and does not in any way imply any limitation on the functionality and scope of the embodiments of the present disclosure.

[0906] like Fig.51As shown, computing device 5100 includes a general computing device 5100. Computing device 5100 may include at least one or more processors or processing units 5110, memory 5120, storage unit 5130, one or more communication units 5140, one or more input devices 5150, and one or more output devices 5160.

[0907] In some embodiments, the computing device 5100 can be implemented as any user terminal or server terminal with computing capabilities. The server terminal can be a server provided by a service provider, a large computing device, etc. The user terminal can be, for example, any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet computer, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It can be expected that the computing device 5100 can support any type of interface with the user (such as a "wearable" circuit, etc.).

[0908] The processing unit 5110 may be a physical or virtual processor and may implement various processes based on a program stored in the memory 5120. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the computing device 5100. The processing unit 5110 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0909] The computing device 5100 typically includes various computer storage media. Such media can be any media accessible to the computing device 5100, including but not limited to volatile and non-volatile media, or removable and non-removable media. The memory 5120 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), or flash memory), or any combination thereof. The storage unit 5130 can be any removable or non-removable medium, and can include machine-readable media, such as a memory, a flash drive, a disk, or other media, which can be used to store information and / or data and can be accessed in the computing device 5100.

[0910] The computing device 5100 may also include additional removable / non-removable, volatile / non-volatile storage media. Fig.51Not shown in the figure, but a magnetic disk drive for reading and / or writing a removable and nonvolatile magnetic disk and an optical disk drive for reading and / or writing a removable nonvolatile optical disk may be provided. In this case, each drive may be connected to a bus (not shown) via one or more data medium interfaces.

[0911] The communication unit 5140 communicates with another computing device via a communication medium. In addition, the functions of the components in the computing device 5100 can be implemented by a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 5100 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general-purpose network nodes.

[0912] The input device 5150 may be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. The output device 5160 may be one or more of various output devices, such as a display, a speaker, a printer, etc. Through the communication unit 5140, the computing device 5100 may also communicate with one or more external devices (not shown) such as storage devices and display devices, one or more of which enable a user to interact with the computing device 5100, or any device (such as a network card, a modem, etc.) that enables the computing device 5100 to communicate with one or more other computing devices if necessary. Such communication may be performed via an input / output (I / O) interface (not shown).

[0913] In some embodiments, some or all components of the computing device 5100 may also be arranged in a cloud computing architecture rather than being integrated in a single device. In a cloud computing architecture, each component can be provided remotely and work together to implement the functions described in the present disclosure. In some embodiments, cloud computing provides computing, software, data access and storage services, which does not require the end user to understand the physical location or configuration of the system or hardware that provides these services. In various embodiments, cloud computing uses a suitable protocol to provide services through a wide area network (such as the Internet). For example, a cloud computing provider provides applications through a wide area network, which can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data can be stored on a server at a remote location. The computing resources in a cloud computing environment can be merged or distributed at the location of a remote data center. Cloud computing infrastructure can provide services through a shared data center, although they act as a single access point for users. Therefore, cloud computing architecture can be used to provide components and functions described herein from a service provider at a remote location. Alternatively, they can be provided from a conventional server or directly installed or otherwise installed on a client device.

[0914] The computing device 5100 may be used to implement video encoding / decoding in embodiments of the present invention. The memory 5120 may include one or more video encoding / decoding modules 5125 having one or more program instructions. These modules may be accessed and executed by the processing unit 5110 to perform the functions of various embodiments described herein.

[0915] In an example embodiment performing video encoding and decoding, the input device 5150 may receive video data as input to be encoded 5170. For example, the video data may be processed by the video encoding and decoding module 5125 to generate an encoded bitstream. The encoded bitstream may be provided as output 5180 via the output device 5160.

[0916] In an example embodiment performing video decoding, input device 5150 may receive an encoded bitstream as input 5170. The encoded bitstream may be processed, for example, by video codec module 5125 to generate decoded video data. The decoded video data may be provided as output 5180 via output device 5160.

[0917] Although the present disclosure has been specifically shown and described with reference to its preferred embodiments, it will be appreciated by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present application, as defined by the appended claims. These changes are intended to be covered by the scope of the present application. Therefore, the foregoing description of the embodiments of the present application is not intended to be limiting.

Claims

1. A video processing method, comprising: For conversion between a video unit of a video and a bitstream of the video unit, applying a combination of intra block copy (IBC) and intra prediction (CIBIP) to the video unit; Derived a prediction of the video unit by combining the IBC prediction signal and the intra prediction signal; as well as Based on the prediction of the video unit, the conversion is performed.

2. The method of claim 1, wherein the prediction of the video unit is obtained by: P(x, y) = w ip1 * IP1(x, y) + w ip2 * IP2(x, y) +... + w ipn * IP n (x, y) + w ibc1 * IBC1(x, y) + w ibc2 * IBC2(x, y) +... + w ibcm * IBC m (x, y), and Where P(x, y) represents the prediction of the video unit, IP k represents the prediction generated by the k-th intra prediction, IBC j is the prediction generated by the jth IBC prediction, W ipk and W ibcj represents the corresponding weight value, and k and j are integers.

3. The method according to claim 1, wherein the intra prediction signal is generated by at least one of the following: Angular intra prediction mode, Direct current (DC) mode, Plane mode, Cross-Component Prediction (CCLM) mode, Multi-model CCLM mode, Left CCLM mode, or Go to CCLM mode.

4. The method according to claim 1, wherein the intra prediction mode of the intra prediction signal is encoded and decoded using one of the following items to indicate the intra prediction mode: Most Probable Model (MPM), Template-based intra mode derivation (TIMD), or Decoder-side intra mode derivation (DIMD).

5. The method of claim 1, wherein a set of intra prediction modes are allowed to be used in the CIBCIP. The method according to claim 1 , wherein an IBC merge mode is used in the CIBCIP.

7. The method of claim 6, wherein one or more block vector (BV) candidates in an IBC merge list are allowed to be used in the CIBIP.

8. The method of claim 6, wherein one or more BV offsets are used for the CIBCIP.

9. The method of claim 8, wherein the one or more BV offsets are added to the BV candidates before the BV candidates are used to obtain the IBC prediction signal.

10. The method of claim 8, wherein the one or more BV offsets are indicated or derived.

11. The method of claim 6, wherein the IBC merge mode comprises at least one of the following: Conventional IBC merge mode, IBC with Block Vector Difference (IBC-MBVD) merge mode, or IBC - Template Matching (TM) Merge Mode.

12. The method of claim 1, wherein an IBC Advanced Motion Vector Prediction (AMVP) mode is used in the CIBCIP.

13. The method of claim 12, wherein one or more BV predictors in the IBC AMVP list are allowed to be used in the CIBCIP.

14. The method of claim 12, wherein a block vector difference (BVD) used in the CIBCIP is indicated in the same manner as in the IBC mode.

15. The method of claim 12, wherein the BVD used in the CIBCIP is predefined.

16. The method of claim 12, wherein the BVD used in the CIBCIP is derived using codec information of the video unit.

17. The method of claim 1, wherein at least one of the following items for obtaining the IBC prediction signal is indicated: a merge index indicating a BV candidate in an IBC merge list, or a BVP index indicating a BV predictor in an IBC AMVP list.

18. The method according to claim 17, wherein the binarization or signaling manner of the merge index or the BVP index is the same as that in the IBC mode.

19. The method of claim 17, wherein at least one of the merge index or the BVP index is predefined.

20. The method of claim 19, wherein the merge index is 0 or 1, or Wherein the BVP index is 0 or 1.

21. The method of claim 17, wherein at least one of the merge index or the BVP index is derived using codec information of the video unit.

22. The method of claim 17, wherein at least one of the merge index or the BVP index is derived using template matching.

23. The method of claim 1, wherein the construction of the IBC Merge List or IBC AMVP List used in the CIBIP is the same as that used in the IBC Mode, or The construction of the IBC merge list or IBC AMVP list used in the CIBIP is different from that used in the IBC mode.

24. The method of claim 1, wherein a first number of BV candidates for the CIBCIP in an IBC merge list is less than or equal to the number of BV candidates for IBC in the IBC merge list, and wherein the first number is an integer greater than 0 and less than or equal to the second number, and / or Wherein a first number of BV candidates for the CIBCIP in the AMVP list is less than or equal to the number of BV candidates for IBC in the AMVP list, and wherein the first number is an integer greater than 0 and less than or equal to the second number.

25. The method of claim 24, wherein the first number is one of: 1, 2, 3, 4, 5, or 6.

26. The method of claim 24, wherein the first N BV candidates of the IBC merge list are used for the CIBCIP, and / or Wherein the top N BV candidates of the AMVP list are used for the CIBCIP, and wherein N is an integer.

27. The method of claim 1, wherein template matching is used to derive / refine the BV used to obtain the IBC prediction signal.

28. The method of claim 27, wherein a BV offset is derived using template matching, the BV offset being added to the BV candidate in the IBC merge list.

29. The method of claim 27, wherein the block vector difference (BVD) is derived using a template matching-based approach.

30. The method of claim 27, wherein the sign of the BVD is derived using a template matching based approach.

31. The method of claim 27, wherein an intra prediction mode or an intra prediction method for obtaining the intra prediction signal is used in the template matching to derive / refine the BV.

32. The method of claim 1, wherein a BV list is reordered prior to use in the CIBCIP.

33. The method of claim 32, wherein template matching or bilateral matching cost is used for reordering of the BV list.

34. The method of claim 32, wherein during construction of the BV list for the CIBCIP, template matching or bilateral matching is used.

35. The method of claim 32, wherein the BV list comprises an IBC consolidated list or an IBC AMVP list.

36. The method of claim 32, wherein the BV list for the CIBCIP is reordered in the same manner as for the IBC mode, or The reordering method of the BV list for the CIBIP is different from the reordering method for the IBC mode.

37. The method of claim 36, wherein the number of BV candidates in the BV list for reordering for the CIBCIP is less than or equal to the number of BV candidates for reordering for IBC mode.

38. The method of claim 37, wherein if IBC merge mode is used for the CIBCIP, the number of BV candidates is one of 1, 2, 3 or 4, or Wherein if IBC AMVP is used for the CIBCIP, the number of BV candidates is one of 1, 2 or 3.

39. The method according to claim 1, wherein the intra prediction comprises a conventional intra prediction mode or another intra prediction mode, wherein the other intra prediction mode obtains a prediction block by using samples in the following items excluding IBC: Current strip, Current tile, Current sub-image, The current image, or Other video units.

40. The method of claim 39, wherein the intra prediction signal is obtained using one of the following: one or more predefined intra prediction modes, One or more of the MPMs, an intra prediction mode derived using a block vector used to obtain said IBC prediction signal, One or more intra prediction modes derived using a template-based approach, One or more intra prediction modes derived using neighboring samples or gradients of neighboring samples, Intra-frame sub-partitioning (ISP), Matrix Weighted Intra Prediction (MIP), Multiple reference line intra prediction (MRL), or Intra template matching prediction (Intra TMP).

41. The method of claim 40, wherein the one or more predefined intra prediction modes include at least one of the following: Plane mode, DC mode, Horizontal mode, or Vertical mode.

42. The method of claim 1, wherein weight parameters for combining the IBC prediction signal and the intra prediction signal are indicated or derived.

43. The method of claim 42, wherein a set of weight parameters is constructed and an index indicating the weight parameters is indicated.

44. The method of claim 42, wherein the weight parameters are derived using the codec information.

45. The method of claim 44, wherein the codec information comprises a codec mode of a neighboring unit.

46. ​​The method of claim 45, wherein the weight parameter depends on whether one or more neighboring units are coded using intra prediction or IBC mode.

47. The method of claim 44, wherein the codec information includes an intra prediction mode used to obtain the intra prediction signal.

48. The method of claim 44, wherein the codec information comprises at least one of: a block size or a block dimension of the video unit, or a block size or a block dimension of a neighboring video unit.

49. The method of claim 42, wherein the weight parameters are derived using a template matching approach.

50. The method of claim 42, wherein the weight parameters are predefined.

51. The method of claim 1, wherein the reference area of ​​the CIBCIP is less than or equal to the reference area of ​​the IBC.

52. The method of claim 51, wherein the reference area of ​​the CIBCIP depends on codec information of intra prediction.

53. The method of claim 51, wherein the reference region of the CIBCIP depends on an intra prediction mode.

54. The method of claim 51, wherein the reference region of the CIBCIP is different from the reference region of the IBC.

55. The method according to claim 1, wherein whether to apply the CIBCIP to the video unit and / or the manner in which the CIBCIP is applied to the video unit depends on codec information, wherein the codec information comprises at least one of the following: Whether IBC or intra prediction is allowed, block dimensions and / or block size, Block depth, Strip type, Image type, Partition tree type, Time layer identifier, block location, or Color components.

56. The method of claim 1, wherein the indication of the CIBCIP is derived dynamically.

57. The method of claim 1, wherein the indication of the CIBCIP is conditional, wherein the condition comprises at least one of: Whether IBC or intra prediction is allowed, block dimensions and / or block size, Block depth, Strip type, Image type, Partition tree type, Time layer identifier, block location, or Color components.

58. The method of claim 1, wherein whether the video unit is encoded and decoded using the CIBCIP is indicated using one or more syntax elements.

59. The method of claim 58, wherein the one or more syntax elements are binarized using one of: Fixed length codec, Truncated unary codec, Unary codec, EG codec, or The encoded flag.

60. The method of claim 58, wherein the one or more syntax elements are bypass coded or context coded.

61. The method of claim 60, wherein the context depends on coded information, and The encoded information includes at least one of the following: Block dimensions, Block size, Strip type, Image type, Information about neighboring blocks, Information about other codecs for the video unit, or Time layer information.

62. The method of claim 58, wherein the indication of the CIBCIP is indicated when the video unit is coded by IBC.

63. The method of claim 58, wherein the one or more syntax elements are indicated before an indication of an IBC-TM mode or an IBC-MBVD mode, or the one or more syntax elements are indicated after an indication of an IBC-TM mode or an IBC-MBVD mode.

64. The method of claim 63, wherein whether to indicate the one or more syntax elements and / or the manner in which the one or more syntax elements are indicated depends on whether at least one of the following is enabled for the video unit: IBC mode, IBC-TM mode, IBC-MBVD mode.

65. The method of claim 58, wherein the one or more syntax elements are indicated at one of: Sequence header, Picture header, Sequence Parameter Set (SPS), Video Parameter Set (VPS), Dependent Parameter Set (DPS), Decoding Capability Information (DCI), Picture Parameter Set (PPS), Adaptation Parameter Set (APS), Strip header, or Tile group header.

66. The method of claim 58, wherein the one or more syntax elements are encoded in a predictive manner.

67. The method of claim 58, wherein the one or more syntax elements of the video unit are predicted by one or more syntax elements of neighboring blocks.

68. The method according to any one of claims 1 to 67, wherein in the CIBCIP a reconstruction reordered IBC (RR-IBC) or a symmetric IBC method is used, or The RR-IBC or symmetric IBC method is disabled in the CIBCIP.

69. The method of claim 1, wherein the flip type of the IBC prediction portion is set to NO_FLIP.

70. The method of any one of claims 1-69, wherein the video unit comprises at least one of: Color components, Prediction Block (PB), Transform Block (TB), Codec Block (CB), Prediction unit PU, Transformation Unit (TU), Codec Tree Block (CTB), Codec Unit (CU), Codec Tree Unit (CTU), CTU line, The CTU group, Strips, Tile, Sub-picture, piece, a sub-region within a block, or An area that contains more than one sample or pixel.

71. The method of any of claims 1-69, wherein an indication of whether and / or how to derive the prediction for the video unit by combining the IBC prediction signal and the intra prediction signal is indicated at one of: Sequence level, Picture group level, Picture level, Stripe level, or Tile group level.

72. The method of any one of claims 1-69, wherein an indication of whether and / or how to derive the prediction for the video unit by combining the IBC prediction signal and the intra prediction signal is indicated in one of: Sequence header, Picture header, Sequence Parameter Set (SPS), Video Parameter Set (VPS), Dependent Parameter Set (DPS), Decoding Capability Information (DCI), Picture Parameter Set (PPS), Adaptation Parameter Set (APS), Strip header, or Tile group header.

73. The method of any one of claims 1-69, further comprising: determining whether and / or how to derive the prediction of the video unit by combining the IBC prediction signal and the intra prediction signal based on at least one of the following: A message indicated in one of the following: DPS, SPS, VPS, PPS, APS, picture header, slice header, tile group header, largest codec unit (LCU), codec unit (CU), LCU row, group of LCUs, TU, PU block, video codec unit, The location of one of the following: CU, PU, ​​TU, block, video codec unit, the block dimensions of the current block and / or its neighboring blocks, the block shapes of the current block and / or its neighboring blocks, a coded mode of the video unit; Instructions for color format, Codec tree structure Strip type, Tile group type, Image type, Color components, Time layer identifier, A profile or level or hierarchy of standards.

74. A video processing method, comprising: Combining a plurality of reference lines for conversion between a video unit of a video and a bitstream of the video unit; deriving an intra prediction of the video unit based on the combined plurality of reference lines; as well as The conversion is performed based on the intra prediction.

75. A method according to claim 74, wherein the number of reference lines and which reference lines are to be combined are one of: predefined, indicated in the bitstream, or dynamically derived, wherein the number of reference lines is an integer number greater than 1.

76. The method of claim 75, wherein the number of reference lines is predefined, and / or The number of the reference lines is 2 or 3.

77. The method of claim 75, wherein the number of reference lines is indicated in one of: Sequence header, Picture header, Sequence Parameter Set (SPS), Video Parameter Set (VPS), Dependent Parameter Set (DPS), Decoding Capability Information (DCI), Picture Parameter Set (PPS), Adaptation Parameter Set (APS), Strip header, or Tile group header.

78. The method of claim 75, wherein the number of reference lines is determined based on codec information.

79. The method of claim 78, wherein the codec information comprises one of the following: Block size, Block dimensions, Block location, Codec mode; Intra prediction mode.

80. The method of claim 75, wherein which reference lines to use in the combination are indicated by a reference line index.

81. The method of claim 80, wherein the reference line index is predefined, or wherein the reference line index is indicated in the bitstream, or The reference line index is derived dynamically.

82. The method of claim 75, wherein one of the reference line indexes is indicated, or the other remaining reference line indexes are indicated.

83. The method of claim 75, wherein one of the reference line indices is indicated and the other remaining reference line indices are predefined or dynamically derived.

84. The method of claim 74, wherein the plurality of reference lines are combined using a weight parameter.

85. The method of claim 84, wherein the combined plurality of reference lines is obtained by: L=W1*L1+W2*L2+...+WN-1*LN, where Li and Wi represent the i-th reference line used in the combination and the corresponding weight, and L represents the combined plurality of reference lines for intra prediction.

86. The method of claim 85, wherein L = (W'1*L1+W'2*L2+...+W'N-1*LN)>>Shift1, wherein W'1+W'2+...+W'N-1=2^Shift1.

87. The method of claim 84, wherein the weight parameters are predefined, or wherein the weight parameter is indicated in the bitstream, or The weight parameters are derived dynamically.

88. The method of claim 84, wherein when reference line La is closer to the video unit than reference line Lb, the corresponding weight parameter Wa or W'a of the reference line La is equal to or greater than Wb or W'b of the reference line Lb.

89. The method of claim 85 or 86, wherein if N=2, then W1=3 / 4 and W2=1 / 4, or W1=5 / 8 and W2=3 / 8, or W1=1 / 2 and W2=1 / 2.

90. The method of claim 85 or 86, wherein if N=3, then W1=1 / 2 and W2=1 / 4 and W3=1 / 4, or W1=5 / 8 and W2=1 / 4 and W3=1 / 8.

91. The method of claim 74, wherein the number of samples in one reference line is the same as the number of samples in another reference line.

92. The method of claim 91, wherein the combination for the sample is obtained as follows: P(x,y)=W1*P(x1,y1)1+W2*P(x2,y2)2+...+WN-1*P,(xN-1,yN-1)N-1,where P(xi,yi)i represents a sample in the i-th reference line, and Wi represents the weight parameter corresponding to the i-th reference line.

93. The method of claim 92, wherein when combining upper portions of the reference lines, samples in different reference lines having the same horizontal position are combined.

94. The method of claim 93, wherein x1 = x2 = ... = xN-1.

95. The method of claim 92, wherein when combining left portions of the reference lines, samples in different reference lines having the same vertical position are combined.

96. A method according to claim 95, wherein y1 = y2 = ... = yN-1.

97. The method of claim 74, wherein the number of samples in one reference line is different than the number of samples in another reference line.

98. The method of claim 97, wherein the number of samples in the plurality of reference lines that are combined is the same as the number of samples in the reference line having the least number of samples.

99. A method according to claim 98, wherein the number of samples used in reference line Lm is greater than the number of samples used in reference line Ln, so as to combine the samples of the combined reference lines, wherein the total number of samples in the reference line Lm is greater than the total number of samples in the reference line Ln.

100. The method of claim 99, wherein two or more samples are used in the reference line Lm and one sample is used in the reference line Ln.

101. The method of claim 98, wherein Sn samples in the reference line Lm are used in combination with Sn samples in the reference line Ln, wherein Sn is an integer and represents the number of samples in the reference line Ln.

102. The method of claim 97, wherein the number of samples in the combined plurality of reference lines is the same as the number of samples of a reference line having a maximum number of samples.

103. A method according to claim 102, wherein if the number of samples in the reference line Ln is less than the number of samples in the reference line Lm, the difference between the number of samples in the reference line Ln and the number of samples in the reference line Lm is filled or derived using the samples in the reference line Ln and is used for the combination.

104. The method of claim 74, wherein the combining of the plurality of reference lines is performed after the derivation of the plurality of reference lines, or wherein said combining of said plurality of reference lines is performed during derivation of said plurality of reference lines.

105. The method of claim 74, wherein the derivation of reference samples used in the combination of the plurality of reference lines is the same as the derivation of reference samples not used in the combination, or The derivation of the reference samples used for the combination among the plurality of reference lines is different from the derivation of the reference samples not used for the combination.

106. The method of claim 105, wherein unavailable reference samples are handled differently in the derivation of reference samples used for the combination in the plurality of reference lines and in the derivation of reference samples not used for the combination.

107. The method of claim 74, wherein after said combining of said plurality of reference lines, reference sample filtering is performed, or Wherein before the combining of the plurality of reference lines, the reference sample filtering is performed.

108. The method of claim 107, wherein the reference sample filtering is different for different reference lines.

109. The method of claim 74, wherein whether to use the combined multiple reference lines to derive the intra-frame prediction of the video unit and / or the manner in which the combined multiple reference lines are used to derive the intra-frame prediction of the video unit depends on codec information.

110. The method of claim 109, wherein the codec information includes one or more intra prediction methods.

111. The method of claim 110, wherein the combined plurality of reference lines are used in at least one of: Conventional intra prediction, Intra-frame sub-partitioning (ISP), Matrix Weighted Intra Prediction (MIP), Multiple reference line intra prediction (MRL), Template-based intra mode derivation (TIMD), or Decoder-side intra mode derivation (DIMD), Conventional chroma intra prediction, A combination of LM and angle of chromaticity, or A method that is in addition to or used to replace the current intra prediction method.

112. The method of claim 109, wherein the codec information includes color components.

113. The method of claim 112, wherein the combined plurality of reference lines are used in intra prediction of a luma component, or The combined plurality of reference lines are used in intra prediction of chrominance components.

114. The method of claim 109, wherein the codec information comprises an intra-prediction mode.

115. The method of claim 114, wherein when DC mode is used, the combined plurality of reference lines are used, or wherein when the planar mode is used, the combined plurality of reference lines are used; or wherein when the angular intra prediction mode is used, the combined plurality of reference lines are used, or wherein when an angular intra prediction mode with a non-integer slope is used, the combined plurality of reference lines are used; or The combined plurality of reference lines are used for more than one intra prediction mode.

116. The method of claim 109, wherein the coded information comprises at least one of: a block size of the video unit; the block dimensions of the video unit, the block size of the neighboring block, or Block dimensions of neighboring blocks.

117. The method of claim 116, wherein when the block size of the video unit is greater than or equal to a first threshold, the combined plurality of reference lines are used, or Wherein when the block size of the video unit block is smaller than a second threshold, the combined plurality of reference lines are used.

118. The method of claim 109, wherein the coded information comprises at least one of: Strip type, Time layer, or Quantization parameter (QP).

119. The method of claim 109, wherein the combined multiple reference lines are not allowed for video units in different CTUs.

120. The method of claim 74, wherein the manner in which the plurality of reference lines are combined is indicated in the bitstream, and / or The manner in which whether to use the combined plurality of reference lines to derive the intra-prediction of the video unit and / or using the combined plurality of reference lines to derive the intra-prediction of the video unit is indicated in a bitstream.

121. The method of any one of claims 74-120, wherein the video unit comprises at least one of: Color components, Prediction Block (PB), Transform Block (TB), Codec Block (CB), Prediction Unit (PU), Transformation Unit (TU), Codec Tree Block (CTB), Codec Unit (CU), Codec Tree Unit (CTU), CTU line, The CTU group, Strips, Tile, sub-image, piece, a sub-region within a block, or An area that contains more than one sample or pixel.

122. The method of any of claims 74-120, wherein an indication of whether and / or how to derive the intra-prediction of the video unit based on the combined plurality of reference lines is indicated at one of: Sequence level, Picture group level, Picture level, Stripe level, or Tile group level.

123. The method of any of claims 74-120, wherein an indication of whether and / or how to derive the intra prediction for the video unit based on the combined plurality of reference lines is indicated in one of: Sequence header, Picture header, Sequence Parameter Set (SPS), Video Parameter Set (VPS), Dependent Parameter Set (DPS), Decoding Capability Information (DCI), Picture Parameter Set (PPS), Adaptation Parameter Set (APS), Strip header, or Tile group header.

124. The method of any one of claims 74-120, further comprising: determining whether and / or how to derive the intra prediction for the video unit based on the combined plurality of reference lines based on at least one of: A message indicated in one of the following: DPS, SPS, VPS, PPS, APS, picture header, slice header, tile group header, largest codec unit (LCU), codec unit (CU), LCU row, group of LCUs, TU, PU block, video codec unit, The location of one of the following: CU, PU, ​​TU, block, video codec unit, the block dimensions of the current block and / or its neighboring blocks, the block shapes of the current block and / or its neighboring blocks, a coded mode of the video unit; Instructions for color format, Encoding tree structure Strip type, Tile group type, Image type, Color components, Time layer identifier, A profile or level or hierarchy of standards.

125. The method of any one of claims 1-124, wherein the converting comprises encoding the video unit into the bitstream.

126. The method of any one of claims 1-124, wherein the converting comprises decoding the video unit from the bitstream.

127. A video processing device comprising a processor and a non-volatile memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform a method according to any one of claims 1-126.

128. A non-transitory computer-readable storage medium storing instructions that cause a processor to execute the method according to any one of claims 1-126.

129. A non-transitory computer-readable storage medium storing a bitstream of a video, the bitstream of the video being generated by a method performed by a video processing device, the method comprising: applying a combination of intra block copy (IBC) and intra prediction (CIBCIP) to video units of the video; deriving a prediction of the video unit by combining an IBC prediction signal and an intra prediction signal; as well as The bitstream is generated based on the prediction of the video unit.

130. A method for storing a bit stream of a video, comprising: applying a combination of intra block copy (IBC) and intra prediction (CIBCIP) to video units of the video; deriving a prediction of the video unit by combining an IBC prediction signal and an intra prediction signal; generating the bitstream based on the prediction of the video unit; as well as The bit stream is stored in a non-transitory computer-readable recording medium.

131. A non-transitory computer-readable storage medium storing a bitstream of a video, the bitstream of the video being generated by a method performed by a video processing device, the method comprising: Combine multiple reference lines; deriving an intra prediction of the video unit based on the combined plurality of reference lines; as well as The bitstream is generated based on the prediction of the video unit.

132. A method for storing a bit stream of a video, comprising: Combine multiple reference lines; deriving an intra prediction of the video unit based on the combined plurality of reference lines; generating the bitstream based on the prediction of the video unit; as well as The bit stream is stored in a non-transitory computer-readable recording medium.

Images