Method and device for video processing and medium

Abstract

Embodiments of the present disclosure provide a solution for video processing. A method for video processing is presented. The method comprises: obtaining a plurality of prediction signals of a video unit of a video using at least one of: a plurality of prediction modes or a plurality of prediction tools, where the plurality of prediction modes comprise two or more prediction modes, and the plurality of prediction coding and decoding tools comprise two or more prediction coding and decoding tools; obtaining at least one of: an output prediction signal for the video unit, a final prediction signal, or a reconstruction by fusing the plurality of prediction signals; and performing the conversion based on the prediction or reconstruction of the video unit.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to video processing techniques, and more specifically, to fusion for intra-frame template matching prediction. Background Technology

[0002] Today, digital video capabilities are being applied to all aspects of people's lives. Various video compression technologies have been proposed for video encoding / decoding, such as MPEG-2, MPEG-4, ITU-TH.263, ITU-TH.264 / MPEG-4 Part 10 Advanced Video Codec (AVC), ITU-TH.265 High Efficiency Video Codec (HEVC) standard, and Multi-Functional Video Codec (VVC) standard. However, the encoding and decoding efficiency of video encoding and decoding technologies is generally expected to be further improved. Summary of the Invention

[0003] Embodiments of this disclosure provide a solution for video processing.

[0004] In a first aspect, a method for video processing is proposed. The method includes: conversion between video units and bitstreams of video units; obtaining multiple prediction signals for the video units using at least one of the following: multiple prediction modes or multiple prediction tools, wherein the multiple prediction modes include two or more prediction modes, and the multiple prediction codec tools include two or more prediction codec tools; obtaining at least one of the following by fusing the multiple prediction signals: an output prediction signal, a final prediction signal, or a reconstruction for the video unit; and performing a conversion based on at least one of the following: the output prediction signal, the final prediction signal, or the reconstruction. In this manner, when the prediction signal used for fusion is obtained from at least two prediction modes, encoding / decoding performance and efficiency are improved.

[0005] In a second aspect, an apparatus for video processing is provided. The apparatus includes a processor and a non-transitory memory having instructions thereon. When executed by the processor, the instructions cause the processor to perform the method according to the first aspect of this disclosure.

[0006] In a third aspect, a non-transitory computer-readable storage medium is proposed. This non-transitory computer-readable storage medium stores instructions that cause a processor to execute the method according to the first aspect of this disclosure.

[0007] In a fourth aspect, another non-transitory computer-readable recording medium is proposed. This non-transitory computer-readable recording medium stores a bitstream of video generated by a method performed by an apparatus for video processing. The method includes: obtaining multiple prediction signals for video units of the video using at least one of the following: multiple prediction modes or multiple prediction tools, wherein the multiple prediction modes include two or more prediction modes, and the multiple prediction encoding / decoding tools include two or more prediction encoding / decoding tools; obtaining at least one of the following by fusing the multiple prediction signals: an output prediction signal, a final prediction signal, or a reconstruction for the video units; and generating a bitstream based on at least one of the following: the output prediction signal, the final prediction signal, or the reconstruction.

[0008] In a fifth aspect, a method for storing a bitstream of video is proposed. The method includes: obtaining multiple prediction signals for video units of the video using at least one of the following: multiple prediction modes or multiple prediction tools, wherein the multiple prediction modes include two or more prediction modes, and the multiple prediction codec tools include two or more prediction codec tools; obtaining at least one of the following by fusing the multiple prediction signals: an output prediction signal, a final prediction signal, or a reconstruction for the video unit; generating a bitstream based on at least one of the following: the output prediction signal, the final prediction signal, or the reconstruction; and storing the bitstream in a non-transitory computer-readable recording medium.

[0009] This synopsis aims to present, in a simplified form, the concept choices further described below in the detailed embodiments. This synopsis is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0010] The above and other objects, features, and advantages of exemplary embodiments of the present disclosure will become more apparent from the following detailed description with reference to the accompanying drawings. In the exemplary embodiments of the present disclosure, the same reference numerals generally refer to the same components.

[0011] Figure 1 A block diagram illustrating an example video codec system according to some embodiments of the present disclosure is shown;

[0012] Figure 2 A block diagram illustrating a first example video encoder according to some embodiments of the present disclosure is shown;

[0013] Figure 3 A block diagram illustrating an example video decoder according to some embodiments of the present disclosure is shown;

[0014] Figure 4 An example of an encoder block diagram is shown;

[0015] Figure 567 intra-frame prediction modes are shown;

[0016] Figure 6 Reference samples for wide-angle intra-frame prediction are shown;

[0017] Figure 7 This illustrates the problem of discontinuities when the orientation exceeds 45°;

[0018] Figure 8 The MMVD search point is shown;

[0019] Figure 9 This is a schematic diagram used for symmetric MVD mode;

[0020] Figure 10 The extended CU region used in BDOF is shown;

[0021] Figure 11 An affine motion model based on control points is shown;

[0022] Figure 12 The affine MVF for each sub-block is shown;

[0023] Figure 13 The location of the inherited affine motion prediction value is shown;

[0024] Figure 14 This demonstrates the inheritance of control point motion vectors;

[0025] Figure 15 The locations of candidate positions for the constructed affine Merge pattern are shown;

[0026] Figure 16 This is a schematic diagram illustrating the use of motion vectors in the proposed combination method;

[0027] Figure 17 The sub-block MV VSB and pixel Δv(i,j) are shown;

[0028] Figure 18A This shows the spatial neighboring block used by ATVMP;

[0029] Figure 18B The paper demonstrates how to derive the motion field of a sub-CU by applying motion displacements from spatial neighbors and scaling motion information from the corresponding co-located sub-CUs.

[0030] Figure 19 Position lighting compensation is shown;

[0031] Figure 20 This indicates that no downsampling was performed on the short side;

[0032] Figure 21 This shows the refinement of motion vectors on the decoding side;

[0033] Figure 22 The diamond-shaped area in the search region is shown;

[0034] Figure 23 The locations of the spatial merge candidates are shown;

[0035] Figure 24 The candidate pairs considered for redundancy checks of spatial merge candidates are shown.

[0036] Figure 25 This is a schematic diagram of motion vector scaling for temporal merge candidates;

[0037] Figure 26 The candidate positions, C0 and C1, for the time-domain Merge candidate are shown;

[0038] Figure 27 The VVC spatial neighboring blocks of the current block are shown;

[0039] Figure 28 This is a schematic diagram of the virtual block in the i-th round of search;

[0040] Figure 29 An example of GPM partitioning grouped at the same angle is shown;

[0041] Figure 30 The unidirectional prediction MV selection for geometric segmentation patterns is shown;

[0042] Figure 31 An exemplary generation of a hybrid weighted w0 using a geometric segmentation pattern is shown;

[0043] Figure 32 The spatial neighboring blocks used to derive spatial merge candidates are shown;

[0044] Figure 33 This demonstrates the execution of template matching over the search area surrounding the initial MV;

[0045] Figure 34 This is a schematic diagram of a sub-block used in OBMC applications;

[0046] Figure 35 The location, type, and transformation type of the SBT are shown;

[0047] Figure 36 A flowchart of a method for video processing according to embodiments of the present disclosure is shown; and

[0048] Figure 37 A block diagram of a computing device in which various embodiments of the present disclosure may be implemented is shown.

[0049] Throughout all the accompanying figures, the same or similar reference numerals generally refer to the same or similar elements. Detailed Implementation

[0050] The principles of this disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are described for illustrative purposes only and to help those skilled in the art understand and implement this disclosure, and do not imply any limitation on the scope of this disclosure. In addition to the methods described below, the disclosure described herein can be implemented in various other ways.

[0051] In the following description and claims, unless otherwise defined, all scientific and technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0052] The terms "an embodiment," "embodiment," "example embodiment," etc., used in this disclosure refer to embodiments that may include specific features, structures, or characteristics, but not every embodiment is required to include that specific feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Moreover, when a specific feature, structure, or characteristic is described in conjunction with an example embodiment, it is claimed that, whether explicitly described or not, such a feature, structure, or characteristic affecting its relation to other embodiments is within the knowledge of those skilled in the art.

[0053] It should be understood that although the terms “first” and “second”, etc., may be used herein to describe various elements, these elements should not be limited to these terms. These terms are used only to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the listed terms.

[0054] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” “having,” “containing,” and / or “comprising” as used herein indicate the presence of the said features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof. Example Environment

[0055] Figure 1This is a block diagram illustrating an example video encoding / decoding system 100 from which the techniques of this disclosure may be utilized. As shown, the video encoding / decoding system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a video encoding device, and the destination device 120 may also be referred to as a video decoding device. In operation, the source device 110 may be configured to generate encoded video data, and the destination device 120 may be configured to decode the encoded video data generated by the source device 110. The source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0056] Video source 112 may include sources such as video capture devices. Examples of video capture devices include, but are not limited to, interfaces for receiving video data from video content providers, computer graphics systems for generating video data, and / or combinations thereof.

[0057] Video data may include one or more images. Video encoder 114 encodes the video data from video source 112 to generate a bitstream. The bitstream may include a sequence of bits forming an encoded representation of the video data. The bitstream may include encoded images and associated data. An encoded image is an encoded representation of an image. Associated data may include sequence parameter sets, image parameter sets, and other syntax structures. I / O interface 116 may include a modulator / demodulator and / or a transmitter. Encoded video data can be directly transmitted to destination device 120 via network 130A through I / O interface 116. Encoded video data may also be stored on storage medium / server 130B for access by destination device 120.

[0058] The destination device 120 may include an I / O interface 126, a video decoder 124, and a display device 122. The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may acquire encoded video data from the source device 110 or the storage medium / server 130B. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to a user. The display device 122 may be integrated with the destination device 120, or it may be external to the destination device 120, which is configured to interface with an external display device.

[0059] The video encoder 114 and the video decoder 124 can operate according to video compression standards, such as the High Efficiency Video Codec (HEVC) standard, the Multi-Functional Video Codec (VVC) standard, and other existing and / or future standards.

[0060] Figure 2This is a block diagram illustrating an example of a video encoder 200 according to some embodiments of the present disclosure. The video encoder 200 may be... Figure 1 An example of a video encoder 114 in system 100 is shown.

[0061] The video encoder 200 can be configured to implement any or all of the technologies disclosed herein. Figure 2 In the example, the video encoder 200 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video encoder 200. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0062] In some embodiments, the video encoder 200 may include a segmentation unit 201, a prediction unit 202, a residual generation unit 207, a transform unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transform unit 211, a reconstruction unit 212, a buffer 213, and an entropy coding unit 214. The prediction unit 202 may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205, and an intra-frame prediction unit 206.

[0063] In other examples, the video encoder 200 may include more, fewer, or different functional components. In one example, the prediction unit 202 may include an intra-block copy (IBC) unit. The IBC unit can perform prediction in an IBC mode, in which at least one reference picture is the picture in which the current video block is located.

[0064] Furthermore, although some components (such as motion estimation unit 204 and motion compensation unit 205) can be integrated, for interpretable purposes, these components are... Figure 2 The examples are shown separately.

[0065] The segmentation unit 201 can segment an image into one or more video blocks. The video encoder 200 and the video decoder 300 can support various video block sizes.

[0066] The mode selection unit 203 can, for example, select one of several coding modes (intra-coding or inter-coding) based on the error result, and provide the resulting intra-coded or inter-coded block to the residual generation unit 207 to generate residual block data, and to the reconstruction unit 212 to reconstruct the coded block for use as a reference image. In some examples, the mode selection unit 203 can select an intra-inter-prediction joint prediction (CIIP) mode, in which prediction is based on inter-prediction signals and intra-prediction signals. In the case of inter-prediction, the mode selection unit 203 can also select a resolution for the block based on the motion vector (e.g., sub-pixel precision or integer pixel precision).

[0067] To perform inter-frame prediction on the current video block, motion estimation unit 204 can generate motion information for the current video block by comparing one or more reference frames from buffer 213 with the current video block. Motion compensation unit 205 can determine the predicted video block for the current video block based on the motion information and decoded samples of images from buffer 213 other than the image associated with the current video block.

[0068] The motion estimation unit 204 and the motion compensation unit 205 can perform different operations on the current video block, for example, depending on whether the current video block is in an I-strip, P-strip, or B-strip. As used herein, an "I-strip" can refer to a portion of an image composed of macroblocks, all of which are based on macroblocks within the same image. Furthermore, as used herein, in some aspects, "P-strip" and "B-strip" can refer to portions of an image composed of macroblocks independent of macroblocks within the same image.

[0069] In some examples, motion estimation unit 204 can perform unidirectional prediction on the current video block, and can search reference images in list 0 or list 1 to find a reference video block for the current video block. Motion estimation unit 204 can then generate a reference index indicating the reference image containing the reference video block in list 0 or list 1, and a motion vector indicating the spatial displacement between the current video block and the reference video block. Motion estimation unit 204 can output the reference index, prediction direction indicator, and motion vector as motion information for the current video block. Motion compensation unit 205 can generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0070] Alternatively, in other examples, motion estimation unit 204 can perform bidirectional prediction on the current video block. Motion estimation unit 204 can search reference images in list 0 to find a reference video block for the current video block, and can also search reference images in list 1 to find another reference video block for the current video block. Motion estimation unit 204 can then generate multiple reference indices indicating the reference images containing multiple reference video blocks in lists 0 and 1, and motion vectors indicating multiple spatial displacements between the multiple reference video blocks and the current video block. Motion estimation unit 204 can output the multiple reference indices and multiple motion vectors of the current video block as motion information for the current video block. Motion compensation unit 205 can generate a predicted video block for the current video block based on the multiple reference video blocks indicated by the motion information of the current video block.

[0071] In some examples, the motion estimation unit 204 can output a complete set of motion information for use in the decoder's decoding process. Alternatively, in some embodiments, the motion estimation unit 204 can reference the motion information of another video block to transmit the motion information of the current video block via a signal. For example, the motion estimation unit 204 can determine that the motion information of the current video block is sufficiently similar to the motion information of neighboring video blocks.

[0072] In one example, the motion estimation unit 204 may indicate a value to the video decoder 300 in the syntax structure associated with the current video block, which indicates that the current video block has the same motion information as another video block.

[0073] In another example, motion estimation unit 204 can identify another video block and motion vector difference (MVD) in the syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. Video decoder 300 can use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0074] As discussed above, the video encoder 200 can transmit motion vectors via signals in a predictive manner. Two examples of predictive signaling techniques that can be implemented by the video encoder 200 include Advanced Motion Vector Prediction (AMVP) and Merge Pattern Signaling.

[0075] Intra-prediction unit 206 can perform intra-prediction on the current video block. When intra-prediction unit 206 performs intra-prediction on the current video block, it can generate prediction data for the current video block based on decoded samples from other video blocks in the same frame. The prediction data for the current video block can include the predicted video block and various syntax elements.

[0076] The residual generation unit 207 can generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) multiple predicted video blocks from the current video block. The residual data for the current video block can include residual video blocks corresponding to different sample components of the samples in the current video block.

[0077] In other examples, such as in skip mode, there may be no residual data for the current video block, and the residual generation unit 207 may not perform the subtraction operation.

[0078] The transform processing unit 208 can generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video blocks associated with the current video block.

[0079] After the transform processing unit 208 generates a transform coefficient video block associated with the current video block, the quantization unit 209 can quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0080] The inverse quantization unit 210 and the inverse transform unit 211 can apply inverse quantization and inverse transform to the transform coefficient video block respectively to reconstruct the residual video block from the transform coefficient video block. The reconstruction unit 212 can add the reconstructed residual video block to the corresponding samples of one or more predicted video blocks generated by the prediction unit 202 to generate a reconstructed video block associated with the current video block for storage in the buffer 213.

[0081] After the video block is reconstructed by reconstruction unit 212, a loop filtering operation can be performed to reduce video block artifacts in the video block.

[0082] Entropy encoding unit 214 can receive data from other functional components of video encoder 200. When entropy encoding unit 214 receives data, it can perform one or more entropy encoding operations to generate entropy-encoded data and output a bitstream including the entropy-encoded data.

[0083] Figure 3 This is a block diagram illustrating an example of a video decoder 300 according to some embodiments of the present disclosure. The video decoder 300 may be... Figure 1 An example of video decoder 124 in system 100 is shown.

[0084] The video decoder 300 can be configured to perform any or all of the technologies disclosed herein. Figure 3 In the example, the video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video decoder 300. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0085] exist Figure 3 In the example, the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra-frame prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, a reconstruction unit 306, and a buffer 307. In some examples, the video decoder 300 can perform a decoding process that is generally contrasted with the encoding process described with respect to the video encoder 200.

[0086] Entropy decoding unit 301 can retrieve the encoded bitstream. The encoded bitstream may include entropy-encoded video data (e.g., encoded blocks of video data). Entropy decoding unit 301 can decode the entropy-encoded video data, and motion compensation unit 302 can determine motion information from the entropy-decoded video data, including motion vectors, motion vector precision, reference picture list indices, and other motion information. Motion compensation unit 302 can determine such information, for example, by performing AMVP and Merge pattern. AMVP is used, which involves deriving several most likely candidates based on data from neighboring PBs and reference pictures. Motion information typically includes horizontal and vertical motion vector displacement values, one or two reference picture indices, and, in the case of a prediction region in a B-strip, an identifier of which reference picture list is associated with each index. As used herein, in some aspects, "Merge pattern" may refer to deriving motion information from spatially or temporally neighboring blocks.

[0087] The motion compensation unit 302 can generate motion compensation blocks, possibly performing interpolation based on an interpolation filter. Identifiers for the interpolation filters used at sub-pixel precision can be included in the syntax elements.

[0088] The motion compensation unit 302 can use the interpolation filter used by the video encoder 200 during the encoding of the video block to calculate the interpolated values ​​for sub-integer pixels of the reference block. The motion compensation unit 302 can determine the interpolation filter used by the video encoder 200 based on the received syntax information, and the motion compensation unit 302 can use the interpolation filter to generate the prediction block.

[0089] Motion compensation unit 302 may use at least some of the syntax information to determine the size of the blocks used to encode (multiple) frames and / or (multiple) stripes of the encoded video sequence, segmentation information describing how each macroblock of the image of the encoded video sequence is segmented, a pattern indicating how each segment is encoded, one or more reference frames (and a list of reference frames) for each inter-frame coded block, and other information for decoding the encoded video sequence. As used herein, in some aspects, a “strip” can refer to a data structure that can be decoded independently of other stripes of the same image in terms of entropy encoding / decoding, signal prediction, and residual signal reconstruction. A strip can be an entire image or a region of an image.

[0090] Intra-prediction unit 303 can use, for example, an intra-prediction mode received in the bitstream to form prediction blocks from spatially adjacent blocks. Dequantization unit 304 dequantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by entropy decoding unit 301. Inverse transform unit 305 applies the inverse transform.

[0091] The reconstruction unit 306 can obtain the decoded block, for example, by adding the residual block to the corresponding predicted block generated by the motion compensation unit 302 or the intra-frame prediction unit 303. If necessary, a deblocking filter can also be used to filter the decoded block to remove block artifacts. The decoded video block is then stored in a buffer 307, which provides a reference block for subsequent motion compensation / intra-frame prediction, and the buffer 307 also generates decoded video for presentation on a display device.

[0092] Some exemplary embodiments of this disclosure will be described in detail below. It should be understood that section headings are used in this document for ease of understanding and not to limit the embodiments disclosed in a section to that section only. Furthermore, although some embodiments are described with reference to multi-functional video codecs or other specific video codecs, the disclosed techniques are also applicable to other video codec techniques. Furthermore, although some embodiments describe video encoding steps in detail, it should be understood that the corresponding decoding steps corresponding to de-encoding will be implemented by the decoder. Additionally, the term "video processing" includes video encoding or compression, video decoding or decompression, and video transcoding, in which video pixels are represented from one compression format to another compression format or at different compression bitrates. 1. Brief Overview This disclosure relates to video codec techniques. Specifically, it relates to intra-frame template matching prediction and its fusion with other codec tools, as well as other codec tools in image / video codecs. This disclosure can be applied to existing video codec standards such as HEVC or VVC. It can also be applied to future video codec standards or video codecs. 2. Introduction Video codec standards have primarily evolved through the development of well-known ITU-T and ISO / IEC standards. ITU-T developed the H.261 and H.263 standards, while ISO / IEC developed MPEG-1 and MPEG-4 Vision. The two organizations jointly developed the H.262 / MPEG-2 video standard, the H.264 / MPEG-4 Advanced Video Codec (AVC) standard, and the H.265 / HEVC standard. Starting with H.262, video codec standards are based on a hybrid video codec architecture, utilizing temporal prediction plus transform coding. To explore future video codec technologies beyond HEVC, the Joint Video Exploration Team (JVET) was jointly established by VCEG and MPEG in 2015. Since then, JVET has adopted many new methods and incorporated them into reference software called the Joint Exploration Model (JEM). In April 2018, the Joint Video Experts Group (JVET) between VCEG (Q6 / 16) and ISO / IEC JTC1 SC29 / WG11 (MPEG) was established to work on the VVC standard, with the goal of reducing the bit rate by 50% compared to HEVC. 2.1. Encoding and decoding process of a typical video codec Figure 4 An example of a VVC encoder block diagram is shown, comprising three loop filtering blocks: Deblocking Filter (DF), Sample Adaptive Compensation (SAO), and ALF. Unlike DF, which uses predefined filters, SAO and ALF utilize the original samples of the current image, reducing the mean square error between the original and reconstructed samples by adding compensation and applying a Finite Impulse Response (FIR) filter, respectively, and by utilizing the encoded / decoded side information through signal transmission compensation and filter coefficients. ALF is located in the last processing stage of each image and can be viewed as a tool attempting to capture and repair artifacts caused by previous stages. 2.2. Intra-mode encoding and decoding with 67 intra-prediction modes To capture arbitrary edge directions presented in natural video, such as Figure 5 As shown, the number of directional intra-prediction modes has been expanded from 33 used in HEVC to 65, while the planar and DC modes remain unchanged. These denser directional intra-prediction modes are applicable to all block sizes and both luma and chroma intra-prediction. In HEVC, each intra-coded block has a square shape, and the length of each side is a power of 2. Therefore, no partitioning operation is needed to generate intra-prediction values ​​using DC mode. In VVC, blocks can have rectangular shapes, which generally requires a partitioning operation for each block. To avoid partitioning operations for DC prediction, only the longer sides are used to calculate the average of non-square blocks. 2.2.1. Wide-angle intra-frame prediction Although 67 modes are defined in VVC, the precise prediction direction for a given intra-prediction mode index depends on the block shape. Regular angular intra-prediction directions are defined clockwise from 45 degrees to -135 degrees. In VVC, for non-square blocks, several regular angular intra-prediction modes are adaptively replaced with wide-angle intra-prediction modes. The replaced modes are transmitted via signaling using the original mode index, which is then remapped to the wide-angle mode index after resolution. The total number of intra-prediction modes remains unchanged at 67, and the intra-mode encoding / decoding method remains unchanged. To support these predicted directions, a top reference of length 2W+1 and a left reference of length 2H+1 are defined, as follows: Figure 6 As shown. The number of modes replaced in the wide-angle directional mode depends on the block aspect ratio. The replaced intra-prediction modes are shown in Table 2-1. Table 2-1 – Intra-prediction modes replaced by wide-angle mode like Figure 7 As shown, in the case of wide-angle intra-frame prediction, two vertically adjacent predicted samples can use two non-adjacent reference samples. Therefore, a low-pass reference sample filter and edge smoothing are applied to wide-angle prediction to reduce the increased gap Δp. α The negative impact of wide-angle mode. If the wide-angle mode represents a non-fractional offset. There are 8 wide-angle modes that satisfy this condition, namely [-14, -12, -10, -6, 72, 76, 78, 80]. When a block is predicted through these modes, the samples in the reference cache are directly copied without applying any interpolation. This modification reduces the number of samples required for smoothing. Furthermore, it aligns the design of non-fractional modes in regular prediction modes with that of wide-angle modes. In VVC, in addition to 4:2:0, 4:2:2 and 4:4:4 chroma formats are also supported. The chroma derivation mode (DM) derivation table for the 4:2:2 chroma format was originally ported from HEVC, with the number of entries expanded from 35 to 67 to align with the expansion of intra-prediction modes. Since the HEVC specification does not support prediction angles below -135 degrees and above 45 degrees, the luma intra-prediction mode range from 2 to 5 is mapped to 2. Therefore, the chroma DM derivation table for the 4:2:2 chroma format is updated by replacing some values ​​in the mapping table entries to more accurately convert the prediction angles for chroma blocks. 2.3. Inter-frame prediction For each inter-frame prediction CU, motion parameters consist of motion vectors, reference picture indices, reference picture list usage indices, and additional information required for generating inter-frame prediction samples using new encoding / decoding features of the VVC. Motion parameters can be transmitted via signaling in an explicit or implicit manner. When a CU is encoded / decoded in skip mode, the CU is associated with a PU and has no significant residual coefficients, no encoded / decoded motion vector increments, or reference picture indices. A Merge mode is defined where motion parameters for the current CU are obtained from neighboring CUs, including spatial and temporal candidates and additional scheduling introduced in the VVC. The Merge mode can be applied to any inter-frame prediction CU, not just skip mode. An alternative to the Merge mode is explicit transmission of motion parameters, where the motion vector for each reference picture list, the corresponding reference picture index, the reference picture list usage flag, and other required information are explicitly transmitted via signaling for each CU. 2.4. Intra-Block Copying (IBC) Intra-Block Copy (IBC) is a tool used in the HEVC extension on SCC. It is well known to significantly improve the encoding and decoding efficiency of screen content material. Since IBC mode is implemented as a block-level encoding and decoding mode, block matching (BM) is performed at the encoder to find the optimal block vector (or motion vector) for each CU. Here, the block vector is used to indicate the displacement from the current block to a reference block that has already been reconstructed within the current frame. The luma block vector of a CU encoded with IBC is integer precise. The chroma block vector is also rounded to integer precision. When combined with AMVR, IBC mode can switch between 1-pixel motion vector precision and 4-pixel motion vector precision. CUs encoded with IBC are considered a third prediction mode, distinct from intra-frame or inter-frame prediction modes. IBC mode is suitable for CUs with a width and height of 64 luma samples or less. On the encoder side, hash-based motion estimation for IBC is performed. The encoder performs RD verification on blocks with a width or height no greater than 16 luminance samples. For non-Merge mode, a block vector search is first performed using a hash-based search. If the hash search does not return valid candidates, a local search based on block matching is performed. In hash-based search, hash key matching (32-bit CRC) between the current block and reference blocks is extended to all allowed block sizes. Hash key calculation for each location in the current image is based on 4x4 sub-blocks. For the larger current block, a hash key match with a reference block is determined when all hash keys of all 4x4 sub-blocks match the hash key at the corresponding reference location. If multiple reference blocks are found to match the hash key of the current block, the block vector cost of each matching reference is calculated, and the one with the lowest cost is selected. In block matching search, the search scope is set to cover both the previous CTU and the current CTU. At the CU level, IBC mode is transmitted via a flag, and it can be transmitted via a flag as either IBC AMVP mode or IBC Skip / Merge mode, as follows: –IBC Skip / Merge Mode: The Merge candidate index is used to indicate which block vector from the list of IBC-encoded blocks from neighboring candidates is used to predict the current block. The Merge list consists of spatial candidates, HMVP candidates, and paired candidates. –IBC AMVP Mode: Block vector differences are encoded and decoded in the same way as motion vector differences. The block vector prediction method uses two candidates as prediction values, one from the left nearest neighbor and one from the top nearest neighbor (if IBC encoded and decoded). When either nearest neighbor is unavailable, the default block vector is used as the prediction value. A flag is transmitted via signaling to indicate the index of the block vector prediction value. 2.5. Merge Pattern with MVD (MMVD) In addition to the Merge mode (where implicitly derived motion information is directly used for generating prediction samples for the current CU), a Merge mode with motion vector difference (MMVD) is also introduced in VVC. After the regular Merge flag is sent, the MMVD flag is immediately transmitted via signal to specify whether the MMVD mode is used for the CU. In MMVD, after selecting a Merge candidate, the Merge candidate is further refined by MVD information transmitted via signals. This further information includes a Merge candidate flag, an index specifying the motion amplitude, and an index indicating the motion direction. In MMVD mode, one of the top two candidates in the Merge list is selected as the MV basis. The MMVD candidate flag is transmitted via signals to specify which Merge candidate to use between the first and second Merge candidates. The distance index specifies the motion amplitude information and indicates a predefined offset from the starting point. For example... Figure 8 As shown, the offset is added to the horizontal or vertical component of the starting MV. The relationship between the distance index and the predefined offset is specified in Table 2-2. Table 2-2 – Relationship between Distance Index and Predefined Offset The direction index indicates the direction of the MVD relative to the starting point. The direction index can represent the four directions shown in Table 2-3. Note that the meaning of the MVD sign can vary depending on the information of the starting MV. When the starting MV is a unidirectional or bidirectional prediction MV and both lists point to the same side of the current image (i.e., both reference POCs are greater than or less than the current image's POC), the sign in Table 2-3 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectional prediction MV and the two MVs point to different sides of the current image (i.e., one reference POC is greater than the current image's POC, and the other reference POC is less than the current image's POC), and the difference in POCs in list 0 is greater than the difference in POCs in list 1, the sign in Table 2-3 specifies the sign of the MV offset added to the list 0 MV component of the starting MV, and the sign of the list 1 MV has the opposite value. Otherwise, if the difference in POCs in list 1 is greater than the difference in POCs in list 0, the sign in Table 2-3 specifies the sign of the MV offset added to the list 1 MV component of the starting MV, and the sign of the list 0 MV has the opposite value. MVD is scaled based on the difference in POC in each direction. If the differences in POC in the two lists are the same, no scaling is needed. Otherwise, if the difference in POC in list 0 is greater than the difference in POC in list 1, then the MVD of list 1 is scaled by defining the difference in POC in L0 as td and the difference in POC in L1 as tb, as shown below. Figure 26 As described in [the document]. If the difference in POC between L1 and L0 is greater than the difference in POC between L0 and L1, then the MVD of list 0 is scaled in the same manner. If the initial MV is unidirectionally predicted, then the MVD is added to the available MV. Table 2-3 – Signs of MV Offsets Defined by Direction Index Directional IDX 00 01 10 11 x-axis + - N / A N / A y-axis N / A N / A + - 2.6. Symmetric MVD Encoding and Decoding In VVC, in addition to the normal one-way and two-way predictive MVD signaling, a symmetric MVD mode for two-way predictive MVD signaling is also applied. In the symmetric MVD mode, the reference image indices of both List 0 and List 1, as well as the motion information of the MVD in List 1, are not transmitted through signals but are derived. The decoding process of the symmetric MVD mode is as follows: 1) At the strip level, the variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1 are derived as follows: – If mvd_l1_zero_flag is 1, then BiDirPredFlag is set to 0. Otherwise, if the most recent reference image in list 0 and the most recent reference image in list 1 form a forward and backward reference pair or a backward and forward reference pair, then BiDirPredFlag is set to 1, and both the reference images in list 0 and list 1 are short-term reference images. Otherwise, BiDirPredFlag is set to 0. 2) At the CU level, if the CU is bidirectional predictive codec and BiDirPredFlag is equal to 1, then the symmetric mode flag indicating whether to use symmetric mode is explicitly indicated by the signal transmission. Figure 9 An example diagram for the symmetric MVD mode is shown. When the symmetric mode flag is true, only mvp_l0_flag, mvp_l1_flag, and MVD0 are explicitly transmitted via signals. The reference indices of list 0 and list 1 are set to be equal to a pair of reference images, respectively. MVD1 is set to be equal to (-MVD0). The final motion vector is shown in the following formula. In the encoder, symmetric MVD motion estimation begins with an initial MV evaluation. A set of initial MV candidates includes MVs obtained from a one-way prediction search, MVs obtained from a two-way prediction search, and MVs from the AMVP list. The MV with the lowest rate-distortion cost is selected as the initial MV for the symmetric MVD motion search. 2.7. Bidirectional Optical Flow (BDOF) The Bidirectional Optical Flow (BDOF) tool is included in VVC. BDOF (formerly known as BIO) was included in JEM. Compared to the JEM version, the BDOF in VVC is a simpler version that requires far fewer computations, especially in terms of the number of multiplications and the size of the multipliers. BDOF is used to refine the bidirectional prediction signal of the CU at the 4x4 sub-block level. BDOF is applied to the CU if all of the following conditions are met: –CU is encoded and decoded using a “true” bidirectional prediction mode, meaning that one of the two reference images is displayed before the current image in the order of display, and the other is displayed after the current image in the order of display. – The distances from the two reference images to the current image (i.e., the difference in point of view) are the same; Both reference images are short-term reference images; –CU is not encoded or decoded using affine mode or SbTMVP Merge mode; –The CU has more than 64 luminance samples; - Both the CU height and CU width are greater than or equal to 8 luminance samples; –BCW weighted index indicates equal weighting; –WP is not enabled for the current CU; –CIIP mode is not used in the current CU. BDOF is applied only to the luminance component. As its name suggests, the BDOF mode is based on the concept of optical flow, which assumes that the motion of the object is smooth. For each 4x4 sub-block, motion refinement (v) is calculated by minimizing the difference between the L0 and L1 predicted samples. x ,v y Motion refinement is then used to adjust the bidirectionally predicted sample values ​​in the 4x4 sub-blocks. The following steps are applied during the BDOF process. First, the horizontal and vertical gradients of the two predicted signals are calculated by directly calculating the difference between two neighboring samples. and Right now, Among them I (k) (i,j) is the sample value at coordinate (i,j) of the predicted signal in list k (k=0,1), and shift1 is calculated based on the luminance bit depth bitDepth as shift1=max(6,bitDepth-6). Then, the autocorrelation and cross-correlation of the gradients S1, S2, S3, S5, and S6 are calculated as follows: S1=∑ (i,j)∈Ω Abs(ψ x (i,j)),S3=∑ (i,j)∈Ω θ(i,j)·Sign(ψ x (i,j)) (2-3) S5=∑ (i,j)∈Ω Abs(ψ y (i,j)),S6=∑ (i,j)∈Ω θ(i,j)·Sign(ψ y (i,j)) in θ(i,j)=(I (1) (i,j)>>n b )-(I (0) (i,j)>>n b ) Where Ω is the 6x6 window surrounding the 4x4 sub-block, and n a and n b The values ​​are set to min(1, bitDepth-11) and min(4, bitDepth-8), respectively. Motion refinement (v) x ,v y Then, the following formula is used to derive the results through cross-correlation and autocorrelation terms: in th′ BIO =2 max(5,BD-7) . It is a floor function, and Based on motion refinement and gradients, the following adjustments are calculated for each sample point in the 4x4 sub-block: Finally, the BDOF samples of CU are calculated by adjusting the bidirectional prediction samples as follows: pred BDOF (x,y)=(I (0) (x,y)+I (1) (x,y)+b(x,y)+o offset >> Displacement (2-7). These values ​​were chosen such that the multiplier in the BDOF process does not exceed 15 bits, and the maximum bit width of the intermediate parameters in the BDOF process is kept within 32 bits. To derive the gradient value, it is necessary to generate some predicted sample points I in the list k (k = 0, 1) outside the current CU boundary. (k) (i,j). For example... Figure 10 As shown, BDOF in VVC uses an extended row / column around the CU boundary. To control the computational complexity of generating prediction samples outside the boundary, prediction samples in the extended region (white area) are generated without interpolation by directly taking reference samples at nearby integer positions (using floor() operations on coordinates), and a normal 8-tap motion-compensated interpolation filter is used to generate prediction samples inside the CU (gray area). These extended sample values ​​are only used in gradient calculations. For the remaining steps in the BDOF process, if any samples and gradient values ​​outside the CU boundary are needed, the samples and gradient values ​​are filled from their nearest neighbors (i.e., repeated). When the width and / or height of a CU is greater than 16 luminance samples, it will be divided into sub-blocks with a width and / or height equal to 16 luminance samples, and the sub-block boundaries will be considered as CU boundaries in the BDOF process. The maximum cell size for the BDOF process is limited to 16x16. The BDOF process can be skipped for each sub-block. The BDOF process is not applied to the sub-block when the SAD between the initial L0 and L1 prediction samples is less than a threshold. The threshold is set to equal to (8*W*(H>>1), where W indicates the sub-block width and H indicates the sub-block height. To avoid the additional complexity of SAD calculation, the SAD between the initial L0 and L1 prediction samples calculated in the DVMR process is reused here. Bidirectional optical flow (BDOF) is disabled if BCW is enabled for the current block, meaning the BCW weighted index indicates unequal weighting. Similarly, BDOF is disabled if WP is enabled for the current block, meaning luma_weight_lx_flag is 1 for either of the two reference images. BDOF is also disabled when the CU is encoded / decoded using symmetric MVD mode or CIIP mode. 2.8. Intra-Frame and Inter-Frame Joint Prediction (CIIP) 2.9. Affine Motion Compensation Prediction In HEVC, only the translational motion model is applied to motion compensation prediction (MCP). In the real world, there are many types of motion, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, block-based affine transformation motion compensation prediction is applied. For example... Figure 11 As shown, the affine motion field of a block is described by motion information from two control points (4 parameters) or three control point motion vectors (6 parameters). For the 4-parameter affine motion model, the motion vector at the sample point (x, y) in the block is derived as follows: For the 6-parameter affine motion model, the motion vector at the sample point (x, y) in the block is derived as follows: Where (mv 0x ,mv 0y (mv) is the motion vector of the top-left control point. 1x ,mv 1y ) is the motion vector of the upper right control point, and (mv 2x ,mv 2y ) is the motion vector of the lower left control point. To simplify motion compensation prediction, block-based affine transformation prediction is applied. To derive the motion vector for each 4x4 luma sub-block, the motion vector of the center sample point of each sub-block is calculated according to the above equation (e.g., ...). Figure 12 (As shown), and rounded to 1 / 16 fractional precision. Then a motion-compensated interpolation filter is applied to generate a prediction for each sub-block with a derived motion vector. The sub-block size for the chroma component is also set to 4x4. The MV of the 4x4 chroma sub-block is calculated as the average of the MVs of the four corresponding 4x4 luma sub-blocks. Similar to translational motion inter-frame prediction, there are two other affine motion inter-frame prediction modes: affine Merge mode and affine... AMVP mode. 2.9.1. Affine Merge Prediction The AF_MERGE mode can be applied to CUs with both width and height greater than or equal to 8. In this mode, the CPMVP of the current CU is generated based on the motion information of spatially neighboring CUs. There can be up to five CPMVP candidates, and a signal transmission index indicates which CPMVP candidate should be used for the current CU. The following three types of CPMVP candidates are used to form the affine merge candidate list: – Inherited affine Merge candidates inferred from the CPMV of neighboring CUs; – Constructed affine Merge candidate CPMVP using translational MV derivation of neighboring CUs; -Zero MV. In VVC, there are at most two inherited affine candidates. These two inherited affine candidates are derived from the affine motion model of neighboring blocks. One inherited affine candidate comes from the left neighboring CU, and the other inherited affine candidate comes from the upper neighboring CU. Candidate blocks are as follows: Figure 13 As shown. For the predicted value on the left, the scan order is A0->A1, and for the predicted value above, the scan order is B0->B1->B2. Only the first inherited candidate from each side is selected. No deduplication check is performed between candidates from two inheritances. When a neighboring affine CU is identified, its control point motion vector is used to derive the CPMVP candidate in the affine Merge list of the current CU. Figure 13 As shown, if the adjacent lower-left block A is encoded and decoded in affine mode, the motion vectors v2, v3, and v4 of the upper-left, upper-right, and lower-left corners of the CU containing block A are obtained. When block A is encoded and decoded using a 4-parameter affine model, the two CPMVs of the current CU are calculated based on v2 and v3. When block A is encoded and decoded using a 6-parameter affine model, the three CPMVs of the current CU are calculated based on v2, v3, and v4. Figure 14 An example of control point motion vector inheritance is shown. The constructed affine candidate is achieved by combining the translational motion information of the neighbors of each control point. The motion information for the control points is derived from... Figure 15 The spatial and temporal nearest neighbors are shown in the derivation. CPMV k (k = 1, 2, 3, 4) represents the k-th control point. For CPMV1, the B2->B3->A2 block is checked, and the MV of the first available block is used. For CPMV2, the B1->B0 block is checked, and for CPMV3, the A1->A0 block is checked. If available, the TMVP is used as CPMV4. After obtaining the motion signatures (MVs) of the four control points, the affine Merge candidate is constructed based on this motion information. The following combinations of control point MVs are used for sequential construction: The following combinations of control point MVs are used: {CPMV1,CPMV2,CPMV3},{CPMV1,CPMV2,CPMV4},{CPMV1,CPMV3,CPMV4},{CPMV2,CPMV3,CPMV4},{CPMV1,CPMV2},{CPMV1,CPMV3}. Combinations of three CPMVs construct a 6-parameter affine merge candidate, and combinations of two CPMVs construct a 4-parameter affine merge candidate. To avoid motion scaling, combinations of control point MVs are discarded if the reference indices of the control points are different. After the inherited affine Merge candidates and the constructed affine Merge candidates are checked, if the list is still not full, a zero MV is inserted at the end of the list. 2.9.2. Affine AMVP Prediction The affine AMVP mode can be applied to CUs with both width and height greater than or equal to 16. An affine flag at the CU level is signaled in the bitstream to indicate whether the affine AMVP mode is used, and another flag is signaled to indicate whether it is a 4-parameter affine or a 6-parameter affine. In this mode, the difference between the current CU's CPVM and its predicted CPMVP is signaled in the bitstream. The affine AMVP candidate list is of size 2 and is generated by sequentially using the following four types of CPVM candidates: – Inherited affine AMVP candidates inferred from the CPMV of neighboring CUs; - Constructed affine AMVP candidate CPMVP derived using translational MV of neighboring CUs; – Translation MV from the neighboring CU; -Zero MV. The checking order for inherited affine AMVP candidates is the same as that for inherited affine Merge candidates. The only difference is that, for AVMP candidates, only affine CUs with the same reference picture as those in the current block are considered. No deduplication process is applied when inserting inherited affine motion predictions into the candidate list. The constructed AMVP candidate is from Figure 15 The specified spatial neighbors are derived using the same checking order as in the affine Merge candidate construction. Furthermore, the reference picture index of neighboring blocks is also checked. The block that is the first to be inter-coded in the checking order and has the same reference picture as the current CU is used. Only one is added. When the current CU is encoded using a 4-parameter affine mode and both mv0 and mv1 are available, they are added as a candidate in the affine AMVP list. When the current CU is encoded using a 6-parameter affine mode and all three CPMVs are available, they are added as a candidate in the affine AMVP list. Otherwise, the constructed AMVP candidates are set to unavailable. If the affine AMVP list still has fewer than 2 candidates after checking the inherited and constructed AMVP candidates, then mv0, mv1, and mv2 will be added sequentially as translation MVs to predict all control point MVs of the current CU, when available. Finally, if the affine AMVP list is still not full, zero MVs are used to fill the affine list. AMVP list. 2.9.3. Affine Motion Information Storage In VVC, the CPMV of an affine CU is stored in a separate cache. The stored CPMV is only used to generate inherited CPMVPs in the affine Merge and Affixed AMVP modes for the most recently encoded / decoded CUs. Subblock MVs derived from the CPMV are used for motion compensation, MV derivation of the Merge / AMVP list for translation MVs, and deblocking. To avoid image row caching for additional CPMVs, affine motion data inheritance from CUs above the CU is treated differently from inheritance from regular neighboring CUs. If the candidate CU for affine motion data inheritance is above... In the CTU line, the bottom left and bottom right sub-blocks MV, instead of CPMV, in the line cache are used for affine MVP derivation. In this way, CPMV is only stored in the local cache. If the candidate CU is a 6-parameter affine encoded / decoded model, the affine model is downgraded to a 4-parameter model. For example... Figure 16 As shown, along the top CTU boundary, the motion vectors of the lower left and lower right sub-blocks of the CU are used for the affine inheritance of the CU in the bottom CTU. 2.9.4. Refinement of Optical Flow Prediction for Affine Modes Compared to pixel-based motion compensation, sub-block-based affine motion compensation saves memory access bandwidth and reduces computational complexity, at the cost of reduced prediction accuracy. To achieve finer-grained motion compensation, Predictive Refinement (PROF) using optical flow is used to refine sub-block-based affine motion compensation predictions without increasing the memory access bandwidth used for motion compensation. In VVC, after sub-block-based affine motion compensation is performed, the luminance prediction samples are refined by adding differences derived from the optical flow equation. PROF is described in the following four steps: Step 1) Affine motion compensation based on sub-blocks is performed to generate sub-block predictions I(i,j). Step 2) Calculate the spatial gradient g of the sub-block prediction at each sample location using a 3-tap filter [-1, 0, 1]. x (i,j) and g y (i,j). The gradient calculation is exactly the same as the gradient calculation in BDOF. g x (i,j)=(I(i+1,j)>>shift1)-(I(i-1,j)>>shift1) (2-10) g y (i,j)=(I(i,j+1)>>shift1)-(I(i,j-1)>>shift1) (2-11) Shift1 is used to control the precision of the gradient. The sub-block (i.e., 4x4) prediction is expanded by one sample on each side for gradient computation. To avoid additional memory bandwidth and additional interpolation computation, those expanded samples on the expanded boundaries are copied from the nearest integer pixel position in the reference image. Step 3) Brightness prediction refinement is calculated using the following optical flow equation. ΔI(i,j)=g x (i,j)*Δv x (i,j)+g y (i,j)*Δv y (i,j) (2-12) where Δv(i,j) is the difference between the sample MV (denoted by v(i,j)) calculated for sample position (i,j) and the sub-block MV of the sub-block to which sample (i,j) belongs, such as Figure 17 As shown, Δv(u,j) is quantized with a precision of 1 / 32 of the brightness sample point. Since the affine model parameters and the sample positions relative to the sub-block center remain unchanged from sub-block to sub-block, Δv(i,j) can be calculated for the first sub-block and reused for other sub-blocks in the same CU. This allows dx(i,j) and dy(i,j) to represent the distance from the sample position (i,j) to the sub-block center (x). SB ,ySB The horizontal and vertical offsets, Δv(x,y), can be derived using the following equation: To maintain accuracy, the center of the sub-block (x) SB ,y SB ) is calculated as ((W) SB -1) / 2,(H SB -1) / 2), where W SB and H SB These are the width and height of the sub-block, respectively. For a 4-parameter affine model For a 6-parameter affine model Where (v 0x ,v 0y ), (v 1x ,v 1y ), (v 2x ,v 2y ) represents the motion vectors of the control points at the top left, top right, and bottom left, and w and h are... The width and height of the CU. Step 4) Finally, the brightness prediction refinement ΔI(i,j) is added to the sub-block prediction I(i,j). The final prediction I' is generated as follows. I′(i,j)= I(i,j)+ΔI(i,j) (2-17) For a CU that has been affinely encoded and decoded, PROF is not applied in two cases: 1) All control points MV are the same, which indicates that the CU has only translational motion; 2) The affine motion parameters are greater than the specified limits, because the sub-block-based affine MC is downgraded to the CU-based MC to avoid large memory access bandwidth requirements. Fast coding methods are applied to reduce the coding complexity of affine motion estimation using PROF. PROF is not applied in the affine motion estimation stage in the following two cases: a) if the CU is not the root block and its parent block does not choose an affine mode as its optimal mode, then PROF is not applied because the probability of the current CU choosing an affine mode as its optimal mode is low; b) if the magnitudes of the four affine parameters (C, D, E, F) are all less than a predefined threshold and the current image is not a low-latency image, then PROF is not applied because the improvement introduced by PROF is small in this case. In this way, affine motion estimation using PROF can be accelerated. 2.10. Sub-block-based temporal motion vector prediction (SbTMVP) VVC supports a sub-block-based temporal motion vector prediction (SbTMVP) method. Similar to temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses the motion field in the co-image to improve motion vector prediction and merge patterns for CUs in the current image. The same co-image used by TMVP is used for SbTVMP. SbTMVP differs from TMVP in the following two main aspects: –TMVP predicts motion at the CU level, but SbTMVP predicts motion at the sub-CU level; – TMVP obtains temporal motion vectors from co-op blocks in the co-op image (the co-op block is the lower right or center block relative to the current CU), while SbTMVP applies motion displacement before obtaining temporal motion information from the co-op image, where the motion displacement is obtained from the motion vector of one of the spatial neighboring blocks from the current CU. The SbTVMP process is as follows: Figure 18A and Figure 18B As shown. SbTMVP predicts the motion vectors of sub-CUs within the current CU in two steps. In the first step, it checks... Figure 18A The spatial nearest neighbor A1 is selected. If A1 has a motion vector that uses a co-located image as its reference image, then that motion vector is chosen as the motion displacement to be applied. If no such motion is identified, the motion displacement is set to (0, 0). In the second step, the motion displacement identified in step 1 is applied (i.e., added to the coordinates of the current block) to the position of the current block. Figure 18B The corresponding image shown obtains motion information (motion vectors and reference indices) at the sub-CU level. Figure 18B The example assumes the motion displacement is set to the motion of block A1. Then, for each sub-CU, the motion information of its corresponding block (the smallest motion grid covering the center sample) in the co-location image is used to derive the motion information for the sub-CU. After the motion information of the co-location sub-CU is identified, it is converted into the motion vector and reference index of the current sub-CU in a manner similar to the TMVP process of HEVC, where temporal motion scaling is applied to align the reference image of the temporal motion vector with the reference image of the current CU. In VVC, a combined sub-block-based Merge list containing both SbTVMP candidates and affine Merge candidates is used for signaling in sub-block-based Merge mode. SbTVMP mode is enabled / disabled via Sequence Parameter Set (SPS) flags. If SbTVMP mode is enabled, the SbTVMP prediction is added as the first entry to the sub-block-based Merge candidate list, followed by the affine Merge candidate. The size of the sub-block-based Merge list is transmitted via signaling in the SPS, and the maximum allowed size of the sub-block-based Merge list in VVC is 5. The sub-CU size used in SbTMVP is fixed at 8x8, and like the affine Merge pattern, the SbTMVP pattern only applies to CUs with both width and height greater than or equal to 8. The encoding logic for the additional SbTMVP Merge candidate is the same as that for other Merge candidates, that is, for each CU in the P-strip or B-strip, an additional RD check is performed to determine whether to use the SbTMVP candidate. 2.11. Adaptive Motion Vector Resolution (AMVR) In HEVC, when `use_integer_mv_flag` in the strip header is equal to 0, the motion vector difference (MVD) (between the CU's motion vector and the predicted motion vector) is transmitted through the signal in quarter-luminance samples. In VVC, a CU-level adaptive motion vector resolution (AMVR) scheme is introduced. AMVR allows the CU's MVD to be encoded and decoded with different precisions, depending on the mode for the current CU (normal AMVP mode or affine mode). In AVMP mode, the current CU's MVD can be adaptively selected as follows: - Normal AMVP mode: quarter brightness sample, half brightness sample, integer brightness sample or four brightness sample. – Affine AMVP mode: quarter luminance sample, integer luminance sample, or 1 / 16 luminance sample. If the current CU has at least one non-zero MVD component, the MVD resolution indication at the CU level is conditionally transmitted via signal. If all MVD components (i.e., both the horizontal and vertical MVD for reference lists L0 and L1) are zero, the quarter-spot luminance sample MVD resolution is presumed. For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether a quarter-luminance sample MVD precision is used for the CU. If the first flag is 0, no further signaling is required, and the quarter-luminance sample MVD precision is used for the current CU. Otherwise, a second flag is signaled to indicate that a half-luminance sample or other MVD precision (integer or quad-luminance sample) is used for the normal AMVP CU. In the case of half-luminance samples, a 6-tap interpolation filter is used instead of the default 8-tap interpolation filter for the half-luminance sample position. Otherwise, a third flag is signaled to indicate whether an integer luminance sample MVD precision or a quad-luminance sample MVD precision is used for the normal AMVP CU. In the case of an affine AMVP CU, the second flag is used to indicate whether an integer luminance sample MVD precision or a 1 / 16 luminance sample MVD precision is used. To ensure that the reconstructed MV has the expected precision (quarter-luminance sample, half-luminance sample, integer luminance sample, or quad-luminance sample), the motion vector prediction value for the CU is rounded to the same precision as the MVD before being added to it. Motion vector predictions are rounded to zero (i.e., negative motion vector predictions are rounded to near positive infinity, and positive motion vector predictions are rounded to near negative infinity). The encoder uses RD checksums to determine the motion vector resolution for the current CU. To avoid always applying RD checksums to each CU... MVD resolution performs four CU-level RD checks. In VTM11, RD checks for MVD accuracy beyond quarter-lumen samples are only invoked under certain conditions. For normal AVMP mode, the RD costs for quarter-lumen sample MVD accuracy and integer lumen sample MV accuracy are first calculated. Then, the integer lumen sample... The RD cost for MVD precision is compared with the RD cost for quarter-luminance sample MVD precision to determine whether further checking of the RD cost for four-luminance sample MVD precision is necessary. If the RD cost for quarter-luminance sample MVD precision is significantly less than the RD cost for integer luminance sample MVD precision, the RD check for four-luminance sample MVD precision is skipped. Then, if the RD cost for integer luminance sample MVD precision is significantly greater than the best RD cost of the previously tested MVD precision, the check for half-luminance sample MVD precision is skipped. For affine AMVP mode, if the affine inter-frame mode is not selected after checking the rate-distortion cost of affine Merge / Skip mode, Merge / Skip mode, quarter-luminance sample MVD precision normal AMVP mode, and quarter-luminance sample MVD precision affine AMVP mode, then 1 / 16 luminance sample MVD precision and 1 pixel MVD precision affine inter-frame modes are not checked. Furthermore, in the 1 / 16 luminance sample and quarter luminance sample MV precision affine inter-frame modes, the affine parameters obtained in the quarter luminance sample MV precision affine inter-frame mode are used as the starting search point. 2.12. Using CU-level weighted bidirectional forecasting (BCW) In HEVC, the bidirectional prediction signal is generated by averaging two prediction signals obtained from two different reference images and / or using two different motion vectors. In VVC, the bidirectional prediction mode is extended beyond simple averaging to allow for a weighted average of the two prediction signals. P bi-pred =((8-w)*P0+w*P1+4)>>3 (2-18) Five weights are allowed in weighted average bidirectional prediction, w∈{-2,3,4,5,10}. For each bidirectional prediction CU, the weight w is determined in one of two ways: 1) for non-Merge CUs, the weighted index is obtained via signal transmission after the motion vector difference; 2) for Merge CUs, the weighted index is inferred from neighboring blocks based on the Merge candidate index. BCW is only applied to CUs with 256 or more luma samples (i.e., CU width multiplied by CU height is greater than or equal to 256). For low-latency images, all 5 weights are used. For non-low-latency images, only 3 weights (w∈{3,4,5}) are used. – At the encoder, fast search algorithms are applied to find weighted indices without significantly increasing encoder complexity. These algorithms are summarized below. For more details, readers can refer to the VTM software. When combined with AMVR, if the current image is a low-latency image, unequal weightings for 1-pixel motion vector precision and 4-pixel motion vector precision are checked only conditionally. – When combined with an affine pattern, an affine ME will be performed for unequal weights if and only if the affine pattern is selected as the current best pattern. – When the two reference images in bidirectional prediction are the same, unequal weighting is checked only under certain conditions. –Unequal weights are not searched when certain conditions are met, depending on the POC distance between the current image and its reference image, the encoding / decoding QP, and the temporal level. The BCW weighted index is encoded and decoded using a context-coded bit followed by a bypass-coded bit. The first context-coded bit indicates whether equal weighting is used; if unequal weighting is used, additional bits are signaled via bypass encoding to indicate which unequal weighting is used. Weighted Prediction (WP) is a codec tool supported by the H.264 / AVC and HEVC standards for efficiently encoding and decoding video content with gradations. Support for WP has also been added to the VVC standard. WP allows weighted parameters (weights and offsets) to be transmitted via signal transmission for each reference picture in each of the reference picture lists L0 and L1. Then, during motion compensation, the corresponding weights and offsets of the reference picture(s) are applied. WP and BCW are designed for different types of video content. To avoid interaction between WP and BCW (which would complicate the VVC decoder design), if the CU uses WP, it is not transmitted via signal transmission. The BCW-weighted index is used, and w is presumed to be 4 (i.e., equal weighting is applied). For Merge CUs, the weighted index is presumed from neighboring blocks based on the Merge candidate index. This can be applied to both normal Merge patterns and inherited affine Merge patterns. For constructed affine Merge patterns, the affine motion information is constructed based on the motion information of up to 3 blocks. The BCW index of a CU using a constructed affine Merge pattern is simply set to be equal to the BCW index of the first control point MV. In VVC, CIIP and BCW cannot be jointly applied to a CU. When a CU is encoded or decoded using CIIP mode, the BCW index of the current CU is set to 2, for example, with equal weighting. 2.13. Local Illumination Compensation (LIC) Local Illumination Compensation (LIC) is an encoding / decoding tool used to address the problem of local illumination variations between the current image and its temporal reference image. LIC is based on a linear model, where a scaling factor and offset are applied to the reference samples to obtain the predicted samples for the current block. Specifically, LIC can be mathematically modeled using the following equation: P(x,y)=α·P r(x+v x ,y+v y )+β Where P(x,y) is the prediction signal of the current block at coordinates (x,y); P r (x+v x ,y+v y ) is composed of the motion vector (v) x ,v y The reference block is pointed to by α; α and β are the corresponding scaling factors and offsets applied to the reference block. Figure 19 The LIC procedure is shown. Figure 19 In this context, when LIC is applied to a block, the Minimum Mean Square Error (LMSE) method is employed to minimize the number of neighboring samples in the current block (i.e., ...). Figure 19 The template T in the image and its corresponding reference sample in the time-domain reference image (i.e., Figure 19 The values ​​of the LIC parameters (i.e., α and β) are derived from the difference between T0 or T1 in the template samples. Additionally, to reduce computational complexity, both the template samples and the reference template samples are downsampled (adaptive downsampling) to derive the LIC parameters; that is, only... Figure 19 The shaded samples in the diagram were used to derive α and β. To improve encoding and decoding performance, such as Figure 20 As shown, no downsampling is performed for the short side. 2.14. Decoder-Side Motion Vector Refinement (DMVR) To improve the accuracy of the MV in the Merge mode, decoder-side motion vector refinement based on bilateral matching (BM) is applied in VVC. In the bidirectional prediction operation, initial values ​​in the reference image list L0 and reference image list L1 are used... The search around the MV is a refined MV. The BM method calculates the distortion between two candidate blocks in the reference image list L0 and list L1. Figure 21 As shown, the SAD between two blocks is calculated based on each MV candidate (e.g., MV0' and MV1') around the initial MV. The MV candidate with the lowest SAD becomes the refined MV and is used to generate the bidirectionally predicted signal. In VVC, the application of DMVR is limited and is only used for encoding and decoding using the following modes and features. CU: - CU-level Merge pattern with bidirectional prediction of MV – Relative to the current image, one reference image is from the past and the other is from the future – The distances (i.e., POC differences) from both reference images to the current image are the same. Both reference images are short-term reference images. –CU has more than 64 luminance samples. - Both CU height and CU width are greater than or equal to 8 luminance samples. –BCW weighted index indicates equal weighting – Do not enable WP for the current block - CIIP mode is not used for the current block. The refined MV derived through the DMVR process was used to generate inter-frame prediction samples and was also used in temporal motion vector prediction for future image encoding and decoding. The original MV was used in the deblocking process and also in spatial motion vector prediction for future CU encoding and decoding. Additional features of DMVR are mentioned in the following sub-entries. 2.14.1. Search Scheme In DVMR, the search point revolves around the initial MV, and the MV offset follows the MV difference mirroring rule. In other words, any point examined by DMVR (represented by the candidate MV pair (MV0, MV1)) obeys the following two equations: MV0′=MV0+MV_offset (2-19) MV1′=MV1-MV_offset (2-20) Where MV_offset represents the refinement offset between the initial MV and the refined MV in one of the reference images. The refinement search range is two integer luminance samples from the initial MV. The search includes an integer sample offset search stage and a fractional sample refinement stage. A 25-point full search is applied to the integer sample offset search. First, the SAD of the initial MV pair is calculated. If the SAD of the initial MV pair is less than a threshold, the integer sample stage of DMVR is terminated. Otherwise, the SAD of the remaining 24 points is calculated and checked in raster scan order. The point with the smallest SAD is selected as the output of the integer sample offset search stage. To reduce the impact of DMVR refinement uncertainties, a bias towards the original MV is proposed during the DMVR process. The SAD between the reference blocks referenced by the initial MV candidates is reduced by 1 / 4 of the SAD value. The integer sample search is followed by fractional sample refinement. To save computational complexity, fractional sample refinement is derived using the parametric error surface equation, rather than through an additional search with SAD comparisons. Fractional sample refinement is conditionally invoked based on the output of the integer sample search phase. Fractional sample refinement is further applied when the integer sample search phase terminates in the first or second iteration with the minimum SAD at the center. In subpixel offset estimation based on parametric error surfaces, the cost at the center location and the costs at the four nearest neighbor locations are used to fit a two-dimensional parabolic error surface equation of the following form. E(x,y)=A(xx min ) 2 +B(yy min ) 2 +C (2-21) Where (x) min ,y min ) corresponds to the fractional position with the minimum cost, and C corresponds to the minimum cost. The above equation is solved by using the costs of the five search points, (x) min ,y min ) is calculated as: x min =(E(-1,0)-E(1,0)) / (2(E(-1,0)+E(1,0)-2E(0,0))) (2-22) y min =(E(0,-1)-E(0,1)) / (2((E(0,-1)+E(0,1)-2E(0,0))) (2-23). x min and y min The value is automatically constrained between -8 and 8 because all cost values ​​are positive, and the minimum value is E(0,0). This corresponds to a half-pixel offset with 1 / 16 pixel MV precision in VVC. The calculated fraction (x min ,y min An integer distance refinement MV is added to obtain a subpixel-precise refinement increment MV. 2.14.2. Bilinear Interpolation and Sample Filling In VVC, the resolution of MV is 1 / 16 of a luma sample. Samples at fractional locations are interpolated using an 8-tap interpolation filter. In DMVR, the search point surrounds an initial fractional pixel MV with an integer sample offset, therefore for... In the DMVR search process, samples at fractional locations need to be interpolated. To reduce computational complexity, a bilinear interpolation filter is used to generate fractional samples for the search process in DMVR. Another important effect of using a bilinear filter is that, utilizing a 2-sample search range, DVMR does not access more reference samples compared to the normal motion compensation process. After obtaining the refined MV through the DMVR search process, a normal 8-tap interpolation filter is applied to generate the final prediction. To avoid accessing more reference samples than the normal MC process, samples that are not needed for the interpolation process based on the original MV but are needed for the interpolation process based on the refined MV are filled from those available samples. 2.14.3. Maximum DMVR Processing Unit: When the width and / or height of a CU is greater than 16 luminance samples, it will be further divided into sub-blocks with a width and / or height equal to 16 luminance samples. The maximum unit size for the DMVR search process is limited to 16x16. 2.15. Multi-pass decoder-side motion vector refinement In this contribution, multi-pass decoder-side motion vector refinement is applied instead of DMVR. In the first pass, bilateral matching (BM) is applied to the codec block. In the second pass, BM is applied to each 16x16 sub-block within the codec block. In the third pass, the motion vectors (MVs) in each 8x8 sub-block are refined by applying bidirectional optical flow (BDOF). The refined MVs are stored for both spatial and temporal motion vector prediction. 2.15.1. First pass – Block-based bilateral matching MV refinement In the first pass, the refined MV is derived by applying BM to the codec block. Similar to decoder-side motion vector refinement (DMVR), the refined MV is searched around the two initial MVs (MV0 and MV1) in the reference picture lists L0 and L1. The refined MVs (MV0_pass1 and MV1_pass1) are derived around the initial MVs based on the minimum bilateral matching cost between the two reference blocks in L0 and L1. BM performs a local search to derive the integer sample precision intDeltaMV and the half-pixel sample precision halfDeltaMv. The local search applies a 3x3 square search pattern to iterate through the search range [-sHor, sHor] in the horizontal direction and the search range [-sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block dimension, and the maximum value of sHor and sVer is 8. The bilateral matching cost is calculated as: bilCost = mvDistanceCost + sadCost. When the block size cbW*cbH is greater than 64, the MRSAD cost function is applied to remove the DC effect of distortion between reference blocks. When bilCost at the center point of the 3x3 search pattern has the minimum cost, the local search of intDeltaMV or halfDeltaMV is terminated. Otherwise, the current minimum cost search point becomes the new center point of the 3x3 search pattern, and the search for the minimum cost continues until it reaches the end of the search range. The existing fractional sample refinement is further applied to derive the final deltaMV. The refined MV after the first pass is then derived as: ·MV0_pass1=MV0+deltaMV ·MV1_pass1=MV1–deltaMV. 2.15.2. Second pass – Sub-block-based bilateral matching MV refinement In the second pass, the refined MV is derived by applying BM to 16x16 grid sub-blocks. For each sub-block, two MVs (MV0_pass1 and MV0_pass1) are obtained in the first pass for the reference image lists L0 and L1. Search around MV1_pass1) for refined MVs. The refined MVs are MV0_pass2(sbIdx2) and... MV1_pass2(sbIdx2) is derived based on the minimum bilateral matching cost between two reference subblocks in L0 and L1. For each subblock, BM performs a full search to derive the integer sample precision intDeltaMV. The full search has a search range of [-sHor, sHor] in the horizontal direction and a search range of [-sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block dimension, and the maximum value of sHor and sVer is 8. The bilateral matching cost is calculated by applying a cost factor to the SATD cost between the two reference subblocks, as follows: bilCost = satdCost * costFactor. The search region (2 * sHor + 1) * (2 * sVer + 1) is divided into... Figure 22 The diagram shows a maximum of five diamond-shaped search regions. Each search region is assigned a cost factor, determined by the distance (intDeltaMV) between each search point and the starting MV, and each diamond region is processed sequentially starting from the center of the search region. Within each region, search points are processed in raster scan order, starting from the top left corner and ending at the bottom right corner. A complete integer-pixel search terminates when the minimum bilCost within the current search region is less than a threshold (equal to sbW*sbH); otherwise, the complete integer-pixel search continues to the next search region until all search points have been checked. Figure 22 The diamond-shaped area in the search area is shown. BM performs a local search to derive the half-sample precision halfDeltaMv. The search pattern and cost function are the same as defined in 2.9.1. The existing VVC DMVR fractional sample refinement is further applied to derive the final deltaMV(sbIdx2). The refined MV from the second pass is then derived as follows: ·MV0_pass2(sbIdx2)=MV0_pass1+deltaMV(sbIdx2) ·MV1_pass2(sbIdx2)=MV1_pass1-deltaMV(sbIdx2). 2.15.3. Third pass – Sub-block based bidirectional optical flow MV refinement In the third pass, the refined MV is derived by applying BDOF to the 8x8 grid sub-blocks. For each 8x8 sub-block, BDOF refinement is applied to derive scaled Vx and Vy without limiting from the refined MV of the parent block in the second pass. The derived bioMv(Vx, Vy) is rounded to 1 / 16 sample precision and limited between -32 and 32. The refined MVs (MV0_pass3(sbIdx3) and MV1_pass3(sbIdx3)) of the third pass are derived as follows: ·MV0_pass3(sbIdx3)=MV0_pass2(sbIdx2)+bioMv ·MV1_pass3(sbIdx3)=MV0_pass2(sbIdx2)–bioMv. 2.16. Sample-based BDOF In sample-based BDOF, instead of block-based derivation of motion refinement (Vx, Vy), it is performed per sample. The encoding / decoding block is divided into 8x8 sub-blocks. For each sub-block, whether to apply BDOF is determined by checking the SAD (Solution Aspect Ratio) between two reference sub-blocks against a threshold. If it is decided to apply BDOF to the sub-block, a sliding 5x5 window is used for each sample in the sub-block, and the existing BDOF process is applied to derive Vx and Vy for each sliding window. The derived motion refinement (Vx, Vy) is applied to adjust the bidirectionally predicted sample values ​​for the center sample of the window. 2.17. Extended Merge Prediction In VVC, the Merge candidate list is constructed by including the following five types of candidates in sequence: (1) Airspace MVP from the airspace adjacent to the CU (2) Temporal MVP from the same CU (3) Historical MVP from FIFO table (4) Paired average MVP (5) Zero MV. The size of the Merge list is transmitted via signaling in the sequence parameter set header, and the maximum allowed size of the Merge list is 6. For each CU encoded in Merge mode, the index of the best Merge candidate is encoded using rounding univariate binarization (TU). The first bit of the Merge index is encoded using the context, and bypass encoding is used for the remaining bits. This section provides the derivation process for the Merge candidates for each category. Similar to HEVC, VVC also supports parallel derivation of the Merge candidate list for all CUs within a given region. 2.17.1. Derivation of Airspace Candidates The derivation of spatial merge candidates in VVC is the same as that in HEVC, except that the positions of the first two merge candidates are swapped. Figure 23 From the candidates shown, a maximum of four Merge candidates can be selected. Figure 23 The airspace is shown The positions of the merged candidates are derived in the order B0, A0, B1, A1, and B2. Position B2 is considered only when one or more CUs at positions B0, A0, B1, and A1 are unavailable (e.g., because it belongs to another stripe or slice) or when it is intra-frame encoded / decoded. After adding the candidate at position A1, a redundancy check is performed on the addition of the remaining candidates. This redundancy check ensures that candidates with the same motion information are not included in the list, thereby improving encoding / decoding efficiency. To reduce computational complexity, not all possible candidate pairs are considered in the aforementioned redundancy check. Instead, only... Figure 24 The system uses arrow links to select pairs, and only adds candidates to the list if the corresponding candidates used for redundancy checks do not have the same motion information. 2.17.2. Time-domain candidate derivation In this step, only one candidate is added to the list. Specifically, in the derivation of this temporal merge candidate, the scaled motion vector is derived based on the corresponding CU belonging to the corresponding reference image. The list of reference images to be used for the derivation of the corresponding CU is explicitly transmitted via signal transmission in the strip header. Figure 25 As shown by the dashed lines, the scaled motion vectors for temporal merge candidates are obtained by scaling the motion vectors from the co-located CU using POC distances tb and td, where tb is defined as the POC difference between the current image and the reference image, and td is defined as the POC difference between the co-located reference image and the co-located image. The reference image index for the temporal merge candidate is set to 0. like Figure 26As shown, the position for the temporal candidate is selected between candidate C0 and C1. If the CU at position C0 is unavailable, intra-frame encoded or decoded, or outside the current line of the CTU, position C1 is used. Otherwise, position C0 is used for the derivation of the temporal merge candidate. 2.17.3. Derivation of Merge Candidates Based on History Historically based MVP (HMVP) merge candidates are added to the merge list after the spatial MVP and TMVP. In this method, motion information from previously encoded / decoded blocks is stored in a table and used as the MVP for the current CU. The table with multiple HMVP candidates is maintained during the encoding / decoding process. The table is reset (cleared) when a new CTU row is encountered. Whenever a CU that is not a sub-block inter-frame encoded / decoded is encountered, the associated motion information is added to the last entry of the table as a new HMVP candidate. The HMVP table size S is set to 6, indicating that a maximum of 6 history-based MVP (HMVP) candidates can be added to the table. When a new motion candidate is inserted into the table, a constrained First-In-First-Out (FIFO) rule is used, where a redundancy check is first applied to find if a duplicate HMVP already exists in the table. If found, the duplicate HMVP is removed from the table, and all subsequent HMVP candidates are shifted forward. HMVP candidates can be used in the Merge candidate list construction process. The latest HMVP candidates in the table are checked sequentially and inserted into the candidate list, following the TMVP candidates. Redundancy checks are applied to HMVP candidates for spatial or temporal Merge candidates. To reduce the number of redundant check operations, the following simplifications are introduced: The number of HMPV candidates used for generating the Merge list is set to (N<=4)? M: (8-N), where N indicates the number of existing candidates in the Merge list, and M indicates the number of available HMVP candidates in the table. The process of building the Merge candidate list from HMVP terminates once the total number of available Merge candidates reaches the maximum allowed number of Merge candidates minus 1. 2.17.4. Derivation of Pairwise Average Merge Candidates Pairwise averaging candidates are generated by averaging predefined candidate pairs from an existing Merge candidate list. These predefined pairs are defined as {(0,1),(0,2),(1,2),(0,3),(1,3),(2,3)}, where the numbers represent the Merge indices in the Merge candidate list. The averaged motion vector is calculated separately for each reference list. If two motion vectors are available in a list, they are averaged even if they point to different reference images; if only one motion vector is available, that vector is used directly; if no motion vector is available, the list remains invalid. When the Merge list is not full after adding pairwise average Merge candidates, zero MVP is inserted at the end until the maximum number of Merge candidates is reached. 2.17.5. Merge Estimation Region The Merge Estimation Region (MER) allows for independent derivation of Merge candidate lists for CUs within the same Merge Estimation Region (MER). Candidate blocks within the same MER as the current CU are not included in the generation of the Merge candidate list for the current CU. Additionally, only when (xCb+cbWidth) >> Log2ParMrgLevel is greater than xCb >> Log2ParMrgLevel and (yCb+cbHeight) >> Log2ParMrgLevel are greater than (yCb>> The update process for the candidate list of motion vector predictions based on history is only performed when Log2ParMrgLevel is called, where (xCb, yCb) is the position of the top-left brightness sample point of the current CU in the image, and (cbWidth, cbHeight) is the CU size. The MER size is selected on the encoder side and is transmitted via signal as log2_parallel_merge_level_minus2 in the sequence parameter set. 2.18. New Merge Candidates 2.18.1. Derivation of Non-Adjacent Merge Candidates In VVC, Figure 27 The five spatially neighboring blocks and one temporally nearest neighbor shown are used to derive the Merge candidate. It is proposed to derive additional Merge candidates from positions not adjacent to the current block using the same pattern as in VVC. To achieve this, for each search round i, a virtual block is generated based on the current block, as follows: First, the relative position of the virtual block to the current block is calculated as follows: Offsetx=-i×gridX, Offsety=-i×gridY Offsetx and Offsety represent the offset of the top-left corner of the virtual block relative to the top-left corner of the current block, and gridX and gridY are the width and height of the search grid. Secondly, the width and height of the virtual block are calculated as follows: newWidth=i×2×gridX+currWidth newHeight=i×2×gridY+currHeight. Where currWidth and currHeight are the width and height of the current block. newWidth and newHeight are the width and height of the new virtual block. gridX and gridY are currently set to currWidth and currHeight, respectively. Figure 28 This illustrates the relationship between the virtual block and the current block. After the virtual block is generated, block A i B i C i D i and E i These can be considered VVC spatial neighbor blocks, and their positions are obtained using the same pattern as the pattern in the VVC. Clearly, if the search round i is 0, the virtual block is the current block. In this case, block A... i B i C i D i and E i It is a spatial neighbor block used in VVC Merge mode. When constructing the Merge candidate list, deduplication is performed to ensure that each element in the Merge candidate list is unique. The maximum number of search rounds is set to 1, which means that five non-adjacent spatial neighbor blocks are utilized. Non-adjacent spatial merge candidates are inserted into the merge list after the temporal merge candidates in the order B1->A1->C1->D1->E1. 2.18.2.STMVP We propose using three spatial merge candidates and one temporal merge candidate to derive the average candidate as the STMVP candidate. The STMVP is inserted before the spatial merge candidate in the upper left corner. STMVP candidates were deduplicated along with all previous Merge candidates in the Merge list. For airspace candidates, the top three candidates in the current Merge candidate list are used. For time-domain candidates, use the same position as the VTM / HEVC co-position. For airspace candidates, the first, second, and third candidates inserted into the current Merge candidate list before STMVP are denoted as F, S, and T. Temporal candidates that share the same position as the VTM / HEVC used in TMVP are denoted as Col. The motion vector (denoted as mvLX) of the STMVP candidate in the prediction direction X is derived as follows: 1) If all four Merge candidates have valid reference indices and are equal to 0 in the prediction direction X (X = 0 or 1), mvLX = (mvLX_F + mvLX_S + mvLX_T + mvLX_Col) >> 2. 2) If the reference indices of three out of the four merge candidates are valid and equal to 0 in the prediction direction X (X = 0 or 1), mvLX=(mvLX_F×3+mvLX_S×3+mvLX_Col×2)>>3 or mvLX=(mvLX_F×3+mvLX_T×3+mvLX_Col×2)>>3 or mvLX=(mvLX_S×3+mvLX_T×3+mvLX_Col×2)>>3. 3) If the reference indices of two of the four Merge candidates are valid and equal to 0 in the prediction direction X (X = 0 or 1), mvLX=(mvLX_F+mvLX_Col)>>1 or mvLX=(mvLX_S+mvLX_Col)>>1 or mvLX=(mvLX_T+mvLX_Col)>>1. Note: STMVP mode is turned off if time-domain candidates are not available. 2.18.3. Merge list size If both non-adjacent Merge candidates and STMVP Merge candidates are considered, the size of the Merge list is signaled in the sequence parameter set header, and the maximum allowed size of the Merge list is 8. 2.19. Geometric Partitioning (GPM) In VVC, geometric partitioning modes are supported for inter-frame prediction. Geometric partitioning modes are transmitted via signaling as a merge mode using CU-level flags. Other merge modes include regular merge mode, MMVD mode, CIIP mode, and sub-block merge mode. A total of 64 partitions are supported for each possible CU size w×h=2. m ×2 n , where m,n∈{3…6} does not include 8x64 and 64x8. Figure 29 An example of GPM partitioning grouped at the same angle is shown. When using this mode, the CU is divided into two parts by a geometrically positioned straight line. Figure 29 The position of the dividing line is mathematically derived from the angle and offset parameters of a specific segment. Each part of the geometric segment in the CU is predicted inter-frame using its own motion; only unidirectional prediction is allowed for each segment, i.e., each part has one motion vector and one reference index. Unidirectional prediction motion constraints are applied to ensure that, as with regular bidirectional prediction, only two motion-compensated predictions are needed for each CU. The unidirectional prediction motion for each segment is derived using the process described in 2.19.1. If a geometric segmentation mode is used for the current CU, the geometric segmentation index (angle and offset) and two merge indices (one for each segment) are further indicated via signal transmission. The number of maximum GPM candidate dimensions is explicitly transmitted in SPS, and the syntax binarization used for the GPM merge index is specified. After predicting each part of the geometric segmentation, the sample values ​​along the geometric segmentation edges are adjusted using a blending process with adaptive weighting as described in 2.19.2. This is the prediction signal for the entire CU, and the transformation and quantization processes are applied to the entire CU as in other prediction modes. Finally, the motion field of the CU predicted using the geometric segmentation mode is stored, as described in 2.19.3. 2.19.1. Construction of One-Way Prediction Candidate List The unidirectional prediction candidate list is directly derived from the Merge candidate list constructed according to the extended Merge prediction process in 2.17. Let n denote the index of the unidirectional prediction motion in the geometric unidirectional prediction candidate list. The LX motion vector of the nth extended Merge candidate (where X equals the parity of n) is used as the nth unidirectional prediction motion vector for the geometric segmentation pattern. These motion vectors in... Figure 30 The value is marked with "x". If the corresponding LX motion vector for the nth extended Merge candidate does not exist, the L(1-X) motion vector of the same candidate is used as the unidirectional predicted motion vector for the geometric segmentation pattern. 2.19.2. Blending along geometrically segmented edges After predicting each segment of the geometric segment using its own motion, a blend is applied to the two predicted signals to derive samples around the segmentation edges. The blend weighting for each location of the CU is derived based on the distance between the individual location and the segmentation edge. The distance from position (x, y) to the segmentation edge is derived as follows: Where i, j are indices for the angle and offset of the geometric segmentation, which depend on the geometric segmentation index of the transmitted signal. ρ x,j and ρ y,j The sign depends on the angle index i. The weighting of each part of the geometric segment is derived as follows: wIdxL(x,y)=partIdx? 32+d(x,y):32-d(x,y) (2-28) w1(x,y) = 1 - w0(x,y) (2-30). partIdx depends on the angle index i. An example of weighted w0 is... Figure 31 It is shown in the middle. 2.19.3. Motion field storage for geometric segmentation patterns Mv1 from the first part of the geometric segmentation, Mv2 from the second part of the geometric segmentation, and Mv, a combination of Mv1 and Mv2, are stored in the motion field of the CU encoded and decoded by the geometric segmentation pattern. The type of motion vector stored for each individual location in the sports field is determined as follows: sType=abs(motionIdx)<32?2:(motionIdx≤0?(1-partIdx):partIdx) (2-31) Where motionIdx equals d(4x+2,4y+2), which is recalculated from equation (2-18). partIdx depends on the angle index i. If sType equals 0 or 1, then Mv0 or Mv1 is stored in the corresponding motion field; otherwise, if sType equals 2, then the combined Mv from Mv0 and Mv2 is stored. The combined Mv is generated using the following procedure: 1) If Mv1 and Mv2 come from different lists of reference images (one from L0 and the other from L1), then Mv1 and Mv2 are simply combined to form a bidirectional predicted motion vector. Otherwise, if Mv1 and Mv2 come from the same list, only the unidirectional predicted motion Mv2 is stored. 2.20. Multiple Hypothesis Prediction In multiple hypothesis prediction (MHP), in inter-frame AMVP mode, regular Merge mode, affine Merge and... In MMVD mode, up to two additional predictions are transmitted via the signal. The resulting overall prediction signal is iteratively accumulated with each additional prediction signal. p n+1 =(1-α) n+1 )p n +α n+1 h n+1 The weighting factor α is specified according to Table 2-4 below. Table 2-4 – Weighting Factors for MHP add_hyp_weight_idx α 0 1 / 4 1 -1 / 8 For inter-frame AMVP mode, MHP is applied only when unequal weighting in BCW is selected in bidirectional prediction mode. Additional assumptions can be either Merge mode or AMVP mode. In Merge mode, motion information is indicated by the Merge index, and the Merge candidate list is the same as in geometric segmentation mode. In AMVP mode, the reference index, MVP index, and MVD are transmitted via signal transmission. 2.21. Non-adjacent airspace candidates Figure 32 The diagram shows the spatial neighbor blocks used to derive spatial merge candidates. Non-adjacent spatial merge candidates are inserted after the TMVP in the regular merge candidate list. The style of spatial merge candidates is shown in... Figure 32 The distance between non-adjacent spatial domain candidates and the current codec block is shown in the diagram. The distance between the current codec block and the candidate spatial domain is based on the width and height of the current codec block. 2.22. Template Matching (TM) Template matching (TM) is a decoder-side MV derivation method used to refine the motion information of the current CU by finding the closest match between a template in the current image (i.e., the top and / or left neighboring blocks of the current CU) and a block in the reference image (i.e., of the same size as the template). For example... Figure 33 As shown, a better MV is searched around the initial motion of the current CU within a search range of [-8, +8] pixels. Two modifications to template matching are proposed: determining the search step size based on AMVR mode, and allowing TM to be cascaded with a bilateral matching process in Merge mode. In AMVP mode, MVP candidates are determined based on template matching error to select the one that minimizes the difference between the current block template and the reference block template. Then, TM performs MV refinement only on that specific MVP candidate. TM refines the MVP candidate using an iterative diamond search, starting with full-pixel MVD precision (or 4 pixels for 4-pixel AMVR mode) within a search range of [-8, +8] pixels. AMVP candidates can be further refined using a cross search with full-pixel MVD precision (or 4 pixels for 4-pixel AMVR mode), followed by half-pixels and quarter-pixels sequentially according to the AMVR mode specified in Table 2-5. This search process ensures that the MVP candidate maintains the same MV precision as indicated by the AMVR mode after the TM process. Table 2-5 – Search Patterns for AMVR and Merge Pattern with AMVR In Merge mode, a similar search method is applied to the Merge candidates indicated by the Merge index. As shown in Table 2-5, TM can proceed up to 1 / 8 pixel MVD precision, or skip those precisions beyond half-pixel MVD precision, depending on whether an alternative interpolation filter is used based on the merged motion information (i.e., used when AMVR is in half-pixel mode). Furthermore, when TM mode is enabled, template matching can operate as a standalone process, or as an additional MV refinement process between the block-based bilateral matching (BM) method and the sub-block-based bilateral matching method, depending on whether BM can be enabled according to its enable condition check. 2.23. Overlapping Block Motion Compensation (OBMC) Overlapping Block Motion Compensation (OBMC) was previously used in H.263. In JEM, unlike H.263, OBMC can be turned on and off using CU-level syntax. When OBMC is used in JEM, it is performed on all motion compensation (MC) block boundaries except for the right and bottom boundaries of the CU. Furthermore, it is applied to both the luma and chroma components. In JEM, MC blocks correspond to codec blocks. When a CU is encoded and decoded using sub-CU modes (including sub-CU merge, affine, and FRUC modes), each sub-block of the CU is an MC block. To handle CU boundaries uniformly, OBMC is performed at the sub-block level on all MC block boundaries, where the sub-block size is set to equal to 4x4, such as... Figure 34 As shown. When OBMC is applied to the current sub-block, in addition to the current motion vector, the motion vectors of the four connected neighboring sub-blocks (if available and not identical to the current motion vector) are also used to derive the prediction block for the current sub-block. These multiple prediction blocks based on multiple motion vectors are combined to generate the final prediction signal for the current sub-block. The predicted block based on the motion vectors of neighboring sub-blocks is represented as P. N , where N represents the index of the neighboring above, below, left, and right sub-blocks, and the predicted block based on the motion vector of the current sub-block is represented as P. C When P N When based on the motion information of neighboring sub-blocks that contain the same motion information as the current sub-block, OBMC does not start from P N Executed. Otherwise, P N Each sample point is added to P C The same point in, namely, P N Four rows / columns were added to P C Weighting factors {1 / 4, 1 / 8, 1 / 16, 1 / 32} were used for P. N Furthermore, weighting factors {3 / 4, 7 / 8, 15 / 16, 31 / 32} were used for P. C An exception is small MC blocks (i.e., when the height or width of the codec block is equal to 4 or the CU is encoded / decoded using sub-CU mode). For small MC blocks, P N Only two rows / columns were added to P C In this case, the weighting factors {1 / 4, 1 / 8} are used for P. N Furthermore, the weighting factors {3 / 4, 7 / 8} were used for P. C For P generated based on the motion vectors of vertical (horizontal) neighboring sub-blocks N P N Samples in the same row (column) are added to P with the same weighting factor. C . In JEM, for CUs with a size of 256 lumen samples or less, a CU-level flag is transmitted via signal transmission to indicate whether OBMC is applied to the current CU. For CUs with a size greater than 256 lumen samples or those not encoded / decoded using AMVP mode, OBMC is applied by default. At the encoder, when OBMC is applied to the CU, its effect is considered during the motion estimation phase. The predicted signal formed by OBMC using motion information from the top and left neighbor blocks is used to compensate for the top and left boundaries of the original signal of the current CU, and then the normal motion estimation process is applied. 2.24. Multiple Transform Selection (MTS) for Core Transformation In addition to DCT-II, which is already used in HEVC, the Multiple Transform Selection (MTS) scheme is used for residual coding and decoding of both inter-frame and intra-frame blocks. It uses several selected transforms from DCT8 / DST7. The newly introduced transform matrices are DST-VII and DCT-VIII. Table 2-6 shows the basis functions of the selected DST / DCTs. Table 2-6 – Transform basis functions for DCT-II / VIII and DSTVII for N-point inputs To maintain the orthogonality of the transformation matrices, the transformation matrices are quantized more precisely than those in HEVC. To keep the intermediate values ​​of the transformation coefficients within the 16-bit range, all coefficients have 10 bits after both the horizontal and vertical transformations. To control the MTS scheme, separate enable flags are specified at the SPS level for intra-frame and inter-frame operations. When MTS is enabled at SPS, a CU-level flag is transmitted via signaling to indicate whether MTS is applied. Here, MTS is applied only to luminance. MTS signaling is skipped when one of the following conditions is met. – The position of the last effective coefficient of luminance TB is less than 1 (i.e., DC only); – The last effective coefficient of brightness TB is located in the MTS zeroing region. If the MTS CU flag is equal to 0, DCT2 is applied in both directions. However, if the MTS CU flag is equal to 1, two additional flags are signaled to indicate the transform type for the horizontal and vertical directions, respectively. The transform and signaling mapping table is shown in Table 2-7. A unified transform selection for ISP and implicit MTS is used by removing intra-frame mode and block shape dependencies. If the current block is in ISP mode, or if the current block is an intra-frame block and both intra-frame explicit MTS and inter-frame explicit MTS are enabled, only DST7 is used for both the horizontal and vertical transform kernels. An 8-bit main transform kernel is used when transform matrix precision is involved. Therefore, all transform kernels used in HEVC remain unchanged, including 4-point DCT-2 and DST-7, 8-point, 16-point, and 32-point DCT-2. In addition, other transform kernels (including 64-point DCT-2, 4-point DCT-8, 8-point, 16-point, and 32-point DST-7 and DCT-8) use an 8-bit main transform kernel. Table 2-7 – Transformation and Signaling Mapping Table To reduce the complexity of large-sized DST-7 and DCT-8 blocks, the high-frequency transform coefficients are zeroed out for DST-7 and DCT-8 blocks with a size (width or height, or both) equal to 32. Only the coefficients in the 16x16 low-frequency region are retained. In HEVC, for example, block residuals can be encoded and decoded using transform skip mode. To avoid redundancy in syntax encoding and decoding, the transform skip flag is not transmitted via signaling when the CU-level MTS_CU_flag is not equal to 0. Note that when LFNST or MIP is activated for the current CU, the implicit MTS transform is set to DCT2. Furthermore, implicit MTS can also be enabled when MTS is enabled for inter-frame encoded blocks. 2.25. Subblock Transformation (SBT) In VTM, a sub-block transform is introduced for inter-frame predicted CUs. In this transform mode, only a sub-part of the residual block is encoded or decoded for the CU. When the inter-frame predicted CU has cu_cbf equal to 1, cu_sbt_flag can be signaled to indicate whether the entire residual block or a sub-part of the residual block is encoded or decoded. In the former case, the inter-frame MTS information is further parsed to determine the transform type of the CU. In the latter case, a portion of the residual block is encoded or decoded using a presumed adaptive transform, and the other portion of the residual block is zeroed out. When SBTs are used in a CU that has undergone inter-frame encoding and decoding, the SBT type and SBT location information are transmitted via signals in the bitstream. For example... Figure 35 As shown, there are two SBT types and two SBT locations. For SBT-V (or SBT-H), the TU width (or height) can be equal to half the CU width (or height) or 1 / 4 of the CU width (or height), resulting in a 2:2 partition or a 1:3 / 3:1 partition. A 2:2 partition is like a binary tree (BT) partition, while a 1:3 / 3:1 partition is like an asymmetric binary tree (ABT) partition. In an ABT partition, only small regions contain non-zero residuals. If one dimension of the CU is 8 in the luma sample, a 1:3 / 3:1 partition along that dimension is not allowed. For a CU, there can be a maximum of 8 SBT modes. Position-dependent transform kernel selection is applied to the luma transform blocks in SBT-V and SBT-H (DCT-2 is always used for chroma TB). Two positions in SBT-H and SBT-V are associated with different kernel transforms. More specifically, the horizontal and vertical transforms for each SBT position are... Figure 35 The transformations are specified in the code. For example, the horizontal and vertical transformations for SBT-V position 0 are DCT-8 and DST-7, respectively. When one side of the residual TU is greater than 32, the transformations in both dimensions are set to DCT-2. Therefore, the sub-block transformations jointly specify the TU slicing, cbf, and horizontal and vertical kernel transformation types of the residual block. Figure 35 The location, type, and transformation type of the SBT are shown. SBT is not applied to CUs encoded and decoded using a combination of inter-frame and intra-frame modes. 3. Problem In current designs of ECM fusion methods, the prediction signals used for fusion are obtained from at most two prediction modes (e.g., for CIIP, the prediction signals are obtained from intra-frame and inter-frame modes; for IBC-CIIP, the prediction signals are obtained from intra-frame and IBC modes), or at most two codec tools (e.g., for MHP, the prediction signals are obtained from regular Merge mode and AMVP mode). When the prediction signals used for fusion are obtained from at least two prediction modes, the codec performance can be improved. 4. Detailed Implementation The detailed solutions below should be considered as examples for explaining general concepts. These solutions should not be interpreted in a narrow sense. Furthermore, these solutions can be combined in any way. Fusion of multiple prediction modes or encoding / decoding tools 1. A method for obtaining output prediction / reconstruction signals for video units by fusing more than one prediction signal, wherein the more than one prediction signal is obtained using two or more prediction modes. The prediction modes used to obtain the prediction signals are denoted as M1, M2, ..., M... N , where N is an integer greater than 2. a. The output prediction can be used as the final prediction, which is added to the residual to obtain the reconstructed signal. b. The output prediction can be further processed to obtain the final prediction. c. In one example, the prediction pattern may include a prediction pattern that uses encoding / decoding information from the current stripe / image for prediction. i. In one example, prediction mode could refer to intra-frame mode, or IBC, or palette. d. In one example, the prediction pattern may include a prediction pattern that uses encoding and decoding information from other stripes / pictures for prediction. i. In one example, the prediction mode could refer to the inter-frame mode. e. In one example, which prediction modes are used to obtain the prediction signal and / or the number (N) of prediction modes can be predefined, transmitted through the signal, or derived. i. In one example, when the fusion method is not allowed for P / B stripes / pictures, the prediction mode (e.g., inter-frame mode) that uses codec information from other stripes / pictures may not be used. ii. In one example, N = 3, or N = 4, or N = 5. f. In one example, at least one prediction signal uses a specific prediction pattern M i It was obtained. i. In one example, when using M i When more than one prediction signal is obtained, the multiple prediction signals can be fused first, and the fused prediction signal can be further fused with other prediction signals, which use M... j Obtain, where M i and M j These are two different prediction models. g. In one example, all predicted signals can be directly fused together to obtain the final predicted signal. i. Alternatively, the prediction signal can be divided into more than one subset, and the prediction signals in one subset can be fused first, and the fused prediction signals from more than one subset can be fused to obtain the final prediction signal. 1) In one example, the determination of how to partition the subset may depend on the prediction pattern. a) In one example, the predicted signals obtained using the same prediction pattern are in the same subset. b) In one example, the predicted signals obtained using the non-inter-frame mode are in the same subset. c) In one example, the predicted signals obtained using IBC and inter-frame modes are in the same subset. h. The determination of whether a video unit is allowed to be encoded and decoded using the proposed fusion method may depend on encoding and decoding information, such as: i. Video unit dimensions and / or video unit size ii. Depth of the video unit. iii. Strip / image type and / or segmentation tree type (single tree, dual tree, or local dual tree) iv. Video unit location v. Color components i. Whether the proposed fusion method is applied to the video unit and / or how the proposed fusion method is applied to the video unit can be transmitted as a signal in the bitstream. i. In one example, the proposed fusion method can be transmitted as a sub-mode of a prediction pattern via a signal. ii. Alternatively, the proposed fusion method can be transmitted via signal as an additional prediction mode. j. In one example, the statement “two prediction signals are fused” could mean obtaining a weighted sum from the signals to obtain the fused prediction signal. i. In one example, the limiting operation can be applied after the weighted sum operation. ii. In one example, a shift operation can be applied after a weighted sum operation. 2. A method is proposed to obtain the final prediction signal and / or reconstruction for a video unit by fusing more than one prediction signal, wherein the more than one prediction signal is obtained using two or more codec tools. The codec tools used to obtain the prediction signal are denoted as T1, T2, ..., TH, where H is an integer greater than 2. a. In one example, the encoding / decoding tool may include an intra-frame prediction method. i. In one example, intra-prediction method can refer to a specific codec tool, such as regular intra-prediction, DIMD, TIMD, MRL, ISP, MIP, intra-TMP, intra-prediction fusion, SGPM, TMRL, PDPC / gradient PDPC, CCLM, MMLM, CCCM, GLM, chroma fusion, or variations thereof. b. In one example, the encoding / decoding tool may include an inter-frame prediction method. i. In one example, inter-frame prediction methods can refer to specific codec tools such as CIIP (e.g., CIIP-plane, CIIP-TIMD, CIIP-TM), BCW (e.g., BCW index derived via TM), MMVD (e.g., MMVD or TM-based reordering for MMVD), template matching (TM), affine (e.g., affine-MMVD, TM-based reordering for affine MMVD), DMVR / multi-pass DMVR, PROF, BDOF or sample-based BDOF, adaptive decoder-side motion vector refinement (ADMVR), OBMC or TM-based OBMC, MHP, GPM (e.g., GPM, GPM-TM, GPM-MMVD, GPM-intra), bilateral / template matching AMVP-Merge mode, or variations thereof. c. In one example, the codec tool may include prediction methods for IBC and / or palettes. i. In one example, the prediction method of IBC can refer to the IBC AMVP mode, which can refer to the ordinary IBC AMVP or the TM-based IBC AMVP or the RR-IBC AMVP mode or other IBC AMVP modes (where the BV prediction value is derived and the BVD is transmitted / derived through the signal), or the IBC Merge mode can refer to the ordinary IBC Merge mode or the IBC-TM Merge mode or the IBC-MBVD mode or the IBC-CIIP or IBC-GPM or IBC-LIC or the subpixel-based IBC, or variations thereof. d. In one example, which codecs are used to obtain the predicted signal and / or the number (H) of codecs can be predefined, transmitted through the signal, or derived. i. In one example, H = 3, or H = 4, or H = 5, H = 6, H = 7, H = 8. e. In one example, the codec tools may include tools for both intra-frame and inter-frame prediction modes. i. Alternatively, the codec tools may include tools for both intra-frame and IBC prediction modes. ii. Alternatively, the encoding / decoding tools may include tools for both intra-frame and intra-frame prediction modes. iii. Alternatively, the encoding / decoding tools may include tools for both inter-frame and IBC prediction modes. iv. Alternatively, the encoding / decoding tools may include tools for both inter-frame and inter-frame prediction modes. v. Alternatively, the encoding / decoding tools may include tools for both IBC and IBC prediction modes. vi. Alternatively, the encoding / decoding tools may include tools for both palette and IBC prediction modes. vii. Alternatively, the encoding / decoding tools may include tools for both palette and intra-prediction modes. viii. Alternatively, the encoding / decoding tools may include tools for both palette and inter-frame prediction modes. f. In one example, at least one prediction signal uses a specific encoding / decoding tool T i It was obtained. i. In one example, when using T i When more than one prediction signal is obtained, the multiple prediction signals can be fused first, and the fused prediction signal can be further fused with other prediction signals. The other prediction signals use T. j Obtain, where T i and T j They are two different encoding and decoding tools. g. In one example, all the predicted signals can be directly fused together to obtain the output predicted signal. i. Alternatively, the prediction signal can be divided into more than one subset, and the prediction signals in one subset can be fused first, and the fused prediction signals from more than one subset are fused to obtain the output prediction signal. 1) In one example, the determination of how to divide the subset can depend on the encoding / decoding tools. a) In one example, the predicted signals obtained using the same encoding / decoding tools are in the same subset. b) In one example, the predicted signals obtained using the intra-frame prediction method are in the same subset. c) In one example, the predicted signals obtained using the inter-frame prediction method are in the same subset. d) In one example, the predicted signals obtained using the IBC prediction method are in the same subset. h. The determination of whether a video unit is allowed to be encoded and decoded using the proposed fusion method may depend on encoding and decoding information, such as: i. Video unit dimensions and / or video unit size ii. Depth of the video unit. iii. Strip / image type and / or segmentation tree type (single tree, dual tree, or local dual tree) iv. Video unit location v. Color components i. Whether the proposed fusion method is applied to the video unit and / or how the proposed fusion method is applied to the video unit can be transmitted as a signal in the bitstream. i. In one example, the proposed fusion method can be transmitted via signal as a sub-mode of an encoding / decoding tool. ii. Alternatively, the proposed fusion method can be transmitted via signal as an additional encoding / decoding tool. 3. A method is proposed to obtain the output prediction signal and / or reconstruction for a video unit by fusing more than one prediction signal, wherein more than one prediction signal is obtained using IBC / intra-frame TMP and inter-frame prediction. 4. In one example, the weighting parameters used for fusion can be predefined, transmitted via signaling, or derived. a. In one example, the weighting parameters can be derived using a template composed of neighboring reconstructed samples. i. In one example, neighboring reconstructed samples can refer to adjacent and / or non-adjacent neighboring samples. ii. In one example, neighboring reconstructed samples may refer to neighboring samples to the left and / or lower left, and / or upper left, and / or above, and / or upper right. iii. In one example, the number of neighboring reconstructed samples and / or the shape / size of the template composed of neighboring reconstructed samples may be predefined, transmitted by a signal, or derived based on encoding / decoding information. 1) In one example, encoding / decoding information can refer to block dimensions / size. iv. In one example, the predicted samples of the template can be used to derive the weighting parameters. 1) In one example, the same prediction method or prediction mode as the current video unit can be applied to the template to generate predicted samples of the template. v. In one example, coordinate information can be used to derive weighted parameters. 1) In one example, coordinate information can refer to the relative position of the sample point to a predefined location (such as the top left position of the template). b. In one example, the weighting parameters can be derived using linear or nonlinear equations. Let N, P... T i P T and R T N represents the number of prediction methods or prediction models, and the template generated by the i-th prediction method or prediction model. i predicted signals, the template's predicted signal, and the template's reconstructed signal. i. In one example, a linear equation could refer to P T =w1*P T 1+w2*P T 2+…+w N *P T N +Bias. 1) In one example, the bias is equal to 0. 2) In one example, the bias is related to w i They were deduced together. 3) As a variant, P T =(w1*P T 1+w2*P T 2+…+w N *P T N +offset + offset) >> displacement, where w i The offset, deflection, and displacement are integers. ii. In one example, by minimizing P on a set of training samples T With R T The differences between them (such as minimizing SSD) can be used to derive w. i And / or bias. 1) A set of training samples can include all the samples of the template. 2) A set of training samples may include at least one sample of the template. 3) In one example, the Least Mean Square (LMS) method can be used. 4) In one example, the LDL method used in CCCM can be used. 5) In one example, Gaussian elimination can be used. 6) In one example, this can be achieved by minimizing P. T With R T Other methods, such as neural networks, can be used to solve for the weighted parameters by considering the differences between them. c. In one example, one or more sets of weighted parameters can be derived. i. In one example, multiple sets of weighted parameters can be derived using different templates. ii. In one example, the set of weighted parameters used may be predefined, transmitted via signal, or derived. iii. In one example, which set of weighting parameters is used may depend on the sample values. 1) For example, if the sample value is greater than (or not less than) T1, the first set of weighted values ​​can be used. 2) For example, if the sample value is less than (or not greater than) T1, the second set of weighted values ​​can be used. 5. In one example, encoding and decoding information can be used to fuse the prediction signal. a. In one example, encoding / decoding information could refer to gradient information. i. In one example, gradient information can be derived using one or more prediction signals. b. In one example, encoding / decoding information can refer to location information. c. In one example, the encoding / decoding information could refer to the neighboring samples of the current sample to be fused. General aspects 6. In the above example, a video unit can refer to a color component / sub-picture / strip / piece / code-decode tree unit (CTU) / CTU line / CTU group / code-decode unit (CU) / prediction unit (PU) / transform unit (TU) / code-decode tree block (CTB) / code-decode block (CB) / prediction block (PB) / transform block (TB) / block / sub-block of a block / sub-region within a block / any other region including more than one sample point or pixel. 7. Whether and how the methods disclosed above are applied can be transmitted via signals at the sequence level / picture group level / picture level / strip level / piece group level, such as in the sequence header / picture header / SPS / VPS / DPS / DCI / PPS / APS / strip header / piece group header. 8. Whether and / or how the methods disclosed above are applied can be used to transmit signals at PB / TB / CB / PU / TU / CU / VPDU / CTU / CTU lines / strips / films / sub-images / other types of areas including more than one sample point or pixel. a. Whether and / or how to apply the methods disclosed above may depend on the encoded / decoded information. Information such as block size, color format, single / double tree partitioning, color components, and strip / image type.

[0093] As used herein, the term "video unit" or "video block" can be a sequence, picture, strip, slice, brick, sub-picture, codec tree unit (CTU) / codec tree block (CTB), CTU / CTB row, one or more codec units (CU) / codec blocks (CB), one or more CTU / CTB, one or more virtual pipeline data units (VPDU), or a sub-region within a picture / strip / slice / brick. The term "reference row" can refer to reconstructed samples of rows and / or columns adjacent or non-adjacent to the current block, which are used to derive intra-prediction of the current video unit via an interpolation filter along a certain direction determined by an intra-prediction mode (e.g., conventional intra-prediction with an intra-prediction mode), or via deriving intra-prediction of the current video unit by weighting the reference samples of the reference row using a matrix or vector (e.g., MIP).

[0094] Figure 36 A flowchart of a method 3600 for video processing according to an embodiment of the present disclosure is shown. Method 3600 is implemented during the conversion between video units of a video and a bitstream of a video.

[0095] At box 3610, for the conversion between video units and video unit bitstreams, multiple prediction signals for the video units are obtained using at least one of the following: multiple prediction modes or multiple prediction tools. Multiple prediction modes include two or more prediction modes. Multiple prediction codecs include two or more prediction codecs.

[0096] In some embodiments, a video unit includes at least one of the following: color components, prediction blocks (PB), transform blocks (TB), codec blocks (CB), prediction units (PU), transform units (TU), codec tree blocks (CTB), codec units (CU), codec tree units (CTU), CTU rows, CTU groups, stripes, slices, sub-pictures, blocks, sub-regions within blocks, or regions including more than one sample point or pixel.

[0097] At box 3620, by fusing multiple prediction signals, at least one of the following is obtained: the output prediction signal for the video unit, the final prediction signal, or the reconstruction.

[0098] At box 3630, the conversion is performed based on at least one of the following: outputting a predicted signal, a final predicted signal, or reconstruction. In some embodiments, the conversion may include encoding video units into a bitstream. Alternatively or additionally, the conversion may include decoding video units from the bitstream. In this way, encoding / decoding performance and efficiency are improved when the predicted signal used for fusion is obtained from at least two prediction modes.

[0099] In some embodiments, multiple prediction signals are obtained using intra-block copy (IBC) prediction and inter-frame prediction. Alternatively, multiple prediction signals are obtained using intra-template matching prediction (IntraTMP) and inter-frame prediction.

[0100] In some embodiments, encoding / decoding information is used to fuse multiple prediction signals. For example, the encoding / decoding information includes at least one of the following: gradient information, position information, or neighboring samples of the current sample to be fused. In some embodiments, the gradient information is derived using one or more prediction signals.

[0101] In some embodiments, fusing multiple prediction signals includes applying a weighted sum operation to the multiple prediction signals to obtain a fused prediction signal. In one example, the expression "two prediction signals are fused" may refer to obtaining a weighted sum from the signals to obtain the fused prediction signal. In some embodiments, a limiting operation is applied after the weighted sum operation. Alternatively or additionally, a shift operation is applied after the weighted sum operation.

[0102] In some embodiments, the multiple prediction modes include prediction modes that use codec information from the current stripe or the current image for prediction. For example, the multiple prediction modes include at least one of the following: intra-frame mode, IBC, or palette.

[0103] In some embodiments, the multiple prediction modes include prediction modes that use encoding / decoding information from other stripes or images for prediction. For example, the multiple prediction modes include inter-frame modes.

[0104] In some embodiments, which prediction modes are used to obtain multiple prediction signals is predefined, transmitted via signals, or derived. Alternatively or additionally, the number of prediction modes is predefined, transmitted via signals, or derived.

[0105] In some embodiments, if the fusion method is not permitted for P-strips or B-strips, prediction modes using codec information from other stripes are not used. Alternatively or additionally, if the fusion method is not permitted for P-images or B-images, prediction modes using codec information from other images are not used. For example, the number of prediction modes in a plurality of prediction modes is one of the following: 3, 4, or 5.

[0106] In some embodiments, all predicted signals are directly fused together to obtain the final predicted signal. In some other embodiments, multiple predicted signals are divided into more than one subset, and the predicted signals in each subset are first fused, and the fused predicted signals from more than one subset are then fused together to obtain the final predicted signal. In some embodiments, the determination of how to divide the multiple predicted signals depends on the prediction mode.

[0107] In some embodiments, the predicted signals obtained using the same prediction mode are in the same subset. In some other embodiments, the predicted signals obtained using a non-inter-frame mode are in the same subset. In some still embodiments, using... The predicted signals obtained from IBC and inter-frame modes are in the same subset.

[0108] In some embodiments, the determination of whether a video unit is permitted to be encoded and decoded using the fusion method depends on encoding and decoding information. For example, the encoding and decoding information includes at least one of the following: video unit dimension, video unit size, video unit depth, stripe type, image type, segmentation tree type, video unit position, or color component.

[0109] In some embodiments, the output prediction signal is used as the final prediction for the video unit, which is added to the residual to obtain the reconstructed signal. In some other embodiments, the output prediction signal is further processed to obtain the final prediction for the video unit.

[0110] In some embodiments, at least one prediction signal is obtained using a prediction mode from a plurality of prediction modes. For example, if multiple prediction signals are obtained using prediction modes, the multiple prediction signals are first fused, and the fused prediction signal is further fused with other prediction signals, which are obtained using another prediction mode. In this case, the prediction mode and the other prediction mode are two different prediction modes.

[0111] In some embodiments, whether and / or how the fusion of multiple prediction signals is applied to a video unit is transmitted via signals in the bitstream. For example, the fusion of multiple prediction signals is transmitted via signals as a sub-mode of a prediction mode. Alternatively or additionally, the fusion of multiple prediction signals is transmitted via signals as a sub-mode of an encoding / decoding tool.

[0112] In some embodiments, the fusion of multiple prediction signals is transmitted via signaling as an additional prediction mode. Alternatively or additionally, the fusion of multiple prediction signals is transmitted via signaling as an additional encoding / decoding tool.

[0113] In some embodiments, the multiple codec tools include tools for both intra-frame and inter-frame prediction modes. For example, the multiple codec tools include tools for both intra-frame and IBC prediction modes. In some embodiments, the multiple codec tools include tools for both intra-frame and intra-frame prediction modes. In some other embodiments, the multiple codec tools include tools for both inter-frame and IBC prediction modes. In some still embodiments, the multiple codec tools include tools for both inter-frame and inter-frame prediction modes. In some embodiments, the multiple codec tools include tools for... Tools for both IBC and IBC prediction modes. In some other embodiments, the multiple codec tools include tools for both palette and IBC prediction modes. In some other embodiments, the multiple codec tools include tools for both palette and intra-frame prediction modes. In some embodiments, the multiple codec tools include tools for both palette and inter-frame prediction modes.

[0114] In some embodiments, multiple codec tools include intra-frame prediction methods. For example, intra-prediction methods include at least one of the following: regular intra-prediction, a variant of regular intra-prediction, decoder-side intra-mode derivation (DIMD), a variant of DIMD, template-based intra-mode derivation (TIMD), a variant of TIMD, multiple reference lines (MRL), a variant of MRL, intra-segmentation (ISP), a variant of ISP, matrix-weighted intra-prediction (MIP), a variant of MIP, intra-TMP, a variant of intra-TMP, intra-prediction fusion, a variant of intra-prediction fusion, spatial geometry segmentation mode (SGPM), a variant of SGPM, template-based multi-reference line intra-prediction (TMRL), a variant of TMRL, position-dependent intra-prediction combination (PDPC), a variant of PDPC, gradient PDPC, a variant of gradient PDPC, cross-component linear model (CCLM), a variant of CCLM, multi-model CCLM (MMLM), a variant of MMLM, convolutional cross-component model (CCCM), a variant of CCCM, gradient linear model (GLM), a variant of GLM, chroma fusion, or a variant of chroma fusion.

[0115] In some embodiments, multiple codec tools include inter-frame prediction methods. For example, inter-frame prediction methods include at least one of the following: Intra-Inter-Frame Joint Prediction (CIIP), variants of CIIP, Bidirectional Prediction (BCW) with Codec Unit (CU) Level Weighting, variants of BCW, Merge Mode with Motion Vector Difference (MMVD), variants of MMVD, Template Matching (TM), variants of TM, Affine, variants of Affine, Decoder-Side Motion Vector Refinement (DMVR), variants of DMVR, Multi-Pass DMVR, variants of Multi-Pass DMVR, Prediction Refinement with Optical Flow (PROF), PRO... Variants of F, Bidirectional Optical Flow (BDOF), Variants of BDOF, Sample-based BDOF, Variants of Sample-based BDOF, Adaptive Decoder-Side Motion Vector Refinement (ADMVR), Variants of ADMVR, Overlapping Block Motion Compensation (OBMC), Variants of OBMC, TM-based OBMC, Variants of TM-based OBMC, Multiple Hypothesis Prediction (MHP), Variants of MHP, Geometric Segmentation Mode (GPM), Variants of GPM, Bilateral Matching (BM) Advanced Motion Vector Prediction (AMVP)-Merge Mode, Variants of BM AMVP-Merge Mode, TM AMVP-Merge Mode, or Variants of TM AMVP-Merge Mode.

[0116] In some embodiments, CIIP includes at least one of the following: CIIP-plane, CIIP-TIMD, or CIIP-TM. Alternatively or additionally, BCE includes a BCE index derived via TM. Alternatively or additionally, MMVD includes TM-based reordering for MMVD. Alternatively or additionally, affine includes affine-MMVD or TM-based reordering for affine MMVD. Alternatively or additionally, GPM includes at least one of the following: GPM, GPM-TM, GPM-MMVD, or GPM-intraframe.

[0117] In some embodiments, multiple codec tools include prediction methods for IBC and / or palettes. In some embodiments, the IBC prediction method includes at least one of the following: IBC AMVP mode, IBC Merge mode, IBC-CIIP mode, IBC-GPM mode, IBC Local Illumination (IBC-LIC) mode, or a subpixel-based IBC mode. For example, the IBC AMVP mode includes at least one of the following: ordinary IBC AMVP mode, TM-based IBC AMVP mode, Reconstruction Reordering (RR)-IBCAMVP mode, or other IBC AMVP modes, wherein block vector (BV) prediction values ​​are derived, and block vector difference (BVD) is transmitted or derived via a signal. In some embodiments, the IBC Merge mode includes at least one of the following: ordinary IBC Merge mode, IBC-TM Merge mode, or IBC Merge mode with block vector difference (IBC-MBVD).

[0118] In some embodiments, which codec tools are used to obtain multiple predicted signals is predefined, transmitted through signals, or derived. Alternatively or additionally, the number of codec tools is predefined, transmitted through signals, or derived. In some embodiments, the number of codec tools is equal to one of the following: 3, 4, 5, 6, 7, or 8.

[0119] In some embodiments, at least one prediction signal is obtained using an encoding / decoding tool. For example, if multiple prediction signals are obtained using an encoding / decoding tool, the multiple prediction signals are first fused together, and the fused prediction signal is further fused together with other prediction signals, which are obtained using another encoding / decoding tool, wherein the encoding / decoding tool and the other encoding / decoding tool are two different encoding / decoding tools.

[0120] In some embodiments, all predicted signals are directly fused together to obtain an output predicted signal. In some embodiments, multiple predicted signals are divided into more than one subset, and the predicted signals in each subset are first fused, and the fused predicted signals from more than one subset are then fused together to obtain an output predicted signal. For example, the determination of how to divide the multiple predictions may depend on the encoding / decoding tool. In some embodiments, predicted signals obtained using the same encoding / decoding tool are in the same subset. In some other embodiments, predicted signals obtained using intra-frame prediction methods are in the same subset.

[0121] In some embodiments, the predicted signals obtained using the inter-frame prediction method are in the same subset. In some other embodiments, the predicted signals obtained using the IBC prediction method are in the same subset.

[0122] In some embodiments, the weighting parameters used for fusion are predefined, transmitted via signaling, or derived. For example, the weighting parameters are derived using a template that includes neighboring reconstructed samples. In some embodiments, neighboring reconstructed samples include at least one of the following: adjacent neighboring samples or non-adjacent neighboring samples. In some other embodiments, neighboring reconstructed samples include one or more of the following: left neighboring sample, lower left neighboring sample, upper left neighboring sample, upper neighboring sample, or upper right neighboring sample.

[0123] In some embodiments, the number of neighboring reconstructed samples is predefined, transmitted via a signal, or derived based on encoding / decoding information. Alternatively or additionally, the shape or size of the template including neighboring reconstructed samples is predefined, transmitted via a signal, or derived based on encoding / decoding information. For example, the encoding / decoding information includes block dimensions or block size.

[0124] In some embodiments, predicted samples of the template are used to derive weighting parameters. In some embodiments, the same prediction method or prediction mode as the current video unit is applied to the template to generate predicted samples of the template.

[0125] In some embodiments, coordinate information is used to derive weighting parameters. For example, coordinate information includes the relative positions of sample points to predefined locations.

[0126] In some embodiments, the weighting parameters are derived using linear or nonlinear equations. For example, the linear equation is P. T =w1*P T 1+w2*P T 2+…+w N *P T N +Bias. In this case, w represents the weighting parameters, N represents the number of prediction methods or prediction models, and P T i P represents the i-th prediction signal of the template generated by the i-th prediction method or prediction mode. T Let R represent the predicted signal of the template. T This represents the reconstruction signal of the template, and the bias is a parameter.

[0127] In some embodiments, the bias value is equal to 0. In some other embodiments, the bias value is derived together with the weighting parameters.

[0128] In some embodiments, the linear equation is P T =(w1*P T 1+w2*P T 2+…+w N *P TN +Bias +Offset) >> Shift. In this case, w represents the weighting parameter, N represents the number of prediction methods or prediction models, and P T i P represents the i-th prediction signal of the template generated by the i-th prediction method or prediction mode. T Let R represent the predicted signal of the template. T The reconstructed signal for the template, w i The offset, deflection, and displacement are integers.

[0129] In some embodiments, by minimizing P on a set of training samples T With R T The difference between them, the method is used to derive w i And / or bias. For example, a set of training samples includes all samples of the template. By way of example, a set of training samples includes at least one sample of the template.

[0130] In some embodiments, the Least Mean Square (LMS) method is used. In some other embodiments, the LDL method used in CCCM is used. In some still embodiments, Gaussian elimination is used. In some embodiments, by minimizing P... T With R T Other methods are used to solve for the weighted parameters by considering the differences between them.

[0131] In some embodiments, one or more sets of weighted parameters are derived. For example, one or more sets of weighted parameters are derived using different templates.

[0132] In some embodiments, which set of weighting parameters is used is predefined, transmitted via signal, or derived. In some other embodiments, which set of weighting parameters is used depends on the sample value. For example, a first set of weighting values ​​is used if the sample value is greater than or not less than a first threshold. Alternatively or additionally, a second set of weighting values ​​is used if the sample value is less than or not greater than the first threshold.

[0133] In some embodiments, an indication of whether at least one of the output prediction signal, final prediction signal, or reconstruction for a video unit is obtained by fusing multiple prediction signals and / or how at least one of the output prediction signal, final prediction signal, or reconstruction for a video unit is obtained by fusing multiple prediction signals is indicated at one of the following: sequence level, picture group level, picture level, stripe level, or slice group level. In some other embodiments, an indication of whether at least one of the output prediction signal, final prediction signal, or reconstruction for a video unit is obtained by fusing multiple prediction signals and / or how at least one of the output prediction signal, final prediction signal, or reconstruction for a video unit is obtained by fusing multiple prediction signals is indicated at one of the following: sequence header, picture header, sequence parameter set (SPS), video parameter set (VPS), dependency parameter set (DPS), decoding capability information (DCI), picture parameter set (PPS), adaptive parameter set (APS), stripe header, or slice group header. In some other embodiments, an indication of whether at least one of the output prediction signal, final prediction signal, or reconstruction for a video unit is obtained by fusing multiple prediction signals and / or how at least one of the output prediction signal, final prediction signal, or reconstruction for a video unit is obtained by fusing multiple prediction signals is included in one of the following: prediction block (PB), transform block (TB), codec block (CB), prediction unit (PU), transform unit (TU), codec unit (CU), virtual pipeline data unit (VPDU), codec tree unit (CTU), CTU row, strip, slice, sub-picture, or region comprising more than one sample point or pixel.

[0134] In some embodiments, method 3600 further includes determining, based on the encoded / decoded information of the video unit, whether at least one of an output prediction signal, a final prediction signal, or a reconstruction for the video unit is obtained by fusing multiple prediction signals and / or how at least one of an output prediction signal, a final prediction signal, or a reconstruction for the video unit is obtained by fusing multiple prediction signals. The encoded / decoded information may include at least one of the following: block size, color format, single-tree and / or dual-tree segmentation, color components, stripe type, or picture type.

[0135] According to another embodiment of this disclosure, a non-transitory computer-readable recording medium is provided. This non-transitory computer-readable recording medium stores a bitstream of video generated by a method performed by an apparatus for video processing. The method includes: obtaining multiple prediction signals for video units of the video using at least one of the following: multiple prediction modes or multiple prediction tools, wherein the multiple prediction modes include two or more prediction modes, and the multiple prediction codec tools include two or more prediction codec tools; obtaining at least one of the following by fusing the multiple prediction signals: an output prediction signal, a final prediction signal, or a reconstruction for the video units; and generating a bitstream based on at least one of the following: the output prediction signal, the final prediction signal, or the reconstruction.

[0136] According to further embodiments of this disclosure, a method for storing a bitstream of video is provided. The method includes: obtaining multiple prediction signals for video units of the video using at least one of the following: multiple prediction modes or multiple prediction tools, wherein the multiple prediction modes include two or more prediction modes, and the multiple prediction codec tools include two or more prediction codec tools; obtaining at least one of the following by fusing the multiple prediction signals: an output prediction signal, a final prediction signal, or a reconstruction for the video units; generating a bitstream based on at least one of the following: the output prediction signal, the final prediction signal, or the reconstruction; and storing the bitstream in a non-transitory computer-readable recording medium.

[0137] Embodiments of this disclosure may be described in light of the following items, and their features may be combined in any reasonable manner.

[0138] Item 1. A video processing method, comprising: conversion between video units of a video and a bitstream of the video units; obtaining multiple prediction signals of the video units using at least one of the following: multiple prediction modes or multiple prediction tools, wherein the multiple prediction modes include two or more prediction modes, and the multiple prediction codecs include two or more prediction codecs; obtaining at least one of the following by fusing the multiple prediction signals: an output prediction signal, a final prediction signal, or a reconstruction for the video units; and performing the conversion based on at least one of the following: the output prediction signal, the final prediction signal, or the reconstruction.

[0139] Item 2. The method according to Item 1, wherein the plurality of prediction signals are obtained using intra-block copy (IBC) prediction and inter-frame prediction, or wherein the plurality of prediction signals are obtained using intra-template matching prediction (IntraTMP) and inter-frame prediction.

[0140] Item 3. The method according to Item 1, wherein encoding / decoding information is used to fuse the plurality of prediction signals.

[0141] Item 4. The method according to Item 3, wherein the encoding / decoding information includes at least one of the following: gradient information, position information, or neighboring samples of the current sample to be fused.

[0142] Item 5. The method according to Item 4, wherein the gradient information is derived using one or more prediction signals.

[0143] Item 6. The method according to Item 1, wherein fusing the plurality of prediction signals comprises: applying a weighted sum operation to the plurality of prediction signals to obtain a fused prediction signal.

[0144] Item 7. The method according to Item 6, wherein a limiting operation is applied after the weighted sum operation, and / or wherein a shift operation is applied after the weighted sum operation.

[0145] Item 8. The method according to Item 1, wherein the plurality of prediction modes includes a prediction mode that uses encoding / decoding information from the current strip or the current image for prediction.

[0146] Item 9. The method according to Item 8, wherein the plurality of prediction modes includes at least one of the following: intra-frame mode, IBC, or palette.

[0147] Item 10. The method according to Item 1, wherein the plurality of prediction modes includes prediction modes that use encoding / decoding information from other stripes or images for prediction.

[0148] Item 11. The method according to Item 10, wherein the plurality of prediction modes includes inter-frame modes.

[0149] Item 12. According to the method of Item 1, wherein which prediction modes are used to obtain the plurality of prediction signals are predefined, or transmitted by signals, or derived, and / or the number of prediction modes is predefined, or transmitted by signals, or derived.

[0150] Item 13. The method according to Item 12, wherein if the fusion method is not permitted for P-strips or B-strips, a prediction mode using codec information from other stripes is not used, and / or wherein if the fusion method is not permitted for P-images or B-images, a prediction mode using codec information from other images is not used.

[0151] Item 14. The method according to Item 13, wherein the number of the prediction modes in the plurality of prediction modes is one of the following: 3, 4 or 5.

[0152] Item 15. The method according to Item 1, wherein all predicted signals are directly fused together to obtain the final predicted signal.

[0153] Item 16. The method according to Item 1, wherein the plurality of prediction signals are divided into more than one subset, and the prediction signals in each subset are first fused, and the fused prediction signals from the more than one subset are fused to obtain a final prediction signal.

[0154] Item 17. The method according to Item 16, wherein the determination of how to divide the plurality of predicted signals depends on the prediction mode.

[0155] Item 18. The method according to Item 17, wherein the predicted signals obtained using the same prediction mode are in the same subset.

[0156] Item 19. The method according to Item 17, wherein the predicted signals obtained using non-inter-frame modes are in the same subset.

[0157] Item 20. The method according to Item 17, wherein the predicted signals obtained using IBC and inter-frame modes are in the same subset.

[0158] Item 21. The method according to Item 1, wherein the determination of whether a video unit is permitted to be encoded or decoded using the fusion method depends on encoding / decoding information.

[0159] Item 22. The method according to Item 21, wherein the encoding / decoding information includes at least one of the following: video unit dimension, video unit size, video unit depth, stripe type, picture type, segmentation tree type, video unit position, or color component.

[0160] Item 23. The method according to Item 1, wherein the output prediction signal is used as the final prediction of the video unit, and the final prediction is added to the residual to obtain the reconstructed signal.

[0161] Item 24. The method according to Item 1, wherein the output prediction signal is further processed to obtain the final prediction of the video unit.

[0162] Item 25. The method according to Item 1, wherein at least one prediction signal is obtained using a prediction mode among the plurality of prediction modes.

[0163] Item 26. The method according to Item 25, wherein if the plurality of predicted signals are obtained using the prediction mode, the plurality of predicted signals are first fused, and the fused predicted signal is further fused with other predicted signals, the other predicted signals being obtained using another prediction mode, wherein the prediction mode and the other prediction mode are two different prediction modes.

[0164] Item 27. The method according to Item 1, wherein the fusion of the plurality of predicted signals is applied to the video unit and / or the fusion of the plurality of predicted signals is applied to the video unit and transmitted in the bitstream via a signal.

[0165] Item 28. The method according to Item 27, wherein the fusion of the plurality of predicted signals is transmitted by signal as a sub-mode of a prediction mode, and / or wherein the fusion of the plurality of predicted signals is transmitted by signal as a sub-mode of a codec tool.

[0166] Item 29. The method according to Item 27, wherein the fusion of the plurality of prediction signals is transmitted via signal as an additional prediction mode, and / or wherein the fusion of the plurality of prediction signals is transmitted via signal as an additional encoding / decoding tool.

[0167] Item 30. The method according to Item 1, wherein the plurality of encoding and decoding tools include tools for both intra-frame and inter-frame prediction modes.

[0168] Item 31. The method according to Item 30, wherein the plurality of encoding and decoding tools include tools for both intra-frame and IBC prediction modes.

[0169] Item 32. The method according to Item 30, wherein the plurality of encoding and decoding tools include tools for both intra-frame and intra-frame prediction modes.

[0170] Item 33. The method according to Item 30, wherein the plurality of encoding and decoding tools include tools for both inter-frame and IBC prediction modes.

[0171] Item 34. The method according to Item 30, wherein the plurality of encoding and decoding tools include tools for both inter-frame and inter-frame prediction modes.

[0172] Item 35. The method according to Item 30, wherein the plurality of encoding and decoding tools include tools for both IBC and IBC prediction modes.

[0173] Item 36. The method according to Item 30, wherein the plurality of encoding and decoding tools include tools for both palette and IBC prediction modes.

[0174] Item 37. The method according to Item 30, wherein the plurality of encoding and decoding tools include tools for both palette and intra-prediction modes.

[0175] Item 38. The method according to Item 30, wherein the plurality of encoding and decoding tools include tools for both palette and inter-frame prediction modes.

[0176] Item 39. The method according to Item 1, wherein the plurality of encoding and decoding tools includes an intra-frame prediction method.

[0177] Item 40. The method according to Item 39, wherein the intra-prediction method comprises at least one of the following: conventional intra-prediction, a variant of conventional intra-prediction, decoder-side intra-mode derivation (DIMD), a variant of DIMD, template-based intra-mode derivation (TIMD), a variant of TIMD, multiple reference lines (MRL), a variant of MRL, intra-segmentation (ISP), a variant of ISP, matrix-weighted intra-prediction (MIP), a variant of MIP, intra-TMP, a variant of intra-TMP, intra-prediction fusion, a variant of intra-prediction fusion, spatial geometry segmentation mode (SGPM), a variant of SGPM, template-based multi-reference line intra-prediction (TMRL), a variant of TMRL, position-dependent intra-prediction combination (PDPC), a variant of PDPC, gradient PDPC, a variant of gradient PDPC, cross-component linear model (CCLM), a variant of CCLM, multi-model CCLM (MMLM), a variant of MMLM, convolutional cross-component model (CCCM), a variant of CCCM, gradient linear model (GLM), a variant of GLM, chroma fusion, or a variant of chroma fusion.

[0178] Item 41. The method according to Item 1, wherein the plurality of encoding and decoding tools includes an inter-frame prediction method.

[0179] Item 42. The method according to Item 41, wherein the inter-frame prediction method comprises at least one of the following: intra-inter-frame joint prediction (CIIP), a variant of CIIP, bidirectional prediction (BCW) weighted at the codec unit (CU) level, a variant of BCW, Merge mode with motion vector difference (MMVD), a variant of MMVD, template matching (TM), a variant of TM, affine, a variant of affine, decoder-side motion vector refinement (DMVR), a variant of DMVR, multi-pass DMVR, a variant of multi-pass DMVR, prediction refinement using optical flow (… PROF), PROF variants, Bidirectional Optical Flow (BDOF), BDOF variants, Sample-based BDOF, Sample-based BDOF variants, Adaptive Decoder-Side Motion Vector Refinement (ADMVR), ADMVR variants, Overlapping Block Motion Compensation (OBMC), OBMC variants, TM-based OBMC, TM-based OBMC variants, Multiple Hypothesis Prediction (MHP), MHP variants, Geometric Segmentation Mode (GPM), GPM variants, Bilateral Matching (BM) Advanced Motion Vector Prediction (AMVP)-Merge mode, BM AMVP-Merge mode variants, TM AMVP-Merge mode or TM AMVP-Merge mode variants.

[0180] Item 43. The method according to Item 42, wherein the CIIP comprises at least one of the following: CIIP-plane, CIIP-TIMD, or CIIP-TM, and / or wherein the BCE comprises a BCE index derived by TM, and / or wherein the MMVD comprises a TM-based reordering for the MMVD, and / or wherein the affine comprises an affine-MMVD or a TM-based reordering for the affine MMVD, and / or wherein the GPM comprises at least one of the following: GPM, GPM-TM, GPM-MMVD, or GPM-intraframe.

[0181] Item 44. The method according to Item 1, wherein the plurality of codec tools include IBC and / or palette prediction methods.

[0182] Item 45. The method according to Item 44, wherein the prediction method of the IBC includes at least one of the following: IBC AMVP mode, IBC Merge mode, IBC-CIIP mode, IBC-GPM mode, IBC Local Illumination (IBC-LIC) mode, or subpixel-based IBC mode.

[0183] Item 46. The method according to Item 45, wherein the IBC AMVP mode includes at least one of the following: a normal IBC AMVP mode, a TM-based IBC AMVP mode, a reconstruction reordering (RR)-IBC AMVP mode, or other IBC AMVP modes, wherein block vector (BV) predictions are derived, and block vector difference (BVD) is transmitted or derived via a signal.

[0184] Item 47. The method according to Item 45, wherein the IBC Merge mode includes at least one of the following: a normal IBC Merge mode, an IBC-TM Merge mode, or an IBC Merge mode with block vector difference (IBC-MBVD).

[0185] Item 48. The method according to Item 1, wherein which encoding / decoding tools are used to obtain the plurality of predicted signals is predefined, or transmitted through signals, or derived, and / or wherein the number of encoding / decoding tools is predefined, or transmitted through signals, or derived.

[0186] Item 49. The method according to Item 48, wherein the number of the encoding / decoding tools is equal to one of the following: 3, 4, 5, 6, 7 or 8.

[0187] Item 50. The method according to Item 1, wherein at least one prediction signal is obtained using an encoding / decoding tool.

[0188] Item 51. The method according to Item 50, wherein if the plurality of prediction signals are obtained using the encoding / decoding tool, the plurality of prediction signals are first fused, and the fused prediction signals are further fused with other prediction signals, the other prediction signals being obtained using another encoding / decoding tool, wherein the encoding / decoding tool and the other encoding / decoding tool are two different encoding / decoding tools.

[0189] Item 52. The method according to Item 1, wherein all prediction signals are directly fused together to obtain the output prediction signal.

[0190] Item 53. The method according to Item 1, wherein the plurality of prediction signals are divided into more than one subset, and the prediction signals in each subset are first fused, and the fused prediction signals from the more than one subset are fused to obtain the output prediction signal.

[0191] Item 54. The method according to Item 53, wherein the determination of how to divide the plurality of predictions may depend on the encoding / decoding tool.

[0192] Item 55. The method according to Item 54, wherein the predicted signals obtained using the same encoding / decoding tools are in the same subset.

[0193] Item 56. The method according to Item 54, wherein the predicted signals obtained using the intra-frame prediction method are in the same subset.

[0194] Item 57. The method according to Item 54, wherein the predicted signals obtained using the inter-frame prediction method are in the same subset.

[0195] Item 58. The method according to Item 54, wherein the predicted signals obtained using the prediction method of IBC are in the same subset.

[0196] Item 59. The method according to Item 1, wherein the weighting parameters used for the fusion are predefined, transmitted by signal, or derived.

[0197] Item 60. The method according to Item 59, wherein the weighting parameters are derived using a template that includes neighboring reconstructed samples.

[0198] Item 61. The method according to Item 60, wherein the neighboring reconstructed samples include at least one of the following: Adjacent neighboring sample points or non-adjacent neighboring sample points.

[0199] Item 62. The method according to Item 60, wherein the neighboring reconstructed sample points include one or more of the following: left neighboring sample point, lower left neighboring sample point, upper left neighboring sample point, upper neighboring sample point, or upper right neighboring sample point.

[0200] Item 63. The method according to Item 60, wherein the number of adjacent reconstructed samples is predefined, transmitted by a signal, or derived based on encoding / decoding information, and / or the shape or size of the template including the adjacent reconstructed samples is predefined, transmitted by a signal, or derived based on encoding / decoding information.

[0201] Item 64. The method according to Item 63, wherein the encoding / decoding information includes block dimension or block size.

[0202] Item 65. The method according to Item 60, wherein the predicted samples of the template are used to derive the weighting parameters.

[0203] Item 66. The method according to Item 60, wherein the same prediction method or prediction mode as the current video unit is applied to the template to generate the predicted samples of the template.

[0204] Item 67. The method according to Item 60, wherein coordinate information is used to derive the weighting parameters.

[0205] Item 68. The method according to Item 67, wherein the coordinate information includes the relative position between the sample point and a predefined location.

[0206] Item 69. The method according to Item 59, wherein the weighting parameters are derived using linear or nonlinear equations.

[0207] Item 70. The method according to Item 69, wherein the linear equation is P T =w1*P T 1+w2*P T 2+…+w N *P T N + bias, where w represents the weighting parameter, N represents the number of prediction methods or prediction models, and P T i P represents the i-th prediction signal of the template generated by the i-th prediction method or prediction mode. T The predicted signal representing the template, and R T The reconstructed signal represents the template, and the bias is a parameter.

[0208] Item 71. The method according to Item 70, wherein the value of the bias is equal to 0, or wherein the value of the bias is derived together with the weighting parameter.

[0209] Item 72. The method according to Item 69, wherein the linear equation is P T =(w1*P T 1+w2*P T 2+…+w N *P T N +Bias+Offset)>>Displacement, where w represents the weighting parameter, N represents the number of prediction methods or prediction models, and P T i P represents the i-th prediction signal of the template generated by the i-th prediction method or prediction mode. T The predicted signal representing the template, and R T The reconstructed signal representing the template, w i The offset, deflection, and displacement are integers.

[0210] Item 73. The method according to Item 70 or 72, wherein by minimizing P on a set of training samples T With R T The difference between them, the method is used to derive w i And / or bias.

[0211] Item 74. The method according to Item 73, wherein the set of training samples comprises all samples of the template, or wherein the set of training samples comprises at least one sample of the template.

[0212] Item 75. The method described in Item 73, wherein the least mean square (LMS) method is used.

[0213] Item 76. The method described in Item 73, wherein the LDL method used in CCCM is employed.

[0214] Item 77. The method described in Item 73, wherein Gaussian elimination is used.

[0215] Item 78. The method according to Item 73, wherein by minimizing P T With R T Other methods are used to solve for the weighted parameters by considering the differences between them.

[0216] Item 79. The method described in Item 69, wherein one or more sets of weighted parameters are derived.

[0217] Item 80. The method according to Item 79, wherein the set or more sets of weighted parameters are derived using different templates.

[0218] Item 81. In the method described in Item 79, which set of weighting parameters is used is predefined, transmitted by signal, or derived.

[0219] Item 82. The method described in Item 79, wherein which set of weighting parameters is used depends on the sample values.

[0220] Item 83. The method according to Item 82, wherein a first set of weighted values ​​is used if the sample value is greater than or not less than a first threshold, and / or wherein a second set of weighted values ​​is used if the sample value is less than or not greater than the first threshold.

[0221] Item 84. The method according to any one of items 1 to 83, wherein the video unit comprises at least one of the following: color components, prediction blocks (PB), transform blocks (TB), codec blocks (CB), prediction units (PU), transform units (TU), codec tree blocks (CTB), codec units (CU), codec tree units (CTU), CTU rows, CTU groups, stripes, slices, sub-pictures, blocks, sub-regions within blocks, or regions comprising more than one sample point or pixel.

[0222] Item 85. The method according to any one of items 1 to 84, wherein an indication of whether at least one of the output prediction signal, the final prediction signal, or the reconstruction for the video unit is obtained by fusing the plurality of prediction signals and / or how at least one of the output prediction signal, the final prediction signal, or the reconstruction for the video unit is obtained by fusing the plurality of prediction signals is indicated at one of the following: sequence level, picture group level, picture level, strip level, or slice group level.

[0223] Item 86. The method according to any one of items 1 to 84, wherein an indication of whether at least one of the output prediction signal, the final prediction signal, or the reconstruction for the video unit is obtained by fusing the plurality of prediction signals and / or how at least one of the output prediction signal, the final prediction signal, or the reconstruction for the video unit is obtained by fusing the plurality of prediction signals is indicated by one of the following: sequence header, picture header, sequence parameter set (SPS), video parameter set (VPS), dependency parameter set (DPS), decoding capability information (DCI), picture parameter set (PPS), adaptive parameter set (APS), strip header, or slice header.

[0224] Item 87. The method according to any one of items 1 to 84, wherein at least one of the output prediction signal, the final prediction signal, or the reconstruction for the video unit is obtained by fusing the plurality of prediction signals and / or how the output prediction signal for the video unit is obtained by fusing the plurality of prediction signals. The indication of the final prediction signal or at least one of the reconstructions is included in one of the following: prediction block (PB), transform block (TB), codec block (CB), prediction unit (PU), transform unit (TU), codec unit (CU), virtual pipeline data unit (VPDU), codec tree unit (CTU), CTU row, strip, slice, sub-picture, or region including more than one sample point or pixel.

[0225] Item 88. The method according to any one of items 1 to 84, further comprising: determining, based on the encoded and decoded information of the video unit, whether at least one of the output prediction signal, the final prediction signal, or the reconstruction for the video unit is obtained by fusing the plurality of prediction signals and / or how at least one of the output prediction signal, the final prediction signal, or the reconstruction for the video unit is obtained by fusing the plurality of prediction signals, wherein the encoded and decoded information includes at least one of the following: block size, color format, single-tree and / or dual-tree segmentation, color components, stripe type, or picture type.

[0226] Item 89. The method according to any one of items 1 to 88, wherein the conversion comprises encoding the video unit into the bitstream.

[0227] Item 90. The method according to any one of items 1 to 88, wherein the conversion includes decoding the video unit from the bitstream.

[0228] Item 91. An apparatus for video processing, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform a method according to any one of Items 1 to 90.

[0229] Item 92. A non-transitory computer-readable storage medium storing instructions that cause a processor to execute the method according to any one of items 1 to 90.

[0230] Item 93. A non-transitory computer-readable recording medium storing a bitstream of video generated by a method performed by means of an apparatus for video processing, wherein the method comprises: obtaining a plurality of prediction signals for video units of the video using at least one of: a plurality of prediction modes or a plurality of prediction tools, wherein the plurality of prediction modes includes two or more prediction modes, and the plurality of prediction codecs includes two or more prediction codecs; obtaining at least one of: an output prediction signal, a final prediction signal, or a reconstruction for the video units by fusing the plurality of prediction signals; and generating the bitstream based on at least one of: the output prediction signal, the final prediction signal, or the reconstruction.

[0231] Item 94. A method for storing a bitstream of video, comprising: obtaining a plurality of prediction signals for video units of the video using at least one of the following: a plurality of prediction modes or a plurality of prediction tools, wherein the plurality of prediction modes includes two or more prediction modes, and the plurality of prediction codecs includes two or more prediction codecs; obtaining at least one of the following by fusing the plurality of prediction signals: an output prediction signal, a final prediction signal, or a reconstruction for the video units; generating the bitstream based on at least one of the following: the output prediction signal, the final prediction signal, or the reconstruction; and storing the bitstream in a non-transitory computer-readable recording medium. Example device

[0232] Figure 37 A block diagram of a computing device 3700 in which various embodiments of the present disclosure may be implemented is shown. The computing device 3700 may be implemented as a source device 110 (or video encoder 114 or 200) or a destination device 120 (or video decoder 124 or 300), or may be included in a source device 110 (or video encoder 114 or 200) or a destination device 120 (or video decoder 124 or 300).

[0233] It should be understood that, Figure 37 The computing device 3700 shown is for illustrative purposes only and is not intended to imply any limitation on the functionality and scope of the embodiments of this disclosure.

[0234] like Figure 37 As shown, computing device 3700 includes general-purpose computing device 3700. Computing device 3700 may include at least one or more processors or processing units 3710, memory 3720, storage unit 3730, one or more communication units 3740, one or more input devices 3750, and one or more output devices 3760.

[0235] In some embodiments, the computing device 3700 can be implemented as any user terminal or server terminal with computing capabilities. The server terminal can be a server, a large computing device, etc., provided by a service provider. The user terminal can be, for example, any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, stations, units, devices, multimedia computers, multimedia tablet computers, internet nodes, communicators, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, and includes accessories and peripherals of these devices, or any combination thereof. It is conceivable that the computing device 3700 can support any type of interface to the user (such as "wearable" circuitry devices, etc.).

[0236] Processing unit 3710 can be a physical processor or a virtual processor, and can perform various processes based on programs stored in memory 3720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capabilities of computing device 3700. Processing unit 3710 may also be referred to as a central processing unit (CPU), microprocessor, controller, or microcontroller.

[0237] Computing device 3700 typically includes various computer storage media. Such media can be any media accessible by computing device 3700, including but not limited to volatile and non-volatile media, or removable and non-removable media. Memory 3720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (such as read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or flash memory) or any combination thereof. Storage cell 3730 can be any removable or non-removable media and may include machine-readable media, such as memory, flash drives, disks, or other media that can be used to store information and / or data and can be accessed within computing device 3700.

[0238] The computing device 3700 may also include additional removable / non-removable storage media, volatile / non-volatile storage media. Although in Figure 37 Not shown, but a disk drive for reading from and / or writing to a removable non-volatile disk, and an optical disc drive for reading from and / or writing to a removable non-volatile optical disc may be provided. In this case, each drive may be connected to a bus (not shown) via one or more data media interfaces.

[0239] Communication unit 3740 communicates with another computing device via a communication medium. Furthermore, the functionality of the components in computing device 3700 can be implemented by a single computing cluster or by multiple computing machines communicating via communication connections. Therefore, computing device 3700 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general-purpose network nodes.

[0240] Input device 3750 can be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 3760 can be one or more of various output devices, such as a monitor, speaker, printer, etc. With the aid of communication unit 3740, computing device 3700 can also communicate with one or more external devices (not shown), such as storage devices and display devices. Computing device 3700 can also communicate with one or more devices that enable a user to interact with computing device 3700, or any device that enables computing device 3700 to communicate with one or more other computing devices (e.g., network card, modem, etc.), if needed. Such communication can be performed via input / output (I / O) interface (not shown).

[0241] In some embodiments, some or all components of computing device 3700 may be arranged in a cloud computing architecture, rather than integrated into a single device. In a cloud computing architecture, components may be provided remotely and may work together to achieve the functionality described herein. In some embodiments, cloud computing provides computing, software, data access, and storage services without requiring the end user to know the physical location or configuration of the system or hardware providing these services. In various embodiments, cloud computing provides services via a wide area network (e.g., the Internet) using suitable protocols. For example, a cloud computing provider provides applications via a wide area network that can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture, along with the corresponding data, may be stored on servers at remote locations. Computing resources in a cloud computing environment may be consolidated or distributed at locations in remote data centers. Cloud computing infrastructure may provide services through shared data centers, although they may appear as a single access point for the user. Therefore, a cloud computing architecture can be used to provide the components and functionality described herein from a service provider at a remote location. Alternatively, the components and functionality described herein may be provided from conventional servers, or directly installed or otherwise installed on client devices.

[0242] In embodiments of this disclosure, computing device 3700 may be used to implement video encoding / decoding. Memory 3720 may include one or more video codec modules 3725 having one or more program instructions. These modules are accessible and executable by processing unit 3710 to perform the functions of the various embodiments described herein.

[0243] In an example embodiment of performing video encoding, input device 3750 may receive video data as input 3770 to be encoded. The video data may be processed, for example, by video codec module 3725 to generate an encoded bitstream. The encoded bitstream may be provided as output 3780 via output device 3760.

[0244] In an example embodiment of performing video decoding, input device 3750 may receive an encoded bitstream as input 3770. The encoded bitstream may be processed, for example, by a video codec module 3725 to generate decoded video data. The decoded video data may be provided as output 3780 via output device 3760.

[0245] While this disclosure has been specifically shown and described with reference to preferred embodiments, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of this application as defined by the appended claims. These variations are intended to be covered by the scope of this application. Therefore, the foregoing description of embodiments of this application is not intended to be limiting.

Claims

1. A method for video processing, comprising: For the conversion between video units and bitstreams of video units, at least one of the following is used to obtain multiple prediction signals for the video units: multiple prediction modes or multiple prediction tools, wherein the multiple prediction modes include two or more prediction modes, and the multiple prediction codec tools include two or more prediction codec tools. By fusing the multiple prediction signals, at least one of the following is obtained: an output prediction signal, a final prediction signal, or a reconstruction for the video unit; and The transformation is performed based on at least one of the following: the output predicted signal, the final predicted signal, or the reconstruction.

2. The method of claim 1, wherein the plurality of predicted signals are obtained using intra-block copy (IBC) prediction and inter-frame prediction, or The multiple prediction signals are obtained using intra-template matching prediction (IntraTMP) and inter-frame prediction.

3. The method of claim 1, wherein the encoding / decoding information is used to fuse the plurality of prediction signals.

4. The method according to claim 3, wherein the encoding / decoding information includes at least one of the following: gradient information, position information, or neighboring samples of the current sample to be fused.

5. The method of claim 4, wherein the gradient information is derived using one or more prediction signals.

6. The method of claim 1, wherein fusing the plurality of prediction signals comprises: The weighted sum operation is applied to the multiple prediction signals to obtain the fused prediction signal.

7. The method of claim 6, wherein the limiting operation is applied after the weighted sum operation, and / or The shift operation is applied after the weighted sum operation.

8. The method of claim 1, wherein the plurality of prediction modes includes a prediction mode that uses encoding / decoding information from the current strip or the current image for prediction.

9. The method of claim 8, wherein the plurality of prediction modes includes at least one of the following: intra-frame mode, IBC, or palette.

10. The method of claim 1, wherein the plurality of prediction modes includes prediction modes that use encoding / decoding information from other stripes or images for prediction.

11. The method of claim 10, wherein the plurality of prediction modes includes inter-frame modes.

12. The method of claim 1, wherein which prediction modes are used to obtain the plurality of prediction signals are predefined, transmitted via signals, or derived, and / or The number of prediction patterns is either predefined, transmitted by signals, or derived.

13. The method of claim 12, wherein if the fusion method is not permitted for P-strips or B-strips, a prediction mode using codec information from other stripes is not used, and / or If the fusion method is not allowed for P-images or B-images, the prediction mode using encoding and decoding information from other images is not used.

14. The method of claim 13, wherein the number of the prediction modes in the plurality of prediction modes is one of the following: 3, 4 or 5.

15. The method of claim 1, wherein all predicted signals are directly fused together to obtain the final predicted signal.

16. The method of claim 1, wherein the plurality of prediction signals are divided into more than one subset, and the prediction signals in each subset are first fused, and the fused prediction signals from the more than one subset are fused to obtain a final prediction signal.

17. The method of claim 16, wherein the determination of how to divide the plurality of predicted signals depends on the prediction mode.

18. The method of claim 17, wherein the predicted signals obtained using the same prediction mode are in the same subset.

19. The method of claim 17, wherein the predicted signals obtained using the non-inter-frame mode are in the same subset.

20. The method of claim 17, wherein the predicted signals obtained using IBC and inter-frame modes are in the same subset.

21. The method of claim 1, wherein the determination of whether a video unit is permitted to be encoded or decoded using the fusion method depends on encoding / decoding information.

22. The method of claim 21, wherein the encoding / decoding information comprises at least one of the following: Video unit dimension, Video unit size, Depth of video unit Strip type, Image type, Segmentation tree type, Video unit location, or Color components.

23. The method of claim 1, wherein the output prediction signal is used as the final prediction of the video unit, and the final prediction is added to the residual to obtain the reconstructed signal.

24. The method of claim 1, wherein the output prediction signal is further processed to obtain the final prediction of the video unit.

25. The method of claim 1, wherein at least one prediction signal is obtained using a prediction mode among the plurality of prediction modes.

26. The method of claim 25, wherein if the plurality of predicted signals are obtained using the prediction mode, the plurality of predicted signals are first fused, and the fused predicted signal is further fused with other predicted signals, the other predicted signals being obtained using another prediction mode, wherein the prediction mode and the other prediction mode are two different prediction modes.

27. The method of claim 1, wherein whether the fusion of the plurality of predicted signals is applied to the video unit and / or how the fusion of the plurality of predicted signals is applied to the video unit is transmitted in the bitstream via a signal.

28. The method of claim 27, wherein the fusion of the plurality of prediction signals is transmitted via a signal as a sub-mode of a prediction mode, and / or The fusion of the multiple predicted signals is transmitted as a sub-mode of an encoding / decoding tool via a signal.

29. The method of claim 27, wherein the fusion of the plurality of prediction signals is transmitted via a signal as an additional prediction mode, and / or The fusion of the multiple prediction signals is transmitted via signal as an additional encoding / decoding tool.

30. The method of claim 1, wherein the plurality of encoding and decoding tools include tools for both intra-frame and inter-frame prediction modes.

31. The method of claim 30, wherein the plurality of encoding and decoding tools include tools for both intra-frame and IBC prediction modes.

32. The method of claim 30, wherein the plurality of encoding and decoding tools include tools for both intra-frame and intra-frame prediction modes.

33. The method of claim 30, wherein the plurality of encoding and decoding tools include tools for both inter-frame and IBC prediction modes.

34. The method of claim 30, wherein the plurality of encoding and decoding tools include tools for both inter-frame and inter-frame prediction modes.

35. The method of claim 30, wherein the plurality of encoding and decoding tools include tools for both IBC and IBC prediction modes.

36. The method of claim 30, wherein the plurality of encoding and decoding tools include tools for both the palette and the IBC prediction mode.

37. The method of claim 30, wherein the plurality of encoding and decoding tools include tools for both palette and intra-prediction modes.

38. The method of claim 30, wherein the plurality of encoding and decoding tools include tools for both palette and inter-frame prediction modes.

39. The method of claim 1, wherein the plurality of encoding and decoding tools includes an intra-frame prediction method.

40. The method of claim 39, wherein the intra-prediction method comprises at least one of the following: conventional intra-prediction, a variant of conventional intra-prediction, decoder-side intra-mode derivation (DIMD), a variant of DIMD, template-based intra-mode derivation (TIMD), a variant of TIMD, multiple reference lines (MRL), a variant of MRL, intra-segmentation (ISP), a variant of ISP, matrix-weighted intra-prediction (MIP), a variant of MIP, intra-TMP, a variant of intra-TMP, intra-prediction fusion, a variant of intra-prediction fusion, spatial geometry segmentation mode (SGPM), a variant of SGPM, template-based multi-reference line intra-prediction (TMRL), a variant of TMRL, position-dependent intra-prediction combination (PDPC), a variant of PDPC, gradient PDPC, a variant of gradient PDPC, cross-component linear model (CCLM), a variant of CCLM, multi-model CCLM (MMLM), a variant of MMLM, convolutional cross-component model (CCCM), a variant of CCCM, gradient linear model (GLM), a variant of GLM, chroma fusion, or a variant of chroma fusion.

41. The method of claim 1, wherein the plurality of encoding and decoding tools includes an inter-frame prediction method.

42. The method of claim 41, wherein the inter-frame prediction method comprises at least one of the following: intra-frame joint prediction (CIIP), a variant of CIIP, bidirectional prediction (BCW) weighted at the codec unit (CU) level, a variant of BCW, Merge mode with motion vector difference (MMVD), a variant of MMVD, template matching (TM), a variant of TM, affine, a variant of affine, decoder-side motion vector refinement (DMVR), a variant of DMVR, multi-pass DMVR, a variant of multi-pass DMVR, prediction refinement using optical flow (PROF), PRO Variants of F, Bidirectional Optical Flow (BDOF), Variants of BDOF, Sample-based BDOF, Variants of Sample-based BDOF, Adaptive Decoder-Side Motion Vector Refinement (ADMVR), Variants of ADMVR, Overlapping Block Motion Compensation (OBMC), Variants of OBMC, TM-based OBMC, Variants of TM-based OBMC, Multiple Hypothesis Prediction (MHP), Variants of MHP, Geometric Segmentation Mode (GPM), Variants of GPM, Bilateral Matching (BM) Advanced Motion Vector Prediction (AMVP)-Merge Mode, Variants of BMAMVP-Merge Mode, TM AMVP-Merge Mode, or Variants of TM AMVP-Merge Mode.

43. The method of claim 42, wherein the CIIP comprises at least one of the following: CIIP-planar, CIIP-TIMD, or CIIP-TM, and / or The BCE includes a BCE index derived via TM, and / or The MMVD includes TM-based reordering for MMVD, and / or The affine includes affine-MMVD or TM-based reordering for affine MMVD, and / or The GPM mentioned therein includes at least one of the following: GPM, GPM-TM, GPM-MMVD, or GPM-intraframe.

44. The method of claim 1, wherein the plurality of encoding and decoding tools include IBC and / or a palette prediction method.

45. The method of claim 44, wherein the IBC prediction method comprises at least one of the following: IBCAMVP mode, IBC Merge mode, IBC-CIIP mode, IBC-GPM mode, IBC Local Illumination (IBC-LIC) mode, or subpixel-based IBC mode.

46. ​​The method of claim 45, wherein the IBC AMVP mode comprises at least one of the following: a normal IBC AMVP mode, a TM-based IBC AMVP mode, a reconstruction reordering (RR)-IBC AMVP mode, or other IBC AMVP modes, wherein block vector (BV) predictions are derived, and block vector difference (BVD) is transmitted or derived via a signal.

47. The method of claim 45, wherein the IBC Merge mode comprises at least one of the following: a normal IBC Merge mode, an IBC-TM Merge mode, or an IBC Merge mode with block vector difference (IBC-MBVD).

48. The method of claim 1, wherein which encoding / decoding tools are used to obtain the plurality of predicted signals are predefined, transmitted via signals, or derived, and / or The number of encoding and decoding tools is either predefined, transmitted via signals, or derived.

49. The method of claim 48, wherein the number of the encoding / decoding tools is equal to one of the following: 3, 4, 5, 6, 7 or 8.

50. The method of claim 1, wherein at least one prediction signal is obtained using an encoding / decoding tool.

51. The method of claim 50, wherein if the plurality of prediction signals are obtained using the encoding / decoding tool, the plurality of prediction signals are first fused, and the fused prediction signals are further fused with other prediction signals, the other prediction signals being obtained using another encoding / decoding tool, wherein the encoding / decoding tool and the other encoding / decoding tool are two different encoding / decoding tools.

52. The method of claim 1, wherein all prediction signals are directly fused together to obtain the output prediction signal.

53. The method of claim 1, wherein the plurality of prediction signals are divided into more than one subset, and the prediction signals in each subset are first fused, and the fused prediction signals from the more than one subset are fused to obtain the output prediction signal.

54. The method of claim 53, wherein the determination of how to divide the plurality of predictions may depend on the encoding / decoding tool.

55. The method of claim 54, wherein the predicted signals obtained using the same encoding / decoding tools are in the same subset.

56. The method of claim 54, wherein the predicted signals obtained using the intra-frame prediction method are in the same subset.

57. The method of claim 54, wherein the predicted signals obtained using the inter-frame prediction method are in the same subset.

58. The method of claim 54, wherein the predicted signals obtained using the prediction method of IBC are in the same subset.

59. The method of claim 1, wherein the weighting parameters used for the fusion are predefined, transmitted via signal, or derived.

60. The method of claim 59, wherein the weighting parameters are derived using a template comprising neighboring reconstructed samples.

61. The method of claim 60, wherein the neighboring reconstructed samples include at least one of the following: adjacent neighboring samples or non-adjacent neighboring samples.

62. The method of claim 60, wherein the neighboring reconstructed sample points include one or more of the following: left neighboring sample point, lower left neighboring sample point, upper left neighboring sample point, upper neighboring sample point, or upper right neighboring sample point.

63. The method of claim 60, wherein the number of neighboring reconstructed samples is predefined, transmitted via signal, or derived based on encoding / decoding information, and / or This includes the fact that the shape or size of the template of the adjacent reconstructed sample points is predefined, transmitted by signal, or derived based on encoding and decoding information.

64. The method of claim 63, wherein the encoding / decoding information includes block dimension or block size.

65. The method of claim 60, wherein the predicted samples of the template are used to derive the weighting parameters.

66. The method of claim 60, wherein the same prediction method or prediction mode as the current video unit is applied to the template to generate the predicted samples of the template.

67. The method of claim 60, wherein coordinate information is used to derive the weighting parameters.

68. The method of claim 67, wherein the coordinate information includes the relative position between the sample point and a predefined position.

69. The method of claim 59, wherein the weighting parameters are derived using linear or nonlinear equations.

70. The method of claim 69, wherein the linear equation is P T =w1*P T 1+w2*P T 2+…+w N *P T N + bias, and Where w represents the weighting parameter, N represents the number of prediction methods or prediction models, and P T i P represents the i-th prediction signal of the template generated by the i-th prediction method or prediction mode. T The predicted signal representing the template, and R T The reconstructed signal represents the template, and the bias is a parameter.

71. The method of claim 70, wherein the value of the bias is equal to 0, or The bias value is derived together with the weighting parameter.

72. The method of claim 69, wherein the linear equation is P T =(w1*P T 1+w2*P T 2+…+w N *P T N +Offset +Shift) >> Displacement Where w represents the weighting parameter, N represents the number of prediction methods or prediction models, and P T i P represents the i-th prediction signal of the template generated by the i-th prediction method or prediction mode. T The predicted signal representing the template, and R T The reconstructed signal representing the template, w i The offset, deflection, and displacement are integers.

73. The method of claim 70 or 72, wherein P is minimized over a set of training samples. T With R T The difference between them, the method is used to derive w i And / or bias.

74. The method of claim 73, wherein the set of training samples comprises all the samples of the template, or The set of training samples includes at least one sample of the template.

75. The method of claim 73, wherein the least mean square (LMS) method is used.

76. The method of claim 73, wherein the LDL method used in CCCM is employed.

77. The method of claim 73, wherein Gaussian elimination is used.

78. The method of claim 73, wherein by minimizing P T With R T Other methods are used to solve for the weighted parameters by considering the differences between them.

79. The method of claim 69, wherein one or more sets of weighting parameters are derived.

80. The method of claim 79, wherein the set of or more sets of weighted parameters are derived using different templates.

81. The method of claim 79, wherein the set of weighting parameters used is predefined, transmitted via signal, or derived.

82. The method of claim 79, wherein which set of weighting parameters is used depends on the sample values.

83. The method of claim 82, wherein if the sample value is greater than or not less than a first threshold, a first set of weighted values ​​is used, and / or If the sample value is less than or not greater than the first threshold, the second set of weighted values ​​is used.

84. The method according to any one of claims 1 to 83, wherein the video unit comprises at least one of the following: Color components, Predicted Block (PB), Transform block (TB), Code Block (CB), Prediction Unit (PU) Transformer Unit (TU) Code-to-decode tree block (CTB), Codec Unit (CU) Code-decode tree unit (CTU), CTU line, CTU group, strip, piece, Sub-images, piece, Sub-regions within a block, or This includes regions containing more than one sample point or pixel.

85. The method according to any one of claims 1 to 84, wherein an indication of whether at least one of the output prediction signal, the final prediction signal, or the reconstruction for the video unit is obtained by fusing the plurality of prediction signals and / or how at least one of the output prediction signal, the final prediction signal, or the reconstruction for the video unit is obtained by fusing the plurality of prediction signals is indicated at one of the following: sequence level, Image group level, Image quality, strip level, or Film series level.

86. The method according to any one of claims 1 to 84, wherein an indication of whether at least one of the output prediction signal, the final prediction signal, or the reconstruction for the video unit is obtained by fusing the plurality of prediction signals and / or how at least one of the output prediction signal, the final prediction signal, or the reconstruction for the video unit is obtained by fusing the plurality of prediction signals is indicated in one of the following: Sequence header, Image header, Sequence Parameter Set (SPS) Video Parameter Set (VPS) Dependency Parameter Set (DPS) Decoding Capability Information (DCI) Image Parameter Set (PPS) Adaptive Parameter Set (APS) strip head, or The beginning of the film.

87. The method according to any one of claims 1 to 84, wherein an indication of whether at least one of the output prediction signal, the final prediction signal, or the reconstruction for the video unit is obtained by fusing the plurality of prediction signals and / or how at least one of the output prediction signal, the final prediction signal, or the reconstruction for the video unit is obtained by fusing the plurality of prediction signals is included in one of the following: Predicted Block (PB), Transform block (TB), Code Block (CB), Prediction Unit (PU) Transformer Unit (TU) Codec Unit (CU) Virtual Pipeline Data Unit (VPDU) Code-decode tree unit (CTU), CTU line, strip, piece, Sub-images, or This includes regions containing more than one sample point or pixel.

88. The method according to any one of claims 1 to 84, further comprising: Based on the encoded and decoded information of the video unit, it is determined whether to obtain at least one of the output prediction signal, the final prediction signal, or the reconstruction for the video unit by fusing the plurality of prediction signals and / or how to obtain at least one of the output prediction signal, the final prediction signal, or the reconstruction for the video unit by fusing the plurality of prediction signals, wherein the encoded and decoded information includes at least one of the following: Block size, Color format, Single-tree and / or dual-tree partitioning, Color components, Strip type, or Image type.

89. The method according to any one of claims 1 to 88, wherein the conversion comprises encoding the video unit into the bitstream.

90. The method according to any one of claims 1 to 88, wherein the conversion comprises decoding the video unit from the bitstream.

91. An apparatus for video processing, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 90.

92. A non-transitory computer-readable storage medium storing instructions that cause a processor to execute the method according to any one of claims 1 to 90.

93. A non-transitory computer-readable recording medium storing a bitstream of video generated by a method performed by means of a video processing apparatus, wherein the method includes: Multiple prediction signals for video units of the video are obtained using at least one of the following: multiple prediction modes or multiple prediction tools, wherein the multiple prediction modes include two or more prediction modes, and the multiple prediction codec tools include two or more prediction codec tools; By fusing the multiple prediction signals, at least one of the following is obtained: an output prediction signal, a final prediction signal, or a reconstruction for the video unit; and The bitstream is generated based on at least one of the following: the output prediction signal, the final prediction signal, or the reconstruction.

94. A method for storing a bitstream of video, comprising: Multiple prediction signals for video units of the video are obtained using at least one of the following: multiple prediction modes or multiple prediction tools, wherein the multiple prediction modes include two or more prediction modes, and the multiple prediction codec tools include two or more prediction codec tools; By fusing the multiple prediction signals, at least one of the following can be obtained: an output prediction signal, a final prediction signal, or a reconstruction for the video unit; The bitstream is generated based on at least one of the following: the output prediction signal, the final prediction signal, or the reconstruction; and The bitstream is stored in a non-transitory computer-readable recording medium.

Images