Decoding method, encoding method, and method for transmitting image information

Abstract

This decoding method comprises: obtaining information on basic prediction; generating a basic prediction block by performing the basic prediction on the basis of the information on the basic prediction; obtaining information on an additional prediction; generating an additional prediction block by performing the additional prediction on the basis of the information on the additional prediction; and generating a final prediction block for a current block on the basis of combination of the basic prediction block and the additional prediction block, wherein the basic prediction may include one of intra prediction, inter prediction, and intra block copy (IBC), and the additional prediction may include at least one of the intra prediction, inter prediction, and intra block copy.

Description

Decoding method, encoding method and transmission method of image information

[0001] The present disclosure relates to a decoding method, an encoding method, and a method for transmitting image information, and relates to a method for processing multiple reference blocks.

[0002] Recently, demand for high-resolution, high-quality images, such as HD (High Definition) and UHD (Ultra High Definition) images, has been increasing across various fields. As image data becomes higher resolution and higher quality, the amount of information transmitted, or bits, increases relative to conventional image data. This increase in information or bits transmitted leads to increased transmission and storage costs.

[0003] Accordingly, a highly efficient image compression technology is required to effectively transmit, store, and play high-resolution, high-quality image information.

[0004] The present disclosure seeks to provide a method and device for performing inter prediction based on a multi-reference mode.

[0005] The present disclosure seeks to provide a signaling method and device for determining a multiple reference mode.

[0006] The present disclosure seeks to provide a derivation method and device for determining a multiple reference mode.

[0007] A decoding method according to one aspect of the present disclosure comprises: obtaining information about a base prediction; generating a base prediction block by performing the base prediction based on the information about the base prediction; obtaining information about an additional prediction; generating an additional prediction block by performing the additional prediction based on the information about the additional prediction; and generating a final prediction block for a current block based on a superposition of the base prediction block and the additional prediction block, wherein the base prediction includes any one of an intra prediction, an inter prediction, or an intra block copy (IBC), and the additional prediction may include at least one of an intra prediction, an inter prediction, or an intra block copy.

[0008] Obtaining information regarding the additional prediction includes: obtaining additional prediction availability information indicating whether the additional prediction is performed; obtaining prediction mode indexing information indicating an additional prediction mode based on the additional prediction availability information; and obtaining additional block generation information for generating the additional prediction block based on the prediction mode indexing information, wherein the block generation information may include at least one of intra block generation information, inter block generation information, or block copy block generation information.

[0009] Obtaining additional block generation information for generating the additional prediction block may include obtaining additional block generation information for generating the additional prediction block based on satisfaction of a predetermined condition. The predetermined condition may include at least one of: whether the additional prediction mode is allowed for the basic prediction mode; whether the additional prediction mode is allowed for a tool applied in the basic prediction mode; whether the additional prediction mode is allowed for the width or height of the current block; whether the additional prediction mode is allowed in correspondence with a quantization parameter of the current block; or whether the additional prediction mode is allowed in correspondence with a temporal layer of the current block.

[0010] Obtaining information regarding the additional prediction includes obtaining additional prediction availability information indicating whether to perform the additional prediction; and inheriting additional block generation information for generating the additional prediction block based on the additional prediction availability information, wherein the block generation information may include at least one of intra block generation information, inter block generation information, or block copy block generation information.

[0011] Inheriting additional block generation information for generating the above additional prediction block may include inheriting additional block generation information of a surrounding block having the same basic prediction mode as the basic prediction mode of the current block.

[0012] Inheriting additional block generation information for generating the above additional prediction block may include deriving an additional prediction mode of a neighboring block having the same basic prediction mode as the basic prediction mode of the current block; and inheriting additional block generation information for an additional prediction mode of a neighboring block selected based on a priority for inheritance of the additional prediction mode.

[0013] Inheriting additional block generation information for generating the additional prediction block may include deriving an additional prediction mode of a neighboring block having the same basic prediction mode as the basic prediction mode of the current block; and inheriting additional block generation information for an additional prediction mode of a neighboring block allowed based on the basic prediction mode of the current block.

[0014] Obtaining information regarding the additional prediction includes obtaining additional prediction availability information indicating whether to perform the additional prediction; and deriving additional block generation information for generating the additional prediction block based on the additional prediction availability information, wherein the block generation information may include at least one of intra block generation information, inter block generation information, or block copy block generation information.

[0015] Inheriting additional block generation information for generating the above additional prediction block may include deriving an additional prediction mode of a neighboring block having the same basic prediction mode as the basic prediction mode of the current block; and inheriting additional block generation information for an additional prediction mode of a neighboring block selected based on a priority for inheritance of the additional prediction mode.

[0016] Inheriting additional block generation information for generating the additional prediction block may include deriving an additional prediction mode of a neighboring block having the same basic prediction mode as the basic prediction mode of the current block; and inheriting additional block generation information for an additional prediction mode of a neighboring block allowed based on the basic prediction mode of the current block.

[0017] Obtaining information regarding the additional prediction includes obtaining additional prediction availability information indicating whether the additional prediction is performed; and deriving additional block generation information for generating the additional prediction block based on the additional prediction availability information, wherein the block generation information may include at least one of intra block generation information, inter block generation information, or block copy block generation information.

[0018] Deriving additional prediction performance information for performing the above additional prediction may include deriving additional prediction performance information for performing the above additional prediction using a template matching method.

[0019] Deriving additional prediction performance information for performing the above additional prediction may include deriving additional prediction performance information for performing the above additional prediction using at least one of a TIMD method or a DIMD method.

[0020] The information regarding the additional prediction may include information regarding a first additional prediction and information regarding a second additional prediction, the additional prediction block may include a first additional prediction block and a second additional prediction block, and generating the final prediction block may include generating the final prediction block for the current block based on a superposition of the base prediction block, the first additional prediction block, and the second additional prediction block.

[0021] A decoding method according to one aspect of the present disclosure comprises encoding information about a base prediction for generating a base prediction block by performing a base prediction; encoding information about an additional prediction for generating an additional prediction block by performing an additional prediction; and encoding merging information for generating a final prediction block for a current block based on merging of the base prediction block and the additional prediction block, wherein the base prediction includes any one of an intra prediction, an inter prediction, or an intra block copy (IBC), and the additional prediction may include at least one of an intra prediction, an inter prediction, or an intra block copy.

[0022] A method for transmitting image information according to one aspect of the present disclosure includes encoding information about a base prediction for generating a base prediction block by performing a base prediction; encoding information about an additional prediction for generating an additional prediction block by performing an additional prediction; encoding combining information for generating a final prediction block for a current block based on combining the base prediction block and the additional prediction block, and transmitting image information including the encoded information about the base prediction, the encoded information about the additional prediction, and the encoded combining information, wherein the base prediction includes one of an intra prediction, an inter prediction, and an intra block copy (IBC), and the additional prediction may include at least one of an intra prediction, an inter prediction, and an intra block copy.

[0023] According to the present disclosure, a method and device for performing inter prediction based on a multi-reference mode can be provided.

[0024] According to the present disclosure, a signaling method and device for determining a multiple reference mode can be provided.

[0025] According to the present disclosure, a derivation method and device for determining a multiple reference mode can be provided.

[0026] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.

[0027] FIG. 1 illustrates a video / image coding system according to the present disclosure.

[0028] FIG. 2 is a schematic block diagram of an encoding device to which an embodiment of the present disclosure can be applied and in which encoding of a video / image signal is performed.

[0029] FIG. 3 is a schematic block diagram of a decoding device to which an embodiment of the present disclosure can be applied and in which decoding of a video / image signal is performed.

[0030] FIG. 4 illustrates an example of an intra-template matching prediction-based video / image encoding method to which an embodiment of the present disclosure can be applied.

[0031] FIG. 5 illustrates an example of an inter-prediction based video / image encoding method to which an embodiment of the present disclosure can be applied.

[0032] FIG. 6 illustrates an example of an inter-prediction based video / image decoding method to which an embodiment of the present disclosure can be applied.

[0033] FIG. 7 exemplarily illustrates an inter prediction procedure to which an embodiment of the present disclosure can be applied.

[0034] FIG. 8 is a diagram illustrating a spatial candidate used in an inter prediction procedure to which an embodiment of the present disclosure can be applied.

[0035] FIG. 9 is a diagram for explaining affine motion prediction used in an inter prediction procedure to which an embodiment of the present disclosure can be applied.

[0036] FIG. 10 is a diagram for explaining template matching to which an embodiment of the present disclosure can be applied.

[0037] FIG. 11 illustrates an inter prediction method performed by a decoding device according to one embodiment of the present disclosure.

[0038] FIG. 12 is a diagram illustrating a reference block used in a multi-reference mode according to one embodiment of the present disclosure.

[0039] FIG. 13 is a diagram illustrating a method for signaling information about an additional reference block used in a multi-reference mode according to one embodiment of the present disclosure.

[0040] FIG. 14 is a diagram illustrating a combination between a mode of a basic block and a mode of an additional reference block according to one embodiment of the present disclosure.

[0041] FIG. 15 is a diagram illustrating a method for signaling information about an additional reference block used in a multi-reference mode according to one embodiment of the present disclosure.

[0042] FIG. 16 is a diagram illustrating a method for deriving additional reference blocks based on template matching used in a multi-reference mode according to one embodiment of the present disclosure.

[0043] FIG. 17 is a diagram illustrating a method for deriving additional reference blocks based on intra-template matching prediction used in a multi-reference mode according to one embodiment of the present disclosure.

[0044] FIG. 18 is a diagram illustrating a method for deriving additional reference blocks based on TIMD used in a multi-reference mode according to one embodiment of the present disclosure.

[0045] FIG. 19 is a diagram illustrating a method for calculating an error (or cost) for deriving an additional reference block in a multi-reference mode according to one embodiment of the present disclosure.

[0046] FIG. 20 illustrates a decoding method according to one embodiment of the present disclosure.

[0047] FIG. 21 illustrates an encoding method according to one embodiment of the present disclosure.

[0048] FIG. 22 is a diagram illustrating an example of a content streaming system to which an embodiment according to the present disclosure can be applied.

[0049] The present disclosure may be modified in various ways and encompasses numerous embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present disclosure to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present disclosure. Throughout the description of each drawing, similar reference numerals have been used to designate similar components.

[0050] While terms such as "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present disclosure, a first component could be referred to as a "second component," and similarly, a second component could also be referred to as a "first component." The term "and / or" includes a combination of multiple related items described herein or any of multiple related items described herein.

[0051] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0052] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present disclosure. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0053] The present disclosure relates to video / image coding. For example, the methods / embodiments disclosed in this specification can be applied to methods disclosed in the versatile video coding (VVC) standard. In addition, the methods / embodiments disclosed in this specification can be applied to methods disclosed in the essential video coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the second generation of audio video coding standard (AVS2), or the next generation of video / image coding standards (e.g., H.267 or H.268).

[0054] This specification presents various embodiments of video / image coding, and unless otherwise stated, the embodiments may be performed in combination with each other.

[0055] In this specification, a video may refer to a set of images over time. A picture generally refers to a unit representing one image at a specific time point, and a slice / tile is a unit that constitutes part of a picture in coding. A slice / tile may include one or more coding tree units (CTUs). A picture may be composed of one or more slices / tiles. A tile is a rectangular area consisting of multiple CTUs within a specific tile column and a specific tile row of a picture. A tile column is a rectangular area of ​​CTUs that has a height equal to the height of the picture and a width specified by the syntax requirements of the picture parameter set. A tile row is a rectangular area of ​​CTUs that has a height specified by the picture parameter set and a width equal to the width of the picture. CTUs within a tile are arranged consecutively according to the CTU raster scan, while tiles within a picture may be arranged consecutively according to the tile raster scan. A slice may contain an integer number of complete tiles or an integer number of contiguous complete CTU rows within a picture, which may be exclusively contained within a single NAL unit. Meanwhile, a picture may be divided into two or more subpictures. A subpicture may be a rectangular region of one or more slices within a picture.

[0056] A pixel, or pel, can refer to the smallest unit that constitutes a picture (or image). Additionally, the term "sample" can be used as a counterpart to a pixel. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luminance component, or only the pixel / pixel value of the chrominance component.

[0057] A unit may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. One unit may include one luma block and two chroma (e.g., cb, cr) blocks. In some cases, the term "unit" may be used interchangeably with terms such as "block" or "area." In general, an MxN block may include a set (or array) of samples (or sample array) or transform coefficients consisting of M columns and N rows.

[0058] As used herein, “A or B” can mean “only A,” “only B,” or “both A and B.” In other words, as used herein, “A or B” can be interpreted as “A and / or B.” For example, as used herein, “A, B or C” can mean “only A,” “only B,” “only C,” or “any combination of A, B, and C.”

[0059] As used herein, a slash ( / ) or a comma can mean "and / or." For example, "A / B" can mean "A and / or B." Accordingly, "A / B" can mean "only A," "only B," or "both A and B." For example, "A, B, C" can mean "A, B, or C."

[0060] In this specification, “at least one of A and B” may mean “only A,” “only B,” or “both A and B.” Additionally, in this specification, the expressions “at least one of A or B” or “at least one of A and / or B” may be interpreted identically to “at least one of A and B.”

[0061] Additionally, in this specification, “at least one of A, B and C” can mean “only A,” “only B,” “only C,” or “any combination of A, B and C.” Additionally, “at least one of A, B or C” or “at least one of A, B and / or C” can mean “at least one of A, B and C.”

[0062] Additionally, parentheses used herein may mean "for example." Specifically, when "prediction (intra-prediction)" is indicated, "intra-prediction" may be suggested as an example of "prediction." In other words, "prediction" in this specification is not limited to "intra-prediction," and "intra-prediction" may be suggested as an example of "prediction." Furthermore, even when "prediction (i.e., intra-prediction)" is indicated, "intra-prediction" may be suggested as an example of "prediction."

[0063] Technical features individually described in a single drawing in this specification may be implemented individually or simultaneously.

[0064] FIG. 1 illustrates a video / image coding system according to the present disclosure.

[0065] Referring to FIG. 1, a video / image coding system may include a first device (source device) and a second device (receiving device).

[0066] A source device can transmit encoded video / image information or data to a receiving device via a digital storage medium or a network in the form of a file or streaming. The source device may include a video source, an encoding device, and a transmitting device. The receiving device may include a receiving device, a decoding device, and a renderer. The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, and the display unit may be configured as a separate device or an external component.

[0067] A video source may obtain video / images through a process of capturing, synthesizing, or generating video / images. The video source may include a video / image capture device and / or a video / image generation device. The video / image capture device may include one or more cameras, a video / image archive containing previously captured video / images, etc. The video / image generation device may include a computer, a tablet, a smartphone, etc., and may (electronically) generate video / images. For example, a virtual video / image may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process of generating related data.

[0068] An encoding device can encode input video / images. The encoding device can perform a series of procedures, such as prediction, transformation, and quantization, to improve compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0069] The transmission unit can transmit encoded video / image information or data output in the form of a bitstream to the receiving unit of a receiving device via a digital storage medium or network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit can include an element for generating a media file via a predetermined file format and an element for transmission via a broadcasting / communication network. The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0070] The decoding device can decode the video / image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoding device.

[0071] The renderer can render decoded video / images. The rendered video / images can be displayed through the display unit.

[0072] FIG. 2 is a schematic block diagram of an encoding device to which an embodiment of the present disclosure can be applied and in which encoding of a video / image signal is performed.

[0073] Referring to FIG. 2, the encoding device (200) may be configured to include an image partitioner (210), a prediction unit (predictor) 220, a residual processor (residual processor) 230, an entropy encoder (entropy encoder) 240, an adder (adder) 250, a filter (filter) 260, and a memory (memory) 270. The prediction unit (220) may include an inter prediction unit (221) and an intra prediction unit (222). The residual processor (230) may include a transformer (transformer) 232, a quantizer (quantizer) 233, a dequantizer (dequantizer) 234, and an inverse transformer (inverse transformer) 235. The residual processing unit (230) may further include a subtractor (231). The addition unit (250) may be called a reconstructor or a recontructed block generator. The image segmentation unit (210), the prediction unit (220), the residual processing unit (230), the entropy encoding unit (240), the addition unit (250), and the filtering unit (260) described above may be configured by one or more hardware components (e.g., an encoding device chipset or processor) according to an embodiment. In addition, the memory (270) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (270) as an internal / external component.

[0074] The image segmentation unit (210) can segment an input image (or picture, frame) input to the encoding device (200) into one or more processing units. For example, the processing unit may be called a coding unit (CU). In this case, the coding unit may be recursively segmented from a coding tree unit (CTU) or a largest coding unit (LCU) according to a QTBTTT (Quad-tree binary-tree ternary-tree) structure.

[0075] For example, a single coding unit may be split into multiple coding units with deeper depths based on a quad-tree structure, a binary tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure may be applied first, and the binary tree structure and / or the ternary structure may be applied later. Alternatively, the binary tree structure may be applied before the quad-tree structure. The coding procedure according to the present specification may be performed based on the final coding unit that is no longer split. In this case, based on coding efficiency according to image characteristics, etc., the largest coding unit may be used directly as the final coding unit, or, if necessary, the coding unit may be recursively split into coding units of lower depths, and the coding unit with the optimal size may be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration, which will be described later.

[0076] As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may each be split or partitioned from the final coding unit described above. The prediction unit may be a unit of sample prediction, and the transform unit may be a unit for deriving a transform coefficient and / or a unit for deriving a residual signal from a transform coefficient.

[0077] The term "unit" may be used interchangeably with terms such as "block" or "area" depending on the case. In general, an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luminance component, or only the pixel / pixel value of the chrominance component. A sample can be used as a term corresponding to a pixel or pel in a picture (or image).

[0078] The encoding device (200) can generate a residual signal (residual block, residual sample array) by subtracting a prediction signal (prediction block, prediction sample array) output from an inter prediction unit (221) or an intra prediction unit (222) from an input video signal (original block, original sample array), and the generated residual signal is transmitted to a conversion unit (232). In this case, a unit that subtracts a prediction signal (prediction block, prediction sample array) from an input video signal (original block, original sample array) within the encoding device (200) may be called a subtraction unit (231).

[0079] The prediction unit (220) can perform a prediction on a block to be processed (hereinafter, referred to as a current block) and generate a predicted block including prediction samples for the current block. The prediction unit (220) can determine whether intra prediction or inter prediction is applied on a current block or CU basis. The prediction unit (220) can generate various information related to prediction, such as prediction mode information, as described later in the description of each prediction mode, and transmit the information to the entropy encoding unit (240). The information related to prediction can be encoded by the entropy encoding unit (240) and output in the form of a bitstream.

[0080] The intra prediction unit (222) can predict the current block by referring to samples in the current picture. The referenced samples may be located in the neighborhood of the current block, or may be located a certain distance away from the current block, depending on the prediction mode. In intra prediction, the prediction modes may include one or more non-directional modes and multiple directional modes. The non-directional mode may include at least one of a DC mode or a planar mode. The directional mode may include 33 directional modes or 65 directional modes depending on the degree of detail in the prediction direction. However, this is only an example, and a greater or lesser number of directional modes may be used depending on the settings. The intra prediction unit (222) may also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0081] The inter prediction unit (221) can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, subblocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring block can include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The above temporal neighboring blocks may be called collocated reference blocks, collocated CUs (colCUs), etc., and the reference pictures including the temporal neighboring blocks may be called collocated pictures (colPic). For example, the inter prediction unit (221) may construct a motion information candidate list based on the neighboring blocks, and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, and for example, in the case of skip mode and merge mode, the inter prediction unit (221) may use the motion information of the neighboring blocks as the motion information of the current block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of the surrounding blocks is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0082] The prediction unit (220) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction for prediction of a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP) mode. In addition, the prediction unit can be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described herein. Palette mode can be viewed as an example of intra coding or intra prediction. When the palette mode is applied, sample values ​​within a picture can be signaled based on information about the palette table and palette index. The prediction signal generated through the prediction unit (220) can be used to generate a restoration signal or a residual signal.

[0083] The transform unit (232) can apply a transform technique to the residual signal to generate transform coefficients. For example, the transform technique can include at least one of a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Karhunen-Loeve Transform (KLT), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transform obtained from a graph when the relationship information between pixels is expressed as a graph. CNT refers to a transform obtained based on generating a prediction signal using all previously restored pixels. In addition, the transform process can be applied to a pixel block having a square size and the same size, or can be applied to a block of a non-square variable size.

[0084] The quantization unit (233) quantizes the transform coefficients and transmits them to the entropy encoding unit (240), and the entropy encoding unit (240) can encode the quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients can be called residual information. The quantization unit (233) can rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on the coefficient scan order, and can also generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form.

[0085] The entropy encoding unit (240) can perform various encoding methods such as exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc. The entropy encoding unit (240) can also encode information necessary for video / image restoration (e.g., values ​​of syntax elements, etc.) together or separately from quantized transform coefficients.

[0086] Encoded information (e.g., encoded video / image information) can be transmitted or stored in the form of a bitstream in units of NAL (network abstraction layer) units. The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. In the present specification, information and / or syntax elements transmitted / signaled from an encoding device to a decoding device may be included in the video / image information. The video / image information may be encoded through the above-described encoding procedure and included in the bitstream. The bitstream may be transmitted via a network or stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media, such as a USB, SD, CD, DVD, Blu-ray, HDD, or SSD. The signal output from the entropy encoding unit (240) may be configured as an internal / external element of the encoding device (200) by a transmitting unit (not shown) and / or a storing unit (not shown), or the transmitting unit may be included in the entropy encoding unit (240).

[0087] The quantized transform coefficients output from the quantization unit (233) can be used to generate a prediction signal. For example, by applying inverse quantization and inverse transformation to the quantized transform coefficients through the inverse quantization unit (234) and the inverse transform unit (235), a residual signal (residual block or residual samples) can be reconstructed. The addition unit (250) can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter prediction unit (221) or the intra prediction unit (222). When there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as a reconstructed block. The addition unit (250) may be called a reconstructor or a reconstructed block generation unit. The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, and can also be used for inter prediction of the next picture after filtering as described below. Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.

[0088] The filtering unit (260) can improve subjective / objective picture quality by applying filtering to the restoration signal. For example, the filtering unit (260) can apply various filtering methods to the restoration picture to generate a modified restoration picture, and store the modified restoration picture in the memory (270), specifically, in the DPB of the memory (270). The various filtering methods can include deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit (260) can generate various information regarding filtering and transmit it to the entropy encoding unit (240). The information regarding filtering can be encoded by the entropy encoding unit (240) and output in the form of a bitstream.

[0089] The modified restored picture transmitted to the memory (270) can be used as a reference picture in the inter prediction unit (221). Through this, when inter prediction is applied, the encoding device can avoid prediction mismatch between the encoding device (200) and the decoding device, and can also improve encoding efficiency.

[0090] The DPB of the memory (270) can store the modified restored picture to be used as a reference picture in the inter prediction unit (221). The memory (270) can store motion information of a block from which motion information in the current picture is derived (or encoded) and / or motion information of blocks in a picture that has already been restored. The stored motion information can be transferred to the inter prediction unit (221) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (270) can store restored samples of restored blocks in the current picture and transfer them to the intra prediction unit (222).

[0091] FIG. 3 is a schematic block diagram of a decoding device to which an embodiment of the present disclosure can be applied and in which decoding of a video / image signal is performed.

[0092] Referring to FIG. 3, the decoding device (300) may be configured to include an entropy decoder (310), a residual processor (320), a predictor (330), an adder (340), a filter (350), and a memory (360). The predictor (330) may include an inter-prediction unit (332) and an intra-prediction unit (331). The residual processor (320) may include a dequantizer (321) and an inverse transformer (321).

[0093] The entropy decoding unit (310), residual processing unit (320), prediction unit (330), addition unit (340), and filtering unit (350) described above may be configured by a single hardware component (e.g., a decoding device chipset or processor) depending on the embodiment. In addition, the memory (360) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (360) as an internal / external component.

[0094] When a bitstream including video / image information is input, the decoding device (300) can restore the image corresponding to the process in which the video / image information is processed in the encoding device of FIG. 2. For example, the decoding device (300) can derive units / blocks based on block division-related information obtained from the bitstream. The decoding device (300) can perform decoding using a processing unit applied in the encoding device. Accordingly, the processing unit of decoding may be a coding unit, and the coding unit may be divided from a coding tree unit or a maximum coding unit according to a quad tree structure, a binary tree structure, and / or a ternary tree structure. One or more transform units may be derived from the coding unit. Then, the restored image signal decoded and output through the decoding device (300) can be reproduced through a reproduction device.

[0095] The decoding device (300) can receive a signal output from the encoding device of FIG. 2 in the form of a bitstream, and the received signal can be decoded through the entropy decoding unit (310). For example, the entropy decoding unit (310) can parse the bitstream to derive information (e.g., video / image information) necessary for image restoration (or picture restoration). The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The decoding device can decode the picture further based on the information on the parameter set and / or the general constraint information. The signaling / received information and / or syntax elements described later in this specification can be decoded through the decoding procedure and obtained from the bitstream. For example, the entropy decoding unit (310) can decode information in a bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the values ​​of syntax elements required for image restoration and the quantized values ​​of transform coefficients for residuals. More specifically, the CABAC entropy decoding method receives a bin corresponding to each syntax element in the bitstream, determines a context model using information of the syntax element to be decoded and decoding information of the surrounding and decoding target blocks or information of symbols / bins decoded in the previous step, and predicts the occurrence probability of the bin according to the determined context model to perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element.At this time, the CABAC entropy decoding method can update the context model using the information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Information regarding prediction among the information decoded by the entropy decoding unit (310) is provided to the prediction unit (inter prediction unit (332) and intra prediction unit (331)), and residual values ​​on which entropy decoding is performed by the entropy decoding unit (310), i.e., quantized transform coefficients and related parameter information, can be input to the residual processing unit (320). The residual processing unit (320) can derive a residual signal (residual block, residual samples, residual sample array). In addition, information regarding filtering among the information decoded by the entropy decoding unit (310) can be provided to the filtering unit (350). Meanwhile, a receiving unit (not shown) that receives a signal output from an encoding device may be further configured as an internal / external element of a decoding device (300), or the receiving unit may be a component of an entropy decoding unit (310).

[0096] Meanwhile, a decoding device according to the present specification may be called a video / video / picture decoding device, and the decoding device may be divided into an information decoding device (video / video / picture information decoding device) and a sample decoding device (video / video / picture sample decoding device). The information decoding device may include the entropy decoding unit (310), and the sample decoding device may include at least one of the inverse quantization unit (321), the inverse transformation unit (322), the addition unit (340), the filtering unit (350), the memory (360), the inter prediction unit (332), and the intra prediction unit (331).

[0097] The inverse quantization unit (321) can inverse quantize the quantized transform coefficients and output the transform coefficients. The inverse quantization unit (321) can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scanning order performed in the encoding device. The inverse quantization unit (321) can perform inverse quantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain transform coefficients.

[0098] In the inverse transform unit (322), the transform coefficients are inversely transformed to obtain a residual signal (residual block, residual sample array).

[0099] The prediction unit (320) can perform a prediction on the current block and generate a predicted block including prediction samples for the current block. The prediction unit (320) can determine whether intra-prediction or inter-prediction is applied to the current block based on the information regarding the prediction output from the entropy decoding unit (310), and can determine a specific intra / inter-prediction mode.

[0100] The prediction unit (320) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit (320) can apply intra prediction or inter prediction for prediction of a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP) mode. In addition, the prediction unit can be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described herein. Palette mode can be viewed as an example of intra coding or intra prediction. When palette mode is applied, information about the palette table and palette index may be included and signaled in the video / image information.

[0101] The intra prediction unit (331) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block, or may be located a certain distance away from the current block, depending on the prediction mode. In intra prediction, the prediction modes may include one or more non-directional modes and multiple directional modes. The intra prediction unit (331) may also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0102] The inter prediction unit (332) can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks can include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (332) can construct a motion information candidate list based on the neighboring blocks, and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter prediction can be performed based on various prediction modes, and information about the prediction can include information indicating an inter prediction mode for the current block.

[0103] The addition unit (340) can generate a restoration signal (restored picture, restoration block, restoration sample array) by adding the acquired residual signal to the prediction signal (prediction block, prediction sample array) output from the prediction unit (including the inter-prediction unit (332) and / or intra-prediction unit (331)). When there is no residual for the block to be processed, such as when skip mode is applied, the prediction block can be used as the restoration block.

[0104] The addition unit (340) may be referred to as a restoration unit or restoration block generation unit. The generated restoration signal may be used for intra prediction of the next processing target block within the current picture, may be output after filtering as described below, or may be used for inter prediction of the next picture. Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0105] The filtering unit (350) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (350) can apply various filtering methods to the restored picture to generate a modified restored picture, and transmit the modified restored picture to the memory (360), specifically, to the DPB of the memory (360). The various filtering methods can include deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0106] The (modified) reconstructed picture stored in the DPB of the memory (360) can be used as a reference picture in the inter prediction unit (332). The memory (360) can store motion information of a block from which motion information is derived (or decoded) in the current picture and / or motion information of blocks in an already reconstructed picture. The stored motion information can be transmitted to the inter prediction unit (332) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (360) can store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra prediction unit (331).

[0107] In this specification, the embodiments described in the filtering unit (260), the inter prediction unit (221), and the intra prediction unit (222) of the encoding device (200) can be applied to the filtering unit (350), the inter prediction unit (332), and the intra prediction unit (331) of the decoding device (300) in the same or corresponding manner, respectively.

[0108] Intra prediction

[0109] The prediction unit of the encoding device / decoding device can derive a reference sample according to the intra prediction mode of the current block among surrounding reference samples of the current block, and can generate a prediction sample of the current block based on the reference sample.

[0110] For example, the prediction unit of the encoding device / decoding device may (i) derive a prediction sample based on an average or interpolation of neighboring reference samples of the current block, and (ii) derive the prediction sample based on a reference sample existing in a specific (prediction) direction with respect to the prediction sample among the neighboring reference samples of the current block. The case of (i) may be called a non-directional mode or a non-angular mode, and the case of (ii) may be called a directional mode or an angular mode.

[0111] Additionally, the prediction unit of the encoding device / decoding device may generate the prediction sample by interpolating the second surrounding sample and the first surrounding sample, which are located in the opposite direction of the prediction direction of the intra prediction mode of the current block based on the prediction sample of the current block among the surrounding reference samples. The above-described case may be referred to as linear interpolation intra prediction (LIP).

[0112] Additionally, the prediction unit of the encoding device / decoding device may derive a temporary prediction sample of the current block based on filtered peripheral reference samples, and may derive a prediction sample of the current block by weighting the temporary prediction sample with at least one reference sample derived according to the intra prediction mode among the existing peripheral reference samples, i.e., unfiltered peripheral reference samples. The above-described case may be referred to as PDPC (Position dependent intra prediction).

[0113] In addition, the prediction unit of the encoding device / decoding device may perform intra prediction encoding by selecting a reference sample line with the highest prediction accuracy among the surrounding multiple reference sample lines of the current block, deriving a prediction sample using the reference sample located in the prediction direction in the corresponding line, and then instructing (signaling) the used reference sample line to the decoding device. The above-described case may be called multi-reference line intra prediction (MRL) or MRL-based intra prediction.

[0114] In addition, the prediction unit of the encoding device / decoding device may divide the current block into vertical or horizontal subpartitions and perform intra prediction based on the same intra prediction mode, and may derive and utilize surrounding reference samples for each subpartition. That is, in this case, the intra prediction mode for the current block is applied equally to the subpartitions, and by deriving and utilizing surrounding reference samples for each subpartition, the intra prediction performance may be improved in some cases. This prediction method may be called intra sub-partitions (ISP) or ISP-based intra prediction.

[0115] Additionally, if the prediction direction based on the prediction sample points between surrounding reference samples, i.e., if the prediction direction points to a fractional sample location, the value of the prediction sample can be derived through interpolation of multiple reference samples located around the prediction direction (around the fractional sample location).

[0116] The intra prediction methods described above may be referred to as intra prediction types, distinct from intra prediction modes. The intra prediction types may be referred to by various terms, such as intra prediction techniques or supplementary intra prediction modes. For example, the intra prediction types (or supplementary intra prediction modes, etc.) may include at least one of the LIP, PDPC, MRL, and ISP described above.

[0117] Information regarding the above intra prediction type can be encoded by an encoding device, included in a bitstream, and signaled to a decoding device. Information regarding the above intra prediction type can be implemented in various forms, such as flag information indicating whether each intra prediction type is applied or index information indicating one of several intra prediction types.

[0118] The MPM (Most Probable Mode) list for deriving the intra prediction mode described above may be configured differently depending on the intra prediction type. Alternatively, the MPM list may be configured in common regardless of the intra prediction type.

[0119] Furthermore, in decoder-side intra-mode derivation (DIMD), intra prediction is derived as a weighted average between a planar predictor and two derived directional predictors. Two angular modes are selected from the Histogram of Gradients (HoG) calculated from adjacent pixels of the current block. Once the two modes are selected, their corresponding predictors and planar predictors are derived, and the weighted average between blocks is then used as the final predictor. The corresponding amplitudes of the HoG for each mode are used to determine the weights.

[0120] The DIMD process is performed before the MPM list is constructed, as the derived intra mode is included in the default list of most probable intra MPMs. The default derived intra mode of a DIMD block is stored with the block and is used to construct the MPM list of adjacent blocks.

[0121] Additionally, for each intra prediction mode in the MPM, the sum of absolute transformed differences (SATD) between the predicted and reconstructed samples of the template is calculated. The first two intra prediction modes with the smallest SATD are selected as the TIMD modes. These two TIMD modes are weighted, and the weighted intra prediction is used to code the current CU. Position-dependent intra prediction combination (PDPC) is included in the derivation of the TIMD modes.

[0122] The costs of the two selected modes are compared against a threshold, and two cost factors satisfying Equation 1 are applied in the test:

[0123] [Mathematical Formula 1]

[0124] costMode2 < 2*costMode1.

[0125] If the condition in Equation 1 is true, fusion is applied, otherwise only Mode 1 is used.

[0126] The weights of the modes are calculated from the SATD cost as in Equation 2:

[0127] [Equation 2]

[0128] weight1 = costMode2 / (costMode1+ costMode2)

[0129] weight2 = 1 - weight1

[0130] FIG. 4 illustrates an example of an intra-template matching prediction-based video / image encoding method to which an embodiment of the present disclosure can be applied.

[0131] Intra-template matching prediction (IntraTMP) is a special intra-prediction mode that copies the optimal prediction block matching the current template and the L-shaped template from the reconstructed portion of the current frame. The encoder searches for the template most similar to the current template within a predefined search range and uses that block as the prediction block. The encoder then signals the use of this mode, and the decoder performs the same prediction operation.

[0132] The prediction signal is generated by matching the L-shaped causal neighborhood of the current block with other blocks in the predefined search region of Fig. 4, which is structured as follows:

[0133] R1: Current CTU

[0134] R2: Top left CTU

[0135] R3: Upper CTU

[0136] R4: Left CTU

[0137] SAD (Sum of absolute differences) is used as the cost function.

[0138] Within each region, the decoder searches for the template with the smallest SAD compared to the current template and uses that block as the predicted block.

[0139] The dimensions of all regions (SearchRange_w, SearchRange_h) are set proportionally to the block dimensions (BlkW, BlkH) to ensure a fixed number of SAD comparisons per pixel, as in Equation 3:

[0140] [Equation 3]

[0141] SearchRange_w = a * BlkW

[0142] SearchRange_h = a * BlkH

[0143] Here, 'a' is a constant that controls the gain / complexity tradeoff. In practice, 'a' can be equal to 5.

[0144] By subsampling the search positions within each search area within the template by two, the number of calculations in the template matching process can be reduced by a quarter. After finding the optimal match, a refinement process is performed. This refinement is achieved through a second template matching search centered on the optimal match with a reduced range. The reduced range is defined as min(BlkW, BlkH) / 2.

[0145] The intra-template matching tool is enabled for CUs with a width and height of 64 or less. The size of this maximum coding unit for intra-template matching is configurable.

[0146] Intra template matching prediction mode is signaled at the coding unit level via a dedicated flag if DIMD is not used in the current coding unit.

[0147] Block vectors (BVs) derived from intra-template matching prediction (IntraTMP) can be used for intra-block copy (IBC). The IntraTMP block vectors (BVs) of stored adjacent blocks are used as spatial block vector (BV) candidates in constructing the IBC candidate list, along with the IBC block vectors (BVs).

[0148] The IntraTMP block vector is stored in the IBC block vector buffer, and the current IBC block can use both the IBC block vector (BV) and the IntraTMP block vector (BV) of the adjacent block as block vector (BV) candidates for the IBC block vector (BV) candidate list.

[0149] Intra block copy (IBC)

[0150] Intra-block copy (IBC) is well known to significantly improve the coding efficiency of screen content. Because IBC mode is implemented as a block-level coding mode, the encoder performs block matching (BM) to find the optimal block vector (or motion vector) for each CU. The block vector is used to indicate the displacement from the current block to a reference block already reconstructed within the current image. The luma block vector of an IBC-coded coding unit has integer precision. The chroma block vector is also rounded to integer precision. When combined with AMVR, IBC mode can switch between 1-pixel and 4-pixel motion vector precision. IBC-coded coding units are treated as a third prediction mode other than intra- or inter-prediction modes. IBC mode can be applied to coding units with both a width and height less than 64 luma samples.

[0151] On the encoder side, hash-based motion estimation is performed for IBC. The encoder performs RD checks on blocks whose width or height does not exceed 16 luma samples. If not in merge mode, a block vector search is performed first using a hash-based search. If the hash search does not return a valid candidate, a block-matching-based local search is performed.

[0152] In hash-based search, the hash key matching (32-bit CRC) between the current block and reference blocks is extended to all allowed block sizes. The hash key calculation for each location in the current picture is based on 4x4 sub-blocks. For larger current blocks, a reference block is considered to match a hash key if the hash keys of all 4x4 sub-blocks match the hash keys of the corresponding reference locations. If the hash keys of multiple reference blocks are found to match the hash key of the current block, the block vector cost of each matching reference is calculated to select the block with the lowest cost.

[0153] In block matching search, the search range is set to include both previous and current CTUs.

[0154] At the coding unit level, IBC mode is signaled with a flag, which can be either IBC AMVP mode or IBC skip / merge mode as follows:

[0155] - IBC Skip / Merge Mode: The merge candidate index is used to indicate which block vector in the list of adjacent candidate IBC coding blocks is used to predict the current block. The merge list consists of spatial, HMVP, and pairwise candidates.

[0156] - IBC AMVP mode: Block vector differences are coded in the same way as motion vector differences. The block vector prediction method uses two candidates as predictors: one from the left neighbor and one from the upper neighbor (if IBC coded). If one of the two neighbors is unavailable, the base block vector is used as the predictor. A flag indicating the block vector predictor index is signaled.

[0157] Inter prediction

[0158] Meanwhile, when inter prediction is applied, the prediction unit of the encoding device / decoding device can perform inter prediction on a block-by-block basis to derive prediction samples. Inter prediction can refer to a prediction derived in a manner dependent on data elements (e.g., sample values, or motion information) of pictures other than the current picture. When inter prediction is applied to the current block, a predicted block (prediction sample array) for the current block can be derived based on a reference block (reference sample array) specified by a motion vector on a reference picture pointed to by a reference picture index.

[0159] At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information of the current block can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information may include a motion vector and / or a reference picture index. The motion information may further include information on the inter prediction type (L0 prediction, L1 prediction, Bi prediction, etc.). When inter prediction is applied, the neighboring blocks may include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture.

[0160] The reference picture including the above reference block and the reference picture including the temporal neighboring block may be the same or different. The temporal neighboring block may be called a collocated reference block, a collocated CU (colCU), etc., and the reference picture including the temporal neighboring block may be called a collocated picture (colPic). For example, a motion information candidate list may be constructed based on the neighboring blocks of the current block, and a flag or index information indicating which candidate is selected (used) to derive the motion vector and / or reference picture index of the current block may be signaled.

[0161] Inter prediction can be performed based on various prediction modes. For example, in the case of skip mode and merge mode, the motion information of the current block may be the same as the motion information of the selected neighboring block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted. In the case of motion vector prediction (MVP) mode, the motion vector of the selected neighboring block is used as a motion vector predictor, and the motion vector difference may be signaled. In this case, the motion vector of the current block can be derived using the sum of the motion vector predictor and the motion vector difference.

[0162] The above motion information may include L0 motion information and / or L1 motion information depending on the inter prediction type (L0 prediction, L1 prediction, Bi prediction, etc.). A motion vector in the L0 direction may be called an L0 motion vector or MVL0, and a motion vector in the L1 direction may be called an L1 motion vector or MVL1. Prediction based on an L0 motion vector may be called an L0 prediction, prediction based on an L1 motion vector may be called an L1 prediction, and prediction based on both the L0 motion vector and the L1 motion vector may be called a bi-prediction (Bi). Here, an L0 motion vector may represent a motion vector associated with a reference picture list L0 (L0), and an L1 motion vector may represent a motion vector associated with a reference picture list L1 (L1). The reference picture list L0 may include pictures preceding the current picture in output order as reference pictures, and the reference picture list L1 may include pictures succeeding the current picture in output order. The preceding pictures may be called forward (reference) pictures, and the succeeding pictures may be called backward (reference) pictures.

[0163] The above reference picture list L0 may further include pictures subsequent to the current picture in output order as reference pictures. In this case, the previous pictures may be indexed first and the subsequent pictures may be indexed next within the reference picture list L0. The above reference picture list L1 may further include pictures previous to the current picture in output order as reference pictures. In this case, the subsequent pictures may be indexed first and the previous pictures may be indexed next within the reference picture list L1. Here, the output order may correspond to a POC (picture order count) order.

[0164] A video / image encoding procedure based on inter prediction may roughly include, for example:

[0165] FIG. 5 illustrates an example of an inter-prediction based video / image encoding method to which an embodiment of the present disclosure can be applied.

[0166] An encoding device can generate prediction samples by performing inter prediction on a current block (S400). The encoding device can derive an inter prediction mode and motion information of the current block, and generate prediction samples of the current block. Here, the inter prediction mode determination, motion information derivation, and prediction sample generation procedures may be performed simultaneously, or one procedure may be performed before the other. For example, the inter prediction unit of the encoding device may include a prediction mode determination unit, a motion information derivation unit, and a prediction sample derivation unit, and the prediction mode determination unit may determine a prediction mode for the current block, the motion information derivation unit may derive motion information of the current block, and the prediction sample derivation unit may derive prediction samples of the current block.

[0167] For example, the inter prediction unit of the encoding device can search for a block similar to the current block within a certain area (search area) of reference pictures through motion estimation, and derive a reference block whose difference from the current block is minimal or below a certain standard. Based on this, a reference picture index indicating a reference picture where the reference block is located can be derived, and a motion vector can be derived based on the positional difference between the reference block and the current block. The encoding device can determine a mode to be applied to the current block among various prediction modes. The encoding device can compare RD costs for the various prediction modes and determine an optimal prediction mode for the current block.

[0168] For example, when skip mode or merge mode is applied to the current block, the encoding device may configure a merge candidate list described below, and derive a reference block among the reference blocks indicated by the merge candidates included in the merge candidate list, the difference from the current block being at least or below a certain standard. In this case, a merge candidate associated with the derived reference block is selected, and merge index information indicating the selected merge candidate may be generated and signaled to a decoding device. Motion information of the current block may be derived using motion information of the selected merge candidate.

[0169] As another example, when the (A)MVP mode is applied to the current block, the encoding device may configure an (A)MVP candidate list described below, and use the motion vector of a motion vector predictor (MVP) candidate selected from among motion vector predictor (MVP) candidates included in the (A)MVP candidate list as the motion vector predictor of the current block. In this case, for example, a motion vector pointing to a reference block derived by the above-described motion estimation may be used as the motion vector of the current block, and a motion vector predictor candidate having a motion vector with the smallest difference from the motion vector of the current block among the motion vector predictor candidates may become the selected motion vector predictor candidate. A motion vector difference (MVD), which is a difference obtained by subtracting the motion vector predictor from the motion vector of the current block, may be derived. In this case, information about the MVD may be signaled to a decoding device. Additionally, when the (A)MVP mode is applied, the value of the reference picture index may be configured as reference picture index information and signaled separately to the decoding device.

[0170] The encoding device can derive residual samples based on the above predicted samples (S410). The encoding device can derive the residual samples by comparing the original samples of the current block with the predicted samples.

[0171] An encoding device can encode image information including prediction information and residual information (S420). The encoding device can output the encoded image information in the form of a bitstream. The prediction information is information related to the prediction procedure and may include prediction mode information (e.g., a skip flag, a merge flag, or a merge index) and / or motion information. The motion information may include candidate selection information (e.g., a merge index, an MVP flag, or an MVP index), which is information for deriving a motion vector. In addition, the motion information may include information about the MVD described above and / or reference picture index information. In addition, the motion information may include information indicating whether L0 prediction, L1 prediction, or bi-prediction is applied. The residual information is information about the residual samples. The residual information may include information about quantized transform coefficients for the residual samples.

[0172] The output bitstream may be stored on a (digital) storage medium and transmitted to a decoding device, or may be transmitted to a decoding device via a network.

[0173] Meanwhile, as described above, the encoding device can generate a reconstructed picture (including reconstructed samples and reconstructed blocks) based on the reference samples and the residual samples. This is to derive the same prediction result as that performed by the decoding device from the encoding device, thereby improving coding efficiency. Accordingly, the encoding device can store the reconstructed picture (or reconstructed samples, reconstructed blocks) in memory and use it as a reference picture for inter prediction. As described above, an in-loop filtering procedure, etc. can be further applied to the reconstructed picture.

[0174] A video / image decoding procedure based on inter prediction may roughly include, for example:

[0175] FIG. 6 illustrates an example of an inter-prediction based video / image decoding method to which an embodiment of the present disclosure can be applied.

[0176] Referring to FIG. 6, the decoding device can perform an operation corresponding to the operation performed by the encoding device. The decoding device can perform a prediction on the current block based on the received prediction information and derive prediction samples.

[0177] Specifically, the decoding device can determine a prediction mode for the current block based on the received prediction information (S500). The decoding device can determine which inter-prediction mode is applied to the current block based on the prediction mode information within the prediction information.

[0178] For example, based on a merge flag, it can be determined whether the merge mode or the (A)MVP mode is applied to the current block. Alternatively, one of various inter prediction mode candidates can be selected based on a mode index. The inter prediction mode candidates can include skip mode, merge mode, and / or (A)MVP mode, or can include various inter prediction modes described below.

[0179] The decoding device can derive motion information of the current block based on the determined inter prediction mode (S510). For example, when skip mode or merge mode is applied to the current block, the decoding device can construct a merge candidate list described below and select one of the merge candidates included in the merge candidate list. The selection can be performed based on the above-described selection information (merge index). Motion information of the current block can be derived using motion information of the selected merge candidate. Motion information of the selected merge candidate can be used as motion information of the current block.

[0180] As another example, when the (A)MVP mode is applied to the current block, the decoding device may construct an (A)MVP candidate list described below, and use the motion vector of an MVP candidate selected from among the MVP (motion vector predictor) candidates included in the (A)MVP candidate list as the MVP of the current block. The selection may be performed based on the above-described selection information (mvp flag or mvp index). In this case, the MVD of the current block may be derived based on the information about the mvp, and the motion vector of the current block may be derived based on the mvp of the current block and the MVD. In addition, the reference picture index of the current block may be derived based on the reference picture index information. A picture indicated by the reference picture index within the reference picture list for the current block may be derived as a reference picture referenced for inter prediction of the current block.

[0181] Meanwhile, as described below, motion information of the current block can be derived without constructing a candidate list, in which case the motion information of the current block can be derived according to the procedure initiated in the prediction mode described below. In this case, constructing a candidate list as described above can be omitted.

[0182] The decoding device may generate prediction samples for the current block based on motion information of the current block (S520). In this case, the reference picture may be derived based on the reference picture index of the current block, and prediction samples of the current block may be derived using samples of a reference block pointed to by the motion vector of the current block on the reference picture. In this case, as described below, a prediction sample filtering procedure may be further performed on all or part of the prediction samples of the current block, depending on the case.

[0183] For example, the inter prediction unit of the decoding device may include a prediction mode determination unit, a motion information derivation unit, and a prediction sample derivation unit, and the prediction mode for the current block may be determined based on prediction mode information received from the prediction mode determination unit, motion information (such as a motion vector and / or a reference picture index) of the current block may be derived based on information about motion information received from the motion information derivation unit, and prediction samples of the current block may be derived from the prediction sample derivation unit.

[0184] The decoding device generates residual samples for the current block based on the received residual information (S530). The decoding device generates restoration samples for the current block based on the prediction samples and the residual samples, and can generate a restoration picture based on these (S540). As described above, an in-loop filtering procedure, etc., may be further applied to the restoration picture.

[0185] FIG. 7 exemplarily illustrates an inter prediction procedure to which an embodiment of the present disclosure may be applied. FIG. 8 is a diagram illustrating spatial candidates used in an inter prediction procedure to which an embodiment of the present disclosure may be applied. FIG. 9 is a diagram illustrating affine motion prediction used in an inter prediction procedure to which an embodiment of the present disclosure may be applied.

[0186] Referring to FIG. 7, the inter prediction procedure described above may include an inter prediction mode determination step, a motion information derivation step according to the determined prediction mode, and a prediction performance (prediction sample generation) step based on the derived motion information. The inter prediction procedure may be performed in an encoding device and a decoding device as described above. In this document, a coding device may include an encoding device and / or a decoding device.

[0187] Referring to FIG. 7, the coding device determines an inter prediction mode for a current block (S600). Various inter prediction modes can be used to predict the current block within a picture. For example, various modes such as merge mode, skip mode, motion vector prediction (MVP) mode, affine mode, sub-block merge mode, and merge with MVD (MMVD) mode can be used. Decoder side motion vector refinement (DMVR) mode, adaptive motion vector resolution (AMVR) mode, bi-prediction with CU-level weight (BCW), bi-directional optical flow (BDOF), etc. can be used as additional modes. In addition, according to one embodiment of the present disclosure, the above-described inter prediction mode can include a multi-hypethesis prediction (MHP) mode. The multi-hypethesis prediction mode represents a method of performing prediction by weighting prediction blocks generated based on additional motion information for bi-directional prediction (or bi-prediction) blocks. The multi-home prediction mode is described in detail below.

[0188] In the present disclosure, the affine mode may be referred to as the affine motion prediction mode. In addition, the MVP mode may be referred to as the advanced motion vector prediction (AMVP) mode. In the present disclosure, motion information candidates derived from some modes and / or some modes may be included as one of the motion information-related candidates of other modes. For example, an HMVP candidate may be added as a merge candidate of the merge / skip mode, or may be added as a motion vector predictor candidate of the AMVP mode. When the HMVP candidate is used as a motion information candidate of the merge mode or the skip mode, the HMVP candidate may be referred to as an HMVP merge candidate.

[0189] Prediction mode information indicating the inter-prediction mode of the current block can be signaled from the encoding device to the decoding device. The prediction mode information can be included in the bitstream and received by the decoding device. The prediction mode information can include index information indicating one of a plurality of candidate modes. Alternatively, the inter-prediction mode can be indicated through hierarchical signaling of flag information.

[0190] In this case, the prediction mode information may include one or more flags. For example, a skip flag may be signaled to indicate whether skip mode is applied, a merge flag may be signaled to indicate whether merge mode is applied when skip mode is not applied, and MVP mode may be indicated to be applied when merge mode is not applied, or a flag for additional distinction may be further signaled. Affine mode may be signaled as an independent mode, or as a mode dependent on merge mode or MVP mode. For example, affine mode may include affine merge mode and affine MVP mode.

[0191] The coding device can derive motion information for the current block (S610). The motion information can be derived based on the inter prediction mode. The coding device can perform inter prediction using the motion information of the current block. The encoding device can derive optimal motion information for the current block through a motion estimation procedure.

[0192] For example, an encoding device can use an original block within an original picture for a current block to search for a similar reference block with a high correlation within a predetermined search range within the reference picture in fractional pixel units, thereby deriving motion information. The similarity of blocks can be derived based on the difference in phase-based sample values. For example, the similarity of blocks can be calculated based on the SAD between the current block (or a template of the current block) and the reference block (or a template of the reference block). In this case, motion information can be derived based on the reference block with the smallest SAD within the search range. The derived motion information can be signaled to a decoding device in various ways based on an inter prediction mode.

[0193] The coding device can perform inter prediction based on motion information for the current block (S620). The coding device can generate prediction sample(s) for the current block based on the motion information. The current block including the prediction samples may be referred to as a predicted block.

[0194] When merge mode is applied, the motion information of the current prediction block is not directly transmitted, but rather the motion information of the surrounding prediction blocks is derived. Accordingly, the motion information of the current prediction block can be indicated by transmitting flag information indicating that merge mode is used and a merge index indicating which surrounding prediction block was used. The above merge mode may be referred to as the regular merge mode.

[0195] The encoder searches for merge candidate blocks used to derive motion information of the current prediction block to perform merge mode. For example, up to five merge candidate blocks may be used, but the present invention is not limited thereto. In addition, the maximum number of merge candidate blocks may be transmitted in the slice header or tile group header, but the present invention is not limited thereto. After finding the merge candidate blocks, the encoder can generate a merge candidate list, and select the merge candidate block with the lowest cost among them as the final merge candidate block.

[0196] For example, the merge candidate list above may use five merge candidate blocks. For example, four spatial merge candidates and one temporal merge candidate may be used. As a specific example, in the case of the spatial merge candidates, the blocks illustrated in FIG. 8 may be used as spatial merge candidates. Hereinafter, the spatial merge candidate or the spatial MVP (Motion Vector Prediction) candidate described below may be referred to as SMVP (Spatial Motion Vector Prediction), and the temporal merge candidate or the temporal MVP (Motion Vector Prediction) candidate described below may be referred to as TMVP (Temporal Motion Vector Prediction).

[0197] The MVP (Motion Vector Prediction) mode may be referred to as the AMVP (advanced motion vector prediction) mode. When the MVP mode is applied, a motion vector predictor (mvp) candidate list may be generated using the motion vectors of the reconstructed spatial neighboring blocks (e.g., the neighboring blocks in FIG. 8) and / or the motion vectors corresponding to the temporal neighboring blocks (or Col blocks). In other words, the motion vectors of the reconstructed spatial neighboring blocks and / or the motion vectors corresponding to the temporal neighboring blocks may be used as motion vector predictor candidates. When paired prediction is applied, an mvp candidate list for deriving L0 motion information and an mvp candidate list for deriving L1 motion information may be generated and used separately.

[0198] The above-described prediction information (or information regarding prediction) may include selection information (e.g., MVP flag or MVP index) indicating an optimal motion vector predictor candidate selected from among the motion vector predictor candidates included in the above list. In this case, the prediction unit may use the selection information to select a motion vector predictor of the current block from among the motion vector predictor candidates included in the motion vector candidate list.

[0199] The prediction unit of the encoding device can obtain a motion vector difference (MVD) between the motion vector of the current block and the motion vector predictor, and can encode this and output it in the form of a bitstream. In other words, the MVD can be obtained by subtracting the motion vector predictor from the motion vector of the current block.

[0200] At this time, the prediction unit of the decoding device can obtain the motion vector difference included in the information regarding the prediction, and derive the motion vector of the current block through the addition of the motion vector difference and the motion vector predictor. The prediction unit of the decoding device can obtain or derive a reference picture index indicating a reference picture, etc., from the information regarding the prediction.

[0201] Existing video coding systems use only a single motion vector to represent the motion of a coded block (i.e., a translation motion model). However, while the translation motion model may represent the optimal motion at the block level, it does not represent the optimal motion for each pixel. Determining the optimal motion vector at the pixel level can improve encoding efficiency.

[0202] To achieve this, affine motion prediction, a method that encodes using an affine motion model, is applied in affine mode. Affine motion prediction can express motion vectors at each pixel level of a block using two, three, or four motion vectors.

[0203] As illustrated in Fig. 9, the affine motion prediction method can determine the motion vector of the pixel position included in the block using two or more control point motion vectors (CPMV). At this time, the set of motion vectors is called an affine motion vector field (MVF).

[0204] A combination of inter prediction and intra prediction may be applied to the current block. An additional flag (e.g., ciip_flag) may be signaled to indicate whether the combined inter / intra prediction (CIIP) mode is applied to the current coding unit. For example, if the coding unit is coded in merge mode, and the coding unit contains more than 64 luma samples (i.e., the coding unit width times the coding unit height is greater than or equal to 64), and both the coding unit width and the coding unit height are less than 128 luma samples, an additional flag is signaled to indicate whether the CIIP mode is applied to the current coding unit. CIIP prediction combines the inter prediction signal and the intra prediction signal. The inter prediction signal P_inter of the CIIP mode is derived using the same inter prediction process as applied in the regular merge mode, and the intra prediction signal P_intra is derived together with the regular intra prediction process in the planar mode. Then, the intra and inter prediction signals are combined using a weighted average, where the weights are calculated according to the coding modes of the upper and left adjacent blocks as follows:

[0205] A. If the top neighbor is available and intra-coded, set isIntraTop to 1, otherwise set isIntraTop to 0;

[0206] B. If the left neighbor is available and intra-coded, set isIntraLeft to 1, otherwise set isIntraLeft to 0;

[0207] C. If (isIntraLeft + isIntraLeft) is 2, set wt to 3;

[0208] D. Otherwise, if (isIntraLeft + isIntraLeft) is 1, set wt to 2;

[0209] E. Otherwise, set wt to 1.

[0210] The CIIP prediction is structured as shown in Equation 4:

[0211] [Equation 4]

[0212]

[0213] FIG. 10 is a diagram for explaining template matching to which an embodiment of the present disclosure can be applied.

[0214] Template Matching (TM) is a decoder-side motion vector derivation method that embodies the motion information of the current coding unit by finding the closest match between a template (i.e., the top and / or left adjacent block of the current coding unit) in the current image and a block (i.e., of the same size as the template) in a reference image. As illustrated in Fig. 10, a better motion vector is searched around the initial motion of the current coding unit within a [-8, +8]-pel search range. The search step size is determined by the AMVR mode, and template matching can be cascaded with a bidirectional matching process in merge mode.

[0215] In AMVP mode, an MVP candidate is determined based on the template matching error, and the template that achieves the minimum difference between the current block template and the reference block template is selected. Template matching is then performed only on this specific MVP candidate to refine the motion vector. Template matching uses an iterative diamond search, starting from full-pel MVD precision (4-pel in 4-pel AMVR mode) within the [-8, +8]-pel search range, to refine this specific MVP candidate.

[0216] Depending on the AMVR mode, a cross-search is performed with full-pel MVD precision (4-pel in the 4-pel AMVR mode), followed by sequential half-pel and quarter-pel searches to further refine the AMVP candidate. This search process ensures that the MVP candidate maintains the same MVD precision as in the AMVR mode even after the TM process. The search process terminates when the difference between the previous minimum cost and the current minimum cost in an iteration is less than a threshold equal to the area of ​​the block.

[0217] [Table 1]

[0218]

[0219] In merge mode, a similar search method is applied to merge candidates indicated by merge indices. As shown in Table 1, template matching can be performed up to 1 / 8-pixel MVD precision or skipped above half-pixel MVD precision, depending on whether an alternative interpolation filter (used when AMVR is in half-pixel mode) is used based on the merged motion information. Furthermore, when template matching mode is enabled, template matching can operate as an independent process or as an additional motion enhancement process between block-based and sub-block-based bidirectional matching (BM) methods, depending on whether the activation status is checked.

[0220] In multi-hypothesis inter prediction (MHP) mode, one or more additional motion-compensated prediction signals are signaled in addition to the existing bi-prediction signal. Consequently, the overall prediction signal is obtained through sample-wise weighted superposition. Using the bi-prediction signal p_bi and the first additional prediction signal h_3, the resulting prediction signal p_3 is obtained as shown in Equation 5:

[0221] [Equation 5]

[0222]

[0223] The weight α can be specified with a new syntax element add_hyp_weight_idx according to the mapping in Table 2:

[0224] [Table 2]

[0225]

[0226] As shown in Equation 6, one or more additional prediction signals may be used. The final overall prediction signal is iteratively accumulated for each additional prediction signal.

[0227] [Equation 6]

[0228]

[0229] As a result, the entire prediction signal is obtained as the last p_n (i.e., the p_n with the largest index n). For example, up to two additional prediction signals can be used (e.g., n is limited to 2).

[0230] The motion parameters of each additional prediction hypothesis can be explicitly signaled by specifying a reference index, a motion vector prediction index, or a motion vector difference, or implicitly signaled by specifying a merge index. A separate multi-hypothesis merge flag can distinguish these two signaling modes.

[0231] For inter AMVP mode, MHP can be applied only when the weights of BCW (Bi-prediction with CU-level weight) in dual prediction mode are not equal.

[0232] Additionally, a combination of MHP and BDOF (Bi-directional optical flow) is possible, but BDOF can only be applied to the bidirectional prediction signal part of the prediction signal (i.e., the common first two hypotheses).

[0233] A predicted block for a current block can be derived based on motion information derived according to an inter prediction mode. The predicted block can include prediction samples (prediction sample array) of the current block. If the motion vector of the current block points to fractional sample units, an interpolation procedure can be performed, through which prediction samples of the current block can be derived based on reference samples of fractional sample units within a reference picture.

[0234] When Affine inter prediction is applied to the current block, prediction samples can be generated based on sample / subblock unit motion vectors (MVs).

[0235] When bi-prediction is applied, prediction samples derived from L0 prediction (i.e., prediction using reference pictures in the reference picture list L0 and MVL0) and L1 prediction (i.e., prediction using reference pictures in the reference picture list L1 and MVL1) can be used as prediction samples of the current block through a weighted sum or weighted average (Bi-prediction with CU-level weight, BCW) (according to phase). When bi-prediction is applied, if the reference pictures used for L0 prediction and the reference pictures used for L1 prediction are located in different temporal directions with respect to the current picture, this can be called bi-directional prediction.

[0236] Based on the derived prediction samples, restored samples and restored pictures can be generated, and then procedures such as in-loop filtering can be performed.

[0237] Hereinafter, a multi-hypothesis prediction mode will be described in detail. As described above, a multi-hypothesis prediction mode refers to a prediction method that uses an additional prediction block (or predictor) to a basic prediction block. The multi-hypothesis prediction mode can be selectively used as one of the various inter prediction modes described above. It goes without saying that the multi-hypothesis prediction mode according to an embodiment of the present disclosure is not limited to these names. In the present specification, the multi-hypothesis prediction mode may also be referred to as a multi-reference mode, a multi-reference prediction, a multi-reference prediction mode, a multi-reference block mode, a Multi-Hyperthesis Prediction (MHP) mode, a multi-hypothesis inter prediction mode, an inter-inter combined prediction mode, a combined inter prediction mode, a combined prediction mode, a multi-inter prediction mode, a multi-prediction mode, an additional reference prediction mode, an additional reference mode, a multi-reference block, etc.

[0238] Example

[0239] FIG. 11 illustrates a prediction method performed by a decoding device according to one embodiment of the present disclosure. The steps or operations illustrated in FIG. 11 are not essential components of the prediction method, and at least some of the steps or operations illustrated in FIG. 11 may be omitted.

[0240] Referring to FIG. 11, a decoding device may generate a base prediction block (or reference block) by performing unidirectional or bidirectional prediction (S700). When a multi-reference mode is applied, the decoding device may generate and combine additional reference blocks in addition to the base reference block generated (or derived) by unidirectional or bidirectional prediction. As an example, a base reference block may refer to a block referenced (or used) to generate (or derive) a base prediction block. As an example, a base reference block may include an L0 reference block and / or an L1 reference block. As an example, an existing prediction block may refer to a block generated (or derived) by combining or weighting basic reference blocks. As an example, a base prediction block may refer to a block obtained by weighting L0 and L1 prediction blocks. In the present disclosure, a base prediction block may be referred to as a base block, an initial prediction block, an initial block, a temporary prediction block, a temporary block, a regular prediction block, a regular block, etc. In this embodiment, the description is mainly based on the case where the basic prediction block is a block obtained by weighting L0 and L1 prediction blocks, but the present disclosure is not limited thereto.

[0241] In other words, in one embodiment, the video decoding device according to the present disclosure can derive a first reference block and a second reference block of the current block by performing bidirectional prediction, and generate a base prediction block by weighting the first reference block and the second reference block. Alternatively, in one embodiment, the video decoding device according to the present disclosure can generate a base prediction block by deriving a third reference block of the current block by performing unidirectional prediction. That is, the base prediction block may be derived by weighting a plurality of reference blocks, or may be derived using a single reference block.

[0242] The decoding device can derive (or generate) an additional reference block (or additional prediction block) based on a multiple reference mode (S710). The decoding device can derive the additional reference block in addition to the base prediction block, and combine (or weight the sum) the base prediction block and the derived additional reference block.

[0243] In one embodiment, when a multi-reference mode is applied, the decoding device may derive and combine additional reference blocks up to a predefined number. In other words, the decoding device may combine (or weight sum) additional reference blocks less than or equal to a predefined number to the base prediction block. For example, the predefined number may be 2. Alternatively, for example, the predefined number may be one of 1, 2, 3, and 4. The predefined number may be referred to as the maximum number of the multi-reference mode.

[0244] Additionally, when multiple additional reference blocks are combined, the multiple additional reference blocks may be sequentially weighted to the base prediction block. For example, when up to two additional reference blocks are generated, the base prediction block and the first additional reference block may be weighted to generate a prediction block, and the generated prediction block and the second additional reference block may be weighted to generate a final prediction block. The prediction block generated by weighting the base prediction block and the first additional reference block may be referred to as an intermediate prediction block.

[0245] Alternatively, when multiple additional reference blocks are combined, the base prediction block and the generated multiple additional reference blocks can be weighted together. That is, after the multiple additional reference blocks are generated, weights can be applied to each of the multiple additional reference blocks and the base prediction block (or L0 reference block and L1 reference block) and weighted together.

[0246] Additionally, as an example, the decoding device can determine whether to apply the multi-reference mode. In this case, a step for determining whether to apply the multi-reference mode may be added prior to step S710. As an example, whether to apply the multi-reference mode may be explicitly signaled or implicitly derived (or determined) by the decoding device.

[0247] In addition, as an embodiment, whether the multiple reference mode is applied can be signaled from the encoding device to the decoding device. For example, a multiple reference mode flag indicating whether the multiple reference mode is applied can be signaled from the encoding device to the decoding device. At this time, a condition for signaling / parsing the multiple reference mode flag can be predefined. The signaling / parsing condition of the multiple reference mode flag can be an availability condition of the multiple reference mode. If the signaling / parsing condition is satisfied, the decoding device can parse the multiple reference mode flag from the bitstream. Alternatively, as an embodiment, whether the multiple reference mode is applied can be derived by the decoding device based on predefined encoding information. As an example, whether the multiple reference mode is applied can be defined in the same way as the availability condition (or signaling / parsing condition) of the multiple reference mode described below.

[0248] In addition, as an embodiment, the decoding device may obtain multi-reference mode information (which may also be referred to as multi-reference mode prediction information) to generate an additional reference block. As an example, the multi-reference mode information may include weight information and / or prediction information. Based on the prediction information, a reference block according to the multi-reference mode, i.e., an additional reference block, may be derived, and the additional reference block derived based on the weight information may be weighted and combined with the base prediction block (or intermediate prediction block). In addition, as an example, the multi-reference mode information may further include a multi-reference mode flag indicating whether the multi-reference mode is applied.

[0249] For example, the prediction information may include mode information used to derive an additional reference block and motion information according to the mode. The mode information may be INTER mode information indicating whether it is a merge mode or an AMVP (or INTER) mode. For example, the mode information may be a merge flag. That is, either the merge mode or the AMVP (or INTER) mode may be used to derive an additional reference block, and a flag syntax element for indicating this may be signaled. Alternatively, a predefined mode among the merge mode or the AMVP (or INTER) mode may be used to derive an additional reference block. Alternatively, either the merge mode or the AMVP (or INTER) mode may be selected based on predefined encoding information.

[0250] For example, when the merge mode is used to derive an additional reference block, the prediction information may include a merge index. The merge index may specify a merge candidate within a merge candidate list. When the AMVP (or INTER) mode is used to derive an additional reference block, the prediction information may include a motion vector predictor flag, a reference index, and motion vector difference information. The motion vector predictor flag may specify a candidate within a motion vector predictor candidate list.

[0251] The decoding device may generate a final prediction block by weighting the base prediction block and the additional reference blocks (S720). As described above, the number of additional reference blocks may be less than or equal to a predefined number. For example, if the number of additional reference blocks is 2, the final prediction block may be a block obtained by weighting the base prediction block and the two additional reference blocks. In one embodiment, the weight information for the weighted sum may be signaled or derived.

[0252] As described above, when multiple additional reference blocks are combined, the multiple additional reference blocks may be sequentially weighted to the base prediction block, or the base prediction block and the multiple additional reference blocks generated may be weighted together.

[0253] Typically, the inter prediction process supports unidirectional prediction or bidirectional prediction, but if it includes more than one prediction block, it can be considered multi-reference prediction (or multi-reference prediction). In other words, multi-reference prediction is a method that utilizes multiple reference blocks (or prediction blocks) for prediction, and the following signaling or derivation methods can be considered.

[0254] As an embodiment, information about an additional reference block may be signaled using a merge index, identical to or similar to the merge mode. Alternatively, information about an additional reference block may be signaled using a reference index, a motion vector predictor (MVP) flag (or index), a motion vector difference, or the like, identical to or similar to the AMVP (or INTER) mode. Alternatively, motion information may be derived by inheriting information about an additional reference block from a neighboring block that has already been decoded. In one embodiment, whether or not information about an additional reference block is signaled may be determined depending on the number of additional reference blocks used. For example, if there is one additional reference block, information about the additional reference block may be signaled, and if there are two additional reference blocks, all or part of information about the additional reference blocks may not be signaled but may be derived on the decoder side.

[0255] According to the multi-reference mode according to the embodiment of the present disclosure, a block having different characteristics from an existing prediction block can be generated by signaling / deriving weight information as well as motion information of the prediction block, and the accuracy of the prediction can be increased by using various reference blocks for prediction.

[0256] FIG. 12 is a diagram illustrating a reference block used in a multi-reference mode according to one embodiment of the present disclosure.

[0257] Referring to Fig. 12, it illustrates a case where multiple reference blocks (prediction blocks) are used for prediction as the multi-reference mode is applied. That is, in Fig. 12, reference blocks P0 and P1 represent basic reference blocks (or regular reference blocks), and reference blocks P2 and P3 represent additional reference blocks.

[0258] FIG. 13 is a diagram illustrating a method for signaling information about an additional reference block used in a multi-reference mode according to one embodiment of the present disclosure. The operations illustrated in FIG. 13 are not essential operations of one embodiment of the present disclosure, and at least some of the operations illustrated in FIG. 13 may be omitted.

[0259] When additional reference blocks may exist and information for each additional reference block is signaled and parsed, the motion information of the added prediction block may be signaled and parsed in the order shown in Fig. 13. Here, MaxNum may indicate the maximum number of additional reference blocks.

[0260] Referring to FIG. 13, a loop may be performed until the number of additional reference blocks reaches the maximum number. If the number of additional reference blocks is less than or equal to the maximum number, mhp_flag may be parsed (signaled). mhp_flag represents a syntax element indicating whether additional reference blocks are used. If additional reference blocks are used, mhp_mrg may be parsed. mhp_mrg represents a syntax element indicating whether merge mode or AMVP (or INTER) mode is used to derive additional reference blocks. If merge mode is applied according to the mhp_mrg value, a merge index and a weight index may be signaled, and if AMVP (or INTER) mode is applied, a reference index, a motion vector predictor index, a motion vector difference data, and a weight index may be signaled.

[0261] As illustrated in Fig. 13, when the maximum number of additional reference blocks that can be generated is MaxNum, the presence or absence of MHP-related syntax can be determined through mhp_flag. For example, when MaxNum is 2, mhp_flag can have values ​​as shown in Table 3 below, and the presence or absence of the first and second additional blocks can be determined based on the values.

[0262] [Table 3]

[0263]

[0264] Referring to Table 3, when mhp_flag is '1', additional reference blocks can exist. Additional reference blocks can be distinguished through mhp_mrg whether it is MHP_MERGE mode or MHP_AMVP (or MHP_INTER) mode. When in MHP_MERGE mode, merge index and weight index can be signaled. When in MHP_AMVP (or MHP_INTER) mode, reference index, motion vector predictor index (or flag), motion vector difference data, and weight index can be signaled.

[0265] The syntax names described in this disclosure are examples, and the names may vary. In addition, in this disclosure, the modes for additional reference blocks when additional reference blocks exist are described as MHP_MERGE and MHP_AMVP (or MHP_INTER), which can be distinguished from the merge mode and AMVP (or INTER) mode, which indicate motion information of regular reference blocks. In particular, the motion vector predictor candidate list for the regular reference block and the motion vector predictor candidate list for the additional reference block can be independently configured. In addition, the MHP_MERGE and MHP_AMVP (or MHP_INTER) modes for the additional reference block can include weight index information.

[0266] In addition, as an embodiment, in addition to the method of applying multiple reference blocks through the signaling described above, multiple reference blocks may be applied through a derivation method. In particular, since merge mode uses motion information inherited from decoded adjacent / non-adjacent blocks, information for multiple reference blocks may also use motion information inherited from adjacent / non-adjacent blocks. For example, when the current block is in merge mode and the adjacent / non-adjacent block for acquiring motion information includes MHP information, the information may be inherited and used to generate additional reference blocks for the current block.

[0267] As an example, as described above, multi-reference block information can be acquired through signaling or a derivation process, and both methods can be applied. For example, when N multi-reference blocks are configured, if there are M multi-reference blocks acquired through a derivation process (M<=N), NM pieces of multi-reference block information can be acquired through signaling.

[0268] Additional reference blocks obtained through signaling or derivation methods can be weighted to generate a final prediction block as follows. For example, when multiple reference blocks are applied, the final prediction block can be calculated as shown in the following mathematical expression 7. Mathematical expression 7 assumes that there are regular reference blocks P0 and P1, and additionally, P2 and P3.

[0269] [Equation 7]

[0270]

[0271] In Equation 7, W0 and W1 represent weights applied to additional reference blocks P2 and P3, respectively. In the first step of Equation 7, a base prediction block can be generated through a weighted sum of regular reference blocks, and in the second and third steps of Equation 7, a weighted sum of additional reference blocks can be performed. As described above, the base prediction block may refer to a regular reference block prior to the weighted sum or may refer to a weighted-sum prediction block.

[0272] In the above, multiple reference blocks are described as being added to the INTER prediction process, but they are not limited to being added to the INTER prediction process. Multiple reference blocks can be included in the INTRA prediction process and the INTRA block copy (IBC) process. In other words, the prediction mode of the base prediction block is not limited to the INTER mode, and INTRA mode or IBC mode is also possible.

[0273] When an additional reference block is included in various regular modes (prediction modes of a regular block), the regular block may include an INTER block (INTER prediction block), an INTRA block (TNTRA prediction block), and an IBC block, and the prediction mode of the additional reference block (hereinafter referred to as 'additional prediction mode') may include not only the MHP_AMVP (or MHP_INTER) mode or the MHP_MERGE mode, but also the MHP_INTRA mode and the MHP_IBC mode. In other words, the prediction mode of the additional reference block (additional prediction mode) is not limited to the INTER mode, and the INTRA mode or the IBC mode is also possible. In one embodiment, the additional prediction mode may be an INTER mode (MHP_INTER mode), an INTRA mode (MHP_INTRA mode), or an IBC mode (MHP_IBC mode).

[0274] As an example, information indicating each additional prediction mode (e.g., a flag or index) may be signaled to determine the prediction mode of each additional reference block. Furthermore, information indicating each additional prediction mode may be acquired at the decoder level through an inheritance method, rather than through signaling. Furthermore, prediction modes of multiple reference blocks may be derived and generated at the decoder level when certain conditions are met, other than through signaling or transmission methods.

[0275] Example 1

[0276] The prediction mode (additional prediction mode) of the additional reference block can include the MHP_INTRA mode, the MHP_IBC mode, or the MHP_INTER mode, and the prediction mode of the additional reference block, i.e., the additional prediction mode, can exist in various combinations when the prediction mode of the basic prediction block, i.e., the basic prediction mode (or base mode), is the INTRA mode, the IBC mode, or the INTER mode.

[0277] As an example, the prediction modes of additional reference blocks including MHP_INTRA mode, MHP_IBC mode and MHP_INTER mode may exist in a limited number depending on the type of the base mode, and may be determined based on the combination with the allowed base mode.

[0278] As an example, the condition for the combination between the prediction mode of the base prediction block (base prediction mode, MHP mode) and the prediction mode of the additional reference block (additional prediction mode) may be the prediction mode of the base prediction block.

[0279] The above conditions for combination between the basic mode and the additional prediction mode may be prediction tools applied to the basic block in the basic INTER mode. The prediction tools of the basic INTER mode may include tools related to the method of generating the prediction block, such as the basic affine mode (e.g., affine AMVP and affine merge), the basic AMVP mode, the basic merge mode, the MMVD mode, the CIIP mode, and the GPM mode.

[0280] The above conditions for the combination between the basic mode and the additional prediction mode may be the mode (planar mode, DC mode, directional mode, etc.) applied to the basic block in the basic INTRA mode. That is, whether MHP_INTRA is allowed or not may be determined based on the mode of the basic INTRA mode. For example, when the mode of the basic INTRA mode is planar mode or DC mode, MHP_INTRA mode may be allowed. In addition, the modes allowed in MHP_INTRA mode may be determined based on the mode of the basic INTRA mode.

[0281] The above condition for the combination between the basic mode and the additional prediction mode may be the number of prediction blocks (or basic reference blocks) used in the basic block. The allowed MHP mode may be determined based on whether unidirectional prediction or bidirectional prediction is used in the basic INTER mode, or whether a mode such as DIMD (Decoder-side intra-mode derivation) or TIMD (template-based intra-mode derivation) with multiple reference blocks is used in the basic INTRA mode.

[0282] The above conditions for combination between the basic mode and the additional prediction mode may be the size of the current block, the width / height of the current block, or the block ratio. In one embodiment, if the width or height of the current block is equal to a reference value, a combination between any one of the basic INTER mode, the basic INTRA mode, or the basic IBC mode and at least one of the MHP_INTER mode, the MHP_INTRA mode, or the MHP_IBC mode is allowed, or if it is greater than the reference value, a combination between any one of the basic INTER mode, the basic INTRA mode, or the basic IBC mode and at least one of the MHP_INTER mode, the MHP_INTRA mode, or the MHP_IBC mode is allowed, or if it is less than the reference value, a combination between any one of the basic INTER mode, the basic INTRA mode, or the basic IBC mode and at least one of the MHP_INTER mode, the MHP_INTRA mode, or the MHP_IBC mode is allowed.

[0283] The above condition for a combination between the base mode and the additional prediction mode may be a quantization parameter (QP). In one embodiment, if the QP is equal to a reference value (e.g., 32), a combination between any one of the base INTER mode, the base INTRA mode, or the base IBC mode and at least one of the MHP_INTER mode, the MHP_INTRA mode, or the MHP_IBC mode is allowed, or if it is greater than the reference value, a combination between any one of the base INTER mode, the base INTRA mode, or the base IBC mode and at least one of the MHP_INTER mode, the MHP_INTRA mode, or the MHP_IBC mode is allowed, or if it is less than the reference value, a combination between any one of the base INTER mode, the base INTRA mode, or the base IBC mode and at least one of the MHP_INTER mode, the MHP_INTRA mode, or the MHP_IBC mode is allowed.

[0284] The above specific condition for the combination between the base mode and the additional prediction mode may be a temporal layer. For example, if the temporal layer is equal to a threshold value (e.g., 4), a combination between any one of the base INTER mode, the base INTRA mode or the base IBC mode and at least one of the MHP_INTER mode, the MHP_INTRA mode or the MHP_IBC mode is allowed, or if it is greater than the threshold value, a combination between any one of the base INTER mode, the base INTRA mode or the base IBC mode and at least one of the MHP_INTER mode, the MHP_INTRA mode or the MHP_IBC mode is allowed, or if it is less than the threshold value, a combination between any one of the base INTER mode, the base INTRA mode or the base IBC mode and at least one of the MHP_INTER mode, the MHP_INTRA mode or the MHP_IBC mode is allowed.

[0285] FIG. 14 is a diagram illustrating a combination between a mode of a basic block and a mode of an additional reference block according to one embodiment of the present disclosure.

[0286] In one embodiment, when an additional reference block exists, INTRA mode and IBC (Intra Block Copy) mode may be supported in addition to INTER mode (AMVP mode, MERGE mode). When INTRA mode as an additional reference block other than a basic block is referred to as MHP_INTRA mode, IBC mode as MHP_IBC mode, and INTER mode as MHP_INTER mode, the combination of a basic block and an additional reference block having each INTER mode, INTRA mode, and IBC mode may be as shown in FIG. 14.

[0287] As illustrated in Fig. 14, for each of the INTER mode, INTRA mode, and IBC mode, there may be MHP_INTRA mode, MHP_IBC mode, and MHP_INTER mode. This is just one example, and the modes of additional reference blocks allowed for each prediction mode may be as follows.

[0288] A. Additional reference blocks in MHP_INTRA mode may be allowed.

[0289] A-1. A limited number of additional reference blocks of MHP_INTRA mode may be included. For example, one additional reference block of MHP_INTRA mode may be permitted.

[0290] A-2. Additional reference blocks in MHP_INTRA mode may be allowed when certain conditions are met.

[0291] B. Additional reference blocks in MHP-IBC mode may be allowed.

[0292] B-1 Additional reference blocks of a limited number of MHP-IBC modes may be included. For example, one additional reference block of MHP_IBC mode may be allowed.

[0293] B-2 MHP_IBC mode may be allowed when certain conditions are met.

[0294] Since B-3 IBC mode is divided into AMVP mode and MERGE mode, MHP_IBC can also be divided into MHP_IBC_AMVP mode and MHP_IBC_MERGE mode. Alternatively, both MHP_IBC_AMVP mode and MHP_IBC_MERGE mode may be allowed, or only a specific mode between MHP_IBC_AMVP mode and MHP_IBC_MERGE mode may be allowed. For example, only MHP_IBC_MERGE mode may be allowed.

[0295] Additional reference blocks in C MHP_INTER mode may be allowed.

[0296] C-1 Additional reference blocks for a limited number of MHP_INTER modes may be included. For example, one MHP_INTER mode may be allowed.

[0297] C-2 Additional reference blocks in MHP_INTRA mode may be allowed when certain conditions are met.

[0298] Since the C-3 basic INTER mode is divided into AMVP mode and MERGE mode, the MHP_INTER mode of the additional reference block can also have MHP_AMVP mode and MHP_MERGE mode. Alternatively, both MHP_AMVP mode and MHP_MERGE mode of the additional reference block may be allowed, or only one of the MHP_AMVP mode and MHP_MERGE mode of the additional reference block may be allowed. For example, only MHP_MERGE mode may be allowed.

[0299] The method described above can generate various combinations of modes. In one embodiment, when the mode of the basic block is INTRA mode, a combination of one additional reference block of MHP_INTRA mode and one additional reference block of MHP_IBC mode can be allowed. In one embodiment, when the mode of the basic block is IBC mode, a combination of one additional reference block of MHP_INTER mode and multiple additional reference blocks of MHP_IBC mode can be allowed.

[0300] FIG. 15 is a diagram illustrating a method for signaling information about an additional reference block used in a multi-reference mode according to one embodiment of the present disclosure. The operations illustrated in FIG. 15 are not essential operations of one embodiment of the present disclosure, and at least some of the operations illustrated in FIG. 15 may be omitted.

[0301] Specifically, FIG. 15 shows an example in which the MHP_INTRA mode, MHP_IBC mode, and MHP_INTER mode of an additional reference block are allowed in a specific mode of a basic block. As an embodiment, when an additional reference block exists and satisfies a specific condition, information (e.g., a flag or an index, etc.) indicating whether the MHP_INTER mode is allowed can be signaled (encoder) / parsed (decoder). For example, a flag mhp_intra_flag indicating whether the MHP_INTRA mode is allowed can be signaled / parsed. The name of the flag mhp_intra_flag is not limited thereto, and flags of various names can be used to indicate whether the MHP_INTRA mode is allowed. For example, when the flag mhp_intra_flag is '1', the additional reference block is classified as the MHP_INTRA mode, and if present, prediction information for an INTRA block as an additional reference block can be signaled / parsed.

[0302] If a specific condition is not satisfied or the flag mhp_intra_flag is '0', information (e.g., a flag or index) indicating whether the MHP_IBC mode is allowed may be signaled / parsed. For example, the flag mhp_ibc_flag indicating whether the MHP_IBC mode is allowed may be signaled / parsed. The name of the flag mhp_ibc_flag is not limited thereto, and flags with various names may be used to indicate whether the MHP_IBC mode is allowed. For example, when the flag mhp_ibc_flag is '1', the additional reference block is identified as the MHP_IBC mode, and prediction information for the IBC block as the additional reference block may be signaled / parsed. For example, prediction information for an MHP_IBC_MERGE block may be signaled / parsed as illustrated in FIG. 15.

[0303] Finally, when certain conditions are not met or the flag mhp_ibc_flag is '0', the additional reference block is identified as MHP_INTER mode, and prediction information for the INTER block as an additional reference block can be signaled / parsed. For example, prediction information for an MHP_MERGE block can be signaled / parsed as illustrated in FIG. 15.

[0304] The method illustrated in FIG. 15 is one example, and various combinations are possible according to the methods listed above. As an example, a flag mhp_intra_flag indicating whether MHP_INTER mode is allowed may be signaled / parsed, and when the flag mhp_intra_flag is '1', prediction information for an MHP_INTRA block may be signaled / parsed, and when the flag mhp_intra_flag is '0', a flag mhp_inter_flag indicating whether MHP_INTER mode is allowed may be signaled / parsed, and when the flag mhp_inter_flag is '1', prediction information for an MHP_INTER block may be signaled / parsed, and when the flag mhp_inter_flag is '0', prediction information for an MHP_IBC block may be signaled / parsed.

[0305] As an example, a flag mhp_ibc_flag indicating whether MHP_IBC mode is allowed may be signaled / parsed, and when the flag mhp_ibc_flag is '1', prediction information for an MHP_IBC block may be signaled / parsed, and when the flag mhp_ibc_flag is '0', a flag mhp_intra_flag indicating whether MHP_INTRA mode is allowed may be signaled / parsed, and when the flag mhp_intra_flag is '1', prediction information for an MHP_INTRA block may be signaled / parsed, and when the flag mhp_intra_flag is '0', prediction information for an MHP_INTER block may be signaled / parsed. As an example, a flag mhp_ibc_flag indicating whether MHP_IBC mode is allowed may be signaled / parsed, and when the flag mhp_ibc_flag is '1', prediction information for an MHP_IBC block may be signaled / parsed, and when the flag mhp_ibc_flag is '0', a flag mhp_inter_flag indicating whether MHP_INTER mode is allowed may be signaled / parsed, and when the flag mhp_inter_flag is '1', prediction information for an MHP_INTER block may be signaled / parsed, and when the flag mhp_inter_flag is '0', prediction information for an MHP_INTRA block may be signaled / parsed.

[0306] As an example, a flag mhp_inter_flag indicating whether MHP_INTER mode is allowed may be signaled / parsed, and when the flag mhp_inter_flag is '1', prediction information for an MHP_INTER block may be signaled / parsed, and when the flag mhp_inter_flag is '0', a flag mhp_intra_flag indicating whether MHP_INTRA mode is allowed may be signaled / parsed, and when the flag mhp_intra_flag is '1', prediction information for an MHP_INTRA block may be signaled / parsed, and when the flag mhp_intra_flag is '0', prediction information for an MHP_IBC block may be signaled / parsed. As an example, a flag mhp_inter_flag indicating whether MHP_INTER mode is allowed may be signaled / parsed, and when the flag mhp_inter_flag is '1', prediction information for an MHP_INTER block may be signaled / parsed, and when the flag mhp_inter_flag is '0', a flag mhp_ibc_flag indicating whether MHP_IBC mode is allowed may be signaled / parsed, and when the flag mhp_ibc_flag is '1', prediction information for an MHP_IBC block may be signaled / parsed, and when the flag mhp_ibc_flag is '0', prediction information for an MHP_INTRA block may be signaled / parsed.

[0307] Specific conditions that allow combinations of each mode (e.g., condition1, condition2 as shown in FIG. 15) may include the following items, and may include one or more conditions.

[0308] A. Certain conditions may be related to the prediction mode of the basic block. That is, whether a certain condition is satisfied may be determined based on whether the prediction mode of the basic block is INTRA mode, IBC mode, or INTER mode. For example, when the prediction mode of the basic block is INTRA mode or IBC mode, the MHP_INTER mode of the additional reference block may be allowed. In another example, when the prediction mode is INTER, MHP_IBC may be allowed.

[0309] A-1. Certain conditions may be related to the prediction tool applied to the basic block in INTER mode. In this case, the prediction tool may be related to the method of generating the prediction block, such as affine mode (affine AMVP and affine merge), AMVP mode, basic merge mode, MMVD mode, CIIP mode, or GPM mode. For example, when the basic block is in AMVP mode, the MHP_INTRA mode of the additional reference block may not be allowed, and as another example, when the basic block is in CIIP mode or GPM mode, the MHP_INTRA mode and / or MHP_IBC mode of the additional reference block may not be allowed.

[0310] A-2 A specific condition may be the prediction mode of the INTRA mode. That is, whether or not the MHP_INTRA of the additional reference block is allowed may be determined based on the prediction mode of the basic block which is the INTRA mode. For example, when the basic block is the INTRA mode and has the planar mode or the DC mode, the MHP_INTRA mode of the additional reference block may be allowed. As another example, when the basic block has an angular mode with a clear directionality (e.g., horizontal mode, vertical mode, or its adjacent directional mode), the MHP_INTRA mode of the additional reference block may not be allowed. In addition, the allowable prediction mode (e.g., planar mode, DC mode, or angular mode) among the MHP_INTRA modes of the additional reference block may be determined based on the prediction mode of the basic block. For example, when the basic block is the INTRA mode and has the angular mode, non-directional modes (e.g., planar mode and DC mode) among the MHP_INTRA modes of the additional reference block may be allowed.

[0311] A-3 A specific condition may be the number of prediction blocks (basic reference blocks) used in the basic block.

[0312] A-3-1 When the basic block is in INTER mode with unidirectional prediction, additional reference blocks in MHP_INTRA mode, MHP_INTER mode, or MHP_IBC mode may be allowed.

[0313] A-3-2 When the undulation block is in INTER mode with bidirectional prediction, a limited number of additional reference blocks (additional reference blocks in MHP_INTRA mode, MHP_INTER mode, or MHP_IBC mode) may be allowed. For example, one MHP_INTRA may be allowed.

[0314] A-3-3 When a basic block contains multiple INTRA blocks, additional reference blocks (MHP_INTRA mode, MHP_INTER mode, or MHP_IBC mode) may not be allowed. For example, a basic block may contain multiple INTRA blocks when the basic block induces multiple prediction modes, such as using DIMD or TIMD, or when the basic block contains different INTRA modes in multiple partitioned regions, such as SGPM.

[0315] A-3-4 When a basic block contains multiple IBC blocks, additional reference blocks (in MHP_INTRA mode, MHP_INTER mode, or MHP_IBC mode) may not be allowed. For example, a basic block may contain multiple IBC blocks if the basic block is a block with two or more block vectors, or if the basic block is an IBC block that allows bidirectional prediction.

[0316] A-4 The specific condition may be the size of the basic block, the width / height ratio of the basic block, or the block ratio. For example, when the size of the basic block (width x height) is less than 1024, MHP_INTRA mode or MHP_IBC mode may be allowed. Another example is when the size of the block (width x height) is less than 64, MHP_IBC or MHP_INTER mode may be allowed.

[0317] A-5 The specific condition may be the quantization parameter (QP) of the basic block. For example, when the QP of the basic block is less than a threshold, the MHP_INTRA mode or MHP_IBC mode of the additional reference block may be allowed. In this case, the threshold value may be 32. In one embodiment, if QP is equal to a threshold value (e.g., 32), a combination between any one of the default INTER mode, the default INTRA mode, or the default IBC mode and at least one of the MHP_INTER mode, the MHP_INTRA mode, or the MHP_IBC mode is allowed, or if it is greater than the threshold value, a combination between any one of the default INTER mode, the default INTRA mode, or the default IBC mode and at least one of the MHP_INTER mode, the MHP_INTRA mode, or the MHP_IBC mode is allowed, or if it is less than the threshold value, a combination between any one of the default INTER mode, the default INTRA mode, or the default IBC mode and at least one of the MHP_INTER mode, the MHP_INTRA mode, or the MHP_IBC mode is allowed.

[0318] A-6 A specific condition may be a temporal layer. For example, when the temporal layer is less than a threshold, MHP_INTRA mode or MHP_IBC mode of the additional reference block may be allowed. In this case, the temporal layer threshold may be 4. For example, if the temporal layer is equal to a threshold value (e.g., 4), a combination between any one of the default INTER mode, the default INTRA mode, or the default IBC mode and at least one of the MHP_INTER mode, the MHP_INTRA mode, or the MHP_IBC mode is allowed, or if it is greater than the threshold value, a combination between any one of the default INTER mode, the default INTRA mode, or the default IBC mode and at least one of the MHP_INTER mode, the MHP_INTRA mode, or the MHP_IBC mode is allowed, or if it is less than the threshold value, a combination between any one of the default INTER mode, the default INTRA mode, or the default IBC mode and at least one of the MHP_INTER mode, the MHP_INTRA mode, or the MHP_IBC mode is allowed.

[0319] This limited allowance of additional reference blocks can help reduce signaling overhead. The proposed method can be used to balance the trade-off between encoder / decoder complexity and performance. Furthermore, the proposed method can utilize only signaling-based information acquisition, only inheritance and derivation-based information acquisition, or a combination of signaling-based and inheritance and derivation-based information acquisition.

[0320] Example 2

[0321] Additional reference blocks can be generated using signaled information or information derived through an inheritance process. Since various combinations of additional reference blocks can exist, compression performance can be improved when additional reference blocks inherited from neighboring blocks have the following constraints. The following constraints can be used for this purpose:

[0322] When the default mode of adjacent / non-adjacent neighboring blocks and the current block are the same, it can be inherited. In one embodiment, when the default prediction mode of the current block is the default INTER mode, information of additional reference blocks can be inherited from neighboring blocks having the default INTER mode. In one embodiment, when the default prediction mode of the current block is the default INTRA mode, information of additional reference blocks can be inherited from neighboring blocks having the default INTRA mode. In one embodiment, when the default prediction mode of the current block is the default IBC mode, information of additional reference blocks can be inherited from neighboring blocks having the default IBC mode.

[0323] The inheritance of additional reference block modes may be determined based on a mode priority determined based on the base mode of the current block. In one embodiment, when a neighboring block includes additional reference blocks of various modes, the order and number of additional reference blocks (or additional prediction modes) inherited from the neighboring block may be determined based on a mode priority determined based on the base mode of the current block.

[0324] Inheritance of additional prediction modes can be determined based on each allowed combination. For example, if only MHP_INTRA and MHP_IBC modes are allowed as additional prediction modes when the default mode is INTER, only information in INTRA and IBC modes among the additional reference blocks included in the surrounding blocks can be inherited.

[0325] Inheritance of additional prediction modes may be determined based on the prediction mode (planar mode, DC mode, or directional mode) of the default INTRA mode. For example, when the default mode of the current block is INTRA mode and has a planar mode or DC mode, information of the MHP_INTRA mode of the surrounding blocks may not be inherited.

[0326] In addition to signaling methods for information about additional prediction modes, methods for inheriting from adjacent / non-adjacent neighboring blocks are described.

[0327] MERGE mode uses motion information from adjacent or non-adjacent blocks without signaling motion information. In multi-reference mode, information regarding additional prediction modes contained in adjacent or non-adjacent blocks can be derived using the same or similar methods as other information.

[0328] As previously explained, additional reference blocks can be generated by signaling information about additional prediction modes (or additional reference blocks) by distinguishing the INTRA mode, IBC (AMVP, MERGE) mode, and INTER (AMVP, MERGE) mode as many times as the number of additional reference blocks allowed. In addition, additional reference blocks of the current block can be generated by distinguishing the additional prediction modes (or additional reference blocks) and inheriting information about the additional prediction modes (or additional reference blocks).

[0329] Inheritance of additional prediction modes can be determined through the following criteria:

[0330] A. Types of basic modes

[0331] When the basic modes of the current block and adjacent / non-adjacent neighboring blocks are the same, the additional prediction modes of the neighboring blocks can be inherited by the current block. Specifically, when the current block is in INTER mode, multi-reference block information (information about additional prediction modes) can be inherited from neighboring blocks in INTER mode. Similarly, when the current block is in IBC mode or INTRA mode, multi-reference block information (information about additional prediction modes) can be inherited from neighboring blocks in IBC mode or INTRA mode, respectively.

[0332] B. Priority by mode

[0333] Information regarding additional prediction modes can be inherited according to a predefined priority order based on the default mode of the current block. Table 4 illustrates an example of the priority order of modes (additional prediction modes) of additional reference blocks based on the default mode (default prediction mode) of the current block. Depending on each default mode, the types and number of additional prediction modes allowed, as well as the priority order of the additional prediction modes, can change.

[0334] [Table 4]

[0335]

[0336] In one embodiment according to Table 4, when the basic mode of the current block is the INTER mode and the modes of additional reference blocks of surrounding blocks having the INTER mode as the basic mode are configured in the order of MHP_IBC mode, MHP_INTRA mode, and MHP_INTER mode, the modes of multiple reference blocks for the current block can be configured in the order of MHP_INTER mode, MHP_IBC mode, and MHP_INTRA mode.

[0337] In another embodiment, if the modes of additional reference blocks of surrounding blocks are configured in the order of MHP_IBC mode (1), MHP_INTRA mode, and MHP_IBC mode (2), and the IBC mode has a high priority, information about the additional prediction mode can be inherited to the current block in the order of MHP_IBC mode (1), MHP_IBC mode (2), and MHP_INTRA mode.

[0338] C. Number of allowed inheritance of additional prediction modes according to basic mode

[0339] Information about additional prediction modes can be inherited according to the allowable number of additional prediction modes for each base mode. In one embodiment, only one prediction mode can be inherited for each prediction mode of each additional reference block. For example, if a neighboring block includes additional reference blocks and includes additional reference blocks of MHP_INTER mode (1), MHP_INTER mode (2), and one MHP_IBC mode, MHP_INTER mode (1) and MHP_IBC mode can be inherited as additional prediction modes of the current block. A neighboring block including MHP_INTER mode (1), MHP_INTER mode (2), and one MHP_IBC mode can have MHP_INTER mode (1), MHP_IBC mode, and MHP_INTER mode (2) inherited. In other words, the allowable number for inheritance of each additional reference block (additional prediction mode) can be applied separately from the allowable number of signaled additional prediction modes.

[0340] D. Allowable combinations with additional prediction modes according to the basic mode

[0341] Inheritance of additional prediction modes may be determined based on the allowable combinations with additional prediction modes according to the base mode. In one embodiment, if only combinations with MHP_INTER mode and MHP_IBC mode are allowed for the base INTER mode, only information of MHP_INTER mode and MHP_IBC mode among information of additional reference blocks included in neighboring blocks may be inherited by the current block. This is an example, and combinations with additional prediction modes that are possible according to the base mode and the number of allowable inheritances according to the base mode within the possible combinations may be determined. In addition, the allowable combinations for inheritance of additional reference blocks (additional prediction modes) and the allowable combinations of signaled additional prediction modes may be independent.

[0342] E. Prediction mode of basic INTRA blocks

[0343] The inheritance of additional prediction modes may be determined based on the prediction mode of the default INTRA mode. For example, when the default mode of the current block is INTRA mode and planar mode or DC mode, information regarding the MHP_INTRA mode of the neighboring blocks may not be inherited. For another example, when the default mode of the current block is INTRA mode and directional mode, information regarding the MHP_INTER mode and MHP_IBC mode of the neighboring blocks may not be inherited. This is just one example, and various combinations may be allowed depending on the prediction mode of the INTRA mode of the current block and the type and prediction mode of the additional reference block of the neighboring blocks.

[0344] Example 3

[0345] Additional reference blocks can be generated using information derived through signaling or inheritance processes. Additionally, additional reference blocks for MHP_INTER mode, MHP_IBC mode, and / or MHP_INTRA mode can be derived based on template matching.

[0346] For example, additional reference blocks in MHP_IBC mode and / or MHP_INTER mode may be generated based on template matching. Template matching may be applied in particular when in MERGE mode (MHP_IBC mode or MHP_INTER mode).

[0347] Additional reference blocks of MHP_INTER mode and / or MHP_IBC mode derived based on template matching may first be searched in integer-pel units based on a base motion vector and / or a base block vector. Thereafter, additional searches may be performed in fractional-pel units based on a position with the smallest error (or cost). Thereafter, an INTER mode block and / or an IBC mode block having a refined motion vector and / or a refined block vector at a position with the smallest error (or cost) may be used as an additional reference block.

[0348] The basic motion vector can be motion information indicated by the MVP index or BVP index in MERGE mode. The search process (initial search process and / or additional search process) can be performed within a predefined search range.

[0349] The error (or cost) can refer to various types of errors or costs, such as the sum of absolute errors, the mean of absolute errors, the sum of squared errors, and the mean of squared errors. The error (or cost) is calculated based on the difference between the template region of the current block and the template region of the predicted block, and can be calculated as SA(T)D (Sum of Absolute Difference), MR-SA(T)D (Mean Reduced-Sum of Absolute Difference), SSE (Sum Squared Error), MSE (Mean Squared Error), etc. The error (or cost) can be a value multiplied by different weights depending on the specific region and location within the search area.

[0350] Additional reference blocks in MHP_INTRA mode can be generated based on template matching. In MHP_INTRA mode, the error (or cost) between the reconstructed sample in the template area of ​​the current block and the predicted sample generated from the reference sample in the template area is calculated, and the mode with the smallest error (or cost) can be selected.

[0351] The prediction mode for calculating the error (or cost) can be determined within the MPM (Most Probable Mode) configured for the MHP_INTRA mode. The MPM may include a planar mode, a horizontal mode, a vertical mode, etc. as basic modes, and may include one or more modes.

[0352] With respect to the block of the above MHP_INTRA mode, the prediction mode of the MHP_INTRA mode can be determined using the following process: a process of calculating a horizontal gradient and a vertical gradient by applying a Sobel filter within a template region of the current block, a process of obtaining a histogram of gradients (HoG) based on the horizontal gradient and the vertical gradient, a process of deriving an angular mode based on the gradient histogram, and a process of using the derived angular mode as a prediction mode for the MHP_INTRA mode.

[0353] The prediction mode for the MHP_INTRA mode based on gradient histogram (HoG) may include one or more modes, depending on the allowed number.

[0354] Below, a specific method for deriving additional reference blocks (or additional prediction modes) without signaling / parsing information about the additional reference blocks (or additional prediction modes) is described.

[0355] FIG. 16 is a diagram illustrating a method for deriving additional reference blocks based on template matching used in a multi-reference mode according to one embodiment of the present disclosure. Specifically, a method for deriving additional reference blocks in MHP_INTER mode and / or additional reference blocks in MHP_IBC mode based on template matching is illustrated.

[0356] When the default mode of the current block is INTER mode, the process described in this embodiment can be utilized. In particular, when the default mode of the current block is MERGE mode, the process described in this embodiment can be applied. However, this is not limited to this, and when the default mode of the current block is AMVP mode, the process described in this embodiment can also be applied.

[0357] Specifically, motion vector refinement (MV refinement) based on template matching can be performed within a search range as illustrated in Fig. 16, using the motion information (motion vector) of the block pointed to by the MVP (Motion Vector Predictor) index as the base motion vector (baseMV). An INTER prediction block as an additional reference block can be generated using the refined motion vector, and the motion vector refinement (MV refinement) process can be performed in the following order.

[0358] A. Search for integer-pel units based on the base motion vector (baseMV) to determine the location with the smallest error (or cost).

[0359] B. If possible, search fractionally by pixel based on the location with the above error (or cost) to determine the location with the smallest error (or cost).

[0360] C. A prediction block is generated by setting the location with the smallest error (or cost) as the refined motion vector (refinedMV) and designating the block with the refined motion vector (refinedMV) as a block in MHP_INTER mode. Here, if the base motion vector (baseMV) and the refined motion vector (refinedMV) are the same, the process of generating an additional reference block can be omitted.

[0361] When the default mode of the current block is IBC mode, the process described in this embodiment can be used. In particular, when the default mode of the current block is MERGE mode, the process described in this embodiment can be applied. However, this is not limited to this, and when the default mode of the current block is AMVP mode, the process described in this embodiment can also be applied.

[0362] Specifically, block vector refinement (BV refinement) based on template matching can be performed within a search range as illustrated in Fig. 16, using the motion information (Block Vector) of the block pointed to by the BVP (Block Vector Predictor) index as the base block vector (baseBV). An IBC prediction block as an additional reference block can be generated using the refined block vector, and the block vector refinement (BV refinement) process can be performed in the following order.

[0363] A. Search for integer-pel units based on the basic block vector (baseBV) to determine the location with the smallest error (or cost).

[0364] B. If possible, search fractionally by pixel based on the location with the above error (or cost) to determine the location with the smallest error (or cost).

[0365] C. A prediction block is generated by setting the location with the smallest error (or cost) as the improved block vector (refinedBV) and designating the block with the improved block vector (refinedBV) as a block in MHP_IBC mode. At this time, when the base block vector (baseBV) and the improved block vector (refinedBV) are identical, the process of generating an additional prediction block can be omitted.

[0366] The search process described above can be performed within a search range, and weights can be applied to the error (or cost) based on each search location or area within the search range during the search process. In one embodiment, the selection rate for a specific area can be increased by calculating as in Equation 8.

[0367] [Equation 8]

[0368] bestCost = λ* cost, (λ < 1.0)

[0369] For example, cost can represent an error (or cost), and λ can be assigned a smaller value the closer it is to the location pointed to by the base motion vector (baseMV) or base block vector (baseBV).

[0370] FIG. 17 is a diagram illustrating a method for deriving an additional reference block based on intra-template matching prediction used in a multi-reference mode according to an embodiment of the present disclosure. FIG. 18 is a diagram illustrating a method for deriving an additional reference block based on TIMD used in a multi-reference mode according to an embodiment of the present disclosure. FIG. 19 is a diagram illustrating a method for calculating an error (or cost) for deriving an additional reference block in a multi-reference mode according to an embodiment of the present disclosure.

[0371] To derive the MHP_IBC block, an intra-template matching prediction (IntraTMP)-based derivation method can be used. In other words, as illustrated in Fig. 17, by utilizing the error (or cost) between the template region of an encoded / decoded adjacent block and the template region of a block pointed to by the block vector of the adjacent block through intra-template matching prediction (IntraTMP), the position with the smallest error (or cost) can be used as the block vector of the MHP_IBC block for the current block.

[0372] When the default mode of the current block is INTRA mode, the process described in this embodiment can be used.

[0373] An INTRA prediction block can be generated as an additional reference block when certain conditions are satisfied, and an MHP_INTRA block can be derived in the following way.

[0374] A. The prediction mode for the MHP_INTRA mode can be used similarly to the TIMD method described above. In other words, as illustrated in FIG. 18, the error (or cost) between the reconstructed sample in the template area of ​​the current block and the predicted sample generated from the reference sample in the template area is calculated, and the mode with the smallest error (or cost) can be selected. The prediction mode for calculating the error (or cost) can be determined within the Most Probable Mode (MPM) configured for the MHP_INTRA mode. The MPM can basically include a planar mode, a horizontal mode, a vertical mode, etc., and can include one or more modes.

[0375] B. The prediction mode for the MHP_INTRA mode can be used similarly to the DIMD method described above. In other words, by applying a Sobel filter within the template region of the current block, horizontal gradients and vertical gradients are calculated, and a Histogram of Gradients (HoG) can be obtained based on the horizontal gradients and the vertical gradients. An angular mode is derived based on the gradient histogram, and the derived angular mode can be used as a prediction mode for the MHP_INTRA mode. The prediction mode for the MHP_INTRA mode based on the Histogram of Gradients (HoG) can include one or more modes depending on the allowed number.

[0376] The error (or cost) used in the process of deriving a block in MHP_INTER mode and / or a block in MHP_IBC mode can be calculated as follows. For example, as illustrated in FIG. 19, based on the error between the adjacent samples (cA, cL) of the current block and the adjacent samples (pA, pL) of the reference block pointed to by the base motion vector (baseMV) or the base block vector (baseBV), a block in MHP_INTER mode and / or MHP_IBC mode can be generated when the error is greater than a threshold.

[0377] Specifically, the error SA between the upper adjacent sample (cA) of the current block and the upper adjacent sample (pA) of the predicted block may be compared with a threshold, or the error SL between the left adjacent sample (cL) of the current block and the left adjacent sample (pL) of the predicted block may be compared with a threshold. In addition, both the error SA associated with the upper adjacent sample and the error SL associated with the left adjacent sample may be compared with the threshold. For convenience of explanation, the error associated with the upper left adjacent sample is not described, but the sample at the corresponding position may be considered in the process of calculating the inter-sample error using the upper adjacent sample, the left adjacent sample, or the upper left adjacent sample.

[0378] The above threshold value can be set by mathematical expressions 9 and 10.

[0379] [Equation 9]

[0380] threshold = width * template height / λ

[0381] [Equation 10]

[0382] threshold = template width * height / λ

[0383] For example, λ can be set to a value satisfying λ > 1.

[0384] Here, the error (or cost) between samples (or sample values) can be calculated by the following methods, and a combination of the following methods can be applied.

[0385] A. The error (or cost) can be considered as the sum of absolute differences (SAD) between samples. In other words, the error (or cost) can be calculated as in Equation 11. In Equation 11, c and p represent adjacent samples of the current block and the predicted block, respectively, and x represents the location of the adjacent sample. In addition, it can be applied to N samples included in the upper, left, or upper / left regions.

[0386] [Equation 11]

[0387]

[0388] B. The error (or cost) can be considered as the Mean Reduced-Sum of Absolute Difference (MR-SAD) between samples. In other words, the error (or cost) can be calculated as in Equation 12. In Equation 12, c and p represent adjacent samples of the current block and the predicted block, respectively, and x represents the location of the adjacent sample. In addition, it can be applied to N samples included in the upper, left, or upper / left regions.

[0389] [Equation 12]

[0390]

[0391] C. The error (or cost) can be considered as the sum squared error (SSE) between samples. In other words, the error (or cost) can be calculated as in Equation 13. In Equation 13, c and p represent adjacent samples of the current block and the predicted block, respectively, and x represents the location of the adjacent sample. In addition, it can be applied to N samples included in the upper, left, or upper / left regions.

[0392] [Equation 13]

[0393]

[0394] D. The error (or cost) can be considered as the mean squared error (MSE) between samples. In other words, the error (or cost) can be calculated as in Equation 14. In Equation 14, c and p represent adjacent samples of the current block and the predicted block, respectively, and x represents the location of the adjacent sample. In addition, it can be applied to N samples included in the upper, left, or upper / left regions.

[0395] [Equation 14]

[0396]

[0397] E. The method for calculating the error (or cost) of each block can be determined variably depending on the characteristics of the block. For example, the error may be calculated by considering MR-SAD for a block with LIC applied. Another example may be the error may be calculated by considering MR-SAD for a block with BCW applied. Another example may be the cost may be calculated by considering SAD for a block with DMVR and BDOF applied.

[0398] FIG. 20 illustrates a decoding method according to one embodiment of the present disclosure. The steps or operations illustrated in FIG. 20 are not essential components of the prediction method, and at least some of the steps or operations illustrated in FIG. 20 may be omitted.

[0399] Referring to FIG. 20, the decoding device can obtain information regarding basic prediction (S800).

[0400] For example, a base prediction can be a series of processes for generating a base prediction sample. The base prediction can be either intra prediction, inter prediction, or IBC.

[0401] The decoding processor of the decoding device can parse information about the base prediction for the current block from the bitstream, inherit it from a neighboring block, or derive it using a specific tool.

[0402] Information about the base prediction may include indexing information specifying the base prediction and / or generation information for generating the base prediction block. For example, the decoding processor may parse the indexing information of the base prediction and the generation information of the base prediction block from the bitstream. As another example, the decoding processor may parse the indexing information of the base prediction and, based on the indexing information, inherit the generation information of the base prediction block from a neighboring block. As another example, the decoding processor may parse the indexing information of the base prediction and, based on the indexing information, derive the generation information of the base prediction block using a tool.

[0403] The decoding device can generate a basic prediction block (S810).

[0404] For example, the decoding processor may generate a base prediction block based on information about the base prediction. The decoding processor may generate the base prediction block based on the generation information of the parsed base prediction block, or based on the generation information of the inherited base prediction block. Furthermore, the decoding processor may generate the base prediction block based on the generation information of the derived base prediction block.

[0405] The decoding device can obtain information regarding additional prediction (S820).

[0406] For example, the decoding processor of a decoding device can obtain information regarding additional predictions. "Additional prediction" broadly refers to a prediction process other than the basic prediction, and narrowly refers to a series of processes for generating additional prediction samples.

[0407] Information about additional prediction may include additional prediction availability information indicating whether additional prediction is performed, prediction mode indexing information indicating an additional prediction mode, and additional block generation information for generating an additional prediction block.

[0408] The decoding processor can obtain prediction mode indexing information indicating an additional prediction mode based on additional prediction availability information, and can obtain block generation information for generating an additional prediction block based on the prediction mode indexing information. The decoding processor can parse the additional prediction availability information, prediction mode indexing information, and additional block generation information from the bitstream.

[0409] The decoding processor may parse additional block generation information for generating an additional prediction block based on satisfaction of a predetermined condition. The predetermined condition may include at least one of: (i) whether the additional prediction mode is allowed for the base prediction mode, (ii) whether the additional prediction mode is allowed for a tool applied to the base prediction mode, (iii) whether the additional prediction mode is allowed for the width or height of the current block, (iv) whether the additional prediction mode is allowed in response to a quantization parameter of the current block, and (v) whether the additional prediction mode is allowed in response to a temporal layer of the current block.

[0410] The decoding processor can inherit block generation information for generating additional prediction blocks based on additional prediction availability information. For example, the decoding processor can inherit additional block generation information of a neighboring block that has the same default prediction mode as the default prediction mode of the current block. The decoding processor can derive additional prediction modes of neighboring blocks that have the same default prediction mode as the default prediction mode of the current block, and inherit additional block generation information for the additional prediction modes of neighboring blocks selected based on a priority for inheritance of the additional prediction modes. The decoding processor can derive additional prediction modes of neighboring blocks that have the same default prediction mode as the default prediction mode of the current block, and inherit additional block generation information for the additional prediction modes of neighboring blocks allowed based on the default prediction mode of the current block.

[0411] The decoding processor can derive block generation information for generating additional prediction blocks based on additional prediction availability information. The decoding processor can derive additional prediction performance information for performing the additional prediction using a template matching method. The decoding processor can derive additional prediction performance information for performing the additional prediction using at least one of the TIMD method or the DIMD method.

[0412] Additionally, the additional prediction may be one or more of intra prediction, inter prediction, or IBC. In other words, multiple additional predictions may be applied. For example, a first additional prediction, a second additional prediction, a third additional prediction, … n-th additional predictions may be applied. The decoding processor may obtain information regarding the first additional prediction and information regarding the second additional prediction.

[0413] The decoding device can generate additional prediction blocks (S830).

[0414] For example, the decoding processor can generate an additional prediction block based on information about the additional prediction. The decoding processor can generate the additional prediction block based on the generation information of the parsed additional prediction block, or generate the base prediction block based on the generation information of the inherited additional prediction block. Furthermore, the decoding processor can generate the additional prediction block based on the generation information of the derived additional prediction block.

[0415] A first additional prediction, a second additional prediction, a third additional prediction, … n-th additional prediction may be applied, and the decoding processor may generate a first additional prediction block and a second additional prediction block.

[0416] The decoding device can generate a final prediction block for the current block based on the polymerization of the base prediction block and the additional prediction block (S840).

[0417] For example, the decoding processor can generate a final prediction block for the current block by weighting or weighting averaging the base prediction block and the additional prediction block.

[0418] A first additional prediction, a second additional prediction, a third additional prediction, … n-th additional prediction may be applied, and the decoding processor may generate a final prediction block for the current block by weighting or weighting the base prediction block, the first additional prediction block, and the second additional prediction block.

[0419] FIG. 21 illustrates an encoding method according to one embodiment of the present disclosure. The steps or operations illustrated in FIG. 21 are not essential components of the prediction method, and at least some of the steps or operations illustrated in FIG. 21 may be omitted.

[0420] The encoding device can encode information about basic prediction for generating a basic prediction block (S900).

[0421] For example, an encoding processor of an encoding device can encode information about a base prediction to generate a base prediction block by performing base prediction.

[0422] The encoding device can encode information regarding additional prediction for generating additional prediction blocks (S910).

[0423] For example, the encoding processor can encode information about additional predictions to generate additional prediction blocks by performing additional predictions.

[0424] The encoding device can encode polymerization information for generating a final prediction block for the current block (S920).

[0425] For example, the encoding processor can encode polymerization information for generating a final prediction block for the current block based on polymerization of the base prediction block and the additional prediction block.

[0426] FIG. 22 is a diagram illustrating an example of a content streaming system to which an embodiment according to the present disclosure can be applied.

[0427] Referring to FIG. 22, a content streaming system to which the embodiment(s) of the present specification are applied may largely include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0428] The encoding server compresses content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data, generates a bitstream, and transmits it to the streaming server. Alternatively, if multimedia input devices such as smartphones, cameras, and camcorders directly generate bitstreams, the encoding server may be omitted.

[0429] The above bitstream can be generated by an encoding method or a bitstream generation method to which the embodiment(s) of the present specification are applied, and the streaming server can temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0430] The streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server acts as an intermediary to inform the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, and the streaming server transmits the multimedia data to the user. At this time, the content streaming system may include a separate control server, in which case the control server controls commands / responses between each device within the content streaming system.

[0431] The streaming server can receive content from a media repository and / or an encoding server. For example, when receiving content from the encoding server, the content can be received in real time. In this case, to provide a smooth streaming service, the streaming server can store the bitstream for a certain period of time.

[0432] Examples of the user devices may include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, HMDs), digital TVs, desktop computers, digital signage, etc.

[0433] Each server within the above content streaming system can be operated as a distributed server, in which case data received from each server can be processed in a distributed manner.

[0434] The claims set forth in this specification may be combined in various ways. For example, the technical features of the method claims of this specification may be combined and implemented as a device, and the technical features of the device claims of this specification may be combined and implemented as a method. Furthermore, the technical features of the method claims and the technical features of the device claims of this specification may be combined and implemented as a device, and the technical features of the method claims and the technical features of the device claims of this specification may be combined and implemented as a method.

[0435] Embodiments according to the present disclosure can be used to encode / decode images.

Claims

1. As a decoding method, Obtain information about the basic prediction; Generating a basic prediction block by performing the basic prediction based on information about the above basic prediction; Obtain information regarding additional predictions; Generating an additional prediction block by performing the additional prediction based on information about the additional prediction; generating a final prediction block for the current block based on the polymerization of the base prediction block and the additional prediction block; The above base prediction includes either intra prediction, inter prediction or intra block copy (IBC), A decoding method wherein the above additional prediction comprises at least one of intra prediction, inter prediction or intra block copy.

2. In paragraph 1, obtaining information regarding the additional prediction comprises: Obtain additional prediction availability information indicating whether the above additional prediction is performed; Obtain prediction mode indexing information indicating an additional prediction mode based on the above additional prediction availability information; Including obtaining additional block generation information for generating the additional prediction block based on the above prediction mode indexing information, A decoding method wherein the above block generation information includes at least one of intra block generation information, inter block generation information, or IBC block generation information.

3. In the second paragraph, obtaining additional block generation information for generating the additional prediction block comprises: Including obtaining additional block generation information for generating the additional prediction block based on satisfaction of a set condition, The above conditions are, Whether the above additional prediction mode is allowed for the above basic prediction mode; The above additional prediction mode is allowed for tools applied in the above basic prediction mode; The above additional prediction mode is allowed for the width or height of the current block; The above additional prediction mode is allowed corresponding to the quantization parameters of the current block; or A decoding method wherein the additional prediction mode comprises at least one of those allowed corresponding to a temporal layer of the current block.

4. In paragraph 1, obtaining information regarding the additional prediction comprises: Obtain additional prediction availability information indicating whether to perform the above additional prediction; Including inheriting additional block generation information for generating the additional prediction block based on the additional prediction availability information, A decoding method wherein the above block generation information includes at least one of intra block generation information, inter block generation information, or block copy block generation information.

5. In the fourth paragraph, inheriting additional block generation information for generating the additional prediction block is A decoding method comprising inheriting additional block generation information of a surrounding block having the same basic prediction mode as the basic prediction mode of the current block.

6. In the fourth paragraph, inheriting additional block generation information for generating the additional prediction block is Derive additional prediction modes of surrounding blocks having the same basic prediction mode as the basic prediction mode of the current block; A decoding method comprising inheriting additional block generation information for additional prediction modes of selected surrounding blocks based on priorities for inheritance of additional prediction modes.

7. In the fourth paragraph, inheriting additional block generation information for generating the additional prediction block, Derive additional prediction modes of surrounding blocks having the same basic prediction mode as the basic prediction mode of the current block; A decoding method comprising inheriting additional block generation information for additional prediction modes of allowed neighboring blocks based on the basic prediction mode of the current block.

8. In paragraph 1, obtaining information regarding the additional prediction comprises: Obtain additional prediction availability information indicating whether to perform the above additional prediction; Including deriving additional block generation information for generating the additional prediction block based on the additional prediction availability information, A decoding method wherein the above block generation information includes at least one of intra block generation information, inter block generation information, or block copy block generation information.

9. In paragraph 8, deriving additional prediction performance information for performing the additional prediction is as follows: A method comprising deriving additional prediction performance information for performing the additional prediction using a template matching method.

10. In the 8th paragraph, deriving additional prediction performance information for performing the additional prediction is as follows: A decoding method comprising deriving additional prediction performance information for performing the additional prediction using at least one of a TIMD method or a DIMD method.

11. In the first paragraph, the information regarding the additional prediction includes information regarding the first additional prediction and information regarding the second additional prediction, The above additional prediction block includes a first additional prediction block and a second additional prediction block, A decoding method wherein generating the final prediction block comprises generating the final prediction block for the current block based on a superposition of the base prediction block, the first additional prediction block, and the second additional prediction block.

12. As an encoding method, Encode information about the base prediction to generate a base prediction block by performing the base prediction; Encode information about additional predictions to generate additional prediction blocks by performing additional predictions; Encoding polymerization information for generating a final prediction block for the current block based on polymerization of the base prediction block and the additional prediction block, The above base prediction includes either intra prediction, inter prediction or intra block copy (IBC), An encoding method wherein the above additional prediction comprises at least one of intra prediction, inter prediction or intra block copy.

13. As a method of transmitting video information, Encode information about the base prediction to generate a base prediction block by performing the base prediction; Encode information about additional predictions to generate additional prediction blocks by performing additional predictions; Encodes the polymerization information for generating a final prediction block for the current block based on the polymerization of the above-mentioned basic prediction block and the above-mentioned additional prediction block, Including transmitting image information including information about the encoded basic prediction, information about the encoded additional prediction, and the encoded polymerization information, The above base prediction includes either intra prediction, inter prediction or intra block copy (IBC), A transmission method wherein the above additional prediction comprises at least one of intra prediction, inter prediction or intra block copy.

Images