Method and computer-readable storage medium

Abstract

A method according to an embodiment may comprise the steps of: acquiring image information from a bitstream; on the basis of the image information, configuring an MVP candidate list including MVP candidates for the current block; and generating a prediction block for the current block on the basis of at least one MVP candidate in the MVP candidate list, wherein the step of configuring the MVP candidate list includes adding, to the MVP candidate list, an MVP candidate having basic motion information and additional motion information, the basic motion information includes basic motion vector information and basic auxiliary information, the additional motion information includes additional motion vector information and additional auxiliary information, and the basic auxiliary information and the additional auxiliary information are changed according to a predetermined condition or a predetermined criterion.

Description

Method and computer-readable storage medium

[0001] The present disclosure relates to a method for encoding / decoding image information, a computer-readable storage medium for storing image information, and a method for transmitting image information.

[0002] Recently, the demand for high-resolution, high-quality images, such as HD (High Definition) images and UHD (Ultra High Definition) images, is increasing in various application fields, and accordingly, high-efficiency image compression technologies are being discussed.

[0003] There are various technologies such as inter prediction technology that predicts pixel values ​​included in the current picture from pictures before or after the current picture, intra prediction technology that predicts pixel values ​​included in the current picture using pixel information within the current picture, and entropy coding technology that assigns short codes to values ​​with high frequency of appearance and long codes to values ​​with low frequency of appearance, and these video compression technologies can be used to effectively compress and transmit or store video data.

[0004] Accordingly, a highly efficient image compression technology is required to effectively transmit, store, and play high-resolution, high-quality image information.

[0005] The present disclosure provides a method for improving inter-screen prediction performance by utilizing various motion vectors by including candidates including multiple motion vectors in the list during the process of constructing a motion vector predictor candidate list in an inter-prediction mode.

[0006] Additionally, a method is provided to improve compression performance by providing diversity in the reference block generation process through a method that can selectively utilize information other than motion vectors.

[0007] According to one embodiment, a method includes the steps of: obtaining image information from a bitstream; constructing an MVP candidate list including MVP candidates for the current block based on the image information; and generating a prediction block for the current block based on at least one MVP candidate in the MVP candidate list; wherein the step of constructing the MVP candidate list includes including an MVP candidate having basic motion information and additional motion information in the MVP candidate list, wherein the basic motion information includes basic motion vector information and basic additional information, and the additional motion information includes additional motion vector information and additional additional information, and the basic additional information and the additional additional information can be changed according to a predetermined condition or a predetermined criterion.

[0008] According to one embodiment, a method includes the steps of: constructing an MVP candidate list including MVP candidates for a current block; generating residual information for the current block based on at least one MVP candidate in the MVP candidate list; and encoding image information including information about a prediction applied to the current block and the residual information; wherein the step of constructing the MVP candidate list includes including an MVP candidate having basic motion information and additional motion information in the MVP candidate list, wherein the basic motion information includes basic motion vector information and basic additional information, and the additional motion information includes additional motion vector information and additional additional information, and the basic additional information and the additional additional information can be changed according to a predetermined condition or a predetermined criterion.

[0009] A computer-readable storage medium storing a bitstream generated by an encoding method according to one embodiment, the encoding method comprising: a step of constructing an MVP candidate list including MVP candidates for a current block; a step of generating residual information for the current block based on at least one MVP candidate in the MVP candidate list; and a step of encoding image information including information about a prediction applied to the current block and the residual information; wherein the step of constructing the MVP candidate list includes including an MVP candidate having basic motion information and additional motion information in the MVP candidate list, wherein the basic motion information includes basic motion vector information and basic additional information, the additional motion information includes additional motion vector information and additional additional information, and the basic additional information and the additional additional information can be changed according to a predetermined condition or a predetermined criterion.

[0010] A method for transmitting data for an image according to one embodiment, the method comprising: obtaining a bitstream for the image, wherein the bitstream is generated based on a step of constructing an MVP candidate list including MVP candidates for a current block, a step of generating residual information for the current block based on at least one MVP candidate in the MVP candidate list, and a step of encoding image information including information about a prediction applied to the current block and the residual information; and a step of transmitting the data including the bitstream; wherein the step of constructing the MVP candidate list includes including an MVP candidate having basic motion information and additional motion information in the MVP candidate list, wherein the basic motion information includes basic motion vector information and basic additional information, the additional motion information includes additional motion vector information and additional additional information, and the basic additional information and the additional additional information can be changed according to a predetermined condition or a predetermined criterion.

[0011] According to the present disclosure, in the process of constructing a motion vector predictor candidate list in an inter prediction mode, inter-screen prediction performance can be improved by utilizing various motion vectors by including candidates including multiple motion vectors in the list.

[0012] Additionally, compression performance can be improved by providing diversity in the reference block generation process through a method that can selectively utilize information other than motion vectors.

[0013] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.

[0014] FIG. 1 illustrates a video / image coding system according to the present disclosure.

[0015] FIG. 2 is a schematic block diagram of an encoding device to which an embodiment of the present disclosure can be applied and in which encoding of a video / image signal is performed.

[0016] FIG. 3 is a schematic block diagram of a decoding device to which an embodiment of the present disclosure can be applied and in which decoding of a video / image signal is performed.

[0017] FIG. 4 illustrates an example of a video / image decoding method to which an embodiment of the present disclosure can be applied.

[0018] FIG. 5 illustrates an example of a video / image encoding method to which an embodiment of the present disclosure can be applied.

[0019] Figure 6 is a diagram showing an example of a search area used in intra template matching.

[0020] FIG. 7 and FIG. 8 illustrate examples of inter prediction-based video / image encoding methods to which embodiments of the present disclosure can be applied.

[0021] FIGS. 9 and 10 illustrate examples of inter-prediction based video / image decoding methods to which embodiments of the present disclosure can be applied.

[0022] FIG. 11 exemplarily illustrates an inter prediction procedure to which an embodiment of the present disclosure can be applied.

[0023] Figure 12 is a diagram showing examples of blocks used to construct a merge candidate list.

[0024] Figure 13 is a diagram showing four movements that can be expressed in the affine movement model.

[0025] Figure 14 is a diagram showing an example of a control point motion vector used in affine motion prediction.

[0026] Figure 15 is an example showing inheritance of control point motion vectors.

[0027] Figure 16 is a drawing showing an example of surrounding blocks for the current block.

[0028] Figure 17 illustrates the process of SbTMVP.

[0029] Figure 18 illustrates an example of GPM segments grouped at the same angle.

[0030] Figure 19 illustrates the left and upper neighboring blocks used in CIIP weight derivation.

[0031] Figure 20 illustrates available IPM candidates for GPM including inter and intra prediction.

[0032] Figure 21 is a diagram illustrating a method for deriving a motion vector from template matching.

[0033] Figure 22 is a diagram showing a case where multiple reference blocks are used in one embodiment.

[0034] FIG. 23 is a flowchart illustrating an example of a method for including an MVP candidate including multiple motion vectors in an MVP candidate list in a decoding method or an encoding method according to one embodiment.

[0035] FIG. 24 is a diagram illustrating an example of a method for constructing a list of MVP candidates in one embodiment.

[0036] FIG. 25 is a diagram illustrating an example of a method for generating a candidate having multiple motion vectors according to one embodiment.

[0037] FIG. 26 is a diagram illustrating an example of a signaling / parsing method when a multi-reference block mode is included as one of the general merge modes in a method according to one embodiment.

[0038] Figure 27 shows the signaling / parsing structure of each detailed mode within the general merge mode.

[0039] Figure 28 shows the signaling / parsing structure when the multi-mvp mode is treated as a separate mode rather than merge mode.

[0040] FIG. 29 is a diagram illustrating an example of a content streaming system to which an embodiment according to the present disclosure can be applied.

[0041] The present disclosure may be modified in various ways and encompasses numerous embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present disclosure to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present disclosure. Throughout the description of each drawing, similar reference numerals have been used to designate similar components.

[0042] While terms such as "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present disclosure, a first component could be referred to as a "second component," and similarly, a second component could also be referred to as a "first component." The term "and / or" includes a combination of multiple related items described herein or any of multiple related items described herein.

[0043] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0044] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present disclosure. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0045] The present disclosure relates to video / image coding. For example, the methods / embodiments disclosed in this specification can be applied to methods disclosed in the versatile video coding (VVC) standard. In addition, the methods / embodiments disclosed in this specification can be applied to methods disclosed in the essential video coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the second generation of audio video coding standard (AVS2), or the next generation of video / image coding standards (e.g., H.267 or H.268).

[0046] This specification presents various embodiments of video / image coding, and unless otherwise stated, the embodiments may be performed in combination with each other.

[0047] In this specification, video may refer to a set of images over time. A picture generally refers to a unit representing one image at a specific time point, and a slice / tile is a unit that constitutes part of a picture in coding. A slice / tile may include one or more coding tree units (CTUs). A picture may be composed of one or more slices / tiles. A tile is a rectangular area consisting of multiple CTUs within a specific tile column and a specific tile row of a picture. A tile column is a rectangular area of ​​CTUs that has a height equal to the height of the picture and a width specified by the syntax requirements of the picture parameter set. A tile row is a rectangular area of ​​CTUs that has a height specified by the picture parameter set and a width equal to the width of the picture. CTUs within a tile are arranged consecutively according to the CTU raster scan, while tiles within a picture may be arranged consecutively according to the tile raster scan. A slice may contain an integer number of complete tiles or an integer number of contiguous complete CTU rows within a picture, which may be exclusively contained within a single NAL unit. Meanwhile, a picture may be divided into two or more subpictures. A subpicture may be a rectangular region of one or more slices within a picture.

[0048] A pixel, or pel, can refer to the smallest unit that constitutes a picture (or image). Additionally, the term "sample" can be used as a counterpart to a pixel. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luminance component, or only the pixel / pixel value of the chrominance component.

[0049] A unit may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. One unit may include one luma block and two chroma (e.g., cb, cr) blocks. In some cases, the term "unit" may be used interchangeably with terms such as "block" or "area." In general, an MxN block may include a set (or array) of samples (or sample array) or transform coefficients consisting of M columns and N rows.

[0050] In this specification, “A or B” can mean “only A,” “only B,” or “both A and B.” In other words, “A or B” in this specification can be interpreted as “A and / or B.” For example, “A, B or C” in this specification can mean “only A,” “only B,” “only C,” or “any combination of A, B, and C.”

[0051] As used herein, a slash ( / ) or a comma can mean "and / or." For example, "A / B" can mean "A and / or B." Accordingly, "A / B" can mean "only A," "only B," or "both A and B." For example, "A, B, C" can mean "A, B, or C."

[0052] In this specification, “at least one of A and B” may mean “only A,” “only B,” or “both A and B.” Additionally, in this specification, the expressions “at least one of A or B” or “at least one of A and / or B” may be interpreted identically to “at least one of A and B.”

[0053] Additionally, in this specification, “at least one of A, B and C” can mean “only A,” “only B,” “only C,” or “any combination of A, B and C.” Additionally, “at least one of A, B or C” or “at least one of A, B and / or C” can mean “at least one of A, B and C.”

[0054] Additionally, parentheses used herein may mean "for example." Specifically, when "prediction (intra-prediction)" is indicated, "intra-prediction" may be suggested as an example of "prediction." In other words, "prediction" in this specification is not limited to "intra-prediction," and "intra-prediction" may be suggested as an example of "prediction." Furthermore, even when "prediction (i.e., intra-prediction)" is indicated, "intra-prediction" may be suggested as an example of "prediction."

[0055] Technical features individually described in a single drawing in this specification may be implemented individually or simultaneously.

[0056] FIG. 1 illustrates a video / image coding system according to the present disclosure.

[0057] Referring to FIG. 1, a video / image coding system may include a first device (source device) and a second device (receiving device).

[0058] A source device can transmit encoded video / image information or data to a receiving device via a digital storage medium or a network in the form of a file or streaming. The source device may include a video source, an encoding device, and a transmitting device. The receiving device may include a receiving device, a decoding device, and a renderer. The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, and the display unit may be configured as a separate device or an external component.

[0059] A video source may obtain video / images through a process of capturing, synthesizing, or generating video / images. The video source may include a video / image capture device and / or a video / image generation device. The video / image capture device may include one or more cameras, a video / image archive containing previously captured video / images, etc. The video / image generation device may include a computer, a tablet, a smartphone, etc., and may (electronically) generate video / images. For example, a virtual video / image may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process of generating related data.

[0060] An encoding device can encode input video / images. The encoding device can perform a series of procedures, such as prediction, transformation, and quantization, to improve compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0061] The transmission unit can transmit encoded video / image information or data output in the form of a bitstream to the receiving unit of a receiving device via a digital storage medium or a network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The storage medium can be a computer-readable storage medium and can store data non-transitory. The transmission unit can include an element for generating a media file via a predetermined file format and an element for transmission via a broadcasting / communication network. The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0062] The decoding device can decode the video / image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoding device.

[0063] The renderer can render decoded video / images. The rendered video / images can be displayed through the display unit.

[0064] FIG. 2 is a schematic block diagram of an encoding device to which an embodiment of the present disclosure can be applied and in which encoding of a video / image signal is performed.

[0065] Referring to FIG. 2, the encoding device (200) may be configured to include an image partitioner (210), a prediction unit (predictor) 220, a residual processor (residual processor) 230, an entropy encoder (entropy encoder) 240, an adder (adder) 250, a filter (filter) 260, and a memory (memory) 270. The prediction unit (220) may include an inter prediction unit (221) and an intra prediction unit (222). The residual processor (230) may include a transformer (transformer) 232, a quantizer (quantizer) 233, a dequantizer (dequantizer) 234, and an inverse transformer (inverse transformer) 235. The residual processing unit (230) may further include a subtractor (231). The addition unit (250) may be called a reconstructor or a recontructed block generator. The image segmentation unit (210), the prediction unit (220), the residual processing unit (230), the entropy encoding unit (240), the addition unit (250), and the filtering unit (260) described above may be configured by one or more hardware components (e.g., an encoding device chipset or processor) according to an embodiment. In addition, the memory (270) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (270) as an internal / external component.

[0066] The image segmentation unit (210) can segment an input image (or picture, frame) input to the encoding device (200) into one or more processing units (PUs). For example, the processing units may be called coding units (CUs). In this case, the coding units may be recursively segmented from a coding tree unit (CTU) or a largest coding unit (LCU) according to a QTBTTT (Quad-Tree Binary-Tree Ternary-Tree) structure.

[0067] For example, a single coding unit may be split into multiple coding units with deeper depths based on a quad-tree structure, a binary tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure may be applied first, and the binary tree structure and / or the ternary structure may be applied later. Alternatively, the binary tree structure may be applied before the quad-tree structure. The coding procedure according to the present specification may be performed based on the final coding unit that is no longer split. In this case, based on coding efficiency according to image characteristics, etc., the largest coding unit may be used directly as the final coding unit, or, if necessary, the coding unit may be recursively split into coding units of lower depths, and the coding unit with the optimal size may be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration, which will be described later.

[0068] As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may each be split or partitioned from the final coding unit described above. The prediction unit may be a unit of sample prediction, and the transform unit may be a unit for deriving a transform coefficient and / or a unit for deriving a residual signal from a transform coefficient.

[0069] The term "unit" may be used interchangeably with terms such as "block" or "area" depending on the case. In general, an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and can represent only a pixel / pixel value of a luminance component or only a pixel / pixel value of a chrominance component. A sample can be used as a term corresponding to a pixel or pel of a picture (or image).

[0070] The encoding device (200) can generate a residual signal (residual block, residual sample array) by subtracting a prediction signal (prediction block, prediction sample array) output from an inter prediction unit (221) or an intra prediction unit (222) from an input video signal (original block, original sample array), and the generated residual signal is transmitted to a conversion unit (232). In this case, a unit that subtracts a prediction signal (prediction block, prediction sample array) from an input video signal (original block, original sample array) within the encoding device (200) may be called a subtraction unit (231).

[0071] The prediction unit (220) can perform a prediction on a block to be processed (hereinafter, referred to as a current block) and generate a predicted block including prediction samples for the current block. The prediction unit (220) can determine whether intra prediction or inter prediction is applied on a current block or CU basis. The prediction unit (220) can generate various information related to prediction, such as prediction mode information, as described later in the description of each prediction mode, and transmit the information to the entropy encoding unit (240). The information related to prediction can be encoded by the entropy encoding unit (240) and output in the form of a bitstream.

[0072] The intra prediction unit (222) can predict the current block by referring to samples within the current picture. The referenced samples, i.e., the reference samples, may be located in the neighborhood of the current block or may be located a certain distance away from the current block depending on the prediction mode. In intra prediction, the prediction modes may include one or more non-directional modes and multiple directional modes. The non-directional mode may include at least one of the DC mode or the planar mode. The directional mode may include 33 directional modes or 65 directional modes depending on the degree of detail in the prediction direction. However, this is merely an example, and a greater or lesser number of directional modes may be used depending on the settings. The intra prediction unit (222) may also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0073] The inter prediction unit (221) can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring block can include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. Temporal neighboring blocks may be called collocated reference blocks, collocated CUs (colCUs), etc., and reference pictures including temporal neighboring blocks may be called collocated pictures (colPic). For example, the inter prediction unit (221) may construct a motion information candidate list based on neighboring blocks, and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, and for example, in the case of skip mode and merge mode, the inter prediction unit (221) may use the motion information of neighboring blocks as the motion information of the current block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of the surrounding blocks is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0074] The prediction unit (220) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction for prediction of a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP) mode. In addition, the prediction unit can be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described herein. Palette mode can be viewed as an example of intra coding or intra prediction. When the palette mode is applied, sample values ​​within a picture can be signaled based on information about the palette table and palette index. The prediction signal generated through the prediction unit (220) can be used to generate a restoration signal or a residual signal.

[0075] The transform unit (232) can apply a transform technique to the residual signal to generate transform coefficients. For example, the transform technique can include at least one of a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Karhunen-Loeve Transform (KLT), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transform obtained from a graph when the relationship information between pixels is expressed as a graph. CNT refers to a transform obtained based on generating a prediction signal using all previously restored pixels. In addition, the transform process can be applied to a pixel block having a square size and the same size, or can be applied to a block of a non-square variable size.

[0076] The quantization unit (233) quantizes the transform coefficients and transmits them to the entropy encoding unit (240), and the entropy encoding unit (240) can encode the quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients can be called residual information. The quantization unit (233) can rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on the coefficient scan order, and can also generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form.

[0077] The entropy encoding unit (240) can perform various encoding methods such as exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc. The entropy encoding unit (240) can also encode information necessary for video / image restoration (e.g., values ​​of syntax elements, etc.) together or separately from quantized transform coefficients.

[0078] Encoded information (e.g., encoded video / image information) can be transmitted or stored in the form of a bitstream in units of NAL (network abstraction layer) units. The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. In the present specification, information and / or syntax elements transmitted / signaled from an encoding device to a decoding device may be included in the video / image information. The video / image information may be encoded through the above-described encoding procedure and included in the bitstream. The bitstream may be transmitted via a network or stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media, such as a USB, SD, CD, DVD, Blu-ray, HDD, or SSD. The signal output from the entropy encoding unit (240) may be configured as an internal / external element of the encoding device (200) by a transmitting unit (not shown) and / or a storing unit (not shown), or the transmitting unit may be included in the entropy encoding unit (240).

[0079] The quantized transform coefficients output from the quantization unit (233) can be used to generate a prediction signal. For example, by applying inverse quantization and inverse transformation to the quantized transform coefficients through the inverse quantization unit (234) and the inverse transform unit (235), a residual signal (residual block or residual samples) can be reconstructed. The addition unit (250) can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter prediction unit (221) or the intra prediction unit (222). When there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as a reconstructed block. The addition unit (250) may be called a reconstructor or a reconstructed block generation unit. The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, and can also be used for inter prediction of the next picture after filtering as described below. Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.

[0080] The filtering unit (260) can improve subjective / objective picture quality by applying filtering to the restoration signal. For example, the filtering unit (260) can apply various filtering methods to the restoration picture to generate a modified restoration picture, and store the modified restoration picture in the memory (270), specifically, in the DPB of the memory (270). The various filtering methods can include deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit (260) can generate various information regarding filtering and transmit it to the entropy encoding unit (240). The information regarding filtering can be encoded by the entropy encoding unit (240) and output in the form of a bitstream.

[0081] The modified restored picture transmitted to the memory (270) can be used as a reference picture in the inter prediction unit (221). Through this, when inter prediction is applied, the encoding device can avoid prediction mismatch between the encoding device (200) and the decoding device, and can also improve encoding efficiency.

[0082] The DPB of the memory (270) can store the modified restored picture to be used as a reference picture in the inter prediction unit (221). The memory (270) can store motion information of a block from which motion information in the current picture is derived (or encoded) and / or motion information of blocks in a picture that has already been restored. The stored motion information can be transferred to the inter prediction unit (221) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (270) can store restored samples of restored blocks in the current picture and transfer them to the intra prediction unit (222).

[0083] Image information output in the form of a bitstream from the encoding device (200) can be transmitted to the decoding device (300).

[0084] FIG. 3 is a schematic block diagram of a decoding device to which an embodiment of the present disclosure can be applied and in which decoding of a video / image signal is performed.

[0085] Image information transmitted in the form of a bitstream from the encoding device (200) can be received by the decoding device (300).

[0086] Referring to FIG. 3, the decoding device (300) may be configured to include an entropy decoder (310), a residual processor (320), a predictor (330), an adder (340), a filter (350), and a memory (360). The predictor (330) may include an inter-prediction unit (332) and an intra-prediction unit (331). The residual processor (320) may include a dequantizer (321) and an inverse transformer (321).

[0087] The entropy decoding unit (310), residual processing unit (320), prediction unit (330), addition unit (340), and filtering unit (350) described above may be configured by a single hardware component (e.g., a decoding device chipset or processor) depending on the embodiment. In addition, the memory (360) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (360) as an internal / external component.

[0088] When a bitstream including video / image information is input, the decoding device (300) can restore the image corresponding to the process in which the video / image information is processed in the encoding device of FIG. 2. For example, the decoding device (300) can derive units / blocks based on block division related information obtained from the bitstream. The decoding device (300) can perform decoding using a processing unit applied in the encoding device. Accordingly, the processing unit of decoding may be a coding unit, and the coding unit may be divided from a coding tree unit or a maximum coding unit according to a quad tree structure, a binary tree structure, and / or a ternary tree structure. One or more transform units may be derived from the coding unit. Then, the restored image signal decoded and output by the decoding device (300) can be reproduced through a reproduction device.

[0089] The decoding device (300) can receive a signal output from the encoding device of FIG. 2 in the form of a bitstream, and the received signal can be decoded through the entropy decoding unit (310). For example, the entropy decoding unit (310) can parse the bitstream to derive information (e.g., video / image information) necessary for image restoration (or picture restoration). The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The decoding device can decode the picture further based on the information on the parameter set and / or the general constraint information. The signaling / received information and / or syntax elements described later in this specification can be decoded and obtained from the bitstream through the decoding procedure. For example, the entropy decoding unit (310) can decode information in a bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the values ​​of syntax elements required for image restoration and the quantized values ​​of transform coefficients for residuals. More specifically, the CABAC entropy decoding method receives a bin corresponding to each syntax element in the bitstream, determines a context model using information of the syntax element to be decoded and decoding information of the surrounding and decoding target blocks or information of symbols / bins decoded in the previous step, and predicts the occurrence probability of the bin according to the determined context model to perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element.At this time, the CABAC entropy decoding method can update the context model using the information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Information regarding prediction among the information decoded by the entropy decoding unit (310) is provided to the prediction unit (inter prediction unit (332) and intra prediction unit (331)), and residual values ​​on which entropy decoding is performed by the entropy decoding unit (310), i.e., quantized transform coefficients and related parameter information, can be input to the residual processing unit (320). The residual processing unit (320) can derive a residual signal (residual block, residual samples, residual sample array). In addition, information regarding filtering among the information decoded by the entropy decoding unit (310) can be provided to the filtering unit (350). Meanwhile, a receiving unit (not shown) that receives a signal output from an encoding device may be further configured as an internal / external element of a decoding device (300), or the receiving unit may be a component of an entropy decoding unit (310).

[0090] Meanwhile, a decoding device according to the present specification may be called a video / video / picture decoding device, and the decoding device may be divided into an information decoding device (video / video / picture information decoding device) and a sample decoding device (video / video / picture sample decoding device). The information decoding device may include the entropy decoding unit (310), and the sample decoding device may include at least one of the inverse quantization unit (321), the inverse transformation unit (322), the addition unit (340), the filtering unit (350), the memory (360), the inter prediction unit (332), and the intra prediction unit (331).

[0091] The inverse quantization unit (321) can inverse quantize the quantized transform coefficients and output the transform coefficients. The inverse quantization unit (321) can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scanning order performed in the encoding device. The inverse quantization unit (321) can perform inverse quantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain transform coefficients.

[0092] In the inverse transform unit (322), the transform coefficients are inversely transformed to obtain a residual signal (residual block, residual sample array).

[0093] The prediction unit (320) can perform a prediction on the current block and generate a predicted block including prediction samples for the current block. The prediction unit (320) can determine whether intra-prediction or inter-prediction is applied to the current block based on the information regarding the prediction output from the entropy decoding unit (310), and can determine a specific intra / inter-prediction mode.

[0094] The prediction unit (320) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit (320) can apply intra prediction or inter prediction for prediction of a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP) mode. In addition, the prediction unit can be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described herein. The palette mode can be viewed as an example of intra coding or intra prediction. When palette mode is applied, information about the palette table and palette index may be signaled and included in the video / image information.

[0095] The intra prediction unit (331) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block, or may be located a certain distance away from the current block, depending on the prediction mode. In intra prediction, the prediction modes may include one or more non-directional modes and multiple directional modes. The intra prediction unit (331) may also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0096] The inter prediction unit (332) can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks can include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (332) can construct a motion information candidate list based on the neighboring blocks, and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter prediction can be performed based on various prediction modes, and information about the prediction can include information indicating an inter prediction mode for the current block.

[0097] The addition unit (340) can generate a restoration signal (restored picture, restoration block, restoration sample array) by adding the acquired residual signal to the prediction signal (prediction block, prediction sample array) output from the prediction unit (including the inter-prediction unit (332) and / or intra-prediction unit (331)). When there is no residual for the block to be processed, such as when skip mode is applied, the prediction block can be used as the restoration block.

[0098] The addition unit (340) may be referred to as a restoration unit or restoration block generation unit. The generated restoration signal may be used for intra prediction of the next processing target block within the current picture, may be output after filtering as described below, or may be used for inter prediction of the next picture. Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0099] The filtering unit (350) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (350) can apply various filtering methods to the restored picture to generate a modified restored picture, and transmit the modified restored picture to the memory (360), specifically, to the DPB of the memory (360). The various filtering methods can include deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0100] The (modified) reconstructed picture stored in the DPB of the memory (360) can be used as a reference picture in the inter prediction unit (332). The memory (360) can store motion information of a block from which motion information in the current picture is derived (or decoded) and / or motion information of blocks in an already reconstructed picture. The stored motion information can be transmitted to the inter prediction unit (332) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (360) can store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra prediction unit (331).

[0101] In this specification, the embodiments described in the filtering unit (260), the inter prediction unit (221), and the intra prediction unit (222) of the encoding device (200) can be applied to the filtering unit (350), the inter prediction unit (332), and the intra prediction unit (331) of the decoding device (300) in the same or corresponding manner, respectively.

[0102] FIG. 4 illustrates an example of a video / image decoding method to which an embodiment of the present disclosure can be applied.

[0103] In image / video coding, the pictures that make up an image / video can be encoded / decoded according to a series of decoding orders. The picture order corresponding to the output order of the decoded pictures can be set differently from the decoding order, and based on this, not only forward prediction but also backward prediction can be performed during inter prediction.

[0104] In FIG. 4, S400 may be performed in the entropy decoding unit (310) of the aforementioned decoding device (300), S410 may be performed in the prediction unit (330), S420 may be performed in the residual processing unit (320), S430 may be performed in the addition unit (340), and S440 may be performed in the filtering unit (350). S400 may include a decoding procedure according to the present disclosure, S410 may include an inter / intra prediction procedure according to the present disclosure, S420 may include a residual processing procedure according to the present disclosure, S430 may include a block / picture restoration procedure according to the present disclosure, and S440 may include an in-loop filtering procedure according to the present disclosure.

[0105] Referring to FIG. 4, the decoding device obtains image / video information from a bitstream (S400), performs prediction based on the obtained image / video information (S410), and restores a picture through residual processing (S420, inverse quantization for quantized transform coefficients, inverse transformation) (S430).

[0106] A modified restored picture can be generated by applying an in-loop filtering procedure (S440) to a restored picture generated through the above restoration procedure, and the modified restored picture can be output as a decoded picture and can be stored in a buffer or memory of a decoding device to be used as a reference picture in an inter prediction procedure when decoding a next picture. In some cases, the in-loop filtering procedure can be omitted, in which case the restored picture can be output as a decoded picture and can be stored in a buffer or memory of a decoding device to be used as a reference picture in an inter prediction procedure when decoding a next picture.

[0107] The in-loop filtering procedure (S440) may include a deblocking filtering procedure, a sample adaptive offset (SAO) procedure, an adaptive loop filter (ALF) procedure, and / or a bi-lateral filter procedure, and some or all of them may be omitted. In addition, one or some of the deblocking filtering procedure, the sample adaptive offset (SAO) procedure, the adaptive loop filter (ALF) procedure, and the bi-lateral filter procedure may be sequentially applied, or all of them may be sequentially applied. For example, the SAO procedure may be performed after the deblocking filtering procedure is applied to the restored picture. Or, for example, the ALF procedure may be performed after the deblocking filtering procedure is applied to the restored picture. This may also be performed in an encoding device.

[0108] FIG. 5 illustrates an example of a video / image encoding method to which an embodiment of the present disclosure can be applied.

[0109] In FIG. 5, the prediction step (S500) may be performed in the prediction unit (220) of the encoding device (200) described above, residual processing (S510) based on the prediction result may be performed in the residual processing unit (230), and the step (S520) of encoding image information including prediction information and residual information may be performed in the entropy encoding unit (240). S500 may include an inter / intra prediction procedure according to the present disclosure, S510 may include a residual processing procedure according to the present disclosure, and S520 may include an encoding procedure according to the present disclosure.

[0110] The encoding procedure may optionally include a procedure for encoding information for picture restoration (e.g., prediction information, residual information, partitioning information, etc.) and outputting it in the form of a bitstream, as well as a procedure for generating a restored picture for the current picture and a procedure for applying in-loop filtering to the restored picture.

[0111] The encoding device (200) can derive (corrected) residual samples from the quantized transform coefficients through the inverse quantization unit (234) and the inverse transformation unit (235), and can generate a restored picture based on the prediction samples and (corrected) residual samples, which are outputs of S500. The restored picture generated in this way can be the same as the restored picture generated by the decoding device (300) described above. A modified restored picture can be generated through an in-loop filtering procedure for the restored picture, which can be stored in a buffer or memory, and, as in the case of the decoding device, can be used as a reference picture in the inter prediction procedure when encoding a subsequent picture.

[0112] As described above, some or all of the in-loop filtering procedure may be omitted in some cases. When the in-loop filtering procedure is performed, (in-loop) filtering-related information (parameters) may be encoded by the entropy encoding unit (240) and output in the form of a bitstream, and the decoding device (300) may perform the in-loop filtering procedure in the same manner as the encoding device based on the filtering-related information.

[0113] Through this in-loop filtering procedure, noise occurring during image / video coding, such as blocking artifacts and ringing artifacts, can be reduced, and subjective / objective image quality can be improved. In addition, by performing the in-loop filtering procedure in both the encoding device (200) and the decoding device (300), the same prediction results can be derived from the encoding device (200) and the decoding device (300), thereby increasing the reliability of picture coding and reducing the amount of data that must be transmitted for picture coding.

[0114] As described above, the picture restoration procedure can be performed not only in the decoding device (300) but also in the encoding device (200). A restoration block can be generated based on intra-prediction / inter-prediction for each block, and a restoration picture including the restoration blocks can be generated. If the current picture / slice / tile group is an I picture / slice / tile group, the blocks included in the current picture / slice / tile group can be restored based only on intra-prediction. On the other hand, if the current picture / slice / tile group is a P or B picture / slice / tile group, the blocks included in the current picture / slice / tile group can be restored based on intra-prediction or inter-prediction. In this case, inter-prediction may be applied to some blocks in the current picture / slice / tile group, and intra-prediction may be applied to some remaining blocks.

[0115] The color component of a picture may include a luma component and a chroma component, and unless explicitly limited in the present disclosure, embodiments according to the present disclosure may be applied to the luma component and the chroma component.

[0116] Meanwhile, when intra prediction is performed, the prediction unit (220, 330) of the encoding device (200) / decoding device (300) can derive a reference sample according to the intra prediction mode of the current block among the surrounding samples of the current block, and can generate a prediction sample of the current block based on the reference sample.

[0117] For example, (i) the prediction sample can be derived based on the average or interpolation of neighboring reference samples of the current block, and (ii) the prediction sample can be derived based on a reference sample existing in a specific (prediction) direction with respect to the prediction sample among the neighboring reference samples of the current block. The case of (i) can be called a non-directional mode or a non-angular mode, and the case of (ii) can be called a directional mode or an angular mode.

[0118] Additionally, linear interpolation intra prediction (LIP) may be applied to perform intra prediction on the current block by linearly interpolating prediction sample values ​​generated based on the intra prediction mode of the current block.

[0119] Additionally, a temporary prediction sample of the current block may be derived based on filtered peripheral reference samples, and a prediction sample of the current block may be derived by weighting at least one reference sample derived according to an intra prediction mode among existing peripheral reference samples, i.e., unfiltered peripheral reference samples, and the temporary prediction sample. Such prediction may be referred to as Position Dependent Intra Prediction Combination (PDPC).

[0120] In addition, intra prediction encoding can be performed by selecting a reference sample line with the highest prediction accuracy among the surrounding multiple reference sample lines of the current block, deriving a prediction sample using the reference sample located in the prediction direction of the selected line, and then instructing (signaling) the used reference sample line to the decoding device. This case can be referred to as multi-reference line intra prediction (MRL) or MRL-based intra prediction.

[0121] Additionally, the current block can be divided into vertical or horizontal subpartitions, and intra prediction can be performed based on the same intra prediction mode, while peripheral reference samples can be derived and utilized for each subpartition. In other words, in this case, the intra prediction mode for the current block is applied equally to the subpartitions, but peripheral reference samples can be derived and utilized for each subpartition, thereby improving intra prediction performance in some cases. This prediction method can be called intra subpartitions (ISP) or ISP-based intra prediction.

[0122] Additionally, if the prediction direction based on the prediction sample points between surrounding reference samples, i.e., if the prediction direction points to a fractional sample location, the value of the prediction sample can be derived through interpolation of multiple reference samples located around the prediction direction (around the fractional sample location).

[0123] The MPM list for deriving the intra prediction mode described above may be configured differently depending on the intra prediction type. Alternatively, the MPM list may be configured in common regardless of the intra prediction type.

[0124] Spatial Geometric Partitioning Mode (SGPM) is an intra mode similar to the GPM inter-coding tool, where two prediction parts are generated during the intra-prediction process. In this mode, a candidate list is created for each entry, containing one partition and two intra-prediction modes. The partition mode and three intra-prediction modes are used to form a combination. The candidate list length can be set to 16, and the selected candidate index can be signaled.

[0125] The candidate list is reordered using templates, where the SAD between the template's prediction and reconstruction is used for alignment. The template size can be fixed to 1.

[0126] For each partition mode, an IPM list for each part is derived using the same intra-inter GPM list derivation. The size of the IPM list can be set to 3. In the list, the TIMD derivation mode can be replaced by two derivation modes in the horizontal and vertical directions.

[0127] SGPM mode can be applied with limited block sizes as follows:

[0128] 4<=width<=64, 4<=height<=64, width <height*8, height<width*8, width*height> =32

[0129] Adaptive blending is also used in spatial GPM, and the blending depth τ can be derived as follows:

[0130] - If min(width, height)==4, 1 / 2 τ is selected

[0131] - else if min(width, height)==8, τ is selected

[0132] - else if min(width, height)==16, 2 τ is selected

[0133] - else if min(width, height)==32, 4 τ is selected

[0134] - else, 8 τ is selected

[0135] Intra Block Copy (IBC) is a method that can significantly improve the coding efficiency of screen content materials. Since IBC mode is implemented as a block-level coding mode, block matching (BM) can be performed in the encoder to find an optimal block vector (or motion vector) for each CU. Here, the block vector can be used to indicate the displacement from the current block to a reference block already reconstructed within the current block. The luma block vector of an IBD-coded CU can be of integer resolution (or precision). The chroma block vector can be rounded to integer resolution. When combined with AMVR, IBC mode can switch between 1-pel (pixel) and 4-pel (pixel) motion vector resolutions. An IBC-coded CU can be treated as a third prediction mode, rather than an intra- or inter-prediction mode. IBC mode can be applied to CUs with both a width and a height of 64 luma samples or less.

[0136] On the encoder side, hash-based motion estimation can be performed for IBC. The encoder can perform a block-by-block (BD) check for blocks whose width and height are no greater than 16 luma samples. For non-merge mode, block vector search can be performed first using a hash-based search. If the hash search does not return a valid candidate, a block-matching-based local search can be performed.

[0137] In hash-based search, the hash key matching (32-bit CRC) between the current block and the reference block can be extended to all allowed block sizes. The hash key calculation for each location in the current picture is based on 4X4 sub-blocks. For larger current blocks, if all the hash keys of the 4X4 sub-blocks match the hash keys of the corresponding reference locations, the hash key can be determined to match that of the reference block.

[0138] When the hash keys of multiple prediction blocks are found to match those of the current block, the block vector cost of each matched reference is calculated and the minimum cost can be selected.

[0139] In block matching searches, the search range can be set to cover both previous and current CTUs. At the CU level, the IBC mode is signaled by a flag, which can be signaled as IBC AMVP mode or IBC Skip / Merge mode as follows:

[0140] - IBC Skip / Merge Mode: A merge candidate index can be used to indicate which block vectors from a list of neighboring candidate IBC coded blocks are used to predict the current block. The merge list can include spatial, HMVP, and pairwise candidates.

[0141] - IBC AMVP mode: Block vector differences can be coded in the same way as motion vector differences. The block vector prediction method can use two candidates as predictors: one from the left neighbor and one from the upper neighbor (if IBC coded). If neither neighbor is available, the default block can be used as the predictor. A flag can be signaled to indicate the block vector predictor index.

[0142] Intra-template matching prediction (IntraTMP) is a special intra-prediction mode that copies the optimal prediction block where the L-shaped template matches the current template within the reconstructed portion of the current frame. Within a predefined search range, the encoder searches the reconstructed region of the current frame for the template most similar to the current template and uses that block as the prediction block. The encoder then signals the use of this mode, and the decoder performs the same prediction operation.

[0143] Figure 6 is a diagram showing an example of a search area used in intra template matching.

[0144] The prediction signal is generated by matching the L-shaped, top-only, or left-only causal neighbors of the current block with other blocks within the predefined search regions of Fig. 6. As illustrated in Fig. 6, there can be a total of six predefined search regions (i.e., R1 to R6), which include not only some of the reconstructed samples within the current CTU located above, left, bottom-left, and top-right of the current block, but also samples reconstructed from the upper CTU and the left CTU:

[0145] The sum of absolute differences (SAD) is used as a cost function.

[0146] A given search order is utilized across six search regions (i.e., R4, R5, R6, R1, R2, R3). Within each region, the decoder generates a list of up to 19 template-matching block vector candidates, sorted in ascending order by template cost (SAD). The supported modes are:

[0147] 1. Single Predictor: A single predictor is selected from the candidate list.

[0148] 2. Fusion of Multiple Predictors: Multiple predictors are fused to derive the final prediction block. The fusion weights can be calculated based on the template matching cost of each predictor, or a weight derivation method based on a Wiener filter can be used.

[0149] 3. Sub-pixel Precision: When using a single predictor, it supports 1 / 2 pixel, 1 / 4 pixel, and 3 / 4 pixel precision, and provides 8 directions each.

[0150] 4. Linear Filter Model: Applies a linear filter learned between the reference template and the current template to the reference block. This mode can be applied to a single predictor that does not use subpixel precision.

[0151] To ensure a fixed number of SAD comparisons per pixel, the sizes of all search regions (SearchRange_w, SearchRange_h) are set to be proportional to the block sizes (BlkW, BlkH). That is:

[0152] SearchRange_w = min(64,a * BlkW)

[0153] SearchRange_h = min(64,a * BlkH)

[0154] Here, 'a' is a constant that controls the trade-off between gain and complexity, and can be set to a = 5.

[0155] To accelerate the template matching process, the search range of each search area can be subsampled by a factor of three. After finding the optimal match, a refinement process is performed. This refinement is achieved through a second template matching search around the optimal match in the reduced range.

[0156] Intra-template matching can be enabled in CUs with a width and height of 64 or less. The maximum CU size for intra-template matching is configurable.

[0157] Intra template matching prediction mode can be signaled at the CU level via a dedicated flag when DIMD is not used for the current CU.

[0158] Meanwhile, when inter prediction is applied, the prediction unit of the encoding device / decoding device can perform inter prediction on a block-by-block basis to derive prediction samples. Inter prediction can refer to a prediction derived in a manner dependent on data elements (e.g., sample values, or motion information) of pictures other than the current picture. When inter prediction is applied to the current block, a predicted block (prediction sample array) for the current block can be derived based on a reference block (reference sample array) specified by a motion vector on a reference picture pointed to by a reference picture index.

[0159] At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information of the current block can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information may include a motion vector and / or a reference picture index. The motion information may further include information on the inter prediction type (L0 prediction, L1 prediction, Bi prediction, etc.). When inter prediction is applied, the neighboring blocks may include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture.

[0160] The reference picture including the above reference block and the reference picture including the temporal neighboring block may be the same or different. The temporal neighboring block may be called a collocated reference block, a collocated CU (colCU), etc., and the reference picture including the temporal neighboring block may be called a collocated picture (colPic). For example, a motion information candidate list may be constructed based on the neighboring blocks of the current block, and a flag or index information indicating which candidate is selected (used) to derive the motion vector and / or reference picture index of the current block may be signaled.

[0161] Inter prediction can be performed based on various prediction modes. For example, in the case of skip mode and merge mode, the motion information of the current block may be the same as the motion information of the selected neighboring block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted. In the case of motion vector prediction (MVP) mode, the motion vector of the selected neighboring block may be used as a motion vector predictor, and the motion vector difference may be signaled. In this case, the motion vector of the current block can be derived using the sum of the motion vector predictor and the motion vector difference.

[0162] The above motion information may include L0 motion information and / or L1 motion information depending on the inter prediction type (L0 prediction, L1 prediction, Bi prediction, etc.). A motion vector in the L0 direction may be called an L0 motion vector or MVL0, and a motion vector in the L1 direction may be called an L1 motion vector or MVL1. Prediction based on an L0 motion vector may be called an L0 prediction, prediction based on an L1 motion vector may be called an L1 prediction, and prediction based on both the L0 motion vector and the L1 motion vector may be called a bi-prediction (Bi). Here, an L0 motion vector may represent a motion vector associated with a reference picture list L0 (L0), and an L1 motion vector may represent a motion vector associated with a reference picture list L1 (L1). The reference picture list L0 may include pictures preceding the current picture in output order as reference pictures, and the reference picture list L1 may include pictures succeeding the current picture in output order. The preceding pictures may be called forward (reference) pictures, and the succeeding pictures may be called backward (reference) pictures.

[0163] The above reference picture list L0 may further include pictures subsequent to the current picture in output order as reference pictures. In this case, the previous pictures may be indexed first and the subsequent pictures may be indexed next within the reference picture list L0. The above reference picture list L1 may further include pictures previous to the current picture in output order as reference pictures. In this case, the subsequent pictures may be indexed first and the previous pictures may be indexed next within the reference picture list L1. Here, the output order may correspond to a POC (picture order count) order.

[0164] FIG. 7 and FIG. 8 illustrate examples of inter prediction-based video / image encoding methods to which embodiments of the present disclosure can be applied.

[0165] Referring to FIG. 7, the encoding device (200) can perform inter prediction for the current block (S600). The encoding device can derive the inter prediction mode and motion information of the current block, and generate prediction samples of the current block. Here, the inter prediction mode determination, motion information derivation, and prediction sample generation procedures may be performed simultaneously, or one procedure may be performed before the other. For example, as illustrated in FIG. 8, the inter prediction unit (221) of the encoding device (200) may include a prediction mode determination unit (221a), a motion information derivation unit (221b), and a prediction sample derivation unit (221c), and the prediction mode determination unit (221a) may determine the prediction mode for the current block, the motion information derivation unit (221b) may derive motion information of the current block, and the prediction sample derivation unit (221c) may derive prediction samples of the current block.

[0166] For example, the inter prediction unit of the encoding device can search for a block similar to the current block within a certain area (search area) of reference pictures through motion estimation, and derive a reference block whose difference from the current block is minimal or below a certain standard. Based on this, a reference picture index indicating a reference picture where the reference block is located can be derived, and a motion vector can be derived based on the positional difference between the reference block and the current block. The encoding device can determine a mode to be applied to the current block among various prediction modes. The encoding device can compare RD costs for the various prediction modes and determine an optimal prediction mode for the current block.

[0167] For example, when skip mode or merge mode is applied to the current block, the encoding device may configure a merge candidate list described below, and derive a reference block among the reference blocks indicated by the merge candidates included in the merge candidate list, wherein the difference between samples from the current block, i.e., the difference in SAD or SATD, is minimal or below a certain standard. In this case, a merge candidate associated with the derived reference block is selected, and merge index information indicating the selected merge candidate may be generated and signaled to the decoding device. Motion information of the current block may be derived using motion information of the selected merge candidate.

[0168] As another example, when the (A)MVP mode is applied to the current block, the encoding device may configure an (A)MVP candidate list described below, and use the motion vector of a motion vector predictor (mvp) candidate selected from among motion vector predictor (mvp) candidates included in the (A)MVP candidate list as the motion vector predictor of the current block. In this case, for example, a motion vector pointing to a reference block derived by the above-described motion estimation may be used as the motion vector of the current block, and a motion vector predictor candidate having a motion vector with the smallest difference from the motion vector of the current block among the motion vector predictor candidates may be the selected motion vector predictor candidate. A Motion Vector Difference (MVD), which is the difference obtained by subtracting the motion vector predictor from the motion vector of the current block, may be derived. In this case, information about the MVD may be signaled to the decoding device. In addition, when the (A)MVP mode is applied, the value of the reference picture index may be configured with reference picture index information and signaled separately to the decoding device.

[0169] The encoding device can derive residual samples based on predicted samples (S610). The encoding device can derive residual samples by comparing the original samples of the current block with the predicted samples.

[0170] An encoding device can encode image information including prediction information and residual information (S620). The encoding device can output the encoded image information in the form of a bitstream. The prediction information is information about prediction and may include prediction mode information (e.g., skip flag, merge flag, or merge index, etc.) and / or motion information. The motion information may include candidate selection information (e.g., merge index, mvp flag, or mvp index), which is information for deriving a motion vector. In addition, the motion information may include information about the above-described MVD and / or reference picture index information. In addition, the motion information may include information indicating whether L0 prediction, L1 prediction, or bi-prediction is applied. The residual information is information about residual samples. The residual information may include information about quantized transform coefficients for the residual samples.

[0171] The output bitstream can be stored on a (digital) storage medium and transmitted to a decoding device, or can be transmitted to a decoding device via a network.

[0172] Meanwhile, as described above, the encoding device can generate a reconstructed picture (including reconstructed samples and reconstructed blocks) based on reference samples and residual samples. This is to derive the same prediction result as that performed by the decoding device from the encoding device, thereby improving coding efficiency. Accordingly, the encoding device can store the reconstructed picture (or reconstructed samples, reconstructed blocks) in memory and use it as a reference picture for inter prediction. As described above, an in-loop filtering procedure, etc. can be further applied to the reconstructed picture.

[0173] FIGS. 9 and 10 illustrate examples of inter-prediction based video / image decoding methods to which embodiments of the present disclosure can be applied.

[0174] A video / image decoding procedure based on inter prediction may roughly include, for example:

[0175] Referring to FIGS. 9 and 10, the decoding device (300) can perform an operation corresponding to the operation performed in the encoding device (200). The decoding device can perform a prediction on the current block based on the received prediction information and derive prediction samples.

[0176] Specifically, the decoding device can determine a prediction mode for the current block based on the received prediction information (S700). The prediction mode determination unit (332a) of the decoding device (300) can determine which inter prediction mode is applied to the current block based on the prediction mode information in the prediction information.

[0177] For example, based on the merge flag, it can be determined whether the current block is subject to merge mode or (A)MVP mode. Alternatively, one of various inter prediction mode candidates can be selected based on the mode index. The inter prediction mode candidates can include skip mode, merge mode, and / or (A)MVP mode, or can include various inter prediction modes described below.

[0178] The decoding device can derive motion information of the current block based on the determined inter prediction mode (S710). For example, when skip mode or merge mode is applied to the current block, the motion information derivation unit (332b) of the decoding device (300) can construct a merge candidate list described below and select one merge candidate from among the merge candidates included in the merge candidate list. This selection can be performed based on the above-described selection information (merge index). Motion information of the current block can be derived using motion information of the selected merge candidate. Motion information of the selected merge candidate can be used as motion information of the current block.

[0179] As another example, when the (A)MVP mode is applied to the current block, the decoding device may construct an (A)MVP candidate list described below, and use the motion vector of an MVP candidate selected from among the MVP (motion vector predictor) candidates included in the (A)MVP candidate list as the MVP of the current block. This selection may be performed based on the selection information (mvp flag or mvp index) described above. In this case, the MVD of the current block may be derived based on information about the MVD, and the motion vector of the current block may be derived based on the MVP and MVD of the current block. In addition, the reference picture index of the current block may be derived based on reference picture index information. A picture indicated by a reference picture index within the reference picture list for the current block may be derived as a reference picture referenced for inter prediction of the current block.

[0180] Meanwhile, the motion information of the current block may be derived without constructing a candidate list, in which case the motion information of the current block may be derived according to the procedure initiated in the prediction mode. In this case, the candidate list construction described above may be omitted.

[0181] The decoding device can generate prediction samples for the current block based on the motion information of the current block (S720). In this case, the prediction sample derivation unit (332c) of the decoding device (300) can derive a reference picture based on the reference picture index of the current block, and derive prediction samples of the current block using samples of the reference block pointed to by the motion vector of the current block on the reference picture. In this case, as described below, a prediction sample filtering procedure may be further performed on all or part of the prediction samples of the current block, depending on the case.

[0182] In other words, the inter prediction unit (332) of the decoding device (300) may include a prediction mode determination unit (332a), a motion information derivation unit (332b), and a prediction sample derivation unit (332c), and may determine a prediction mode for the current block based on the prediction mode information received from the prediction mode determination unit (332a), derive motion information (motion vector and / or reference picture index, etc.) of the current block based on the information about motion information received from the motion information derivation unit (332b), and derive or generate prediction samples of the current block from the prediction sample derivation unit (332c).

[0183] The decoding device generates residual samples for the current block based on the received residual information (S730). The decoding device (300) generates restoration samples for the current block based on the prediction samples and residual samples, and can generate a restoration picture based on these (S740). As described above, in-loop filtering procedures, etc. may be further applied to the restoration picture.

[0184] FIG. 11 exemplarily illustrates an inter prediction procedure to which an embodiment of the present disclosure can be applied.

[0185] Referring to FIG. 11, the inter prediction procedure (S600) as described above may include an inter prediction mode determination step, a motion information derivation step according to the determined prediction mode, and a prediction performance (prediction sample generation) step based on the derived motion information. The inter prediction procedure may be performed in an encoding device and a decoding device as described above. In this document, a coding device may include an encoding device and / or a decoding device.

[0186] Referring to FIG. 11, the coding device determines an inter prediction mode for a current block (S800). Various inter prediction modes can be used to predict the current block within a picture. For example, various modes such as merge mode, skip mode, MVP (Motion Vector Prediction) mode, affine mode, sub-block merge mode, and MMVD (Merge with MVD) mode can be used. Decoder side Motion Vector Refinement (DMVR) mode, Adaptive Motion Vector Resolution (AMVR) mode, bi-prediction with CU-level Weight (BCW), bi-directional optical flow (BDOF), etc. can be used as auxiliary modes. In addition, according to an embodiment, the above-described inter prediction mode can include a multi-hypethesis prediction (MHP) mode. The multi-hypethesis prediction mode represents a method of performing prediction by weighting an additional prediction block generated based on additional motion information to an inter prediction block. The multi-hypethesis prediction mode will be described in detail below.

[0187] In the present disclosure, the affine mode may be referred to as the affine motion prediction mode. In addition, the MVP mode may be referred to as the Advanced Motion Vector Prediction (AMVP) mode. In the present disclosure, motion information candidates derived from some modes and / or some modes may be included as one of the motion information-related candidates of other modes. For example, an HMVP candidate may be added as a merge candidate of the merge / skip mode, or may be added as a motion vector predictor candidate of the AMVP mode. When an HMVP candidate is used as a motion information candidate of the merge mode or the skip mode, the HMVP candidate may be referred to as an HMVP merge candidate.

[0188] Prediction mode information indicating the inter-prediction mode of the current block can be signaled from the encoding device to the decoding device. The prediction mode information can be included in the bitstream and received by the decoding device. The prediction mode information can include index information indicating one of multiple candidate modes. Alternatively, the inter-prediction mode can be indicated through hierarchical signaling of flag information.

[0189] In this case, the prediction mode information may include one or more flags. For example, a skip flag may be signaled to indicate whether skip mode is applied, a merge flag may be signaled to indicate whether merge mode is applied when skip mode is not applied, and MVP mode may be indicated to be applied when merge mode is not applied, or additional flags may be signaled for additional distinction. Affine mode may be signaled as an independent mode, or as a mode dependent on merge mode or MVP mode. For example, an affine mode may include an affine merge mode and an affine MVP mode.

[0190] The coding device can derive motion information for the current block (S810). The motion information can be derived based on the inter-prediction mode determined in the aforementioned step. The coding device can perform inter-prediction using the motion information of the current block. The encoding device can derive optimal motion information for the current block through a motion estimation procedure.

[0191] For example, an encoding device can use an original block within an original picture for a current block to search for a similar reference block with a high correlation within a predetermined search range within the reference picture in fractional pixel units, thereby deriving motion information. The similarity of blocks can be derived based on the difference in phase-based sample values. For example, the similarity of blocks can be calculated based on the SAD between the current block (or a template of the current block) and the reference block (or a template of the reference block). In this case, motion information can be derived based on the reference block with the smallest SAD within the search range. The derived motion information can be signaled to a decoding device in various ways based on an inter prediction mode.

[0192] The coding device can perform inter prediction based on motion information for the current block to generate prediction samples (S820). The current block containing the prediction samples may be referred to as a prediction block.

[0193] Meanwhile, information indicating whether the above-described List0 (L0) prediction, List1 (L1) prediction, or bi-prediction is used for the current block (current coding unit) can be signaled. This information may be called motion prediction direction information, inter-prediction direction information, or inter-prediction indication information, and may be configured / encoded / signaled, for example, in the form of an inter_pred_idc syntax element. That is, the inter_pred_idc syntax element can indicate whether the above-described List0 (L0) prediction, List1 (L1) prediction, or bi-prediction is used for the current block (current coding unit). In this document, for the convenience of explanation, the inter-prediction type (L0 prediction, L1 prediction, or BI prediction) indicated by the inter_pred_idc syntax element may be represented as motion prediction direction. L0 prediction may be represented as pred_L0, L1 prediction as pred_L1, and bi-prediction as pred_BI. For example, depending on the value of the inter_pred_idc syntax element, the prediction type can be indicated as in Table 1 below.

[0194] [Table 1]

[0195]

[0196] As described above, a picture may include one or more slices. A slice may have one of the following types: intra (I) slice, predictive (P) slice, and bi-predictive (B) slice. The slice type may be indicated based on slice type information. For blocks within an I slice, inter prediction is not used for prediction, and only intra prediction can be used. Of course, even in this case, the original sample values ​​can be coded and signaled without prediction. For blocks within a P slice, either intra prediction or inter prediction can be used, and when inter prediction is used, only uni prediction can be used. On the other hand, for blocks within a B slice, either intra prediction or inter prediction can be used, and when inter prediction is used, up to bi prediction can be used.

[0197] L0 and L1 may include reference pictures encoded / decoded before the current picture. For example, L0 may include reference pictures that are before and / or after the current picture in POC order, and L1 may include reference pictures that are after and / or before the current picture in POC order. In this case, L0 may be assigned a relatively lower reference picture index to reference pictures that are before the current picture in POC order, and L1 may be assigned a relatively lower reference picture index to reference pictures that are after the current picture in POC order. For B slices, bi-prediction may be applied, and in this case, either uni-directional bi-prediction or bi-directional bi-prediction may be applied. Bi-directional bi-prediction may be called true bi-prediction.

[0198] Inter prediction can be performed using motion information of the current block. The encoding device can derive optimal motion information for the current block through a motion estimation procedure. For example, the encoding device can search for similar reference blocks with high correlation within a predetermined search range within the reference picture using the original block within the original picture for the current block, in fractional pixel units, and thereby derive motion information. The similarity between blocks can be derived based on the difference in phase-based sample values. For example, the similarity between blocks can be calculated based on the SAD between the current block (or a template of the current block) and the reference block (or a template of the reference block). In this case, motion information can be derived based on the reference block with the smallest SAD within the search range. The derived motion information can be signaled to the decoding device in various ways based on the inter prediction mode.

[0199] When merge mode is applied, the motion information of the current prediction block is not directly transmitted, but the motion information of the surrounding prediction blocks is used to derive the motion information of the current prediction block. Therefore, the motion information of the current prediction block can be indicated by transmitting flag information indicating that merge mode is used and a merge index indicating which surrounding prediction block was used. The merge mode may also be called regular merge mode.

[0200] To perform merge mode, the encoder can search for merge candidate blocks to derive motion information of the current prediction block. For example, up to five merge candidate blocks can be used, but the present invention is not limited thereto. In addition, the maximum number of merge candidate blocks can be transmitted in the slice header or tile group header, but the present invention is not limited thereto. After finding the merge candidate blocks, the encoder can generate a merge candidate list, and select the merge candidate block with the lowest cost among them as the final merge candidate block.

[0201] Figure 12 is a diagram showing examples of blocks used to construct a merge candidate list.

[0202] The present invention provides various embodiments for merge candidate blocks constituting a merge candidate list.

[0203] The merge candidate list may utilize, for example, five merge candidate blocks. For example, four spatial merge candidates and one temporal merge candidate may be utilized. As a specific example, in the case of spatial merge candidates, the blocks illustrated in Fig. 12 may be utilized as spatial merge candidates. Hereinafter, the spatial merge candidates or the spatial MVP candidates described below may be referred to as SMVPs, and the temporal merge candidates or the temporal MVP candidates described below may be referred to as TMVPs.

[0204] The list of merge candidates for the current block can be constructed based on, for example, the following procedure:

[0205] - Insert spatial merge candidates derived by exploring spatial surrounding blocks into the merge candidate list.

[0206] - Insert the temporal merge candidates derived by exploring the temporal surrounding blocks into the merge candidate list.

[0207] - Compare the number of current merge candidates with the maximum number of merge candidates.

[0208] - If the number of current merge candidates is less than the maximum number of merge candidates, insert additional merge candidates into the merge candidate list.

[0209] Specifically, the above-described procedure is explained in detail, the coding device (encoding device / decoding device) searches the spatial neighboring blocks of the current block and inserts the derived spatial merge candidates into the merge candidate list. For example, the spatial neighboring blocks may include the lower left corner neighboring blocks, the left neighboring blocks, the upper right corner neighboring blocks, the upper neighboring blocks, and the upper left corner neighboring blocks of the current block. However, this is merely an example, and in addition to the above-described spatial neighboring blocks, additional neighboring blocks such as the right neighboring blocks, the lower neighboring blocks, and the lower right neighboring blocks may be further used as the spatial neighboring blocks. The coding device may search the spatial neighboring blocks based on priorities to detect available blocks and derive motion information of the detected blocks as spatial merge candidates. For example, the encoder and decoder may search the five blocks illustrated in FIG. 12 in the order of A1, B1, B0, A0, and B2, and sequentially index the available candidates to form a merge candidate list.

[0210] The coding device searches the temporal neighboring blocks of the current block and inserts the derived temporal merge candidates into the merge candidate list. The temporal neighboring blocks may be located on a reference picture that is different from the current picture where the current block is located. The reference picture where the temporal neighboring blocks are located may be called a collocated picture or col picture. The temporal neighboring blocks may be searched in the order of the lower right corner neighboring blocks and the lower right center block of the co-located block of the current block in the col picture.

[0211] Meanwhile, when motion data compression is applied, specific motion information can be stored as representative motion information for each storage unit in a col picture. In this case, there is no need to store motion information for all blocks within a certain storage unit, and thus, a motion data compression effect can be obtained. In this case, a certain storage unit may be predetermined, for example, a 16x16 sample unit or an 8x8 sample unit, or size information for a certain storage unit may be signaled from the encoder to the decoder. When motion data compression is applied, the motion information of a temporal neighboring block may be replaced with representative motion information of a certain storage unit where the temporal neighboring block is located. That is, in terms of implementation, a temporal merge candidate may be derived based on the motion information of a prediction block that covers a position that is arithmetically shifted to the right by a certain value based on the coordinates of the temporal neighboring block (upper left sample position), rather than a prediction block located at the coordinates of the temporal neighboring block. For example, if the certain storage unit is 2 n x2 nIn the case of sample units, if the coordinates of the temporal surrounding block are (xTnb, yTnb), then the modified position is ((xTnb>>n)<<n), (yTnb> >n)< <n))에 위치하는 예측 블록의 움직임 정보가 시간적 머지 후보를 위하여 사용될 수 있다. 구체적으로 예를 들어, 일정 저장 단위가 16x16 샘플 단위인 경우, 시간적 주변 블록의 좌표가 (xTnb, yTnb)라 하면, 수정된 위치인 ((xTnb> >4)<<4), (yTnb>>4)<<4)) motion information of the prediction block located at can be used for the temporal merge candidate. Or, for example, if the storage unit is an 8x8 sample unit and the coordinates of the temporal neighboring block are (xTnb, yTnb), the motion information of the prediction block located at the modified position ((xTnb>>3)<<3), (yTnb>>3)<<3)) can be used for the temporal merge candidate.

[0212] The coding device can compare the current number of merge candidates with the maximum number of merge candidates. The maximum number of merge candidates can be predefined or signaled from the encoder to the decoder. For example, the encoder can generate information regarding the maximum number of merge candidates, encode it, and transmit it to the decoder in bitstream form. Once the maximum number of merge candidates is reached, the subsequent candidate addition process may not proceed.

[0213] If the comparison result shows that the number of current merge candidates is less than the maximum number of merge candidates, the coding device inserts an additional merge candidate into the merge candidate list. The additional merge candidate may include, for example, at least one of the following: history-based merge candidate(s), pair-wise average merge candidate(s), ATMVP, combined bi-predictive merge candidate (if the slice / tile group type of the current slice / tile group is type B), and / or zero-vector merge candidate.

[0214] If the comparison result shows that the number of current merge candidates is not less than the maximum number of merge candidates, the coding device can terminate the construction of the merge candidate list. In this case, the encoder can select the optimal merge candidate among the merge candidates constituting the merge candidate list based on the RD (rate-distortion) cost and signal the selection information (e.g., merge index) indicating the selected merge candidate to the decoder. The decoder can select the optimal merge candidate based on the merge candidate list and the selection information.

[0215] As described above, the motion information of the selected merge candidate can be used as the motion information of the current block, and prediction samples of the current block can be derived based on the motion information of the current block. The encoder can derive residual samples of the current block based on the prediction samples, and can encode residual information about the residual samples and transmit them to the decoder. As described above, the decoder can generate reconstructed samples based on the residual samples and prediction samples derived based on the transmitted residual information, and generate a reconstructed picture based on the reconstructed samples.

[0216] When skip mode is applied, motion information for the current block can be derived in the same way as when merge mode is applied. However, when skip mode is applied, the residual signal for the corresponding block is omitted, so the predicted samples can be used directly as restored samples.

[0217] History-based merge candidates derivation (HMVP) merge candidates can be added to the merge list after spatial and temporal merge candidates. In this method, motion information of previously encoded blocks is stored in a table and used as the MVP of the current coding unit (CU). The table containing multiple HMVP candidates is maintained during encoding and decoding processes. The table is initialized (emptied) when a new coding tree unit (CTU) row begins. Whenever there is a non-subblock inter-coded CU, the associated motion information is added to the last entry of the table as a new HMVP candidate.

[0218] In VVC, the size S of the HMVP table is set to 5, indicating that up to five history-based MVP (HMVP) candidates can be added to the table. When inserting a new move candidate into the table, a limited first-in, first-out (FIFO) rule is applied, and a duplicate check is performed first to determine if there are identical HMVPs in the table. If an identical HMVP is found, it is removed from the table, and all subsequent HMVP candidates are moved forward.

[0219] HMVP candidates can be used in the merge candidate list construction process. The most recent HMVP candidates in the table are sequentially examined and inserted into the merge candidate list after the TMVP candidates. HMVP candidates are also checked for overlap with spatial and temporal merge candidates.

[0220] To reduce the number of redundant check operations, the following simplifications are introduced:

[0221] The number of HMVP candidates used to generate the merge list is set to M if (N ≤ 4), otherwise to (8 - N), where N represents the number of existing candidates in the merge list and M represents the number of available HMVP candidates in the table.

[0222] When the total number of possible merge candidates reaches the maximum number of allowed merge candidates minus 1, the process of constructing the merge candidate list using HMVP ends.

[0223] In this specification, a pair-wise average merge candidate may be referred to as a pair-wise average candidate or a pair-wise candidate. The pair-wise average candidate is generated by averaging predefined pair-wise candidates within an existing merge candidate list, and the predefined pairs are defined as {(0, 1), (0, 2), (1, 2), (0, 3), (1, 3), (2, 3)}. Here, the numbers represent merge indices within the merge candidate list.

[0224] The average motion vector is calculated individually for each reference list. If both motion vectors are available in a single reference list, the average is calculated even if they point to different reference pictures. If only one motion vector is available, that motion vector is used directly. If no motion vectors are available, the list remains invalid.

[0225] If the merge list is not filled after adding pairwise average merge candidates, a zero MVP is inserted at the end of the list until the maximum number of merge candidates is reached.

[0226] This section describes the MMVD mode (Merge mode with MVD) mentioned above. In addition to Merge mode, which directly uses implicitly derived motion information for the prediction samples of the current block, Merge mode using the difference between motion vectors (MMVD) can also be used. Since similar motion information derivation methods are used in Skip mode and Merge mode, MMVD can also be applied to Skip mode. After signaling the Skip flag and Merge flag, MMVD (e.g., mmvd_flag) flag information can be signaled to indicate whether MMVD mode is to be used for the current block.

[0227] In MMVD mode, after a merge candidate is selected, the candidate can be further refined based on the signaled MVD information. When MMVD is applied to the current block (i.e., when mmvd_flag is 1), additional information about MMVD can be signaled. The additional information can include a merge candidate flag (e.g., mmvd_merge_flag) indicating whether the first or second candidate in the merge candidate list is used with motion vector difference, a distance index indicating the motion magnitude (e.g., mmvd_distance_idx), and a direction index indicating the motion direction (e.g., mmvd_direction_idx). In MMVD mode, one of the first two candidates in the merge list can be selected to be used as the MV reference. The merge candidate flag is signaled to indicate which candidate is used.

[0228] The distance index specifies motion size information and represents a predefined offset from the starting point. The offset is added to the horizontal or vertical component of the starting MV. The relationship between the distance index and the predefined offset is shown in Table 2 below.

[0229] [Table 2]

[0230]

[0231] Here, if slice_fpel_mmvd_enabled_flag is 1, it indicates that the merge mode with motion vector difference uses integer sample precision in the current slice. If slice_fpel_mmvd_enabled_flag is 0, it indicates that the merge mode with motion vector difference can use fractional sample precision in the current slice. The slice_fpel_mmvd_enabled_flag syntax element can be signaled via the slice header or included in the slice header.

[0232] The direction index indicates the direction of the MVD with respect to the starting point. The direction index can indicate one of four directions, as shown in Table 3 below. The meaning of the MVD symbol may vary depending on the information of the starting MV. If the starting MV is a non-predicted MV or both lists are bidirectional MVs pointing to the same side of the current picture (i.e., the POCs of both references are both greater than or less than the POC of the current picture), the sign in Table 3 below can indicate the sign of the MV offset added to the starting MV. If the starting MVs are bidirectional predictive MVs and the two MVs point to different sides of the current picture (i.e., the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture), the sign in Table 3 below indicates the sign of the MV offset added to the MV element of List 0 of the starting MV, and the sign of List 1 MV has the opposite value.

[0233] [Table 3]

[0234]

[0235] The two elements of the merge plus MVD offset MmvdOffset[x0][y0] can be derived as follows:

[0236] - MmvdOffset[ x0 ][ y0 ]

[0000] = ( MmvdDistance[ x0 ][ y0 ] << 2 ) * MmvdSign[ x0 ][ y0 ][0]

[0237] - MmvdOffset[ x0 ][ y0 ]

[0001] = ( MmvdDistance[ x0 ][ y0 ] << 2 ) * MmvdSign[ x0 ][ y0 ][1]

[0238] Meanwhile, the MVP (Motion Vector Prediction) mode may also be referred to as the AMVP (advanced motion vector prediction) mode. When the MVP mode is applied, a motion vector predictor (mvp) candidate list can be generated using the motion vectors of the reconstructed spatial neighboring blocks and / or the motion vectors corresponding to the temporal neighboring blocks (or Col blocks). That is, the motion vectors of the reconstructed spatial neighboring blocks and / or the motion vectors corresponding to the temporal neighboring blocks can be used as motion vector predictor candidates. When paired prediction is applied, an mvp candidate list for deriving L0 motion information and an mvp candidate list for deriving L1 motion information can be generated and used separately.

[0239] The above-described prediction information (or information regarding prediction) may include selection information (e.g., MVP flag or MVP index) indicating an optimal motion vector predictor candidate selected from among the motion vector predictor candidates included in the list. In this case, the prediction unit of the decoding device may use the selection information to select a motion vector predictor of the current block from among the motion vector predictor candidates included in the motion vector candidate list.

[0240] The prediction unit of the encoding device can obtain a motion vector difference (MVD) between the motion vector of the current block and the motion vector predictor, and can encode and output it in the form of a bitstream. That is, the MVD can be obtained as a value obtained by subtracting the motion vector predictor from the motion vector of the current block. At this time, the prediction unit of the decoding device can obtain the motion vector difference included in the information regarding the prediction, and derive the motion vector of the current block through the addition of the motion vector difference and the motion vector predictor. The prediction unit of the decoding device can obtain or derive a reference picture index indicating a reference picture, etc. from the information regarding the prediction. For example, a motion vector predictor candidate list can be configured as follows:

[0241] - Search for spatial candidate blocks for motion vector prediction and insert them into the prediction candidate list.

[0242] - Check if the number of spatial candidate blocks is less than 2

[0243] - If the number of spatial candidate blocks is less than 2, search for temporal candidate blocks and insert them into the prediction candidate list.

[0244] - If no temporal candidate block is available, use zero motion vector.

[0245] - If the number of spatial candidate blocks is not less than 2, the construction of the motion vector predictor candidate list is terminated.

[0246] Meanwhile, when the MVP mode is applied, the reference picture index can be explicitly signaled. In this case, the reference picture index for L0 prediction (refidxL0) and the reference picture index for L1 prediction (refidxL1) can be signaled separately. For example, when the MVP mode is applied and bi-prediction (BI prediction) is applied, both information about refidxL0 and information about refidxL1 can be signaled.

[0247] When the MVP mode is applied, information about the Motion Vector Difference (MVD) derived from the encoding device as described above can be signaled or encoded and transmitted to the decoding device. The information about the MVD can include, for example, information indicating x and y components for the MVD absolute value and sign. In this case, information indicating whether the MVD absolute value is greater than 0 and greater than 1, and the MVD remainder can be signaled in stages. For example, information indicating whether the MVD absolute value is greater than 1 can be signaled only when the value of the flag information indicating whether the MVD absolute value is greater than 0 is 1.

[0248] For example, information about an MVD can be encoded in an encoding device and signaled to a decoding device using the following syntax:

[0249] [Table 4]

[0250]

[0251] For example, MVD[compIdx] can be derived based on abs_mvd_greater0_flag[compIdx] * ( abs_mvd_minus2[compIdx] + 2 ) * ( 1 2 * mvd_sign_flag[compIdx]). Here, compIdx (or cpIdx) represents the index of each component and can have the value 0 or 1. A compIdx value of 0 can represent the x component, and a compIdx value of 1 can represent the y component. However, this is just an example, and values ​​for each component can be represented using a coordinate system other than the x, y coordinate system.

[0252] Meanwhile, MVD for L0 prediction (MVDL0) and MVD for L1 prediction (MVDL1) may be signaled separately, and information about MVD may include information about MVDL0 and / or information about MVDL1. For example, if MVP mode and BI prediction are applied to the current block, information about MVDLO and information about MVDL1 may both be signaled.

[0253] Meanwhile, when BI prediction is applied, symmetric MVD may be used considering coding efficiency. In this case, signaling of some of the motion information may be omitted. For example, when symmetric MVD is applied to the current block, information about refidxL0, information about refidxL1, and information about MVDL1 may not be signaled from the encoding device to the decoding device, but may be derived internally. For example, when MVP mode and BI prediction are applied to the current block, flag information indicating whether symmetric MVD is applied (e.g., symmetric MVD flag information or sym_mvd_flag syntax element) may be signaled, and when the value of the flag information is 1, the decoding device may determine that symmetric MVD is applied to the current block.

[0254] When the symmetric MVD mode is applied (i.e., when the value of the symmetric MVD flag information is 1), information about mvp_l0_flag, mvp_l1_flag, and MVDL0 may be explicitly signaled, and signaling of information about refidxL0, information about refidxL1, and information about MVDL1 may be omitted and derived internally as described above. For example, refidxL0 may be derived as an index pointing to a previous reference picture that is closest to the current picture in POC order within reference picture list 0 (which may be referred to as list 0 or L0). refidxL1 may be derived as an index pointing to a subsequent reference picture that is closest to the current picture in POC order within reference picture list 1 (which may be referred to as list 1 or L1). Or, for example, both refidxL0 and refidxL1 may be derived as 0. Or, for example, the refidxL0 and refidxL1 may be derived as the minimum indexes having the same POC difference in relation to the current picture. As a specific example, when [POC of the current picture] - [POC of the first reference picture indicated by refidxL0] is referred to as the first POC difference, and [POC of the second reference picture indicated by refidxL1] is referred to as the second POC difference, only when the first POC difference and the second POC difference are the same, the value of refidxL0 pointing to the first reference picture may be derived as the value of refidxL0 of the current block, and the value of refidxL1 pointing to the second reference picture may be derived as the value of refidxL1 of the current block.Also, for example, if there are multiple sets in which the first POC difference and the second POC difference are the same, refidxL0 and refidxL1 of the set with the minimum difference can be derived as refidxL0 and refidxL1 of the current block.

[0255] MVDL1 can be derived from -MVDL0. For example, the final MV for the current block can be derived as shown in Equation 1 below.

[0256] [Formula 1]

[0257]

[0258] Existing video coding systems use only a single motion vector (using a translation motion model) to represent the motion of a coded block. While this method can optimally represent motion at the block level, it does not accurately represent the optimal motion for each pixel. Therefore, determining the optimal motion vector at the pixel level can improve encoding efficiency. To achieve this, we describe an affine motion prediction method that uses an affine motion model to encode the data.

[0259] Figure 13 is a diagram showing four movements that can be expressed in the affine movement model.

[0260] Affine motion prediction methods can express motion vectors at each pixel unit of a block using two, three, or four motion vectors.

[0261] The affine motion model can express four types of motion, as illustrated in Fig. 13. The affine motion model that expresses three types of motion (translation, scale, and rotation) among the motions that the affine motion model can express is called a similarity (or simplified) affine motion model, and the proposed methods are described below based on the similarity affine motion model. However, the disclosed embodiments are not limited to the motion model.

[0262] Figure 14 is a diagram showing an example of a control point motion vector used in affine motion prediction.

[0263] As illustrated in Figure 14, affine motion prediction can determine the motion vector of the pixel position included in the block using two or more control point motion vectors (CPMV). At this time, the set of motion vectors is called an affine motion vector field (MVF) and can be determined by the equations below.

[0264] For the 4-parameter affine motion model, the motion vector at the sample location (x,y) of the block can be derived by Equation 2 below.

[0265] [Formula 2]

[0266]

[0267] For the 6-parameter affine motion model, the motion vector at the sample location (x,y) of the block can be derived by Equation 3 below.

[0268] [Formula 3]

[0269]

[0270] Here is the CPMV of the CP at the top-left corner of the coding block. is the CPMV of the CP at the top-right corner location, is the CPMV of the CP at the bottom-left corner location. And W corresponds to the width of the current block, H corresponds to the height of the current block, is the motion vector at position {x, y}.

[0271] In the encoding / decoding process, the affine MVF can be determined in units of pixels or pre-defined subblocks. When determined in units of pixels, a motion vector is obtained based on each pixel value, and when determined in units of subblocks, the motion vector of the corresponding block is obtained based on the pixel value at the center of the subblock (the lower right side of the center, i.e., the lower right sample among the four central samples). For example, as illustrated in Fig. 14, it is possible for the affine MVF to be determined in units of 4*4 subblocks. However, this is merely an example, and the size of the subblock applied to affine prediction can be varied.

[0272] When affine prediction is available, the motion models applicable to the current block can include three models: a translational motion model, a 4-parameter affine motion model, and a 6-parameter affine motion model. Here, the translational motion model can represent a model in which existing block-level motion vectors are used, the 4-parameter affine motion model can represent a model in which two CPMVs are used, and the 6-parameter affine motion model can represent a model in which three CPMVs are used.

[0273] Affine motion prediction may include affine MVP (or affine inter) mode and affine merge. In affine motion prediction, the motion vectors of the current block can be derived on a sample-by-sample or sub-block-by-subblock basis.

[0274] In affine merge mode, CPMV can be determined based on the affine motion model of neighboring blocks coded using affine motion prediction. Neighboring blocks affine-coded in search order can be used in affine merge mode. If one or more neighboring blocks are coded using affine motion prediction, the current block can be coded using affine merge.

[0275] That is, when the affine merge mode is applied, the CPMVs of the current block can be derived using the CPMVs of the surrounding blocks. In this case, the CPMVs of the surrounding blocks can be used as the CPMVs of the current block as they are, or the CPMVs of the surrounding blocks can be modified based on the size of the surrounding blocks and the size of the current block and then used as the CPMVs of the current block.

[0276] Meanwhile, in the case of affine merge where MV is derived in units of subblocks, it can be called subblock merge mode, and this can be indicated based on the merge subblock flag (merge_subblock_flag (value 1)). In this case, the affine merging candidate list described later can also be called a subblock merging candidate list. In this case, the subblock merging candidate list can further include a candidate derived by SbTMVP. In this case, the candidate derived by sbTMVP can be used as the candidate for index 0 of the subblock merging candidate list. In other words, the candidate derived by sbTMVP can be positioned before the inherited affine candidates and constructed affine candidates described later in the subblock merging candidate list.

[0277] When affine merge mode is applied, an affine merge candidate list may be constructed to derive CPMVs for the current block. The affine merge candidate list may include, for example, at least one of the following candidates:

[0278] 1) Inherited affine candidates

[0279] 2) Constructed affine candidates

[0280] 3) Zero MVs candidates

[0281] Here, the inherited affine candidate is a candidate derived based on the CPMVs of the surrounding blocks when the surrounding blocks are coded in affine mode, the constructed affine candidate is a candidate derived by constructing CPMVs based on the MVs of the corresponding CP surrounding blocks for each CPMV unit, and the zero MVs candidate can represent a candidate composed of CPMVs whose value is 0. For example, the zero MVs candidate can be optionally inserted into the candidate list when the number of current candidates is less than the number of maximum candidates.

[0282] Figure 15 is an example showing inheritance of control point motion vectors.

[0283] Two inherited affine candidates can be derived from the affine motion model of the surrounding blocks: one from the left surrounding CUs, and the other from the upper CUs.

[0284] Referring back to Figure 12 described above, for the left predictor, the scanning order is A0 -> A1, and for the upper predictor, the scanning order is B0 -> B1 -> B2. Only the first inherited candidate on each side can be selected, and no pruning check is performed between the two inherited candidates.

[0285] Once the surrounding affine CUs are identified, the control point motion vectors are used to derive CPMVP candidates from the affine merge list of the current CU. Referring to Figure 15, when the surrounding lower-left block A is coded in affine mode, the motion vectors v2, v3, and v4 of the upper-left corner, upper-right corner, and lower-left corner of the CU containing block A are obtained. When block A is coded with a 4-parameter affine model, two CPMVs of the current CU are calculated according to v2 and v3. When block A is coded with a 6-parameter affine model, three CPMVs of the current CU are calculated according to v2, v3, and v4.

[0286] Figure 16 is a drawing showing an example of surrounding blocks for the current block.

[0287] The constructed affine candidate refers to a candidate constructed by combining neighbor translational motion information of each control point. Motion information for the control point can be derived from the specified spatial and temporal surroundings, as illustrated in Fig. 16.

[0288] CPMV k (k=1, 2, 3, 4) represents the kth control point. For CPMV1, blocks B2 -> B3 -> A2 are checked in that order, and the MV of the first available block is used. For CPMV2, blocks B1 -> B0 are checked in that order, and for CPMV3, blocks A1 -> A0 are checked in that order. TMVP can be used as CPMV4 if available.

[0289] After acquiring the MVs for the four control points, affine merge candidates can be constructed based on motion information. The following combinations of control point MVs can be used sequentially for construction:

[0290] {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4},

[0291] {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}

[0292] A combination of three CPMVs forms a six-parameter affine merge candidate, and a combination of two CPMVs forms a four-parameter affine merge candidate. To avoid the motion scaling procedure, if the reference indices of the control points are different, the corresponding combination of control point MVs is not used.

[0293] Subblock-based temporal motion vector prediction (SbTMVP) can be used. Similar to Temporal Motion Vector Prediction (TMVP), SbTMVP uses the motion fields of collocated pictures to improve motion vector prediction and merge mode for CUs of the current picture.

[0294] The same collocated pictures used in TMVP are also used in SbTMVP.

[0295] SbTMVP differs from TMVP in two main aspects:

[0296] 1. TMVP predicts motion at the CU level, whereas SbTMVP predicts motion at the sub-CU level.

[0297] 2. While TMVP obtains the motion vector of a collocated block from a collocated picture (a collocated block is a block located at the bottom right or center (bottom-right center) of the current CU), SbTMVP obtains temporal motion information from a collocated picture after applying a motion shift. Here, the motion shift is obtained from the motion vector of one of the spatially adjacent blocks of the current CU.

[0298] Figure 17 illustrates the process of SbTMVP.

[0299] SbTMVP predicts the motion vectors of sub-CUs within the current CU in two steps. In the first step, the spatially adjacent block A1, illustrated in (a) of Figure 17, is examined. If A1 has a motion vector that uses the collocated picture as a reference picture, this motion vector (which may be referred to as a temporal motion vector (tempVM)) is selected as the motion shift to be applied. If such a motion is not identified, the motion shift is set to (0,0).

[0300] In the second step, the motion shift identified in the first step is applied (i.e., added to the coordinates of the current block) to obtain motion information (motion vectors and reference indices) at the sub-CU level from the collocated picture, as illustrated in (b) of Fig. 17. In the example of (b) of Fig. 17, it is assumed that the motion shift is set to the motion of block A1. Then, for each sub-CU, the motion information of the corresponding block (the minimum motion grid covering the center sample) in the collocated picture is used to derive the motion information of the sub-CU. The center sample (the bottom-right center sample) can correspond to the bottom-right sample among the four center samples in the sub-CU if the horizontal and vertical lengths of the sub-block are even. After the motion information of the collocated sub-CU is identified, it is converted into the motion vector and reference index of the current sub-CU in a manner similar to the TMVP process of HEVC.

[0301] In the second step, the motion shift identified in the first step is applied (i.e., added to the coordinates of the current block) to obtain motion information (motion vectors and reference indices) at the sub-CU level from the collocated picture, as illustrated in (b) of Fig. 17. In the example of (b) of Fig. 17, it is assumed that the motion shift is set to the motion of block A1. Then, for each sub-CU, the motion information of the corresponding block (the minimum motion grid covering the center sample) in the collocated picture is used to derive the motion information of the sub-CU. The center sample (the bottom right center sample) may correspond to the bottom right sample among the four center samples in the sub-CU when the horizontal and vertical lengths of the sub-block are even.

[0302] After the motion information of the collocated sub-CU is identified, it is converted into the motion vector and reference index of the current sub-CU in a manner similar to the TMVP process of HEVC. At this time, temporal motion scaling may be applied, which aligns the reference picture of the temporal motion vector with the reference picture of the current CU.

[0303] Bilinear interpolation and sample padding are explained. The resolution of a motion vector (MV) can be 1 / 16 of a luma sample unit. Samples corresponding to fractional positions are interpolated using an 8-tap interpolation filter. In DMVR, search points with integer sample offsets are set around the initial fractional-pel MV, so samples corresponding to the fractional positions must be interpolated during the DMVR search process.

[0304] To reduce computational complexity, DMVR's search process uses a bilinear interpolation filter to generate fractional samples. Another important benefit is that, even with a two-sample search range, the use of a bilinear filter eliminates the need to access more reference samples than with a typical motion compensation process.

[0305] After obtaining a refined MV through the DMVR search process, a standard 8-tap interpolation filter is applied to generate the final prediction. To avoid accessing more reference samples than the typical motion compensation process, samples not required for the original MV reference interpolation process but required for the refined MV reference interpolation process are padded from available samples.

[0306] Geometric partitioning mode (GPM) can be supported for inter prediction. GPM is a type of merge mode that can be signaled using CU-level flags. Other merge modes can include regular merge mode, MMVD mode, CIIP mode, and sub-block merge mode. Each possible CU size is wxh = 2. m x 2 n A total of 64 partitions can be supported, where m,n ∈ {36}, and 8x64 and 64x8 are excluded.

[0307] When this mode is used, a CU can be divided into two parts by a geometrically positioned straight line. The location of the division line can be mathematically derived from the angle and offset parameters of a specific partition. Each part of the geometric partition of the CU can be inter-predicted using its own motion. Only a single prediction is allowed for each partition, i.e., each part has one motion vector and one reference index. A uni-prediction motion constraint can be applied to ensure that only two motion-compensated predictions are required for each CU, similar to conventional bi-prediction.

[0308] Figure 18 illustrates an example of GPM segments grouped at the same angle.

[0309] A single predicted motion for each partition can be derived using a process as illustrated in Fig. 18.

[0310] When GPM is used for the current block, a geometric partition index indicating the partition mode of the geometric partition (angle and offset) and two merge indices (one for each partition) can be additionally signaled. The maximum number of GPM candidate sizes can be explicitly signaled in the SPS, and the syntax binarization of the GPM merge indices can be specified. After predicting each part of the geometric partition, the sample values ​​along the geometric partitioning edge can be adjusted using adaptive weighted blending processing, such as blending along the geometric partitioning edge described below. This is a prediction signal for the entire CU, and the transform and quantization processes can be applied to the entire CU as in other prediction modes. Finally, the motion field of the CU predicted using GPM can be stored in the motion field storage for the GPM geometric partitioning mode.

[0311] Combined inter and intra prediction (CIIP) may be applied to the current block. An additional flag (e.g., ciip_flag) may be signaled to indicate whether the CIIP mode applies to the current CU. For example, when the CU is coded in merge mode, if the CU contains at least 64 luma samples (i.e., the product of the CU width and the CU height is greater than or equal to 64), and both the CU width and the CU height are less than 128 luma samples, the additional flag may be signaled to indicate whether the CIIP mode applies to the current CU.

[0312] CIIP prediction combines inter-prediction signals and intra-prediction signals. The inter-prediction signal P_inter in CIIP mode can be derived using the same inter-prediction process applied in regular merge mode, and the intra-prediction signal P_intra can be derived according to the regular intra-prediction process in planar mode. The intra- and inter-prediction signals can then be combined using a weighted average. Figure 19 illustrates neighboring blocks used in CIIP weight derivation. Here, the weight values ​​can be calculated as follows depending on the coding modes of the left and upper neighboring blocks:

[0313] - If the upper neighbor is available and intra-coded, isIntraTop is set to 1, otherwise isIntraTop is set to 0.

[0314] - If the left neighbor is available and intra-coded, isIntraLeft is set to 1, otherwise isIntraLeft is set to 0.

[0315] - If (isIntraTop + isIntraLeft) is 2, wt is set to 3

[0316] - Otherwise, if (isIntraTop + isIntraLeft) is 1, wt is set to 2

[0317] - Otherwise, wt is set to 1

[0318] The CIIP prediction can be constructed as shown in Equation 4 below.

[0319] [Formula 4]

[0320]

[0321] In a GPM including inter and intra prediction, final predicted samples can be generated by weighting the inter-predicted samples and intra-predicted samples for each GPM-separated region. The inter-predicted samples are derived from the inter-GPM, while the intra-predicted samples can be derived from an intra-prediction mode (IPM) candidate list and an index signaled from an encoding device. For example, the IPM candidate list size can be predefined as 3.

[0322] Figure 20 illustrates available IPM candidates for GPM including inter and intra prediction.

[0323] Available IPM candidates can be parallel angular mode (Parallel mode), perpendicular angular mode (Perpendicular mode), and planar mode with respect to the GPM block boundary, as illustrated in (a), (b), and (c) of FIG. 20. Furthermore, as illustrated in (d) of FIG. 20, GPMs including inter and intra prediction can be restricted to reduce signaling overhead for IPMs and prevent an increase in the intra prediction circuit size of the hardware decoder. In addition, direct motion vector and IPM storage in the GPM blending region can be introduced to further improve coding performance.

[0324] In DIMD and adjacent mode-based IPM derivation, parallel modes can be registered first. Therefore, if there are no identical IPM candidates in the list, up to two IPM candidates derived from the decoder-side intra-mode derivation (DIMD) method and / or neighboring blocks can be registered.

[0325] In deriving the neighbor mode, there are up to five locations for the available neighbor blocks, but this may be limited by the GPM block boundary angles already used in GPM using template matching (GPM-TM).

[0326] Table 5 shows the locations of available neighboring blocks for deriving IPM candidates according to the GPM block boundary angle. A and L can represent the upper and left sides of the predicted block.

[0327] [Table 5]

[0328]

[0329] GPM-Intra can be combined with GPM-MMVD (GPM with merge with motion vector difference). To further improve coding performance, TIMD can be used to generate IPM candidates for GPM-Intra. Parallel modes can be registered first, followed by TIMD, DIMD, and IPM candidates from neighboring blocks.

[0330] Template Matching (TM) is a decoder-side motion vector (MV) derivation method that finds the optimal match between a template of the current coding unit (CU) within the current picture (i.e., the upper and / or left adjacent block of the current CU) and a block within a reference picture (i.e., a block of the same size as the template) to refine the motion information of the current coding unit (CU).

[0331] Figure 21 is a diagram illustrating a method for deriving motion vectors from template matching. For example, as illustrated in Figure 21, a more appropriate motion vector is searched for around the initial motion of the current CU within the [-8, +8] pel search range. Furthermore, the search step size may be determined according to the AMVR mode, and TM may be applied continuously with the bidirectional matching process in merge mode.

[0332] In AMVP mode, the MVP candidate is determined based on the template matching error to select the candidate with the smallest difference between the template of the current block and the template of the reference block, and then TM is performed only on the MVP candidate for MV refinement. The TM refines the MVP candidate using an iterative diamond search within the pel search range [-8, +8] starting from full-pel MVD precision (or 4-pel precision in 4-pel AMVR mode). The AMVP candidate can be further refined by a cross search using full-pel MVD precision (or 4-pel precision in 4-pel AMVR mode), and then sequentially refined to half-pel and quarter-pel precision according to Table 6 specified in AMVR mode.

[0333] [Table 6]

[0334]

[0335] This search process ensures that AMVP candidates maintain the same MV precision specified by the AMVR mode even after the TM process. During the search process, if the difference between the previous minimum cost and the current minimum cost in the iteration is less than a threshold equal to the block area, the search process ends.

[0336] In Merge mode, a similar search method is applied to merge candidates specified by the merge index. As shown in Table 6 above, TM can be performed up to 1 / 8 pel MVD precision, or steps exceeding the half-pel MVD precision can be skipped. This depends on whether the alternative interpolation filter used when AMVR is in half-pel mode is used based on the merged motion information.

[0337] Additionally, when TM mode is enabled, template matching can operate as an independent process or as an additional MV refinement process between block-based and sub-block-based bidirectional matching (BM) methods. This depends on whether BM satisfies the activation conditions and can be applied.

[0338] In the multi-hypothesis inter prediction mode, one or more additional motion-compensated prediction signals are signaled in addition to the existing bidirectional prediction signal, and the resulting overall prediction signal can be obtained by sample-wise weighted superposition. Bidirectional prediction signal p bi And using the first additional inter prediction signal / assumption h3, the resulting prediction signal p3 can be obtained as follows.

[0339] [Formula 5]

[0340]

[0341] The weight factor α can be specified by a new syntax element add_hyp_weight_idx according to the mapping shown in Table 7 below.

[0342] [Table 7]

[0343]

[0344] Similarly, one or more additional prediction signals can be used. The resulting overall prediction signal can be iteratively accumulated with each additional signal, as in Equation 6 below.

[0345] [Formula 6]

[0346]

[0347] The resulting overall prediction signal is the final p n (i.e. p with the largest index n n) can be obtained. For example, up to two additional prediction signals can be used (i.e., n is limited to 2).

[0348] The motion parameters of each additional prediction hypothesis can be explicitly signaled by specifying the reference index, the motion vector predictor index, and the motion vector difference, or implicitly signaled by specifying the merge index. A separate multi-hypothesis merge flag can distinguish these two signaling modes.

[0349] Local Illumination Compensation (LIC) is an inter-prediction technique that models the local illumination variation between a current block and its predicted block as a function of the illumination variation between the current block template and a reference block template. The parameters of this function can be represented by the scale α and the offset β, which form a linear equation for compensating for the illumination variation, i.e., α*p[x]+β, where p[x] is the reference sample pointed to by the MV at position x in the reference picture. When wrap-around motion compensation is enabled, the MV should be clipped to account for the wrap-around offset.

[0350] Since α and β can be derived based on the current block template and the reference block template, no encoding overhead is required for them, and only the LIC flag is signaled to indicate the use of LIC in AMVP mode.

[0351] Local light compensation is used in inter-CUs in the following aspects:

[0352] - Intra-neighbor samples can be used to derive LIC parameters.

[0353] - LIC is disabled for blocks with less than 32 luma samples.

[0354] - For both non-subblock mode and affine mode, LIC parameter derivation is performed based on template block samples corresponding to the current CU, and not based on some template block samples corresponding to the upper left 16Х16 units.

[0355] - Samples of the reference block template are generated by performing motion compensation (MC) using the block MV without rounding the MV to integer pixel units.

[0356] For a pairwise prediction inter-CU, two sets of LIC parameters are derived separately for the L0 and L1 prediction samples, respectively. An iterative approach is used to derive the L0 and L1 LIC parameters. Specifically, the L0 LIC parameters are first derived by minimizing the difference between the L0 template prediction T0 and template T. Then, the samples of template T are updated by subtracting the corresponding samples of T0. Next, the L1 parameters are computed by minimizing the difference between the L1 template prediction T1 and the updated template. Finally, the L0 parameters are computed again in the same manner.

[0357] Meanwhile, when bi-prediction is applied to the current block, prediction samples can be derived based on a weighted average. Applying a weighted average to bi-prediction is called BCW (Bi-prediction with CU-level weight). When bi-prediction is applied, a bi-prediction signal (bi-prediction samples) can be derived through a weighted average of the L0 prediction signal and the L1 prediction signal, as shown in Equation 7 below.

[0358] [Formula 7]

[0359]

[0360] For example, when applying a weighted average to a pairwise prediction, five weights are allowed, and the available weights can be set to w ∈ {-2, 3, 4, 5, 10}. For each pairwise prediction CU, the weight w can be determined in one of two ways:

[0361] -For non-merge CUs, the weight index is signaled after motion vector difference.

[0362] - For merge CUs, the weight index is inferred based on the surrounding blocks and the selected merge candidate index.

[0363] Weighted average biprediction is applied only to CUs with more than 256 luma samples (i.e., CUs with width X height greater than 256). For low-delay pictures, all five weights are used, while for non-low-delay pictures, only three weights (w ∈ {3, 4, 5}) are used.

[0364] In the encoding device (200), a fast search algorithm can be applied to quickly search the weight index without increasing complexity. The related algorithm can be summarized as follows:

[0365] -When combined with AMVR (Adaptive Motion Vector Resolution), unequal weights are conditionally checked for 1-pel and 4-pel motion vector precision only when the current picture is low-delay.

[0366] - When combined with affine mode, Affine ME (motion estimation) is performed on asymmetric weights only when affine mode is selected as the current optimal mode.

[0367] -If the two reference pictures are identical, the asymmetric weights are only checked conditionally.

[0368] - Also, asymmetric weights are not explored if certain conditions are met, such as the difference in POC between the current picture and the reference picture, the coding QP, and the temporal level.

[0369] The BCW weight index is coded using a single context-coded bin followed by bypass-coded bins. The first context-coded bin indicates whether equal weights are used. If asymmetric weights are used, additional bins can be signaled using bypass to indicate the use of asymmetric weights.

[0370] Weighted Prediction (WP) is a coding tool for efficiently encoding video content that includes fading. WP can signal weights and offsets for each reference picture in each list (L0, L1), and the weights and offsets of the corresponding reference pictures are applied during motion compensation. WP and BCW are technologies designed to suit different types of video content. To reduce decoder design complexity, the BCW weight index is not signaled when a CU uses WP, in which case the weight w is assumed to be 4 (equal weight).

[0371] For merge CUs, the weight index is inferred based on the surrounding blocks and the selected merge candidate index. This approach can be applied not only to the standard merge mode but also to the inherited affine merge mode. In the constructed affine merge mode, affine motion information is constructed based on motion information obtained from up to three blocks. In this case, the procedure for deriving the BCW index of a CU is as follows:

[0372] 1) Divide the BCW index range {0,1,2,3,4} into three groups {0}, {1,2,3}, {4}. If the BCW indices of all control points belong to the same group, proceed to step 2. Otherwise, the BCW index of the currently configured candidate is set to 2.

[0373] 2) If two or more control points have the same BCW index, the corresponding index value is assigned to the candidate. Otherwise, the BCW index of the currently configured candidate is set to 2.

[0374] The disclosed embodiments provide a method for enhancing compression performance by utilizing various motion vectors applied to the inter-frame prediction process. This allows for predictions beyond unidirectional or bidirectional prediction. In one embodiment, a motion vector predictor (MVP) containing multiple motion vectors is allowed, allowing for various motion vectors, and the accuracy of the predicted block can be enhanced through a weighted sum between the prediction blocks indicated by each motion vector.

[0375] According to one embodiment, in the process of constructing MVP candidates, a candidate having multiple motion vectors may be included in the candidate list. The candidate having multiple motion vectors may refer to a candidate that includes an additional motion vector in addition to a unidirectional or bidirectional motion vector (a base motion vector). In the disclosed embodiment, bidirectional means bi-directional, which may refer to the L0 direction and the L1 direction. Therefore, it does not necessarily mean only different directions with respect to the current picture, but also includes cases where they are in the same direction. In other words, bi-directional prediction or bi-directional motion vectors may refer to bi-prediction or bi-predictive motion vectors.

[0376] The number of additional motion vectors may be predefined or signaled.

[0377] The above multiple motion vectors can be divided into basic (regular) motion vectors and additional motion vectors, and each motion vector can be included as part of the basic motion information and additional motion information. Each motion information can include BCW index, LIC flag, and interpolation filter information, etc., so that during the process of generating the final prediction block, the information can be changed and stored and utilized according to the characteristics of the block.

[0378] In the process of generating the final prediction block, an additional weight index may be signaled for a weighted sum between the base prediction block and the additional prediction block. The weight index may be modified by changing the signaled weight value and applying it based on factors such as the block size and shape, whether bidirectional prediction is performed, the BCW index, and the LIC flag.

[0379] The above mode having multiple motion vectors can operate as a prediction mode and as a part of the general merge mode.

[0380] Alternatively, the mode having the multiple motion vectors may be signaled as a single prediction mode, separate from the merge mode and the inter mode.

[0381] In one embodiment, a method for including candidates with multiple motion vectors in the process of constructing MVP candidates in inter-screen prediction mode is described. In inter-screen prediction mode, motion vector predictors are used in various modes, such as AMVP mode and merge mode. Since higher predictor accuracy contributes to improved compression performance, constructing a diverse set of candidates is effective.

[0382] According to one embodiment, the MVP candidate having multiple motion vectors can be applied to not only the AMVP mode or the merge mode, but also the affine mode, the AMVP-merge mode, the SMVD (Symmetric MVD), the sbTMVP, the MMVD mode, the GPM mode, the GPM-Inter / Intra mode, the CIIP mode, the TM mode, etc. However, the modes / tools listed above are merely examples to which the MVP candidate having multiple motion vectors according to the disclosed embodiment can be applied, and among the modes or tools not listed above, if there is a mode or tool that uses the MVP candidate, the MVP candidate having multiple motion vectors according to the disclosed embodiment can be applied.

[0383] As mentioned above, the number of multiple motion vectors included in an MVP candidate can be determined based on the number of additional motion vectors defined in advance. For example, referring to the syntax in Table 8 below, information indicating whether multiple motion vectors are supported or allowed in SPS (e.g., sps_multiple_predictor_enabled_flag) and information indicating the maximum number of motion vectors that can be added when multiple motion vectors are allowed, i.e., when the flag is '1' (e.g., sps_max_num_additional_predictor_minus1) can be signaled. This means that 'sps_max_num_additional_predictor_minus1 + 1' additional motion information can be included in addition to the unidirectional or bidirectional motion information that an existing block can have.

[0384] [Table 8]

[0385]

[0386] It is self-evident that the location of information indicating whether multiple motion vectors are allowed and the maximum number of motion vectors that can be added can be included in a higher-order parameter set other than SPS, such as VPS, PPS, APS, PH, SH, etc. For example, if a flag indicating whether multiple motion vectors are allowed is located in PPS, whether multiple motion vectors are allowed can be determined for each picture, not for each sequence.

[0387] In addition, information indicating whether the above multiple motion vectors are allowed and information indicating the maximum number of motion vectors that can be added may be located at different levels so that the maximum number of multiple motion vectors can be specified differently depending on the application unit. For example, as in the example of Table 8, sps_multiple_predictor_enabled_flag indicating whether multiple motion vectors are allowed exists in SPS, and sps_max_num_additional_predictor indicating the maximum number of multiple motion vectors can be signaled at different levels such as picture, slice, CTU, and CU so that different maximum numbers of multiple motion vectors can be variably specified for each unit where sps_max_num_additional_predictor is located.

[0388] Additionally, the maximum number of multiple motion vectors can be set according to a predefined value without being separately signaled. For example, as shown in the syntax of Table 9 below, when the value of information indicating whether multiple motion vectors are allowed (e.g., sps_multiple_predictor_enabled_flag) is 1, the maximum number of motion vectors that can be added can be predefined as a specific integer value without separate signaling.

[0389] [Table 9]

[0390]

[0391] For example, a specific value of 1 or 2 can be used as the maximum number of motion vectors that can be added. This means that one or two additional motion information can be included in addition to the unidirectional or bidirectional motion information that an existing block can have.

[0392] Figure 22 is a diagram showing a case where multiple reference blocks are used in one embodiment.

[0393] In one embodiment, a unidirectional or bidirectional motion vector may be defined as a basic (regular) motion vector, and a reference block obtained using the basic motion vector may be defined as a basic reference block. In addition, a motion vector added here may be defined as an additional motion vector, and a reference block obtained using the motion vector may be defined as an additional reference block. Multiple motion vectors may be defined as including a basic motion vector and an additional motion vector, and multiple reference blocks may be defined as including a basic reference block and an additional reference block.

[0394] Additionally, in some cases, multiple reference blocks or multiple prediction blocks may mean additional reference blocks or additional prediction blocks, and multiple motion vectors may mean additional motion vectors.

[0395] Additionally, a prediction block generated based on a basic reference block may be referred to as a basic prediction block, a prediction block generated based on an additional reference block may be referred to as an additional prediction block, and a multi-prediction block including the basic prediction block and the additional prediction blocks may be referred to as a multi-prediction block. In addition, a prediction block generated based on the basic prediction block and the additional prediction blocks (e.g., weighted sum or weighted average) may be referred to as a final prediction block.

[0396] Alternatively, they may all be referred to as base reference blocks, additional reference blocks, or multiple reference blocks until the final predicted block for the current block is generated.

[0397] Alternatively, a base reference block, multiple reference blocks, or additional reference blocks may be used in combination with a base prediction block, multiple prediction blocks, or additional prediction blocks.

[0398] Referring to FIG. 22, when multiple motion vectors are allowed for the current block C, bidirectional basic reference blocks P0 and P1 can be acquired, and additional reference blocks P2 and P3 can be acquired based on additional motion vectors. For example, a weighted sum can be applied to the basic reference blocks and the additional reference blocks to generate a final prediction block.

[0399] For convenience of explanation, it is assumed that two reference blocks are acquired for each prediction direction, but of course, the number of reference blocks acquired for each direction may change.

[0400] FIG. 23 is a flowchart illustrating an example of a method for including an MVP candidate including multiple motion vectors in an MVP candidate list, according to a decoding method or encoding method according to one embodiment. The decoding method or encoding method of FIG. 23 may be performed by the aforementioned decoding device (300) or encoding device (200).

[0401] Referring to FIG. 23, a decoding method or an encoding method according to one embodiment may include the steps of determining a prediction mode applied to a current block as a prediction mode using MVP (S900), constructing an MVP candidate list including MVP candidates having multiple motion vectors (S910), and generating a prediction block based on the final MVP candidate (S920).

[0402] In the case of a decoding method, image information can be obtained from a bitstream, and an MVP candidate list including MVP candidates for the current block can be constructed based on the image information. Specifically, the image information obtained from the bitstream can include information regarding prediction. Based on whether the prediction mode indicated by the information regarding prediction is a prediction mode using MVP, such as the aforementioned AMVP mode or merge mode, the prediction mode applied to the current block can be determined as a prediction mode using MVP.

[0403] Image information obtained from a bitstream may include information indicating whether multiple motion vectors are supported or allowed (e.g., sps_multiple_predictor_enabled_flag). Based on the information indicating that multiple motion vectors are allowed (e.g., the value of sps_multiple_predictor_enabled_flag is 1), an MVP candidate list including MVP candidates having multiple motion vectors may be constructed (S910).

[0404] A prediction block is generated based on the final MVP candidate in the MVP candidate list (S920). The final MVP candidate refers to an MVP candidate used for prediction of the current block among the MVP candidates in the MVP candidate list. The encoding device (200) can encode information (e.g., index information) indicating the final MVP candidate and transmit it to the decoding device (300) in the form of a bitstream. The decoding device (300) can obtain the corresponding information from the transmitted bitstream and derive the final MVP candidate. If the final MVP candidate has multiple motion vectors, a prediction block can be generated using an additional reference block together with a basic reference block. For example, a prediction block can be generated by applying weights to the basic reference block and the additional reference block. Alternatively, a final prediction block can be generated by applying weights to the basic prediction block and the additional prediction block.

[0405] Meanwhile, it goes without saying that the description of the above-described decoding method and encoding method can be applied to one embodiment as well. For example, a decoding method according to one embodiment can generate a reconstructed block based on residual information (which can be omitted depending on the prediction mode) included in the generated prediction block and image information after generating the prediction block. An encoding method according to one embodiment can generate residual information (which can be omitted depending on the prediction mode) based on the generated prediction block, and encode image information including information about the prediction and the residual information, and transmit it in the form of a bitstream.

[0406] FIG. 24 is a diagram illustrating an example of a method for constructing an MVP candidate list in the disclosed embodiment.

[0407] An example of Fig. 24 illustrates the process of constructing an MVP candidate list in the basic merge mode. Referring to Fig. 24, spatial neighboring candidates (S911), temporal candidates (S912), non-adjacent spatial neighboring candidates (S913), HMVP candidates (S914), candidates with multiple motion vectors (S915), and pairwise candidates (S916) may be included in the MVP candidate list.

[0408] However, the types, order, and number of MVP candidates illustrated in FIG. 24 are merely examples applicable to one embodiment, and the types, order, or number may be changed, or existing MVP candidates may be replaced with MVP candidates having multiple motion vectors. For example, an MVP candidate including multiple motion vectors may be included in the MVP candidate list, replacing a pairwise average candidate.

[0409] Meanwhile, after the MVP candidate list is constructed (S910), a reordering based on template matching costs may be performed on each candidate within the MVP candidate list. In the process of calculating the template matching cost for reordering candidates, the cost of a unidirectional reference block may be calculated based on the difference between the adjacent sample of the current block and the adjacent sample of the reference block, and the cost of a bidirectional reference block may be calculated based on the difference between the adjacent sample of the current block and the adjacent sample of each reference block.

[0410] For candidates containing multiple motion vectors, the cost can be calculated using adjacent samples of the current block and adjacent samples of reference blocks indicated by each motion vector, or the complexity of the calculation can be reduced by using only some of the reference blocks. In this case, if the final prediction block is determined by a weighted sum for each reference block, the cost can be calculated by applying the weight for each reference block to the adjacent samples of each reference block. If only some of the reference blocks are used, the cost can be calculated by applying a corrected form of weight. For example, if only some of the reference blocks are used for cost calculation, a method can be applied to increase the weight for the reference blocks and decrease the weights of the reference blocks not used for cost calculation. However, this is only one example, and it is of course true that various other methods of applying a corrected form of weight can also be applied.

[0411] In addition, in the process of constructing a list of MVP candidates, a method of constructing multiple candidates including multiple motion vectors, and then reordering and selecting only some of the candidates based on template matching cost can be applied. The process of calculating the template matching cost can be applied to the same cost calculation method as the example described above. In addition, since a candidate including multiple motion vectors means that it includes at least two reference blocks, reordering and selecting only some of the candidates can be performed by calculating the bilateral matching cost using each prediction block. The cost can be calculated based on the difference value between each reference block, and it can be calculated for multiple reference blocks, or it can be calculated only for the selected reference blocks by selecting some reference blocks.

[0412] Meanwhile, in the process of constructing the MVP candidate list, a redundancy check between the MVP candidate with multiple motion vectors and other candidates can be performed as follows:

[0413] - Duplication can be determined based on the number of reference blocks for each candidate.

[0414] - If each candidate to be compared has the same number of reference blocks, the presence or absence of overlap can be determined using the reference index and motion vector representing the motion information of each candidate.

[0415] - If there are multiple candidates with multiple motion vectors in the MVP candidate list, and a weighted sum between reference blocks obtained from each motion vector can be applied, and the weights between the reference blocks of each candidate are different, they can be considered as non-overlapping candidates.

[0416] When constructing multiple candidates with multiple motion vectors, the motion vectors contained in each candidate can be restricted to include at least one motion vector different from the motion vectors contained in the preceding candidate with multiple motion vectors in the list. This allows for determining overlap based solely on motion information between candidates, without a weighted overlap check.

[0417] When applying the above method to merge mode and AMVP mode, signaling can be done as follows.

[0418] The MVP candidate having the above multiple motion vectors can be applied to each tool (Sub-block based MERGE mode, CIIP mode, GPM mode, etc.) that can be combined with the existing merge mode and the merge mode, and it is possible to include a candidate having multiple motion vectors in the process of constructing the MVP candidate list of each mode.

[0419] Additionally, it is also possible to implement as another merge mode within the general merge mode. For example, as a merge mode having multiple motion vectors, a flag indicating the mode (e.g., multi_pred_merge_flag) can be signaled. When the value of the flag is '1', the MVP candidate list can be composed of candidates having multiple motion vectors, and a merge index indicating a specific candidate within the MVP candidate list can be signaled. When the process of reordering and selecting only some candidates using the template-based cost or the bilateral-based cost described above is performed, the merge index can be omitted or indicate a candidate within the list that includes only some candidates.

[0420] Candidates with multiple motion vectors can also be applied to AMVP mode. In AMVP mode, MVP candidate lists can be constructed for each direction, so it is possible to include candidates with multiple motion vectors in each MVP candidate list in the L0 and L1 directions.

[0421] Figure 25 is a diagram showing an example of an MVP candidate list configured for each direction in AMVP mode.

[0422] When allowing additional motion vectors in AMVP mode, the number of motion vectors that can be had in each direction can be limited to 2 to 4 motion vectors in the end. In the example of Fig. 25, Case 1 represents a case in which a total of 4 MVPs are obtained through a combination of candidate CAND[2] with multiple motion vectors in the L0 direction MVP candidate list and candidate CAND[2] with multiple motion vectors in the L1 direction MVP candidate list. Case 2 represents a case in which a total of 3 MVPs are obtained through a combination of candidate CAND[0] with a single motion vector in the L0 direction MVP candidate list and candidate CAND[2] with multiple motion vectors in the L1 direction MVP candidate list.

[0423] The application method in the above AMVP mode can be changed as follows in consideration of encoding / decoding efficiency. Specifically, additional motion vectors for the AMVP mode can be applied only to blocks to which bidirectional prediction is applied. In addition, a limited number of multiple motion vectors can be allowed by configuring a candidate having multiple motion vectors only in the process of configuring MVP candidates in a specific direction (L0 or L1). That is, by allowing n (n is a natural number greater than or equal to 1) additional motion vectors only in a specific prediction direction, the total number of additional motion vectors can be limited to n.

[0424] Meanwhile, it is also possible to construct a candidate having multiple motion vectors without signaling whether or not multiple motion vectors are allowed, and it is also possible to signal a flag indicating the mode to allow multiple motion vectors. In the latter case, if the mode flag is signaled, and the flag has a value of '1', an MVP candidate having multiple motion vectors can be included in the MVP candidate list. The flag may mean that a candidate having multiple motion vectors consisting of the number of allowed motion vectors among the MVP candidates can be constructed, or it may mean that an MVP candidate list consisting only of candidates having multiple motion vectors can be constructed. Alternatively, it may mean the number of multiple motion vectors that can be included in the MVP candidate list. In addition, mvp_index or mvp_flag, which points to a specific candidate in the MVP candidate list, may be signaled, and when the process of reordering and selecting only some candidates using the template-based cost or bilateral-based cost is performed, mvp_index or mvp_flag may be omitted or may point to a candidate in the list that only includes some candidates.

[0425] Meanwhile, in AMVP mode, if the final MVP candidate, i.e., the MVP candidate used for predicting the current block, is a candidate with multiple motion vectors, MVD information corresponding to each motion vector can be signaled. Similarly to the conventional AMVP mode, the final MV can be calculated using the derived MVP information and the signaled MVD information. At this time, the signaling method for the MVD information can be applied in various ways, such as indicating the magnitude of the vector or deriving it from information based on distance and direction by indexing the MVD similarly to the MMVD. Furthermore, for an MVP with multiple motion vectors, the MVD information can be reduced in signaling information by signaling only some information, limited to the basic (regular) motion vector, rather than the additional motion vector. In other words, the additional motion vector predictor included in the MVP candidate can be utilized as a motion vector as is, without the MVD. Alternatively, when the final MVP candidate has multiple motion vectors, variations are possible, such as transmitting MVD information for some motion vectors and not transmitting MVD information for some motion vectors, or not transmitting the entire MVD information.

[0426] The method described in the disclosed embodiment can be similarly applied to AMVP-merge. Additional motion vectors can be included in each direction of the AMVP-merge mode, and it is possible to allow additional motion vectors in a modified way, such as applying them only to each of the AMVP mode or merge mode directions. When including additional motion vectors in the AMVP mode direction, the candidate indicated by mvp_index or mvp_flag can include multiple motion vectors. In this case, modifications such as additionally signaling MVD information or not signaling the additional motion vectors are possible, and when including additional motion vectors in the merge mode direction, the candidate indicated by the merge index can include multiple motion vectors.

[0427] Meanwhile, in the process of constructing an MVP candidate list including the aforementioned multiple motion vectors, various motion information can be utilized. The MVP candidates included in the MVP candidate list have basic motion information and additional motion information. The basic motion information includes not only basic motion vector information and reference picture index information, but also basic additional information, and the additional motion information may include not only additional motion vector information and reference picture index information, but also additional additional information. At this time, the additional information may include BCW index information, LIC information (e.g., LIC flag and LIC parameter information), interpolation filter information, etc., and such additional information may be changed and applied according to set conditions or criteria.

[0428] While this additional information can be utilized to improve the accuracy of reference blocks, since it is determined when a reference block is unidirectional or bidirectional, it may actually increase the inaccuracy of the reference block when applied to blocks with two or more multiple motion vectors. Therefore, one embodiment provides a method for selectively utilizing information other than motion vectors, thereby providing diversity in the reference block generation process, thereby improving compression performance.

[0429] In a method for selectively utilizing information other than motion vectors, a BCW index containing information for weighted summation of bidirectional prediction blocks can be modified and applied. When a BCW index included in basic motion information and a BCW index included in additional motion information exist, whether or not to modify the BCW index value can be adaptively determined as follows:

[0430] - Whether to change the BCW index can be determined based on whether the difference between the two weights indicated by the BCW index is less than or equal to a threshold. For example, if the weight indicated by the BCW index is Assuming that there is a difference in weight between each reference block, The BCW index value can be maintained only when it is greater than a predefined threshold value, and changed to a default value otherwise. In this case, the predefined threshold value can have a value of, for example, 2 / 8. In addition, the default value of the BCW index can be set to have the same weight for each direction, such as (4 / 8: 4 / 8). This condition can be applied to the basic motion information and / or the additional motion information. In addition, the method can be modified, such as applying it only when the basic motion information and the additional motion information have different BCW indices.

[0431] - It is possible to determine to apply only one piece of information among each candidate using a predefined method. For example, the BCW index included in the additional motion information can be set to the default value so that the weighted sum is not applied. As another example, the priority of the basic motion information and the additional motion information can be determined, and the BCW index value of the motion information with a higher priority can be followed, and the BCW index included in the motion information with a lower priority can be set to the default value. The priority can be determined by the order in the MVP candidate list when two MVP candidates in the MVP candidate list are combined to form a multi-MVP. That is, since the motion information of a candidate earlier in the MVP candidate list is more reliable, the BCW index value of the earlier candidate can be applied as is, and the BCW index value included in the later candidate can be changed to the default value and applied. In addition, the priority can also be determined using the difference in weights between each reference block. For example, if the difference in weights is greater than a threshold, the corresponding BCW index value can be applied as is, and if not, the BCW index value can be changed to the default value and applied. In addition, the above method can be modified, such as by applying it only when the basic motion information and the additional motion information each have different BCW indices.

[0432] In a method of selectively utilizing information other than motion vectors, it is possible to selectively utilize LIC information, such as LIC flag and LIC parameter information, which predicts and applies the degree of light change from adjacent samples between the current block and the reference block. For example, when a block includes multiple motion vectors, the LIC flag and LIC parameter information of each MVP candidate can be changed and applied. When the LIC flag and LIC parameter information included in the basic motion information and the LIC flag and LIC parameter information included in the additional motion information exist, whether or not the LIC flag and LIC parameter information values ​​are modified can be adaptively determined as follows. The LIC model can be expressed as in Equation 8 below, where a represents a weight and b represents an offset:

[0433] [Formula 8]

[0434]

[0435] - Assuming that there is a weight of the LIC parameter (for example, MIN < weight <= MAX, MIN = 0, MAX = 32), whether to change the LIC information can be determined based on whether the weight of the LIC parameter is less than or equal to a predefined threshold. For example, the LIC parameter value can be maintained only when the weight is greater than the threshold, and changed to a default value otherwise. In this case, the predefined weight threshold can have a value of (MAX >> 3). The above condition can be applied to the basic motion information and / or the additional motion information. In addition, the method can be modified, such as applying it only when the basic motion information and the additional motion information have different LIC flags. Alternatively, the method can be modified according to the prediction direction of each motion information. For example, it can be applied when it is a unidirectional prediction.

[0436] - Assuming that there is an offset of the LIC parameter (for example, MIN <= offset < MAX, MIN = -1024, MAX = 1024), the LIC parameter value can be maintained only when the offset is out of a predefined threshold range, and set to a default value otherwise. In this case, the threshold range of the offset can be defined as ((MIN >> 6) <= offset < (MAX >> 6)), that is, -16 <= offset < 16. The offset condition can be applied in combination with the weight condition, and the condition can be applied to the basic motion information and / or the additional motion information. In addition, the method can be modified, such as applying it only when the basic motion information and the additional motion information have different LIC flags. Or, it can be modified according to the prediction direction of each motion information. For example, it can be applied when it is a unidirectional prediction.

[0437] - It can be determined to apply only the information of one candidate among each candidate using a predefined method. For example, LIC can be not applied by setting the LIC flag included in the additional motion information to FALSE (default value). If it includes parameters (weight, offset) other than the LIC flag, the parameters can be changed to the default values ​​(weight = 1, offset = 0) and applied. As another example, the priority of the basic motion information and the additional motion information can be determined, and the value of the motion information with a higher priority can be applied as is, and the LIC information (e.g., the LIC flag and LIC parameter) included in the motion information with a lower priority can be changed to the default values ​​and applied. For example, the priority can be determined based on the order of each MVP candidate in the MVP candidate list when two MVP candidates in the MVP candidate list are combined to form a multi-MVP. In other words, since the motion information of a candidate earlier in the MVP candidate list is more reliable, only the motion information of that candidate can be applied as is, and the LIC flag and LIC parameter values ​​included in a later candidate can be changed to the default values ​​and applied. In addition, the above priority can also be determined by using the difference between the weights and offsets of the LIC parameters between each reference block. For example, if the difference between the weights and offsets is greater than a threshold, the LIC parameters can be applied as is, and if not, the LIC parameters can be changed to default values ​​and applied. In addition, the above method can be modified, such as applying it only when the basic motion information and the additional motion information each have different LIC flags and LIC parameters. Alternatively, it can be modified according to the prediction direction of each motion information. For example, it can be applied when there is a unidirectional prediction.

[0438] In a method of selectively utilizing information other than motion vectors, when a block includes multiple motion vectors, interpolation filter information may be changed. The interpolation filter defines filter coefficients for each predefined phase, and in particular, when the motion vector indicates a half-pel position, another interpolation filter for the position may be defined. However, this is just one example, and the interpolation filter may define one of multiple coefficients by assigning a flag or index. When interpolation filter information exists, including interpolation information for the half-pel and additional interpolation filter information, the interpolation filter information included in the basic motion information and the interpolation filter information included in the additional motion information may be adaptively determined as follows:

[0439] - It can be determined to apply information of only one candidate among each candidate using a predefined method. For example, motion compensation can be performed by setting the interpolation filter information included in the additional motion information to a default value. As another example, the priority of the basic motion information and the additional motion information can be determined, and the value of the motion information with a higher priority can be applied as is, and the interpolation filter information included in the motion information with a lower priority can be changed to a default value and applied. For example, when two MVP candidates in the MVP candidate list are combined to form a multi-MVP, the priority can be determined based on the order in the candidate list. In other words, since the motion information of a candidate earlier in the list is more reliable, the interpolation filter information of that candidate can be applied as is, and the interpolation filter information included in a later candidate can be changed to a default value and applied. In addition, the default value in the above example can mean a predefined existing interpolation filter, such as 8-tap or 12-tap. In addition, the method can be modified, such as applying it only when the basic motion information and the additional motion information each have different interpolation filter information. Alternatively, it can be applied only when the number of taps of the interpolation filter containing the basic motion information and the additional motion information are different.

[0440] - When a block includes multiple motion vectors, another predefined interpolation filter may be used for the mode other than the interpolation filter information included in the basic motion information and the additional motion information. For example, a predefined interpolation filter having the same tap and the same coefficient may be applied to the basic motion information and the additional motion information to perform motion compensation. At this time, a filter different from the default interpolation filter may exist. This is an example, and the basic motion information and the additional motion information may include another predefined interpolation filter, but these may be different filters. At this time, one or more other filters different from the default interpolation filter may exist.

[0441] As described above, when multiple reference blocks exist, each reference block is divided into a basic reference block and an additional reference block, and an average sum between each reference block can be applied to generate a final prediction block. As the image becomes more diverse, reference blocks of various shapes may be required, and weighted sum can play a role in generating these various shapes of reference blocks. Hereinafter, a method of signaling / parsing an index of weight information required for weighted sum of multiple reference blocks is described. In the disclosed embodiment, signaling of information may include encoding of the information, and parsing of the information may include obtaining the information from a bitstream. In addition, even if there is no separate description, a description of signaling some information may include parsing the information, and a description of parsing some information may include signaling the information.

[0442] A mode including multiple reference blocks exists as a single prediction mode, and weight index information can be signaled / parsed when that prediction mode is used.

[0443] FIG. 26 is a diagram illustrating an example of a signaling / parsing method when a multi-reference block mode is included as one of the general merge modes in a method according to one embodiment.

[0444] Referring to FIG. 26, as an example, whether the regular merge mode is applied can be defined by the merge flag (regular_merge_flag), and whether the multi-reference block mode is applied can be defined by the multi-mvp flag (multi_mvp_flag). When the flag is TRUE, an additional merge index (additional_merge_idx) for indicating an additional reference block can be signaled / parsed, and a weight index (weight_idx) for weighted sum between the basic reference block and the additional reference block can be signaled / parsed. However, this is just one example, and the signaling / parsing order or position of the multi-mvp flag can be changed.

[0445] Additionally, it is possible that the signaling / parsing of the additional merge index is omitted. If the signaling / parsing of the additional merge index is omitted, it can be derived as a modified form of the merge index (merge_idx). For example, it can be calculated as additional_merge_idx = merge_idx + offset. In this case, the offset can be specified as a constant such as 1 or 2. In another example, the additional merge index (additional_merge_idx) may not be used, and the merge index (merge_idx) may be used as an index of a candidate that includes both the basic reference block and the additional reference block.

[0446] When the multi-mvp flag (multi_mvp_flag) is TRUE, the weight index (weight_idx) signaled / parsed can exist in the range of 0 to N-1, where N is a positive integer. For example, N can be 4. The weight represented by each index can be used as a weight for the weighted sum between the basic reference block and the additional reference block. That is, it can be used for the weighted sum between the basic reference block (the resulting block to which the weighted sum or the average sum is applied in case of bidirectional prediction) and the additional reference block (the resulting block to which the weighted sum or the average sum is applied in case of bidirectional prediction).

[0447] The weights can be set as follows depending on the characteristics of the current block:

[0448] - Weight information can be assigned based on the size and shape of the current block. For example, if the block size (e.g., the product of the block's width and height) is larger than a predefined threshold, the weight of the base reference block can be set higher. In this case, the threshold value can have a value of 128 or 256. However, the above threshold value is merely an example, and other threshold values ​​may be used.

[0449] Weighting information can be derived based on whether unidirectional or bidirectional prediction is applied to the base reference block and additional reference blocks. For example, if unidirectional and bidirectional prediction blocks are mixed, the weight of the bidirectional prediction block can be set higher.

[0450] - Weight information can be derived based on the BCW index, LIC flag, and parameters included in the basic and additional motion information. For example, the weight of a prediction block that includes information with a non-default BCW index can be set higher. Similarly, the weight of a prediction block that includes motion information with a LIC flag that is TRUE or a LIC parameter that is not a default value can be set higher.

[0451] - Weight information can be derived based on the distance (POC Difference) between the reference picture and the current picture included in the basic motion information and the additional motion information. For example, the weight of a prediction block containing motion information with a small POC difference can be set higher. In this case, if bidirectional prediction is applied to the basic prediction block or the additional prediction block, the average value of the POC difference between the reference picture in each direction and the current picture can be used. Alternatively, the POC difference with a reference picture located at a closer distance among the two reference pictures can also be used.

[0452] - The weights may be derived to different values ​​considering the number of taps or the type of filter of the interpolation filter that the basic motion information and the additional motion information contain. For example, the weight of a prediction block that contains information with a long number of taps of the interpolation filter may be set higher. As another example, when the interpolation filters applied to the basic prediction block and the additional prediction block have different characteristics, such as a smoothing filter and a sharpening filter, the weight of the prediction block that contains the smoothing filter information may be set higher.

[0453] Meanwhile, a mode that includes multiple reference blocks (multi-mvp mode) may be included as one of the merge modes, or may be treated as a new mode distinct from the merge mode.

[0454] The general merge mode can be branched into subblock merge mode, mmvd merge mode, regular merge mode, GPM mode, CIIP mode, etc., and the multi-mvp mode can be included in the general merge mode as a mode distinct from the above modes.

[0455] Figure 27 shows the signaling / parsing structure of each detailed mode within the general merge mode.

[0456] Referring to Figure 27, in the case of a non-subblock merge mode within the general merge mode, the regular merge mode can be signaled as being divided into mmvd mode, multi-mvp mode, and regular merge mode. In the case of a non-regular merge mode, it can be CIIP mode or GPM mode.

[0457] Figure 28 shows the signaling / parsing structure when the multi-mvp mode is treated as a separate mode rather than merge mode.

[0458] In one embodiment, the prediction mode may include an inter mode, a merge mode, and a blended mode including a multi-mvp mode. However, it should be noted that the blended mode is merely an example and may be referred to by other names.

[0459] According to the example of Fig. 28, sub-block merge mode, regular merge mode, and MMVD mode can be signaled / parsed as general merge mode, and GPM mode, CIIP, and multi-mvp mode can be signaled / parsed as mixed mode when not in general merge mode. In addition, affine mode, SMVD mode, and normal inter mode can be signaled / parsed as inter mode when not in general merge mode or mixed mode. This can reduce the signaling bits of mixed mode.

[0460] Whether each mode is applied can be indicated by a flag indicating whether the corresponding mode is applied, or can be derived from a flag indicating whether another mode is applied. For example, a blend flag (e.g., blend_flag) indicating whether the blend mode is applied can be signaled / parsed, a GPM flag (e.g., gpm_flag) indicating whether GPM is applied when the blend flag indicates TRUE can be signaled / parsed, and a CIIP flag (e.g., ciip_flag) indicating whether CIIP is applied when the GPM flag indicates FALSE can be signaled / parsed. When the CIIP flag indicates FALSE, it can be determined that the multi-mvp mode is applied.

[0461] The examples or embodiments described so far can be combined with each other, and changes required by the combination of embodiments can also be included within the scope of the disclosed invention or disclosed embodiments.

[0462] FIG. 29 is a diagram illustrating an example of a content streaming system to which an embodiment according to the present disclosure can be applied.

[0463] Referring to FIG. 29, a content streaming system to which the embodiment(s) of the present specification are applied may largely include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0464] The encoding server compresses content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data, generates a bitstream, and transmits it to the streaming server. Alternatively, if multimedia input devices such as smartphones, cameras, and camcorders directly generate bitstreams, the encoding server may be omitted.

[0465] The above bitstream can be generated by an encoding method or a bitstream generation method to which the embodiment(s) of the present specification are applied, and the streaming server can temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0466] The streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server acts as an intermediary to inform the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, and the streaming server transmits the multimedia data to the user. At this time, the content streaming system may include a separate control server, in which case the control server controls commands / responses between each device within the content streaming system.

[0467] The streaming server can receive content from a media repository and / or an encoding server. For example, when receiving content from the encoding server, the content can be received in real time. In this case, to provide a smooth streaming service, the streaming server can store the bitstream for a certain period of time.

[0468] Examples of the user devices may include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, HMDs), digital TVs, desktop computers, digital signage, etc.

[0469] Each server within the above content streaming system can be operated as a distributed server, in which case data received from each server can be processed in a distributed manner.

[0470] The claims set forth in this specification may be combined in various ways. For example, the technical features of the method claims of this specification may be combined and implemented as a device, and the technical features of the device claims of this specification may be combined and implemented as a method. Furthermore, the technical features of the method claims and the technical features of the device claims of this specification may be combined and implemented as a device, and the technical features of the method claims and the technical features of the device claims of this specification may be combined and implemented as a method.

[0471] Embodiments according to the present disclosure can be used to encode / decode images.

Claims

1. A step of obtaining image information from a bitstream; A step of constructing an MVP candidate list including MVP candidates for the current block based on the above image information; and A step of generating a prediction block for the current block based on at least one MVP candidate in the MVP candidate list; The steps for forming the above MVP candidate list are: Including an MVP candidate having basic movement information and additional movement information in the list of MVP candidates, The above basic motion information includes basic motion vector information and basic additional information, The above additional motion information includes additional motion vector information and additional additional information, A method in which the above basic additional information and the above additional additional information are changed according to set conditions or set criteria.

2. In paragraph 1, The above basic additional information and the above additional additional information are, A method, each of which includes at least one of BCW (Bi-prediction with CU-level weight) index information, LIC (Local Illuminatin Compensation) information, and interpolation filter information.

3. In paragraph 2, A method in which whether to change the BCW index is determined based on whether the difference between the two weights indicated by the BCW index is less than or equal to a threshold value.

4. In paragraph 2, A method in which the BCW index included in the above additional movement information is changed to a predetermined default value.

5. In paragraph 2, A method in which BCW index information included in high-priority motion information among the above-mentioned basic motion information and the above-mentioned additional motion information is maintained, and BCW index information included in low-priority motion information is changed to a predetermined default value.

6. In paragraph 2, A method in which it is determined whether or not to change the LIC information based on whether the weight or offset of the LIC parameter included in the LIC information is within a predetermined range.

7. In paragraph 2, A method in which LIC information or interpolation filter information included in high-priority motion information among the above-mentioned basic motion information and the above-mentioned additional motion information is maintained, and LIC information or interpolation filter information included in low-priority motion information is changed to a predetermined default value.

8. In paragraph 1, The step of generating a prediction block for the current block above is: Including weighting a basic prediction block generated by the basic motion information and an additional prediction block generated by the additional motion information, The weight information applied to the above weighted sum is: A method, wherein the method is determined based on the size or shape of the current block, whether bidirectional prediction is applied to the current block, or BCW information, LIC information, interpolation filter information included in the basic motion information and the additional motion information, or the distance between the reference picture and the current picture.

9. A step of constructing an MVP candidate list including MVP candidates for the current block; A step of generating residual information for the current block based on at least one MVP candidate in the MVP candidate list; and A step of encoding image information including information about prediction applied to the current block and the residual information; The steps for forming the above MVP candidate list are: Including an MVP candidate having basic movement information and additional movement information in the list of MVP candidates, The above basic motion information includes basic motion vector information and basic additional information, The above additional motion information includes additional motion vector information and additional additional information, A method in which the above basic additional information and the above additional additional information are changed according to set conditions or set criteria.

10. In paragraph 9, The above basic additional information and the above additional additional information are, A method, each of which includes at least one of BCW (Bi-prediction with CU-level weight) index information, LIC (Local Illuminatin Compensation) information, and interpolation filter information.

11. In paragraph 10, A method in which whether to change the BCW index is determined based on whether the difference between the two weights indicated by the BCW index is less than or equal to a threshold value.

12. In paragraph 10, A method in which the BCW index included in the above additional movement information is changed to a predetermined default value.

13. In paragraph 10, A method in which BCW index information included in high-priority motion information among the above-mentioned basic motion information and the above-mentioned additional motion information is maintained, and BCW index information included in low-priority motion information is changed to a predetermined default value.

14. In paragraph 10, A method in which it is determined whether or not to change the LIC information based on whether the weight or offset of the LIC parameter included in the LIC information is within a predetermined range.

15. In paragraph 10, A method in which LIC information or interpolation filter information included in high-priority motion information among the above-mentioned basic motion information and the above-mentioned additional motion information is maintained, and LIC information or interpolation filter information included in low-priority motion information is changed to a predetermined default value.

16. In paragraph 10, The step of generating residual information for the current block is: Including weighting a basic prediction block generated by the basic motion information and an additional prediction block generated by the additional motion information, The weight information applied to the above weighted sum is: A method, wherein the method is determined based on the size or shape of the current block, whether bidirectional prediction is applied to the current block, or BCW information, LIC information, interpolation filter information included in the basic motion information and the additional motion information, or the distance between the reference picture and the current picture.

17. In a computer-readable storage medium in which a bitstream generated by an encoding method is stored, The above encoding method is, A step of constructing an MVP candidate list including MVP candidates for the current block; A step of generating residual information for the current block based on at least one MVP candidate in the MVP candidate list; and A step of encoding image information including information about prediction applied to the current block and the residual information; The steps for forming the above MVP candidate list are: Including an MVP candidate having basic movement information and additional movement information in the list of MVP candidates, The above basic motion information includes basic motion vector information and basic additional information, The above additional motion information includes additional motion vector information and additional additional information, A storage medium in which the above basic additional information and the above additional additional information are changed according to set conditions or set standards.

18. In a method of transmitting data about an image, A step of obtaining a bitstream for the image, wherein the bitstream is generated based on the steps of: configuring an MVP candidate list including MVP candidates for a current block; generating residual information for the current block based on at least one MVP candidate in the MVP candidate list; and encoding image information including information about a prediction applied to the current block and the residual information; and A step of transmitting the data including the bitstream; The steps for forming the above MVP candidate list are: Including an MVP candidate having basic movement information and additional movement information in the list of MVP candidates, The above basic motion information includes basic motion vector information and basic additional information, The above additional motion information includes additional motion vector information and additional additional information, A method in which the above basic additional information and the above additional additional information are changed according to set conditions or set criteria.

Images