Image decoding method and apparatus for deriving weight index information for generating prediction samples

Abstract

According to the disclosure of this document, when the inter-frame prediction type of the current block indicates bidirectional prediction, weight index information of candidates in the merge candidate list or the sub-block merge candidate list can be derived, thereby improving coding efficiency.

Description

Technical Field

[0001] The present disclosure relates to an image decoding method and apparatus for deriving weight index information for generating prediction samples. Background Art

[0002] Recently, there has been an increasing demand for high-resolution, high-quality images / videos, such as 4K or 8K ultra-high-definition (UHD) images / videos, in various fields. As image / video resolution or quality becomes higher, relatively more information or bits are transmitted compared to conventional image / video data. Therefore, if image / video data is transmitted via a medium such as an existing wired / wireless broadband line or stored in a conventional storage medium, the cost of transmission and storage is likely to increase.

[0003] In addition, there is growing interest and demand for virtual reality (VR) and artificial reality (AR) content and immersive media such as holograms; and there is also growing broadcasting of images / videos that exhibit image / video characteristics that are different from actual images / videos (e.g., game images / videos).

[0004] Therefore, highly efficient image / video compression technology is required to effectively compress and transmit, store, or play high-resolution, high-quality images / videos exhibiting various characteristics as described above. Summary of the Invention

[0005] Technical issues

[0006] The present disclosure provides a method and apparatus for improving image coding efficiency.

[0007] The present disclosure also provides a method and apparatus for deriving weight index information for generating prediction samples in inter-frame prediction.

[0008] The present disclosure also provides a method and apparatus for deriving weight index information about candidates in an affine merge candidate list during bidirectional prediction.

[0009] Technical Solution

[0010] In one aspect of the present disclosure, an image decoding method performed by a decoding device is provided. The image decoding method includes: receiving image information including inter-frame prediction mode information through a bitstream; generating a merge candidate list of a current block based on the inter-frame prediction mode information; deriving motion information about the current block based on a candidate selected from the merge candidate list; generating L0 prediction samples and L1 prediction samples of the current block based on the motion information; and generating prediction samples of the current block based on the L0 prediction samples, the L1 prediction samples, and weight information, wherein the weight information is derived based on weight index information about the selected candidate. The candidate includes an affine merge candidate, and the affine merge candidate includes a control point motion vector (CPMV). When the affine merge candidate includes CPMV0 of control point 0 (CP0) located on the upper left side of the current block, weight index information about the affine merge candidate is derived based on the 0th weight index information about the CP0. When the affine merge candidate does not include CPMV0 of the CP0 located on the upper left side of the current block, weight index information about the affine merge candidate is derived based on first weight index information about CP1 located on the upper right side of the current block.

[0011] In another aspect of the present disclosure, an image encoding method performed by an encoding device is provided. The image encoding method includes the following steps: determining an inter prediction mode of a current block and generating inter prediction mode information indicating the inter prediction mode; generating a merge candidate list for the current block based on the inter prediction mode; generating selection information indicating one of the candidates included in the merge candidate list; and encoding image information including the inter prediction mode information and the selection information. The candidates include an affine merge candidate, and the affine merge candidate includes a control point motion vector (CPMV). When the affine merge candidate includes CPMV0 of control point 0 (CP0) located on the upper left side of the current block, weight index information about the affine merge candidate is expressed based on the 0th weight index information about the CP0. When the affine merge candidate does not include CPMV0 of the CP0 located on the upper left side of the current block, weight index information about the affine merge candidate is expressed based on the first weight index information about control point 1 (CP1) located on the upper right side of the current block.

[0012] In another aspect of the present disclosure, a computer-readable storage medium storing encoding information that causes an image decoding device to implement an image decoding method is provided. The image decoding method includes the following steps: obtaining image information including inter-frame prediction mode information from a bitstream; generating a merge candidate list for a current block based on the inter-frame prediction mode information; selecting a candidate from the candidates included in the merge candidate list; deriving motion information about the current block based on the selected candidate; generating L0 prediction samples and L1 prediction samples of the current block based on the motion information; and generating prediction samples of the current block based on the L0 prediction samples, the L1 prediction samples, and weight information. The weight information is derived based on weight index information about the selected candidate. The candidate includes an affine merge candidate, and the affine merge candidate includes a control point motion vector (CPMV). When the affine merge candidate includes CPMV0 of control point 0 (CP0) located on the upper left side of the current block, weight index information about the affine merge candidate is derived based on the 0th weight index information about the CP0. When the affine merge candidate does not include CPMV0 of the CP0 located at the upper left side of the current block, weight index information about the affine merge candidate is derived based on first weight index information about the control point 1 (CP1).

[0013] Technical Effects

[0014] According to the present disclosure, the overall image / video compression efficiency can be improved.

[0015] According to the present disclosure, motion vector candidates can be efficiently constructed during inter prediction.

[0016] According to the present disclosure, weight-based bidirectional prediction can be performed efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a diagram schematically illustrating an example of a video / image encoding system to which an embodiment of the present disclosure can be applied.

[0018] Figure 2 is a diagram schematically illustrating a configuration of a video / image encoding device to which an embodiment of the present disclosure can be applied.

[0019] Figure 3 is a diagram schematically illustrating a configuration of a video / image decoding device to which an embodiment of the present disclosure can be applied.

[0020] Figure 4 is a diagram illustrating an example of an inter-frame prediction process.

[0021] Figure 5 This is a diagram for explaining the merge mode in inter-frame prediction.

[0022] Figure 6 is a diagram illustrating an example of motion represented by an affine motion model.

[0023] Figure 7a and Figure 7b is a diagram illustrating an example of a control point motion vector (CPMV) used for affine motion prediction.

[0024] Figure 8 3 is a diagram illustrating an example of determining an affine MVF in units of sub-blocks.

[0025] Figure 9 A diagram for explaining the affine merge mode in inter-frame prediction.

[0026] Figure 10 3 is a diagram for explaining the positions of candidates in the affine merge mode.

[0027] Figure 11 is a diagram for explaining subblock-based temporal motion vector prediction (SbTMVP) in inter prediction.

[0028] Figure 12 and Figure 13 is a diagram schematically illustrating an example of a video / image encoding method and related components according to an embodiment of the present disclosure.

[0029] Figure 14 and Figure 15 is a diagram schematically illustrating an example of an image / video decoding method and related components according to an embodiment of the present disclosure.

[0030] Figure 16 is a diagram illustrating an example of a content streaming system to which an embodiment of the present disclosure is applied. DETAILED DESCRIPTION

[0031] The present disclosure can be modified in various ways and has several exemplary embodiments. Therefore, specific exemplary embodiments of the present disclosure will be illustrated in the accompanying drawings and described in detail. However, this is not intended to limit the present disclosure to specific embodiments. The terms used in this specification are only used to describe specific exemplary embodiments, rather than to limit the present disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form. It will be understood that the terms "including", "having" and the like used in this specification specify the presence of features, numbers, steps, operations, components, parts or combinations thereof set forth in this specification, but do not exclude the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0032] In addition, in order to facilitate the description of different feature functions, each component in the drawings described in this disclosure is illustrated independently, which does not mean that each component is implemented as separate hardware or separate software. For example, two or more components among the components can be combined to form a component, or a component can be divided into multiple components. Embodiments in which each component is integrated and / or separated are also included in the scope of this disclosure.

[0033] In the present disclosure, "A or B" may mean "only A", "only B", or "both A and B". In other words, "A or B" in the present disclosure may be interpreted as "A and / or B". For example, in the present disclosure, "A, B, or C" means "only A", "only B", "only C", or "any one of A, B, and C, and any combination thereof".

[0034] As used in this disclosure, a slash ( / ) or a comma (,) may mean "and / or". For example, "A / B" may mean "A and / or B". Thus, "A / B" may mean "only A", "only B", or "both A and B". For example, "A, B, C" may mean "A, B, or C".

[0035] In this document, “at least one of A and B” may mean “only A”, “only B”, or “both A and B”. In addition, in this document, the expression “at least one of A or B” or “at least one of A and / or B” may be interpreted as the same as “at least one of A and B”.

[0036] In addition, in this document, "at least one of A, B, and C" may mean "only A", "only B", "only C", or "any combination of A, B, and C". In addition, "at least one of A, B, or C" or "at least one of A, B, and / or C" may mean "at least one of A, B, and C".

[0037] In addition, the parentheses used in this document may mean "for example." Specifically, when the expression "prediction (intra-frame prediction)" is used, it can indicate that "intra-frame prediction" is proposed as an example of "prediction." In other words, the term "prediction" in this document is not limited to "intra-frame prediction" and can indicate that "intra-frame prediction" is proposed as an example of "prediction." In addition, even when the expression "prediction (i.e., intra-frame prediction)" is used, it can indicate that "intra-frame prediction" is proposed as an example of "prediction."

[0038] In this document, technical features independently described in one drawing may be implemented independently or may be implemented simultaneously.

[0039] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in the entire drawings, like reference numerals are used to indicate like elements, and the same description of like elements may be omitted.

[0040] Figure 1 An example of a video / image encoding system to which the embodiments of the present disclosure can be applied is illustrated.

[0041] Reference Figure 1 The video / image coding system may include a first device (source device) and a second device (receiver device). The source device may send the coded video / image information or data to the receive device in the form of a file or stream transmission via a digital storage medium or a network.

[0042] The source device may include a video source, an encoding device, and a transmitter. The receiving device may include a receiver, a decoding device, and a renderer. The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display, and the display may be configured as a separate device or an external component.

[0043] A video source may acquire video / images through a process of capturing, synthesizing, or generating video / images. A video source may include a video / image capture device and / or a video / image generation device. For example, a video / image capture device may include one or more cameras, a video / image archive including previously captured video / images, etc. For example, a video / image generation device may include a computer, a tablet computer, and a smartphone, and may (electronically) generate video / images. For example, a virtual video / image may be generated by a computer, etc. In this case, the video / image capture process may be replaced by a process of generating relevant data.

[0044] An encoding device encodes input video / images. For compression and coding efficiency, the encoding device performs a series of processes such as prediction, transformation, and quantization. The encoded data (encoded video / image information) is output as a bitstream.

[0045] The transmitter can transmit the encoded image / image information or data, output as a bitstream, in the form of a file or stream to a receiver of a receiving device via a digital storage medium or network. Digital storage media may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter may include components for generating a media file in a predetermined file format and may also include components for transmission via a broadcast / communication network. The receiver may receive / extract the bitstream and transmit the received bitstream to a decoding device.

[0046] The decoding device may decode a video / image by performing a series of processes corresponding to the operations of the encoding device, such as inverse quantization, inverse transformation, and prediction.

[0047] The renderer may render the decoded video / image, and the rendered video / image may be displayed on a display.

[0048] The present disclosure relates to video / image coding. For example, the methods / implementations disclosed in the present disclosure can be applied to methods disclosed in the Versatile Video Coding (VVC) standard, the Essential Video Coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the second-generation Audio Video Coding standard (AVS2), or next-generation video / image coding standards (e.g., H.267 or H.268, etc.).

[0049] This document proposes various embodiments of video / image coding, and unless otherwise mentioned, the above embodiments may also be performed in combination with each other.

[0050] In this document, video may refer to a series of images over time. A picture generally refers to a unit that represents an image at a specific time frame, and a slice / tile refers to a unit that constitutes a part of a picture from a coding perspective. A slice / tile may include one or more coding tree units (CTUs). A picture may be composed of one or more slices / tiles.

[0051] A tile is a rectangular area of ​​CTUs within a specific tile column and a specific tile row in a picture. A tile column is a rectangular area of ​​CTUs whose height is equal to the height of the picture and whose width is specified by the syntax elements in the picture parameter set. A tile row is a rectangular area of ​​CTUs whose height is specified by the syntax elements in the picture parameter set and whose width is equal to the width of the picture. Tile scan is a specific sequential ordering of the CTUs that partition a picture, where the CTUs are ordered consecutively in a CTU raster scan within a tile, and the tiles in a picture are ordered consecutively in a raster scan of the tiles of the picture. A slice may include multiple complete tiles of a picture that may be contained in one NAL unit or multiple consecutive CTU rows in a tile. In this document, tile group and slice may be used interchangeably. For example, in this document, a tile group / tile group header may be referred to as a slice / slice header.

[0052] In addition, a picture can be divided into two or more sub-pictures. A sub-picture can be a rectangular area of ​​one or more slices within a picture.

[0053] A pixel or a picture element (pel) may refer to the smallest unit constituting a picture (or image). In addition, "sample" may be used as a term corresponding to a pixel. A sample may generally represent a pixel or a pixel value, and may represent only a pixel / pixel value of a luminance component, or only a pixel / pixel value of a chrominance component.

[0054] A unit may represent a basic unit of image processing. A unit may include at least one of a specific area of ​​a picture and information related to the area. A unit may include a luminance block and two chrominance (e.g., CB, CR) blocks. In some cases, a unit may be used interchangeably with terms such as block or area. In general, an M×N block may include M columns and N rows of samples (or sample arrays) or a set (or array) of transform coefficients. Alternatively, a sample may refer to a pixel value in the spatial domain, and when such a pixel value is transformed into the frequency domain, it may refer to a transform coefficient in the frequency domain.

[0055] Figure 2 This figure schematically illustrates the configuration of a video / image encoding device to which embodiments of this document can be applied. Hereinafter, a so-called encoding device may include an image encoding device and / or a video encoding device. Furthermore, a so-called image encoding method / device may include a video encoding method / device. Alternatively, a so-called video encoding method / device may include an image encoding method / device.

[0056] Reference Figure 2 , the encoding device 200 includes an image segmenter 210, a predictor 220, a residual processor 230 and an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 may include an inter-frame predictor 221 and an intra-frame predictor 222. The residual processor 230 may include a transformer 232, a quantizer 233, an inverse quantizer 234, and an inverse transformer 235. The residual processor 230 may also include a subtractor 231. The adder 250 may be referred to as a reconstructor or a reconstructed block generator. According to an embodiment, the image segmenter 210, the predictor 220, the residual processor 230, the entropy encoder 240, the adder 250, and the filter 260 may be configured by at least one hardware component (e.g., an encoder chipset or processor). In addition, the memory 270 may include a decoded picture buffer (DPB) or may be configured by a digital storage medium. The hardware components may also include the memory 270 as an internal / external component.

[0057] The image splitter 210 may split the input image (or picture or frame) input to the encoding device 200 into one or more processors. For example, a processor may be referred to as a coding unit (CU). In this case, the coding unit may be recursively split from a coding tree unit (CTU) or a maximum coding unit (LCU) according to a quadtree, binary tree, and ternary tree (QTBTTT) structure. For example, a coding unit may be split into multiple coding units of greater depth based on a quadtree structure, a binary tree structure, and / or a ternary structure. In this case, for example, the quadtree structure may be applied first, followed by the binary tree structure and / or ternary structure. Alternatively, the binary tree structure may be applied first. The encoding process according to the present disclosure may be performed based on the final coding unit that is no longer split. In this case, the maximum coding unit may be used as the final coding unit based on image characteristics, coding efficiency, etc., or, if necessary, the coding unit may be recursively split into coding units of greater depth, and the coding unit of the optimal size may be used as the final coding unit. Here, the encoding process may include prediction, transformation, and reconstruction processes (described later). As another example, the processor may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may be split or divided from the final coding unit. The prediction unit may be a unit for sample prediction, and the transform unit may be a unit for deriving transform coefficients and / or a unit for deriving a residual signal from the transform coefficients.

[0058] In some cases, the term "unit" may be used interchangeably with terms such as "block" or "region." In general, an M×N block may represent a set of samples or transform coefficients consisting of M columns and N rows. A sample may generally represent a pixel or pixel value, and may represent only the pixel / pixel value of the luma component or only the pixel / pixel value of the chroma component. A sample may be used as a term corresponding to a picture (or image) of a pixel or pixel.

[0059] The encoding device 200 can subtract the prediction signal (prediction block, prediction sample array) output from the inter-frame predictor 221 or the intra-frame predictor 222 from the input image signal (original block, original sample array) to generate a residual signal (residual block, residual sample array), and the generated residual signal is sent to the transformer 232. In this case, as illustrated, the unit in the encoding device 200 for subtracting the prediction signal (prediction block, prediction sample array) from the input image signal (original block, original sample array) can be referred to as a subtractor 231. The predictor can perform prediction on a processing target block (hereinafter referred to as a current block) and generate a prediction block including prediction samples of the current block. The predictor can determine whether to apply intra-frame prediction or inter-frame prediction in units of the current block or CU. The predictor can generate various information related to prediction, such as prediction mode information, and send the generated information to the entropy encoder 240, as described below when describing each prediction mode. The information related to the prediction can be encoded by the entropy encoder 240 and output in the form of a bitstream.

[0060] The intra-frame predictor 222 can predict the current block with reference to samples in the current picture. Depending on the prediction mode, the referenced samples may be located near the current block or may be spaced apart. In intra-frame prediction, the prediction mode may include multiple non-directional modes and multiple directional modes. For example, the non-directional mode may include a DC mode and a planar mode. For example, depending on the level of detail of the prediction direction, the directional mode may include 33 directional prediction modes or 65 directional prediction modes. However, this is merely an example, and more or fewer directional prediction modes may be used depending on the settings. The intra-frame predictor 222 may use the prediction mode applied to the neighboring blocks to determine the prediction mode applied to the current block.

[0061] The inter-frame predictor 221 can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector in a reference picture. To reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. It can also include information about the inter-frame prediction direction (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter-frame prediction, neighboring blocks can include spatially neighboring blocks in the current picture and temporally neighboring blocks in the reference picture. The reference picture containing the reference block and the reference picture containing the temporally neighboring block can be the same or different. Temporally neighboring blocks can be referred to as collocated reference blocks, collocated CUs (colCUs), etc., and the reference picture containing temporally neighboring blocks can be referred to as collocated pictures (colPics). For example, the inter-frame predictor 221 can configure a motion information candidate list based on the neighboring blocks and generate information indicating which candidate is used to derive the motion vector and / or reference picture index for the current block. Inter-frame prediction can be performed based on various prediction modes. For example, in the case of skip mode and merge mode, the inter-frame predictor 221 can use the motion information of the neighboring block as the motion information of the current block. In skip mode, unlike merge mode, a residual signal may not be transmitted. In the case of motion vector prediction (MVP) mode, the motion vector of the neighboring block may be used as a motion vector predictor, and the motion vector of the current block may be indicated by signaling the motion vector difference.

[0062] The predictor 220 can generate a prediction signal based on various prediction methods described below. For example, the predictor can apply intra prediction or inter prediction to the prediction of a block, and can apply intra prediction and inter prediction simultaneously. This can be referred to as combined inter and intra prediction (CIIP). In addition, the predictor can predict the block based on an intra-block copy (IBC) prediction mode or based on a palette mode. The IBC prediction mode or palette mode can be used for image / video coding of content such as games, for example, screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in terms of deriving reference blocks within the current picture. That is, IBC can use at least one of the inter prediction techniques described in this document. The palette mode can be considered an example of intra coding or intra prediction. When the palette mode is applied, the sample values ​​in the picture can be signaled based on information about the palette table and the palette index.

[0063] The prediction signal generated by the predictor (including the inter-frame predictor 221 and / or the intra-frame predictor 222) can be used to generate a reconstructed signal or can be used to generate a residual signal. The transformer 232 can generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen-Loeve transform (KLT), a graph-based transform (GBT), or a conditional nonlinear transform (CNT). Here, GBT refers to a transform obtained from a curve graph when the relationship information between pixels is represented by a curve graph. CNT refers to a transform obtained based on a prediction signal generated using all previously reconstructed pixels. In addition, the transform process can be applied to blocks of pixels of the same size as a square, or can be applied to blocks of variable size rather than square.

[0064] The quantizer 233 quantizes the transform coefficients and transmits them to the entropy encoder 240. The entropy encoder 240 encodes the quantized signal (information about the quantized transform coefficients) and outputs the encoded signal as a bitstream. The information about the quantized transform coefficients can be referred to as residual information. The quantizer 233 can rearrange the quantized transform coefficients in block form into a one-dimensional vector form based on the coefficient scanning order, and can generate information about the transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. The entropy encoder 240 can perform various encoding methods such as (for example) exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). The entropy encoder 240 can encode information necessary for video / image reconstruction (for example, syntax element values, etc.) in addition to the quantized transform coefficients, together or separately. The encoded information (for example, the encoded video / image information) can be transmitted or stored in the form of a bitstream in units of network abstraction layer (NAL) units. The video / image information may also include information about various parameter sets such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may also include conventional constraint information. In this document, information and / or syntax elements sent / signaled from the encoding device to the decoding device may be included in the video / image information. The video / image information may be encoded through the above-mentioned encoding process and included in the bitstream. The bitstream may be sent over a network or may be stored in a digital storage medium. Here, the network may include a broadcast network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. A sending unit (not shown) and / or a storage unit (not shown) for sending or storing a signal output from the entropy encoder 240 may be configured as an internal / external element of the encoding device 200, or the sending unit may be included in the entropy encoder 240.

[0065] The quantized transform coefficients output from the quantizer 233 can be used to generate a prediction signal. For example, the residual signal (residual block or residual sample) can be reconstructed by applying inverse quantization and inverse transform to the quantized transform coefficients using the inverse quantizer 234 and the inverse transformer 235. The adder 250 can add the reconstructed residual signal to the prediction signal output from the inter-frame predictor 221 or the intra-frame predictor 222 to generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array). When there is no residual of the processing target block, such as when the skip mode is applied, the prediction block can be used as a reconstructed block. The adder 250 can be referred to as a recovery unit or a recovery block generator. The generated reconstructed signal can be used for intra-frame prediction of the next processing target block in the current picture, and can be used for inter-frame prediction of the next picture after filtering as described below.

[0066] Furthermore, luma mapping with chroma scaling (LMCS) may be applied during the picture encoding and / or reconstruction process.

[0067] The filter 260 can improve the subjective / objective image quality by applying filtering to the reconstructed signal. For example, the filter 260 can generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture, and store the modified reconstructed picture in the memory 270, specifically, in the DPB of the memory 270. Various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filter 260 can generate various types of information related to filtering and transmit the generated information to the entropy encoder 240, as described later when each filtering method is described. The information related to filtering can be encoded by the entropy encoder 240 and output in the form of a bitstream.

[0068] The modified reconstructed picture transmitted to the memory 270 may be used as a reference picture in the inter predictor 221. When inter prediction is applied by the encoding apparatus, prediction mismatch between the encoding apparatus 200 and the decoding apparatus may be avoided, and encoding efficiency may be improved.

[0069] The DPB of the memory 270 can store the corrected reconstructed picture for use as a reference picture in the inter-frame predictor 221. The memory 270 can store the motion information of the block from which the motion information in the current picture is derived (or encoded) and / or the motion information of the block in the reconstructed picture. The stored motion information can be transmitted to the inter-frame predictor 221 to be used as the motion information of the spatially adjacent block or the motion information of the temporally adjacent block. The memory 270 can store the reconstructed samples of the reconstructed block in the current picture and can transmit the reconstructed samples to the intra-frame predictor 222.

[0070] In addition, in this document, at least one of quantization / dequantization and / or transform / inverse transform may be omitted. When quantization / dequantization is omitted, the quantized transform coefficient may be referred to as a transform coefficient. When transform / inverse transform is omitted, the transform coefficient may be referred to as a coefficient or a residual coefficient, or may still be referred to as a transform coefficient for consistency of expression.

[0071] In addition, in this document, quantized transform coefficients and transform coefficients may be referred to as transform coefficients and scaled transform coefficients, respectively. In this case, residual information may include information about the transform coefficients, and the information about the transform coefficients may be signaled via residual coding syntax. The transform coefficients may be derived based on the residual information (or information about the transform coefficients), and the scaled transform coefficients may be derived by inverse transforming (scaling) the transform coefficients. Residual samples may be derived based on the inverse transform (transform) of the scaled transform coefficients. This may also be applied / expressed in other parts of this document.

[0072] Figure 3 is a diagram schematically illustrating the configuration of a video / image decoding device to which the disclosure of this document can be applied.

[0073] Reference Figure 3 , the decoding device 300 may include and be configured with an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an intra-frame predictor 331 and an inter-frame predictor 332. The residual processor 320 may include an inverse quantizer 321 and an inverse transformer 322. According to an embodiment, the entropy decoder 310, the residual processor 320, the predictor 330, the adder 340, and the filter 350 described above may be configured by one or more hardware components (e.g., a decoder chipset or processor). In addition, the memory 360 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware components may also include the memory 360 as an internal / external component.

[0074] When a bit stream including video / image information is input, the decoding device 300 may generate a Figure 2 The image is reconstructed by processing the video / image information in the encoding device illustrated in . For example, the decoding device 300 can derive the unit / block based on the block segmentation related information obtained from the bit stream. The decoding device 300 can perform decoding using the processing unit applied to the encoding device. Therefore, the processing unit for decoding can be, for example, a coding unit, and the coding unit can be divided from the coding tree unit or the maximum coding unit according to a quadtree structure, a binary tree structure and / or a ternary tree structure. One or more transform units can be derived from the coding unit. In addition, the reconstructed image signal decoded and output by the decoding device 300 can be reproduced by a reproduction device.

[0075] The decoding device 300 may receive the data from the Figure 2 The received signal is output by the encoding device, and the entropy decoder 310 can decode the received signal. For example, the entropy decoder 310 can parse the bitstream to derive information necessary for image reconstruction (or picture reconstruction) (e.g., video / image information). The video / image information may also include information about various parameter sets such as the Adaptation Parameter Set (APS), Picture Parameter Set (PPS), Sequence Parameter Set (SPS), or Video Parameter Set (VPS). In addition, the video / image information may also include general constraint information. The decoding device can also decode the picture based on the information about the parameter sets and / or general constraint information. The signaled / received information and / or syntax elements described in this document can then be decoded and obtained from the bitstream through a decoding process. For example, the entropy decoder 310 decodes the information in the bitstream based on a coding method such as Exponential Golomb coding, Context-Adaptive Variable Length Coding (CAVLC), or Context-Adaptive Binary Arithmetic Coding (CABAC), and outputs syntax elements necessary for image reconstruction and quantized values ​​of the transform coefficients for the residual. More specifically, the CABAC entropy decoding method can receive a bin corresponding to each syntax element in the bitstream, determine the context model by using the decoding target syntax element information, the decoding information of the decoding target block, or the information of the symbol / bin decoded in the previous level, and perform arithmetic decoding on the bin by predicting the probability of the bin appearing according to the determined context model, and generate a symbol corresponding to the value of each syntax element. In this case, the CABAC entropy decoding method can update the context model by using the information of the decoded symbol / bin of the context model for the next symbol / bin after determining the context model. The information related to the prediction among the information decoded by the entropy decoder 310 can be provided to the predictor (inter-frame predictor 332 and intra-frame predictor 331), and the residual value (i.e., quantized transform coefficients and related parameter information) that has been entropy decoded in the entropy decoder 310 can be input to the residual processor 320.

[0076] The inverse quantizer 321 may inversely quantize the quantized transform coefficients and output the transform coefficients. The inverse quantizer 321 may rearrange the quantized transform coefficients in a two-dimensional block format. In this case, the rearrangement may be performed based on the coefficient scanning order performed in the encoding device. The inverse quantizer 321 may inversely quantize the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain the transform coefficients.

[0077] The inverse transformer 322 inversely transforms the transform coefficients to obtain a residual signal (residual block, residual sample array).

[0078] The predictor may perform prediction on the current block and generate a prediction block including prediction samples of the current block. The predictor may determine whether to apply intra prediction or inter prediction to the current block based on the information on prediction output from the entropy decoder 310 and may determine a specific intra / inter prediction mode.

[0079] The predictor 330 can generate a prediction signal based on various prediction methods to be described later. For example, the predictor can apply intra-frame prediction or inter-frame prediction to the prediction of a block, and can apply intra-frame prediction and inter-frame prediction simultaneously. This can be referred to as combined inter-frame and intra-frame prediction (CIIP). In addition, the predictor can predict the block based on an intra-block copy (IBC) prediction mode or a palette mode. The IBC prediction mode or palette mode can be used for image / video coding of content such as games, for example, screen content coding (SCC). IBC can basically perform prediction within the current picture, but can be performed similarly to inter-frame prediction in terms of deriving reference blocks within the current picture. That is, IBC can use at least one of the inter-frame prediction techniques described in this document. The palette mode can be considered an example of intra-frame coding or intra-frame prediction. When the palette mode is applied, information about the palette table and palette index can be included in the video / image information and signaled.

[0080] The intra-frame predictor 331 can predict the current block by referencing samples in the current picture. Depending on the prediction mode, the referenced samples may be located near the current block or may be located separately from the current block. In intra-frame prediction, prediction modes may include multiple non-directional modes and multiple directional modes. The intra-frame predictor 331 may determine the prediction mode to be applied to the current block by using the prediction modes applied to neighboring blocks.

[0081] The inter-frame predictor 332 can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can also include information about the inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter-frame prediction, the neighboring blocks can include spatial neighboring blocks in the current picture and temporal neighboring blocks in the reference picture. For example, the inter-frame predictor 332 can construct a motion information candidate list based on the neighboring blocks and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter-frame prediction can be performed based on various prediction modes, and the prediction information can include information indicating the inter-frame prediction mode for the current block.

[0082] The adder 340 can generate a reconstructed signal (reconstructed picture, reconstructed block, or reconstructed sample array) by adding the obtained residual signal to the prediction signal (prediction block or prediction sample array) output from the predictor (including the inter-frame predictor 332 and / or the intra-frame predictor 331). If the residual of the target block is not processed (such as when skip mode is applied), the prediction block can be used as the reconstructed block.

[0083] The adder 340 may be referred to as a reconstructor or a reconstructed block generator. The generated reconstructed signal may be used for intra-frame prediction of the next block to be processed in the current picture, and as described later, may also be output through filtering or may also be used for inter-frame prediction of the next picture.

[0084] In addition, luminance mapping with chroma scaling (LMCS) can also be applied to the picture decoding process.

[0085] The filter 350 can improve the subjective / objective image quality by applying filtering to the reconstructed signal. For example, the filter 350 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image, and store the modified reconstructed image in the memory 360, specifically, in the DPB of the memory 360. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0086] The (modified) reconstructed picture stored in the DPB of the memory 360 can be used as a reference picture in the inter-frame predictor 332. The memory 360 can store the motion information of the block from which the motion information in the current picture is derived (or decoded) and / or the motion information of the block in the reconstructed picture. The stored motion information can be transmitted to the inter-frame predictor 332 to be used as the motion information of the spatially adjacent block or the motion information of the temporally adjacent block. The memory 360 can store the reconstructed samples of the reconstructed block in the current picture and transmit the reconstructed samples to the intra-frame predictor 331.

[0087] In the present disclosure, the embodiments described in the filter 260 , the inter predictor 221 , and the intra predictor 222 of the encoding apparatus 200 may be applied equally or correspondingly to the filter 350 , the inter predictor 332 , and the intra predictor 331 .

[0088] In addition, as described above, when performing video encoding, prediction is performed to improve compression efficiency. Accordingly, a prediction block including prediction samples for a current block as a block to be encoded (i.e., an encoding target block) can be generated. Here, the prediction block includes prediction samples in the spatial domain (or pixel domain). The prediction block is derived in the same manner in the encoding device and the decoding device, and the encoding device can signal information about the residual between the original block and the prediction block (residual information) instead of the original sample value of the original block to the decoding device, thereby improving image coding efficiency. The decoding device can derive a residual block including residual samples based on the residual information, add the residual block and the prediction block to generate a reconstructed block including reconstructed samples, and generate a reconstructed picture including the reconstructed block.

[0089] Residual information can be generated through a transformation and quantization process. For example, the encoding device can derive a residual block between the original block and the prediction block, perform a transformation process on the residual samples (residual sample array) included in the residual block to derive a transformation coefficient, perform a quantization process on the transformation coefficient to derive a quantized transformation coefficient, and signal the relevant residual information to the decoding device (via a bitstream). Here, the residual information may include value information, position information, transformation technology, transformation kernel, quantization parameter, etc. of the quantized transformation coefficient. The decoding device can perform an inverse quantization / inverse transformation process based on the residual information and derive residual samples (or residual blocks). The decoding device can generate a reconstructed picture based on the prediction block and the residual block. In addition, in order to provide a reference for inter-frame prediction of subsequent pictures, the encoding device can also perform inverse quantization / inverse transformation on the quantized transformation coefficient to derive a residual block and generate a reconstructed picture based on this.

[0090] Figure 4 is a diagram illustrating an example of an inter-frame prediction process.

[0091] Reference Figure 4 The inter-frame prediction process may include determining an inter-frame prediction mode, deriving motion information according to the determined prediction mode, and performing prediction (generating prediction samples) based on the derived motion information. The inter-frame prediction process may be performed by the encoding device and decoding device as described above. In the present disclosure, the encoding device may include an encoding device and / or a decoding device.

[0092] Reference Figure 4In operation S400, the encoding device determines an inter-frame prediction mode for the current block. Various inter-frame prediction modes can be used to predict the current block in the picture. For example, various modes such as merge mode, skip mode, motion vector prediction (MVP) mode, affine mode, sub-block merge mode, and merge with MVD (MMVD) mode can be used. Decoder-side motion vector refinement (DMVR) mode, adaptive motion vector resolution (AMVR) mode, bidirectional prediction with CU-level weights (BCW), bidirectional optical flow (BDOF), etc. can be further used as auxiliary modes or can be used instead. Affine mode can be referred to as affine motion prediction mode. MVP mode can be referred to as advanced motion vector prediction (AMVP) mode. In the present disclosure, some modes and / or motion information candidates derived from some modes can be included as one of the motion information-related candidates of other modes. For example, an HMVP candidate can be added as a merge candidate for merge / skip mode, or can be added as an MVP candidate for MVP mode. When an HMVP candidate is used as a motion information candidate for merge mode or skip mode, the HMVP candidate can be referred to as an HMVP merge candidate.

[0093] Prediction mode information indicating the inter-frame prediction mode of the current block can be signaled from the encoding device to the decoding device. The prediction mode information can be included in the bitstream and received by the decoding device. The prediction mode information may include index information indicating one of a plurality of candidate modes. Alternatively, the inter-frame prediction mode may be indicated by hierarchical signaling of flag information. In this case, the prediction mode information may include one or more flags. For example, a skip flag is signaled to indicate whether the skip mode is applied; when the skip mode is not applied, a merge flag is signaled to indicate whether the merge mode is applied; and when the merge mode is not applied, the application of the MVP mode may be indicated or a flag for additional classification may be further signaled. The affine mode may be signaled as an independent mode, or may be signaled as a mode that depends on the merge mode or the MVP mode. For example, the affine mode may include an affine merge mode and an affine MVP mode.

[0094] In operation S410, the encoding apparatus derives motion information about a current block. The motion information may be derived based on an inter prediction mode.

[0095] The encoding device can use the motion information about the current block to perform inter-frame prediction. The encoding device can derive the best motion information about the current block through a motion estimation process. For example, the encoding device can search for a similar reference block with high correlation in units of fractional pixels within a predetermined search range in the reference picture by using the original block in the original picture of the current block, and can derive motion information using the similar reference block. The similarity of the blocks can be derived based on the difference between the sample values ​​based on the phase. For example, the similarity of the blocks can be calculated based on the SAD between the current block (or the template of the current block) and the reference block (or the template of the reference block). In this case, the motion information can be derived based on the reference block with the smallest SAD in the search area. The derived motion information can be signaled to the decoding device according to any of various methods based on the inter-frame prediction mode.

[0096] In operation S420, the encoding device performs inter-frame prediction based on the motion information about the current block. The encoding device may derive a prediction sample of the current block based on the motion information. The current block including the prediction sample may be referred to as a prediction block.

[0097] Figure 5 This is a diagram for explaining the merge mode in inter-frame prediction.

[0098] When the merge mode is applied, the motion information about the current prediction block is not directly sent, but the motion information about the neighboring prediction blocks is used to derive the motion information about the current prediction block. Therefore, the motion information about the current prediction block can be indicated by sending flag information indicating the use of the merge mode and a merge index indicating which nearby prediction block to use. The merge mode can be called the regular merge mode. For example, when the value of the regular_merge_flag syntax element is 1, the merge mode can be applied.

[0099] In order to perform the merge mode, the encoding device needs to search for a merge candidate block for deriving motion information about the current prediction block. For example, up to five merge candidate blocks can be used, but the embodiments of the present disclosure are not limited to this. In addition, the maximum number of merge candidate blocks can be sent in the slice header or the tile group header, but the embodiments of the present disclosure are not limited to this. After finding the merge candidate blocks, the encoding device can generate a merge candidate list and can select the merge candidate block with the minimum cost among the merge candidate blocks as the final merge candidate block.

[0100] The present disclosure may provide various implementations of merging candidate blocks constituting a merging candidate list.

[0101] For example, the merge candidate list can use five merge candidate blocks. For example, four spatial merge candidates and one temporal merge candidate can be used. As a specific example, in the case of spatial merge candidates, Figure 4 The blocks illustrated in may be used as spatial merging candidates. Hereinafter, a spatial merging candidate or a spatial MVP candidate to be described later may be referred to as an SMVP, and a temporal merging candidate or a temporal MVP candidate to be described later may be referred to as a TMVP.

[0102] For example, the merge candidate list of the current block may be constructed based on the following process.

[0103] The encoding device (encoding device / decoding device) may insert the spatial merge candidates derived by searching the spatial neighboring blocks of the current block into the merge candidate list. For example, the spatial neighboring blocks may include the lower left neighboring block, the left neighboring block, the upper right neighboring block, the upper neighboring block, and the upper left neighboring block of the current block. However, this is merely an example, and in addition to the above-mentioned spatial neighboring blocks, other neighboring blocks such as the right neighboring block, the lower neighboring block, and the lower right neighboring block may be further used as spatial neighboring blocks. The encoding device may detect available blocks by searching the spatial neighboring blocks based on priority, and may derive motion information about the detected blocks as spatial merge candidates. For example, the encoding device or the decoding device may search in the order of A1→B1B0→A0→B2. Figure 5 The five blocks shown in are used and the available candidates are sequentially indexed to construct a merge candidate list.

[0104] The encoding device may search for temporally neighboring blocks of the current block and insert the derived temporal merge candidates into the merge candidate list. The temporally neighboring blocks may be at a reference picture that is a different picture from the current picture in which the current block is located. The reference picture in which the temporally neighboring blocks are located may be referred to as a co-located picture or a collocated picture. The temporally neighboring blocks may be searched for on the collocated picture in the order of the bottom-right neighboring block and the bottom-right center block of the co-located block of the current block. In addition, when motion data compression is applied, specific motion information may be stored as representative motion information in each predetermined storage unit in the collocated picture. In this case, it is not necessary to store the motion information about all blocks in the predetermined storage unit, and accordingly, a motion data compression effect can be obtained. In this case, the predetermined storage unit may be predetermined as, for example, a unit of 16×16 samples or a unit of 8×8 samples, or the size information about the predetermined storage unit may be signaled from the encoding device to the decoding device. When motion data compression is applied, the motion information about the temporally neighboring blocks may be replaced with the representative motion information on the predetermined storage unit in which the temporally neighboring blocks are located. That is, in this case, from the perspective of implementation, instead of the prediction block at the coordinates of the temporally neighboring blocks, the motion information of the prediction block covering the arithmetic left shift position after arithmetic right shift by a certain value based on the coordinates (top-left sample position) of the temporally neighboring blocks may be used to derive the temporal merge candidates. For example, when the predetermined storage unit is a unit of 2n×2n samples, if the coordinates of the temporally neighboring block are (xTnb, yTnb), the motion information of the prediction block at the corrected position ((xTnb>>n)<<n), (yTnb>>n)<<n)) may be used for the temporal merge candidates. Specifically, when the predetermined storage unit is a unit of 16×16 samples, if the coordinates of the temporally neighboring block are (xTnb, yTnb), the motion information of the prediction block at the corrected position ((xTnb>>4)<<4), (yTnb>>4)<<4)) may be used for the temporal merge candidates. Alternatively, when the predetermined storage unit is a unit of 8×8 samples, if the coordinates of the temporally neighboring block are (xTnb, yTnb), the motion information of the prediction block at the corrected position ((xTnb>>3)<<3), (yTnb>>3)<<3)) may be used for the temporal merge candidates.

[0105] The encoding device may check whether the number of current merge candidates is less than the maximum number of merge candidates. The maximum number of merge candidates may be predefined or signaled from the encoding device to the decoding device. For example, the encoding device may generate and encode information about the maximum number of merge candidates and send the information to the decoder in the form of a bitstream. When the maximum number of merge candidates is full, the subsequent candidate addition process may not be performed.

[0106] As a result of the check, when the current number of merge candidates is less than the maximum number of merge candidates, the encoding device may insert an additional merge candidate into the merge candidate list. For example, the additional merge candidate may include at least one of a history-based merge candidate, a pairwise average merge candidate, an ATMVP, a combined bidirectional prediction merge candidate (when the slice / tile group type of the current slice / tile group is type B), and / or a zero vector merge candidate, which will be described later.

[0107] As a result of the check, when the current number of merge candidates is not less than the maximum number of merge candidates, the encoding device may terminate the construction of the merge candidate list. In this case, the encoding device may select the optimal merge candidate from the merge candidates constituting the merge candidate list based on the rate-distortion (RD) cost, and signal selection information (e.g., a merge index) indicating the selected merge candidate to the decoding device. The decoding device may select the optimal merge candidate based on the merge candidate list and the selection information.

[0108] As described above, the motion information about the selected merge candidate may be used as the motion information about the current block, and the prediction samples of the current block may be derived based on the motion information about the current block. The encoding device may derive residual samples of the current block based on the prediction samples, and may signal residual information about the residual samples to the decoding device. As described above, the decoding device may generate reconstructed samples based on the residual samples derived based on the residual information and the prediction samples, and may generate a reconstructed picture based on the residual samples.

[0109] When skip mode is applied, motion information about the current block can be derived in the same manner as when merge mode is applied. However, when skip mode is applied, the residual signal of the corresponding block is omitted, so the prediction sample can be directly used as the reconstructed sample. For example, when the value of the cu_skip_flag syntax element is 1, skip mode can be applied.

[0110] In addition, pairwise average merge candidates may be referred to as pairwise average candidates or pairwise candidates. Pairwise average candidates may be generated by averaging predefined candidate pairs in an existing merge candidate list. In addition, the predefined pairs may be defined as {(0,1), (0,2), (1,2), (0,3), (1,3), (2,3)}. Here, the number may indicate the merge index of the merge candidate list. The averaged motion vector may be calculated separately for each reference list. For example, when two motion vectors are available in one list, they may be averaged even if the two motion vectors point to different reference pictures. For example, when only one motion vector is available, one motion vector may be used directly. For example, when no available motion vector exists, the list may remain invalid.

[0111] For example, when the merge candidate list is not full even after adding pairwise average merge candidates, that is, when the current number of merge candidates in the merge candidate list is less than the maximum number of merge candidates, a zero vector (zero MVP) may be inserted last until the maximum number of merge candidates appears. That is, a zero vector may be inserted until the number of current merge candidates in the merge candidate list reaches the maximum number of merge candidates.

[0112] Furthermore, conventionally, only a single motion vector can be used to express the motion of a coding block. Specifically, a translational motion model can be used. However, while this method may express optimal motion per block, it does not guarantee the actual optimal movement of each sample. Determining the optimal motion vector per sample can improve coding efficiency. For this purpose, an affine motion model can be used. The affine motion prediction method, which uses the affine motion model for coding, can efficiently express the four types of motion described below.

[0113] Figure 6 is a diagram illustrating an example of motion expressed by an affine motion model.

[0114] Reference Figure 6 , the motion that can be expressed by the affine motion model can include translation motion, scaling motion, rotation motion and shear motion. That is, through affine motion prediction, it is possible to efficiently express not only Figure 6 The image (part of) shown in the figure moves in a plane over time, and also expresses a scaling motion in which (part of) the image is scaled over time, a rotational motion in which (part of) the image is rotated over time, and a shearing motion in which (part of) the image is deformed into a parallelogram over time.

[0115] The encoding / decoding device can predict the distorted shape of an image based on the motion vector at the control point (CP) of the current block through affine motion prediction. By doing so, the encoding / decoding device can increase the accuracy of the prediction, thereby improving the performance of compressed images. In addition, the motion vector of at least one control point of the current block can be derived using the motion vector of a neighboring block of the current block, thereby reducing the amount of data related to the additional information added and improving the efficiency of inter-frame prediction.

[0116] An affine motion model that expresses three motions (translation, scaling, and rotation) among the motions that can be expressed by the affine motion model may be referred to as a similar (or simplified) affine motion model. However, the affine motion model is not limited to the above-mentioned motion model.

[0117] The affine motion prediction method can use two, three, or four motion vectors to express a motion vector in each sample unit of a block.

[0118] Figure 7aand Figure 7b is a diagram illustrating an example of a control point motion vector (CPMV) used for affine motion prediction.

[0119] In affine motion prediction, two or more control point motion vectors may be used to determine the motion vector of a sample position included in a block. In this case, the set of motion vectors may be referred to as an affine motion vector field (MVF).

[0120] For example, Figure 7a A case where two CPMVs are used, which may be referred to as a 4-parameter affine model, may be exemplified. In this case, for example, a motion vector at a sample position (x, y) may be determined as in Equation 1.

[0121] [Formula 1]

[0122]

[0123] For example, Figure 7b A case of using three CPMVs, which may be referred to as a 6-parameter affine model, may be exemplified. In this case, for example, a motion vector at a sample position (x, y) may be determined as in Equation 2.

[0124] [Formula 2]

[0125]

[0126] In Equation 1 and Equation 2, {v x ,v y} can indicate the motion vector at position (x, y). In addition, {v 0x ,v 0y} may indicate the CPMV of the control point (CP) at the upper left corner of the coding block, {v 1x ,v 1y} may indicate the CPMV of the CP at the upper right corner of the coding block, and {v 2x ,v 2y} may indicate the CPMV of the CP at the lower left corner of the coding block. In addition, W may indicate the width of the current block, and H may indicate the height of the current block.

[0127] Figure 8 is a diagram illustrating an example in which the affine MVF is determined in units of subblocks.

[0128] In the encoding / decoding process, the affine MVF can be determined in units of samples or in units of predefined subblocks. For example, if the affine MVF is determined in units of samples, a motion vector can be obtained based on each sample value. Alternatively, for example, when the affine MVF is determined in units of subblocks, the motion vector of the corresponding block can be obtained based on the sample value at the center of the subblock (which is to the lower right of the center, i.e., the lower right sample among the four center samples). In other words, the motion vector of the current block in affine motion estimation can be derived in units of samples or subblocks.

[0129] exist Figure 8 In the case of , the affine MVF is determined in units of 4×4 sub-blocks, but the size of the sub-block can be modified differently.

[0130] That is, when affine prediction is available, the three motion models applicable to the current block may include a translational motion model, a 4-parameter affine motion model, and a 6-parameter affine motion model. Here, the translational motion model may refer to a model in which an existing block-unit motion vector is used, the 4-parameter affine motion model may refer to a model in which two CPMVs are used, and the 6-parameter affine motion model may refer to a model in which three CPMVs are used.

[0131] Furthermore, affine motion prediction may include an affine MVP (or affine inter) mode or an affine merge mode.

[0132] Figure 9 A diagram for explaining the affine merge mode in inter-frame prediction.

[0133] For example, in affine merge mode, the CPMV can be determined based on the affine motion model of the neighboring blocks encoded using affine motion prediction. For example, neighboring blocks encoded as affine motion predictions in search order can be used in affine merge mode. That is, when at least one of the neighboring blocks is encoded using affine motion prediction, the current block can be encoded in affine merge mode. Here, the refined merge mode may be referred to as AF_MERGE.

[0134] When the affine merge mode is applied, the CPMV of the neighboring block can be used to derive the CPMV of the current block. In this case, the CPMV of the neighboring block can be used as the CPMV of the current block as it is, or the CPMV of the neighboring block can be modified based on the size of the neighboring block and the size of the current block and used as the CPMV of the current block.

[0135] On the other hand, in the case of an affine merge mode in which a motion vector (MV) is derived in units of sub-blocks, this may be referred to as a sub-block merge mode, and the sub-block merge mode may be indicated based on a sub-block merge flag (or merge_subblock_flag syntax element). Alternatively, when the value of the merge_subblock_flag syntax element is 1, it may be indicated that the sub-block merge mode is applied. In this case, the affine merge candidate list to be described later may be referred to as a sub-block merge candidate list. In this case, the sub-block merge candidate list may also include a candidate derived by the SbTMVP to be described later. In this case, the candidate derived by the SbTMVP may be used as a candidate for index 0 of the sub-block merge candidate list. In other words, the candidate derived from the SbTMVP may be before the inherited affine candidate or the constructed affine candidate to be described later in the sub-block merge candidate list.

[0136] When the affine merge mode is applied, an affine merge candidate list may be constructed to derive the CPMV of the current block. For example, the affine merge candidate list may include at least one of the following candidates. 1) Inherited affine merge candidate. 2) Constructed affine merge candidate. 3) Zero motion vector candidate (or zero vector). Here, the inherited affine merge candidate is a candidate derived based on the CPMV of a neighboring block when encoding the neighboring block in the affine mode, the constructed affine merge candidate is a candidate derived by constructing a CPMV based on the MV of the neighboring block of the corresponding CP in units of each CPMV, and the zero motion vector candidate may indicate a candidate consisting of a CPMV whose value is 0.

[0137] For example, the affine merge candidate list can be constructed as follows.

[0138] There can be up to two inherited affine candidates, and the inherited affine candidates can be derived from the affine motion model of the neighboring blocks. The neighboring blocks can include a left neighboring block and an upper neighboring block. The candidate blocks can be as follows Figure 4 The scanning order of the left predictor may be A1→A0, and the scanning order of the upper predictor may be B1→B0→B2. Only one inherited candidate may be selected from each of the left and upper sides. Pruning check may not be performed between two inherited candidates.

[0139] When a neighboring affine block is identified, the control point motion vector of the identified block can be used to derive the CPMVP candidate in the affine merge list of the current block. In this case, the neighboring affine block can indicate a block encoded in an affine prediction mode among the neighboring blocks of the current block. For example, referring to Figure 8When encoding the lower left neighboring block A in affine prediction mode, the motion vector v2 of the upper left corner of the neighboring block A, the motion vector v3 of the upper right corner of the neighboring block A, and the motion vector v4 of the lower left corner of the neighboring block A can be obtained. When encoding the neighboring block A using a 4-parameter affine motion model, two CPMVs of the current block can be calculated based on v2 and v3. When encoding the neighboring block A using a 6-parameter affine motion model, three CPMVs of the current block can be calculated based on v2, v3, and v4.

[0140] Figure 10 3 is a diagram for explaining the positions of candidates in the affine merge mode.

[0141] The constructed affine candidate may refer to a candidate constructed by combining neighboring translation motion information about each control point. Motion information about a control point may be derived from a specified spatial neighborhood and temporal neighborhood. CPMVk (k=0, 1, 2, 3) may indicate the kth control point.

[0142] Reference Figure 10 For CPMV0, blocks can be checked in the order B2 → B3 → A2, and the motion vector of the first available block can be used. For CPMV1, blocks can be checked in the order B1 → B0. For CPMV2, blocks can be checked in the order A1 → A0. If available, the temporal motion vector predictor (TMVP) can be used as CPMV3.

[0143] After obtaining the motion vectors of the four control points, an affine merge candidate can be generated based on the obtained motion information. The combination of the control point motion vectors can correspond to any one of {CPMV0, CPMV1, CPMV2}, {CPMV0, CPMV1, CPMV3}, {CPMV0, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV3}, {CPMV0, CPMV1}, and {CPMV0, CPMV2}.

[0144] A combination of three CPMVs can constitute a 6-parameter affine merge candidate, and a combination of two CPMVs can constitute a 4-parameter affine merge candidate. To avoid motion scaling processing, relevant combinations of control point motion vectors can be discarded when the reference indices of the control points are different.

[0145] Figure 11 is a diagram for explaining subblock-based temporal motion vector prediction (SbTMVP) in inter prediction.

[0146] In addition, a sub-block based temporal motion vector prediction (SbTMVP) method may also be used. For example, SbTMVP may be referred to as advanced temporal motion vector prediction (ATMVP). SbTMVP may use the motion field in a collocated picture to improve the motion vector prediction and merge mode of the CU in the current picture. Here, a collocated picture may be referred to as a collocated picture.

[0147] For example, SbTMVP can predict motion at the sub-block (or sub-CU) level. In addition, SbTMVP can apply motion shifting before obtaining temporal motion information from the collocated picture. Here, the motion shifting can be obtained from the motion vector of one of the spatially neighboring blocks of the current block.

[0148] SbTMVP can predict the motion vector of a sub-block (or sub-CU) in the current block (or CU) according to two steps.

[0149] In the first step, you can Figure 4 The spatially neighboring blocks are tested in the order A1, B1, B0, and A0 in FIG. The first spatially neighboring block having a motion vector using the collocated picture as its reference picture may be checked, and this motion vector may be selected as the motion shift to be applied. When no such motion is detected from the spatially neighboring blocks, the motion shift may be set to (0, 0).

[0150] In the second step, the motion shift checked in the first step can be applied to obtain sub-block level motion information (motion vector and reference index) from the collocated picture. For example, the motion shift can be added to the coordinates of the current block. For example, the motion shift can be set to Figure 4 In this case, for each sub-block, motion information about the sub-block can be derived using motion information about the corresponding block in the collocated picture. Temporal motion scaling can be applied to align the reference picture of the temporal motion vector with the reference picture of the current block.

[0151] A sub-block based merge list including a combination of SbTVMP candidates and affine merge candidates can be used for signaling of affine merge mode. Here, affine merge mode can be referred to as sub-block based merge mode. Depending on the flag included in the sequence parameter set (SPS), SbTVMP mode can be available or unavailable. When SbTMVP mode is available, the SbTMVP predictor can be added as the first entry in the sub-block based merge candidate list, and the affine merge candidate can follow. The maximum allowable size of affine merge candidates can be 5.

[0152] The size of the sub-CU (or sub-block) used in SbTMVP can be fixed to 8×8, and as in affine merge mode, SbTMVP mode can be applied only to blocks with both width and height of 8 or greater. The encoding logic of the additional SbTMVP merge candidate can be the same as the encoding logic of other merge candidates. That is, for each CU in a P or B slice, an RD check using an additional rate-distortion (RD) cost can be performed to determine whether to use the SbTMVP candidate.

[0153] Furthermore, a prediction block for the current block can be derived based on motion information derived from a prediction mode. The prediction block may include prediction samples (an array of prediction samples) for the current block. When the motion vector of the current block indicates a fractional sample unit, an interpolation process may be performed. Accordingly, prediction samples for the current block may be derived based on fractional sample unit reference samples in a reference picture. When affine inter prediction (affine prediction mode) is applied to the current block, prediction samples may be generated based on sample / sub-block units MV. When bidirectional prediction is applied, prediction samples may be derived as a weighted sum (based on phase) or weighted average of prediction samples derived based on L0 prediction (i.e., prediction using reference pictures in reference picture list L0 and MVL0) and prediction samples derived based on L1 prediction (i.e., prediction using reference pictures in reference picture list L1 and MVL1). Here, a motion vector in the L0 direction may be referred to as an L0 motion vector or MVL0, and a motion vector in the L1 direction may be referred to as an L1 motion vector or MVL1. In the case of applying bidirectional prediction, when the reference picture for L0 prediction and the reference picture for L1 prediction are in different temporal directions relative to the current picture (ie, corresponding to the case of bidirectional direction or bidirectional prediction), this can be called true bidirectional prediction.

[0154] In addition, as described above, reconstructed samples and reconstructed pictures may be generated based on the derived prediction samples, and then, processes such as in-loop filtering may be performed.

[0155] In addition, when bidirectional prediction is applied to the current block, the prediction sample can be derived based on weighted averaging. For example, bidirectional prediction using weighted averaging can be referred to as bidirectional prediction with CU-level weights (BCW), bidirectional prediction with weighted averaging (BWA), or bidirectional prediction with weighted averaging.

[0156] Conventionally, a bidirectional prediction signal (i.e., a bidirectional prediction sample) can be derived by a simple average of an L0 prediction signal (L0 prediction sample) and an L1 prediction signal. That is, a bidirectional prediction sample can be derived as an average of an L0 prediction sample based on an L0 reference picture and MVL0 and an L1 prediction sample based on an L1 reference picture and MVL1. However, when bidirectional prediction is applied, a bidirectional prediction signal (i.e., a bidirectional prediction sample) can be derived by a weighted average of an L0 prediction signal and an L1 prediction signal as follows. For example, a bidirectional prediction signal (i.e., a bidirectional prediction sample) can be derived as shown in Formula 3.

[0157] [Formula 3]

[0158] P bi-pred =((8-w)*P0+w*P1+4)>>3

[0159] In Formula 3, Pbi-pred may indicate the value of the bidirectional prediction signal, that is, the predicted sample value derived by applying the bidirectional prediction, and w may indicate a weight. In addition, P0 may indicate the value of the L0 prediction signal, that is, the predicted sample value derived by applying the L0 prediction, and P1 may indicate the value of the L1 prediction signal, that is, the predicted sample value derived by applying the L1 prediction.

[0160] For example, five weights may be allowed in weighted average bidirectional prediction. For example, the five weights w may include -2, 3, 4, 5, or 10. That is, the weight w may be determined as one of the weight candidates including -2, 3, 4, 5, or 10. For each CU to which bidirectional prediction is applied, the weight w may be determined by one of two methods. In the first method, the weight index may be signaled after the motion vector difference of the non-merged CU. In the second method, the weight index of the merged CU may be inferred from the neighboring blocks based on the merge candidate index.

[0161] For example, weighted average bidirectional prediction can be applied to a CU with 256 or more luma samples. That is, when the product of the width and height of the CU is greater than or equal to 256, weighted average bidirectional prediction can be applied. In the case of a low-latency picture, five weights can be used, while in the case of a non-low-latency picture, three weights can be used. For example, the three weights can include 3, 4, or 5.

[0162] For example, in an encoding device, a fast search algorithm can be applied to find weight indices without significantly increasing the complexity of the encoding device. The algorithm can be summarized as follows. For example, when the current picture is a low-latency picture when combined with adaptive motion vector resolution (AMVR) (when AMVR is used as an inter-frame prediction mode), non-equal weights can be conditionally checked for 1-pixel and 4-pixel motion vector accuracy. For example, when combined with affine (when the affine prediction mode is used as the inter-frame prediction mode), affine motion estimation (ME) can be performed with non-equal weights when the affine prediction mode is currently selected as the optimal mode. For example, when two reference pictures for bidirectional prediction are the same, non-equal weights can be conditionally checked. For example, when specific conditions are met based on the POC distance, encoding quantization parameter (QP), and temporal level between the current picture and the reference picture, the search for non-equal weights can be omitted.

[0163] For example, a BCW weight index may be encoded using one context coding bin followed by a bypass coding bin. The first context coding bin may indicate whether equal weights are used. When unequal weights are used based on the first context coding bin, bypass coding may be used to signal additional bins to indicate the unequal weights to be used.

[0164] Also, when bidirectional prediction is applied, weight information for generating a prediction sample may be derived based on weight index information about a candidate selected from among candidates included in the merge candidate list.

[0165] According to an embodiment of the present disclosure, when constructing motion vector candidates for merge mode, weight index information about temporal motion vector candidates can be derived as follows. For example, when the temporal motion vector candidate uses bidirectional prediction, weight index information about weighted averaging can be derived. That is, when the inter-frame prediction type is bidirectional prediction, weight index information about the temporal merge candidate (or temporal motion vector candidate) in the merge candidate list can be derived.

[0166] For example, the weight index information about the weighted average related to the temporal motion vector candidate can always be derived as 0. Here, the weight index information being 0 may mean that the weight of each reference direction (i.e., the L0 prediction direction and the L1 prediction direction in the bidirectional prediction) is the same. For example, the process for deriving the motion vector of the luma component for the merge mode can be shown in Table 1 below.

[0167] [Table 1]

[0168]

[0169]

[0170]

[0171]

[0172]

[0173] Referring to Table 1, gbiIdx may indicate a bidirectional prediction weight index, and gbiIdxCol may indicate a bidirectional prediction weight index for a temporal merge candidate (e.g., a temporal motion vector candidate in a merge candidate list). In the process of deriving the motion vector for the luma component of the merge mode (Table 3 of 8.4.2.2), gbiIdxCol may be derived as 0. That is, the weight index of the temporal motion vector candidate may be derived as 0.

[0174] Alternatively, the weight index of the weighted average of the temporal motion vector candidates can be derived based on the weight index information about the collocated blocks. Here, the collocated blocks can be referred to as collocated blocks, co-located blocks, or co-located reference blocks, and the collocated blocks can refer to blocks at the same position as the current block on the reference picture. For example, the process for deriving the motion vector for the luma component for merge mode can be shown in Table 2 below.

[0175] [Table 2]

[0176]

[0177]

[0178]

[0179]

[0180]

[0181] Referring to Table 2, gbiIdx may indicate a bidirectional prediction weight index, and gbiIdxCol may indicate a bidirectional prediction weight index for a temporal merge candidate (e.g., a temporal motion vector candidate in a merge candidate list). In deriving a motion vector for the luma component in merge mode, when the slice type or tile group type is B (Table 4 of Table 8.4.2.2), gbiIdxCol may be derived as gbiIdxCol. That is, the weight index of the temporal motion vector candidate may be derived as the weight index of the collocated block.

[0182] In addition, according to another embodiment of the present disclosure, when the motion vector candidates for the merge mode are constructed in units of sub-blocks, the weight index of the weighted average of the temporal motion vector candidates can be derived. Here, the merge mode in units of sub-blocks can be referred to as an affine merge mode (in units of sub-blocks). The temporal motion vector candidate can indicate a temporal motion vector candidate based on a sub-block and can be referred to as an SbTMVP (or ATMVP) candidate. That is, when the inter prediction type is bidirectional prediction, weight index information about the SbTMVP candidate (or sub-block-based temporal motion vector candidate) in the affine merge candidate list or the sub-block merge candidate list can be derived.

[0183] For example, the weight index information regarding the weighted average of the sub-block-based temporal motion vector candidates may always be derived as 0. Here, the weight index information of 0 may mean that the weights of the reference directions (i.e., the L0 prediction direction and the L1 prediction direction in the bidirectional prediction) are the same. For example, the process for deriving the motion vector and the reference index in the sub-block merge mode and the process for deriving the sub-block-based temporal merge candidate may be as shown in Tables 3 and 4.

[0184] [Table 3]

[0185]

[0186]

[0187]

[0188]

[0189]

[0190]

[0191] [Table 4]

[0192]

[0193]

[0194] Referring to Tables 3 and 4 above, gbiIdx may indicate a bidirectional prediction weight index, gbiIdxSbCol may indicate a bidirectional prediction weight index of a subblock-based temporal merge candidate (e.g., a temporal motion vector candidate in a subblock-based merge candidate list), and in the process for deriving a subblock-based temporal merge candidate (8.4.4.3), gbiIdxSbCol may be derived as 0. That is, the weight index of the subblock-based temporal motion vector candidate may be derived as 0.

[0195] Alternatively, weight index information for weighted averaging of subblock-based temporal motion vector candidates may be derived based on weight index information about the temporal center block. For example, the temporal center block may indicate a subblock or sample located at the center of a collocated block or a collocated block, and specifically, may indicate a subblock or sample located to the lower right of the four center subblocks or samples of the collocated block. In this case, for example, the process for deriving motion vectors and reference indices in subblock merging mode, the process for deriving subblock-based temporal merging candidates, and the process for deriving basic motion information for subblock-based temporal merging may be shown in Tables 5, 6, and 7.

[0196] [Table 5]

[0197]

[0198]

[0199]

[0200]

[0201]

[0202]

[0203]

[0204] [Table 6]

[0205]

[0206]

[0207]

[0208]

[0209] [Table 7]

[0210]

[0211]

[0212]

[0213]

[0214] Referring to Tables 5, 6, and 7, gbiIdx may indicate a bidirectional prediction weight index, and gbiIdxSbCol may indicate a bidirectional prediction weight index of a subblock-based temporal merge candidate (e.g., a temporal motion vector candidate in a subblock-based merge candidate list). In the process for deriving basic motion information about subblock-based temporal merging (8.4.4.4), gbiIdxSbCol may be derived as gbiIdxcolCb. That is, the weight index of the subblock-based temporal motion vector candidate may be derived as the weight index of the temporal center block. For example, the temporal center block may indicate a subblock or sample at the center of a collocated block or a collocated block, and specifically, may indicate a subblock or sample at the lower right side of the four center subblocks or samples of the collocated block.

[0215] Alternatively, weight index information of a weighted average of sub-block-based temporal motion vector candidates can be derived based on weight index information in units of each sub-block, and when a sub-block is not available, the weight index information can be derived based on the weight index information of the temporal center block. For example, the temporal center block may indicate a sub-block or sample at the center of a collocated block or a collocated block, and specifically, may indicate a sub-block or sample at the lower right side of the four center sub-blocks or samples of the collocated block. For example, in this case, the process for deriving a motion vector and a reference index in a sub-block merge mode, the process for deriving a sub-block-based temporal merge candidate, and the process for deriving basic motion information for sub-block-based temporal merge can be shown in Tables 8, 9, and 10.

[0216] [Table 8]

[0217]

[0218]

[0219]

[0220]

[0221]

[0222]

[0223]

[0224] [Table 9]

[0225]

[0226]

[0227]

[0228]

[0229] [Table 10]

[0230]

[0231]

[0232]

[0233]

[0234] Referring to Tables 8, 9, and 10, gbiIdx may indicate a bidirectional prediction weight index, and gbiIdxSbCol may indicate a bidirectional prediction weight index for a subblock-based temporal merge candidate (e.g., a temporal motion vector candidate in a subblock-based merge candidate list). In the process for deriving basic motion information for subblock-based temporal merging (8.4.4.3), gbiIdxSbCol may be derived as gbiIdxcolCb. Alternatively, in the process for deriving basic motion information for subblock-based temporal merging according to a condition (e.g., when both availableFlagL0SbCol and availableFlagL1SbCol are 0) (8.4.4.3), gbiIdxSbCol may be derived as ctrgbiIdx, and in the process for deriving basic motion information for subblock-based temporal merging (8.4.4.4), ctrgbiIdx may be derived as gbiIdxSbCol. That is, the weight index of the sub-block-based temporal motion vector candidate can be derived as a weight index per sub-block, or when the sub-block is not available, it can be derived as a weight index of the temporal center block. For example, the temporal center block may indicate a sub-block or sample at the center of a collocated block or a collocated block, and specifically, may indicate a sub-block or sample at the lower right side of the four center sub-blocks or samples of the collocated block.

[0235] In addition, according to another embodiment of the present disclosure, when constructing motion vector candidates for merge mode, weight index information about paired candidates can be derived. For example, paired candidates can be included in the merge candidate list, and weight index information about the weighted average of the paired candidates can be derived. Paired candidates can be derived based on other merge candidates in the merge candidate list, and when the paired candidates use bidirectional prediction, the weight index of the weighted average can be derived. That is, when the inter-frame prediction type is bidirectional prediction, weight index information about the paired candidates in the merge candidate list can be derived.

[0236] The pair candidate may be derived based on the other two merge candidates (eg, cand0 and cand1) among the candidates included in the merge candidate list.

[0237] For example, the weight index information about the paired candidate can be derived based on the weight index information about either of the two merge candidates (e.g., merge candidate cand0 or merge candidate cand1). For example, the weight index information about the candidate using bidirectional prediction among the two merge candidates can be used to derive the weight index information about the paired candidate.

[0238] Alternatively, when the weight index information about each of the other two merge candidates is the same as the first weight index information, the weight index information about the paired candidate can be derived based on the first weight index information. In addition, when the weight index information about each of the other two merge candidates is different, the weight index information about the paired candidate can be derived based on the default weight index information. The default weight index information may correspond to weight index information for assigning the same weight to each of the L0 prediction sample and the L1 prediction sample.

[0239] Alternatively, when the weight index information about each of the other two merge candidates is the same as the first weight index information, the weight index information about the paired candidate can be derived based on the first weight index information. In addition, when the weight index information about each of the other two merge candidates is different, the weight index information about the paired candidate includes default weight index information among the weight index information about each of the other two candidates. The default weight index information may correspond to weight index information for assigning the same weight to each of the L0 prediction sample and the L1 prediction sample.

[0240] In addition, according to another embodiment of the present disclosure, when the motion vector candidate for the merge mode is constructed in units of sub-blocks, weight index information of the weighted average of the temporal motion vector candidates can be derived. Here, the merge mode in units of sub-blocks can be referred to as an affine merge mode (in units of sub-blocks). The temporal motion vector candidate can indicate a temporal motion vector candidate based on a sub-block and can be referred to as an SbTMVP (or ATMVP) candidate. The weight index information about the SbTMVP candidate can be derived based on the weight index information about the left neighboring block of the current block. That is, when the candidate derived by SbTMVP uses bidirectional prediction, the weight index of the left neighboring block of the current block can be derived as a weight index of the merge mode based on the sub-block.

[0241] For example, since the SbTMVP candidate can be derived from the collocated block based on the spatially adjacent left block (or left neighboring block) of the current block, the weight index of the left neighboring block can be considered reliable. Therefore, the weight index of the SbTMVP candidate can be derived as the weight index of the left neighboring block.

[0242] Furthermore, according to another embodiment of the present disclosure, when constructing motion vector candidates for affine merge mode, weight index information regarding weighted averaging can be derived when the affine merge candidate uses bidirectional prediction. That is, when the inter prediction type is bidirectional prediction, weight index information regarding the candidates in the affine merge candidate list or the subblock merge candidate list can be derived.

[0243] For example, among the affine merge candidates, the constructed affine merge candidate can derive CP0, CP1, CP2, or CP3 candidates based on the spatial neighboring blocks or temporal neighboring blocks of the current block to indicate the candidate for deriving the MVF as an affine model. For example, CP0 can indicate a control point located at the upper left sample position of the current block, CP1 can indicate a control point located at the upper right sample position of the current block, and CP2 can indicate the lower left sample position of the current block. In addition, CP3 can indicate a control point located at the lower right sample position of the current block.

[0244] For example, a constructed affine merge candidate among the affine merge candidates may be generated based on a combination of corresponding control points of the current block such as {CP0, CP1, CP2}, {CP0, CP1, CP3}, {CP0, CP2, CP3}, {CP1, CP2, CP3}, {CP0, CP1}, and {CP0, CP2}. For example, the affine merge candidate may include at least one of {CPMV0, CPMV1, CPMV2}, {CPMV0, CPMV1, CPMV3}, {CPMV0, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV3}, {CPMV0, CPMV1}, and {CPMV0, CPMV2}, and CPMV0, CPMV1, CPMV2, and CPMV3 may correspond to motion vectors of CP0, CP1, CP2, and CP3, respectively.

[0245] In one embodiment, when the affine merge candidate includes CPMV0 of control point 0 (CP0) located on the upper left side of the current block, weight index information about the affine merge candidate can be derived based on the 0th weight index information about CP0. The 0th weight index information may correspond to the weight index information of the block used to derive CPMV0 among the neighboring blocks of CP0. In this case, the neighboring blocks of CP0 may include the upper left neighboring block of the current block, the left neighboring block adjacent to the lower side of the upper left neighboring block, and the upper neighboring block adjacent to the right side of the upper left neighboring block.

[0246] On the other hand, when the affine merge candidate does not include CPMV0 of CP0 located on the upper left side of the current block, weight index information about the affine merge candidate can be derived based on first weight index information about control point 1 (CP1) located on the upper right side of the current block. The first weight index information may correspond to weight index information of a block used to derive CPMV1 among neighboring blocks of CP1. In this case, the neighboring blocks of CP1 may include the upper right neighboring block of the current block and the upper neighboring block adjacent to the left side of the upper right neighboring block.

[0247] For each of {CPMV0, CPMV1, CPMV2}, {CPMV0, CPMV1, CPMV3}, {CPMV0, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV3}, {CPMV0, CPMV1} and {CPMV0, CPMV2}, weight index information about the affine merge candidate can be derived based on the weight index information about the block used to derive the first CPMV.

[0248] According to another embodiment for deriving weight index information about an affine merge candidate, when the weight index information about CP0 located on the upper left side of the current block and the weight index information about CP1 located on the right side of the current block are identical to each other, the weight index information about the affine merge candidate can be derived based on the 0th weight index information about CP0. The 0th weight index information may correspond to the weight index information of the block used to derive CPMV0 among the neighboring blocks of CP0. On the other hand, when the weight index information about CP0 located on the upper left side of the current block and the weight index information about CP1 located on the upper right side of the current block are different from each other, the weight index information about the affine merge candidate can be derived based on the default weight index information. The default weight index information may correspond to the weight index information for assigning the same weight to each of the L0 prediction sample and the L1 prediction sample.

[0249] According to another embodiment of deriving weight index information about affine merge candidates, the weight index information about the affine merge candidates can be derived as the weight index of the candidate with the highest frequency of occurrence among the weight indexes of each candidate. For example, the weight index of the candidate block determined as the motion vector in CP0 among the CP0 candidate blocks, the weight index of the candidate block determined as the motion vector in CP1 among the CP1 candidate blocks, the weight index of the candidate block determined as the motion vector in CP2 among the CP2 candidate blocks, and / or the weight index of the candidate block determined as the motion vector in CP3 among the CP3 candidate blocks, whichever weight index overlaps the most, can be derived as the weight index of the affine merge candidate.

[0250] For example, CP0 and CP1 may be used as control points, CP0, CP1, and CP2 may be used, and CP3 may not be used. However, for example, when a CP3 candidate of an affine block (a block encoded in an affine prediction mode) is to be used, the method of deriving a weight index in a temporal candidate block described in the above embodiment may be used.

[0251] Figure 12 and Figure 13 is a diagram schematically illustrating an example of a video / image encoding method and related components according to an embodiment of the present disclosure.

[0252] Figure 12 The method disclosed in Figure 2 or Figure 13 Specifically, for example, Figure 12 Operations S1200 to S1220 may be performed by Figure 13 The predictor 220 of the encoding device 200 performs, and Figure 12 Operation S1230 may be performed by Figure 13 The entropy encoder 240 of the encoding device 200 performs. In addition, although Figure 12 Not shown in the example, but can be Figure 12 The predictor 220 of the encoding device 200 derives predicted samples or prediction related information, the residual processor 230 of the encoding device 200 can derive residual information from the original samples or predicted samples, and the entropy encoder 240 of the encoding device 200 can generate a bitstream from the residual information or prediction related information. Figure 12 The method disclosed in may include the embodiments described above in the present disclosure.

[0253] Reference Figure 12 In operation S1200, the encoding device may determine an inter-frame prediction mode of the current block and generate inter-frame prediction mode information indicating the inter-frame prediction mode. For example, the encoding device may determine a merge mode, an affine (merge) mode, or a subblock merge mode as the inter-frame prediction mode to be applied to the current block and may generate inter-frame prediction mode information indicating the determined inter-frame prediction mode.

[0254] In operation S1210, the encoding device may generate a merge candidate list for the current block based on the inter prediction mode. For example, the encoding device may generate the merge candidate list according to the determined inter prediction mode. In this case, when the determined inter prediction mode is the affine merge mode or the subblock merge mode, the merge candidate list may be referred to as an affine merge candidate list or a subblock merge candidate list, but may also be simply referred to as a merge candidate list.

[0255] For example, candidates may be inserted into the merge candidate list until the number of candidates in the merge candidate list becomes the maximum number of candidates. Here, the candidate may indicate a candidate or candidate block for deriving motion information (or motion vector) of the current block. For example, the candidate block may be derived by searching the neighboring blocks of the current block. For example, the neighboring blocks may include spatial neighboring blocks and / or temporal neighboring blocks of the current block, the spatial neighboring blocks may be preferentially searched to derive (spatial merging) candidates, and then the temporal neighboring blocks may be searched to derive (temporal merging) candidates, and the derived candidates may be inserted into the merge candidate list. For example, when the number of candidates in the merge candidate list is less than the maximum number of candidates in the merge candidate list even after the candidate is inserted, additional candidates may be inserted. For example, the additional candidates include at least one of a history-based merge candidate, a pairwise average merge candidate, an ATMVP, and a combined bidirectional prediction merge candidate (when the slice / tile group type of the current slice / tile group is type B), and / or a zero vector merge candidate.

[0256] Alternatively, for example, the candidate may be inserted into the affine merge candidate list until the number of candidates in the affine merge candidate list becomes the maximum number of candidates. Here, the candidate may include the control point motion vector (CPMV) of the current block. Alternatively, the candidate may indicate a candidate or candidate block for deriving the CPMV. The CPMV may indicate a motion vector at a control point (CP) of the current block. For example, the number of CPs may be 2, 3, or 4, and the CP may be at least a portion of the upper left side (or upper left corner), upper right side (or upper right corner), lower left side (or lower left corner), or lower right side (or lower right corner) of the current block, and there may be only one CP at each position.

[0257] For example, candidates can be derived by searching neighboring blocks of the current block (or neighboring blocks of the CP of the current block). For example, the affine merge candidate list may include at least one of an inherited affine merge candidate, a constructed affine merge candidate, and a zero motion vector candidate. For example, in the affine merge candidate list, the inherited affine merge candidate may be inserted first, and then the constructed affine merge candidate may be inserted. In addition, when the number of candidates in the affine merge candidate list is less than the maximum number of candidates even after the affine merge candidate constructed in the affine merge candidate list is inserted, the remainder may be filled with a zero motion vector candidate. Here, the zero motion vector candidate may be referred to as a zero vector. For example, the affine merge candidate list may be a list based on an affine merge mode that derives motion vectors in units of samples, or may be a list based on an affine merge mode that derives motion vectors in units of sub-blocks. In this case, the affine merge candidate list may be referred to as a sub-block merge candidate list, and the sub-block merge candidate list may also include a candidate derived from the SbTMVP (or SbTMVP candidate). For example, when an SbTMVP candidate is included in a sub-block merge candidate list, it may be located before an inherited affine merge candidate and a constructed affine merge candidate in the sub-block merge candidate list.

[0258] In operation S1220, the encoding device may generate selection information indicating one of the candidates included in the merge candidate list. For example, the merge candidate list may include at least one of a spatial merge candidate, a temporal merge candidate, a paired candidate, or a zero vector candidate, and one of these candidates may be selected for inter-frame prediction of the current block. Alternatively, for example, the sub-block merge candidate list may include at least one of an inherited affine merge candidate, a constructed affine merge candidate, an SbTMVP candidate, or a zero vector candidate, and one of these candidates may be selected for inter-frame prediction of the current block.

[0259] For example, the selection information may include index information indicating a selected candidate in the merge candidate list. For example, the selection information may be referred to as merge index information or sub-block merge index information.

[0260] In addition, the encoding device may generate inter-frame prediction type information indicating the inter-frame prediction type of the current block as bidirectional prediction. For example, the inter-frame prediction type of the current block may be determined as bidirectional prediction among L0 prediction, L1 prediction, or bidirectional prediction, and inter-frame prediction type information indicating this determination may be generated. In this case, L0 prediction may indicate prediction based on reference picture list 0, L1 prediction may indicate prediction based on reference picture list 1, and bidirectional prediction may indicate prediction based on reference picture list 0 and reference picture list 1. For example, the encoding device may generate inter-frame prediction type information based on the inter-frame prediction type. For example, the inter-frame prediction type information may include an inter_pred_idc syntax element.

[0261] In operation S1230, the encoding device may encode image information including inter-frame prediction mode information and selection information. For example, the image information may be referred to as video information. The image information may include various information according to the above-mentioned embodiments of the present disclosure. For example, the image information may include at least a portion of prediction-related information or residual-related information. For example, the prediction-related information may include at least a portion of inter-frame prediction mode information, selection information, and inter-frame prediction type information. For example, the encoding device may generate a bitstream or encoding information by encoding all or part of the image information including the above-mentioned information (or syntax elements). Alternatively, the encoding device may output it in the form of a bitstream. In addition, the bitstream or encoding information may be sent to a decoding device via a network or a storage medium.

[0262] Despite Figure 12 Although not illustrated in the figure, the encoding device may, for example, generate prediction samples of the current block. Alternatively, for example, the encoding device may generate prediction samples of the current block based on the selected candidate. Alternatively, for example, the encoding device may derive motion information based on the selected candidate and generate prediction samples of the current block based on the motion information. For example, the encoding device may generate L0 prediction samples and L1 prediction samples based on bidirectional prediction, and may generate prediction samples of the current block based on the L0 prediction samples and the L1 prediction samples. In this case, the prediction samples of the current block may be generated from the L0 prediction samples and the L1 prediction samples using weight index information (or weight information) for bidirectional prediction. In this case, the weight information may be expressed based on the weight index information.

[0263] In other words, for example, the encoding device may generate L0 prediction samples and L1 prediction samples of the current block based on the selected candidate. For example, when the inter prediction type of the current block is determined to be bidirectional prediction, the current block may be predicted using reference picture list 0 and reference picture list 1. For example, the L0 prediction samples may represent prediction samples of the current block derived based on reference picture list 0, and the L1 prediction samples may represent prediction samples of the current block derived based on reference picture list 1.

[0264] For example, the candidate may include a spatial merging candidate. For example, when the selected candidate is a spatial merging candidate, L0 motion information and L1 motion information may be derived based on the spatial merging candidate, and L0 prediction samples and L1 prediction samples may be generated based on the derived L0 motion information and L1 motion information.

[0265] For example, the candidate may include a temporal merging candidate. For example, when the selected candidate is a temporal merging candidate, L0 motion information and L1 motion information may be derived based on the temporal merging candidate, and L0 prediction samples and L1 prediction samples may be generated based on the derived L0 motion information and L1 motion information.

[0266] For example, the candidate may include a paired candidate. For example, when the selected candidate is a paired candidate, L0 motion information and L1 motion information may be derived based on the paired candidate, and L0 prediction samples and L1 prediction samples may be generated based on the derived L0 motion information and L1 motion information. For example, the paired candidate may be derived based on two other merge candidates from among the candidates included in the merge candidate list.

[0267] Alternatively, for example, the merge candidate list may be a subblock merge candidate list, and an affine merge candidate, a subblock merge candidate, or an SbTMVP candidate may be selected. Here, an affine merge candidate in subblock units may be referred to as a subblock merge candidate.

[0268] For example, the candidate may include a subblock merge candidate. For example, when the selected candidate is a subblock merge candidate, L0 motion information and L1 motion information may be derived based on the subblock merge candidate, and L0 prediction samples and L1 prediction samples may be generated based on the derived L0 motion information and L1 motion information. For example, the subblock merge candidate may include a control point motion vector (CPMV), and the L0 prediction samples and L1 prediction samples may be generated by performing prediction in subblock units based on the CPMV.

[0269] Here, the CPMV may be indicated based on a block among neighboring blocks of a control point (CP) of the current block. For example, the number of CPs may be 2, 3, or 4, and the CP may be located at least partially on the upper left side (or upper left corner), upper right side (or upper right corner), lower left side (or lower left corner), or lower right side (or lower right corner) of the current block, and only one CP may exist at each position.

[0270] For example, the CP may be CP0 located on the upper left side of the current block. In this case, the neighboring blocks may include the upper left neighboring block of the current block, the left neighboring block adjacent to the lower side of the upper left neighboring block, and the upper neighboring block adjacent to the right side of the upper left neighboring block. Alternatively, the neighboring blocks may include Figure 10 Block A2, block B2 or block B3 in.

[0271] Alternatively, for example, the CP may be CP1 located on the right side of the current block. In this case, the neighboring blocks may include the upper right neighboring block of the current block and the upper neighboring block adjacent to the left side of the upper right neighboring block of the current block. Alternatively, the neighboring blocks may include Figure 10 Block B0 or ​​block B1 in.

[0272] Alternatively, for example, the CP may be CP2 located at the lower left side of the current block. In this case, the neighboring blocks may include the lower left corner neighboring block of the current block and the left neighboring block adjacent to the upper side of the lower left corner neighboring block. Alternatively, the neighboring blocks may include Figure 10 Block A0 or block A1 in.

[0273] Alternatively, for example, the CP may be CP3 located at the lower right side of the current block. In this case, CP3 may also be referred to as RB. In this case, the adjacent block may include a collocated block of the current block or a lower right corner adjacent block of the collocated block. In this case, the collocated block may include a block located at the same position as the current block in a reference picture different from the current picture in which the current block is located. Alternatively, the adjacent block may include Figure 10 Block T in .

[0274] Alternatively, for example, the candidate may include an SbTMVP candidate. For example, when the selected candidate is an SbTMVP candidate, L0 motion information and L1 motion information may be derived based on the left neighboring block of the current block, and based on this, L0 prediction samples and L1 prediction samples may be generated. For example, the L0 prediction samples and L1 prediction samples may be generated by performing prediction in units of subblocks.

[0275] For example, the L0 motion information may include an L0 reference picture index, an L0 motion vector, etc., and the L1 motion information may include an L1 reference picture index, an L1 motion vector, etc. The L0 reference picture index may include information indicating a reference picture in reference picture list 0, and the L1 reference picture index may include information indicating a reference picture in reference picture list 1.

[0276] For example, the encoding device may generate a prediction sample of the current block based on the L0 prediction sample, the L1 prediction sample and the weight information. For example, the weight information may be displayed based on the weight index information. The weight index information may indicate weight index information about bidirectional prediction. For example, the weight information may include information about the weighted average of the L0 prediction sample or the L1 prediction sample. That is, the weight index information may indicate index information about the weight used for weighted averaging, and the weight index information may be generated in the process of generating the prediction sample based on the weighted average. For example, the weight index information may include information indicating any one of three or five weights. For example, the weighted average may represent a weighted average in bidirectional prediction (BCW) with CU-level weights or bidirectional prediction (BWA) with weighted average.

[0277] For example, the candidate may include a temporal merge candidate, and the weight index information about the temporal merge candidate may be represented by 0. That is, the weight index information about the temporal merge candidate may be represented by 0. Here, the weight index information of 0 may mean that the weight of each reference direction (that is, the L0 prediction direction and the L1 prediction direction in the bidirectional prediction) is the same. Alternatively, for example, the candidate may include a temporal merge candidate, and the weight index information may be indicated based on the weight index information about the collocated block. That is, the weight index information about the temporal merge candidate may be indicated based on the weight index information about the collocated block. Here, the collocated block may include a block at the same position as the current block in a reference picture different from the current picture in which the current block is located.

[0278] For example, the candidate may include a paired candidate, and the weight index information may be indicated based on the weight index information about one of the other two candidates in the merge candidate list used to derive the paired candidate. That is, the weight index information about the paired candidate may be indicated based on the weight index information about one of the other two candidates in the merge candidate list used to derive the paired candidate.

[0279] For example, the candidate may include a paired candidate, and the paired candidate may be indicated based on the other two candidates among the candidates. When the weight index information about each of the other two candidates is the same as the first weight index information, the weight index information about the paired candidate may be indicated based on the first weight index information. When the weight index information about each of the other two candidates is different, the weight index information about the paired candidate may be indicated based on the default weight index information, and in this case, the default weight index information may correspond to the weight index information for assigning the same weight to each of the L0 prediction sample and the L1 prediction sample.

[0280] For example, the candidate may include a paired candidate, and the paired candidate may be indicated based on the other two candidates among the candidates. When the weight index information about each of the other two candidates is the same as the first weight index information, the weight index information about the paired candidate may be indicated based on the first weight index information. When the weight index information about each of the other two candidates is different, the weight index information may be indicated based on the weight index information of each of the other two candidates that is not the default weight index information. The default weight index information may correspond to weight index information for assigning the same weight to each of the L0 prediction sample and the L1 prediction sample.

[0281] For example, the merge candidate list may be a subblock merge candidate list, and an affine merge candidate, a subblock merge candidate, or an SbTMVP candidate may be selected. Here, an affine merge candidate in a subblock unit may be referred to as a subblock merge candidate.

[0282] For example, the candidates include affine merging candidates, and the affine merging candidates may include control point motion vectors (CPMVs).

[0283] For example, when the affine merge candidate includes CPMV0 of control point 0 (CP0) located on the upper left side of the current block, weight index information about the affine merge candidate may be expressed based on the 0th weight index information about CP0. When the affine merge candidate does not include CPMV0 of CP0 located on the upper left side of the current block, weight index information about the affine merge candidate may be expressed based on the first weight index information of control point 1 (CP1) located on the upper right side of the current block.

[0284] The 0th weight index information may correspond to the weight index information of the block used to derive CPMV0 among the neighboring blocks of CP0, and the neighboring blocks of CP0 may include the upper left neighboring block of the current block, the left neighboring block adjacent to the lower side of the upper left neighboring block, and the upper neighboring block adjacent to the right side of the upper left neighboring block.

[0285] The first weight index information may correspond to weight index information of a block for deriving CPMV1 among neighboring blocks of CP1, and the neighboring blocks of CP1 may include an upper right neighboring block of the current block and an upper neighboring block adjacent to the left side of the upper right neighboring block.

[0286] Alternatively, for example, the candidate may include an SbTMVP candidate, and the weight index information about the SbTMVP candidate may be indicated based on the weight index information about the left neighboring block of the current block. That is, the weight index information about the SbTMVP candidate may be indicated based on the weight index information about the left neighboring block.

[0287] Alternatively, for example, the candidate may include an SbTMVP candidate, and the weight index information about the SbTMVP candidate may be represented by 0. That is, the weight index information about the SbTMVP candidate may be represented by 0. Here, the weight index information of 0 may mean that the weight of each reference direction (ie, the L0 prediction direction and the L1 prediction direction in the bidirectional prediction) is the same.

[0288] Alternatively, for example, the candidate may include an SbTMVP candidate, and the weight index information may be indicated based on the weight index information about the center block in the collocated block. That is, the weight index information about the SbTMVP candidate may be indicated based on the weight index information about the center block in the collocated block. Here, the collocated block may include a block in a reference picture different from the current picture in which the current block is located that is co-located with the current block, and the center block may include the lower right sub-block among the four sub-blocks located at the center of the collocated block.

[0289] Alternatively, for example, the candidate may include an SbTMVP candidate, and the weight index information may be indicated based on the weight index information about each sub-block in the collocated block. That is, the weight index information about the SbTMVP candidate may be indicated based on the weight index information about each sub-block in the collocated block.

[0290] Alternatively, although Figure 12 Although not illustrated in the example, the encoding device may, for example, derive residual samples based on the prediction samples and the original samples. In this case, residual-related information may be derived based on the residual samples. Residual samples may be derived based on the residual-related information. Reconstructed samples may be generated based on the residual samples and the prediction samples. A reconstructed block and a reconstructed picture may be derived based on the reconstructed samples. Alternatively, for example, the encoding device may encode image information including residual-related information or prediction-related information.

[0291] For example, the encoding device may generate a bitstream or encoding information by encoding all or part of the image information including the above information (or syntax elements). Alternatively, the encoding device may output it in the form of a bitstream. In addition, the bitstream or encoding information may be sent to a decoding device via a network or storage medium. Alternatively, the bitstream or encoding information may be stored in a computer-readable storage medium and the bitstream or encoding information may be generated using the above image encoding method.

[0292] Figure 14 and Figure 15 is a diagram schematically illustrating an example of a video / image decoding method and related components according to an embodiment of the present disclosure.

[0293] Figure 14 The method disclosed in Figure 3 or Figure 15 Specifically, for example, Figure 14 Operation S1400 may be performed by Figure 15 The entropy decoder 310 of the decoding device 300 in the embodiment of the present invention performs the following steps: Figure 14 Operations S1410 to S1440 may be performed by Figure 15 The predictor 330 of the decoding device 300 in FIG. Figure 14 Not shown in the example, but can be Figure 15 The entropy decoder 310 of the decoding device 300 derives prediction-related information or residual information from the bitstream, the residual processor 320 of the decoding device 300 can derive residual samples from the residual information, the predictor 330 of the decoding device 300 can derive prediction samples from the prediction-related information, and the adder 340 of the decoding device 300 can derive a reconstructed block or a reconstructed picture from the residual samples or the prediction samples. Figure 14 The method disclosed in may include the embodiments described above in the present disclosure.

[0294] Reference Figure 14 In operation S1400, a decoding device may receive image information including inter-frame prediction mode information through a bitstream. For example, the image information may be referred to as video information. The image information may include various information according to the above-described embodiments of the present disclosure. For example, the image information may include at least a portion of prediction-related information or residual-related information.

[0295] For example, the prediction-related information may include inter-frame prediction mode information or inter-frame prediction type information. For example, the inter-frame prediction mode information may include information indicating at least one of various inter-frame prediction modes. For example, various modes such as merge mode, skip mode, motion vector prediction (MVP) mode, affine mode, sub-block merge mode, or merge with MVD (MMVD) mode may be used. In addition, decoder-side motion vector refinement (DMVR) mode, adaptive motion vector resolution (AMVR) mode, BCW (bidirectional prediction with CU-level weights) or BDOF (bidirectional optical flow), etc. may be further used as auxiliary modes, or may be used instead. For example, the inter-frame prediction type information may include an inter_pred_idc syntax element. Alternatively, the inter-frame prediction type information may include information indicating any one of L0 prediction, L1 prediction, and bidirectional prediction.

[0296] In operation S1410, the encoding device may generate a merge candidate list for the current block based on the inter-frame prediction mode information. For example, the decoding device may determine the inter-frame prediction mode of the current block as a merge mode, an affine (merge) mode, or a sub-block merge mode based on the inter-frame prediction mode information, and may generate a merge candidate list based on the determined inter-frame prediction mode. In this case, when the inter-frame prediction mode is determined to be an affine merge mode or a sub-block merge mode, the merge candidate list may be referred to as an affine merge candidate list or a sub-block merge candidate list, but may also be simply referred to as a merge candidate list.

[0297] For example, candidates may be inserted into the merge candidate list until the number of candidates in the merge candidate list becomes the maximum number of candidates. Here, the candidate may indicate a candidate or candidate block for deriving motion information (or motion vector) of the current block. For example, the candidate block may be derived by searching the neighboring blocks of the current block. For example, the neighboring blocks may include spatial neighboring blocks and / or temporal neighboring blocks of the current block, the spatial neighboring blocks may be preferentially searched to derive (spatial merging) candidates, and then the temporal neighboring blocks may be searched to derive (temporal merging) candidates, and the derived candidates may be inserted into the merge candidate list. For example, when the number of candidates in the merge candidate list is less than the maximum number of candidates in the merge candidate list even after the candidate is inserted, additional candidates may be inserted. For example, the additional candidates include at least one of a history-based merge candidate, a pairwise average merge candidate, an ATMVP, and a combined bidirectional prediction merge candidate (when the slice / tile group type of the current slice / tile group is type B), and / or a zero vector merge candidate.

[0298] Alternatively, for example, the candidate may be inserted into the affine merge candidate list until the number of candidates in the affine merge candidate list becomes the maximum number of candidates. Here, the candidate may include the control point motion vector (CPMV) of the current block. Alternatively, the candidate may indicate a candidate or candidate block for deriving the CPMV. The CPMV may indicate a motion vector at a control point (CP) of the current block. For example, the number of CPs may be 2, 3, or 4, and the CP may be at least a portion of the upper left side (or upper left corner), upper right side (or upper right corner), lower left side (or lower left corner), or lower right side (or lower right corner) of the current block, and there may be only one CP at each position.

[0299] For example, a candidate block can be derived by searching for neighboring blocks of the current block (or neighboring blocks of the CP of the current block). For example, the affine merge candidate list may include at least one of an inherited affine merge candidate, a constructed affine merge candidate, and a zero motion vector candidate. For example, in the affine merge candidate list, the inherited affine merge candidate may be inserted first, and then the constructed affine merge candidate may be inserted. In addition, when the number of candidates in the affine merge candidate list is less than the maximum number of candidates even after the affine merge candidate constructed in the affine merge candidate list is inserted, the remainder may be filled with a zero motion vector candidate. Here, the zero motion vector candidate may be referred to as a zero vector. For example, the affine merge candidate list may be a list based on an affine merge mode that derives motion vectors in units of samples, or may be a list based on an affine merge mode that derives motion vectors in units of sub-blocks. In this case, the affine merge candidate list may be referred to as a sub-block merge candidate list, and the sub-block merge candidate list may also include a candidate derived from the SbTMVP (or SbTMVP candidate). For example, when an SbTMVP candidate is included in a sub-block merge candidate list, it may be located before an inherited affine merge candidate and a constructed affine merge candidate in the sub-block merge candidate list.

[0300] In operation S1420, the decoding device may derive motion information of the current block based on the candidate selected from the merge candidate list. For example, the merge candidate list may include at least one of a spatial merge candidate, a temporal merge candidate, a paired candidate, or a zero vector candidate, and one of these candidates may be selected for inter-frame prediction of the current block. Alternatively, for example, the sub-block merge candidate list may include at least one of an inherited affine merge candidate, a constructed affine merge candidate, an SbTMVP candidate, or a zero vector candidate, and one of these candidates may be selected for inter-frame prediction of the current block. For example, the selected candidate may be selected from the merge candidate list based on selection information. For example, the selection information may include index information indicating the selected candidate in the merge candidate list. For example, the selection information may be referred to as merge index information or sub-block merge index information. For example, the selection information may be included in the image information. Alternatively, the selection information may be included in the inter-frame prediction mode information.

[0301] In operation S1430, the decoding device may generate an L0 prediction sample and an L1 prediction sample of the current block based on the motion information. For example, when the inter-frame prediction type is derived as a bidirectional prediction, the decoding device may derive the L0 motion information and the L1 motion information based on the selected candidate. The decoding device may derive the inter-frame prediction type of the current block as a bidirectional prediction based on the inter-frame prediction type information. For example, the inter-frame prediction type of the current block may be derived as a bidirectional prediction among L0 prediction, L1 prediction, or bidirectional prediction based on the inter-frame prediction type information. In this case, the L0 prediction may indicate a prediction based on reference picture list 0, the L1 prediction may indicate a prediction based on reference picture list 1, and the bidirectional prediction may indicate a prediction based on reference picture list 0 and reference picture list 1. For example, the inter-frame prediction type information may include an inter_pred_idc syntax element.

[0302] For example, the L0 motion information may include an L0 reference picture index and an L0 motion vector, and the L1 motion information may include an L1 reference picture index and an L1 motion vector. The L0 reference picture index may include information indicating a reference picture in reference picture list 0, and the L1 reference picture index may include information indicating a reference picture in reference picture list 1.

[0303] For example, the candidate may include a spatial merging candidate. For example, when the selected candidate is a spatial merging candidate, L0 motion information and L1 motion information may be derived based on the spatial merging candidate, and L0 prediction samples and L1 prediction samples may be generated based on the derived L0 motion information and L1 motion information.

[0304] For example, the candidate may include a temporal merging candidate. For example, when the selected candidate is a temporal merging candidate, L0 motion information and L1 motion information may be derived based on the temporal merging candidate, and L0 prediction samples and L1 prediction samples may be generated based on the derived L0 motion information and L1 motion information.

[0305] For example, the candidate may include a paired candidate. For example, when the selected candidate is a paired candidate, L0 motion information and L1 motion information may be derived based on the paired candidate, and L0 prediction samples and L1 prediction samples may be generated based on the derived L0 motion information and L1 motion information. For example, the paired candidate may be derived based on two other candidates among the candidates included in the merge candidate list.

[0306] Alternatively, for example, the merge candidate list may be a subblock merge candidate list, and an affine merge candidate, a subblock merge candidate, or an SbTMVP candidate may be selected. Here, an affine merge candidate in subblock units may be referred to as a subblock merge candidate.

[0307] For example, the candidate may include an affine merge candidate. For example, when the selected candidate is an affine merge candidate, L0 motion information and L1 motion information may be derived based on the affine merge candidate, and L0 prediction samples and L1 prediction samples may be generated based on the affine merge candidate. For example, the affine merge candidate may include a control point motion vector (CPMV), and the L0 prediction sample and L1 prediction sample may be generated by performing prediction in units of subblocks based on the CPMV.

[0308] Here, the CPMV can be derived based on one of the neighboring blocks of the control point (CP) of the current block. For example, the number of CPs can be 2, 3, or 4, and the CP can be located at least partially on the upper left side (or upper left corner), upper right side (or upper right corner), lower left side (or lower left corner), or lower right side (or lower right corner) of the current block, and only one CP can exist at each position.

[0309] For example, the CP may be CP0 located on the upper left side of the current block. In this case, the neighboring blocks may include the upper left neighboring block of the current block, the left neighboring block adjacent to the lower side of the upper left neighboring block, and the upper neighboring block adjacent to the right side of the upper left neighboring block. Alternatively, the neighboring blocks may include Figure 10 Block A2, block B2 or block B3 in.

[0310] Alternatively, for example, the CP may be CP1 located on the right side of the current block. In this case, the neighboring blocks may include the upper right neighboring block of the current block and the upper neighboring block adjacent to the left side of the upper right neighboring block of the current block. Alternatively, the neighboring blocks may include Figure 10 Block B0 or ​​block B1 in.

[0311] Alternatively, for example, the CP may be CP2 located at the lower left side of the current block. In this case, the neighboring blocks may include the lower left corner neighboring block of the current block and the left neighboring block adjacent to the upper side of the lower left corner neighboring block. Alternatively, the neighboring blocks may include Figure 10 Block A0 or block A1 in.

[0312] Alternatively, for example, the CP may be CP3 located at the lower right side of the current block. In this case, CP3 may also be referred to as RB. In this case, the adjacent block may include a collocated block of the current block or a lower right corner adjacent block of the collocated block. In this case, the collocated block may include a block located at the same position as the current block in a reference picture different from the current picture in which the current block is located. Alternatively, the adjacent block may include Figure 10 Block T in .

[0313] Alternatively, for example, the candidate may include an SbTMVP candidate. For example, when the selected candidate is an SbTMVP candidate, L0 motion information and L1 motion information may be derived based on the left neighboring block of the current block, and L0 prediction samples and L1 prediction samples may be generated based on the L0 motion information and L1 motion information. For example, the L0 prediction samples and L1 prediction samples may be generated by performing prediction in sub-block units.

[0314] In operation S1440, the decoding device may generate a prediction sample of the current block based on the L0 prediction sample, the L1 prediction sample and the weight information. For example, the weight information may be derived based on the weight index information of the candidate selected from the candidates included in the merge candidate list. For example, the weight information may include information about the weighted average of the L0 prediction sample or the L1 prediction sample. That is, the weight index information may indicate index information about the weight used for weighted averaging, and the weighted average may be obtained based on the weighted index information. For example, the weight index information may include information indicating any one of three or five weights. For example, the weighted average may indicate a weighted average in bidirectional prediction (BCW) with CU-level weight or bidirectional prediction (BWA) with weighted average.

[0315] For example, the candidate may include a temporal merging candidate, and the weight index information about the temporal merging candidate may be derived as 0. That is, the weight index information about the temporal merging candidate may be derived as 0. Here, the weight index information of 0 may mean that the weight of each reference direction (i.e., the L0 prediction direction and the L1 prediction direction in the bidirectional prediction) is the same.

[0316] For example, the candidate may include a temporal merge candidate, and the weight index information about the temporal merge candidate may be derived based on the weight index information about the collocated block. That is, the weight index information about the temporal merge candidate may be derived based on the weight index information about the collocated block. Here, the collocated block may include a block in a reference picture different from the current picture in which the current block is located and located at the same position as the current block.

[0317] For example, the candidate may include a paired candidate, and the weight index information may be derived as weight index information about one of the other two candidates in the merge candidate list used to derive the paired candidate. That is, the weight index information about the paired candidate may be derived as weight index information about one of the other two candidates in the merge candidate list used to derive the paired candidate.

[0318] For example, the candidate may include a paired candidate, and the paired candidate may be derived based on the other two candidates among the candidates. When the weight index information about each of the other two candidates is the same as the first weight index information, the weight index information for the paired candidate may be derived based on the first weight index information. When the weight index information about each of the other two candidates is different, the weight index information about the paired candidate may be derived based on the default weight index information, and in this case, the default weight index information may correspond to the weight index information for assigning the same weight to each of the L0 prediction sample and the L1 prediction sample.

[0319] For example, the candidate may include a paired candidate, and the paired candidate may be derived based on the other two candidates among the candidates. When the weight index information about each of the other two candidates is the same as the first weight index information, the weight index information for the paired candidate may be derived based on the first weight index information. When the weight index information about each of the other two candidates is different, the weight index information may be derived based on the weight index information of each of the other two candidates that is not the default weight index information. The default weight index information may correspond to weight index information for assigning the same weight to each of the L0 prediction sample and the L1 prediction sample.

[0320] For example, the merge candidate list may be a subblock merge candidate list, and an affine merge candidate, a subblock merge candidate, or an SbTMVP candidate may be selected. Here, an affine merge candidate in a subblock unit may be referred to as a subblock merge candidate.

[0321] For example, the candidates include affine merging candidates, and the affine merging candidates may include control point motion vectors (CPMVs).

[0322] For example, when the affine merge candidate includes CPMV0 of control point 0 (CP0) located on the upper left side of the current block, the weight index information about the affine merge candidate can be derived based on the 0th weight index information about CP0. When the affine merge candidate does not include CPMV0 of CP0 located on the upper left side of the current block, the weight index information about the affine merge candidate can be derived based on the first weight index information about the control point (CP1) located on the upper right side of the current block.

[0323] The 0th weight index information may correspond to the weight index information of the block used to derive CPMV0 among the neighboring blocks of CP0, and the neighboring blocks of CP0 may include the upper left neighboring block of the current block, the left neighboring block adjacent to the lower side of the upper left neighboring block, and the upper neighboring block adjacent to the right side of the upper left neighboring block.

[0324] The first weight index information may correspond to weight index information of a block for deriving CPMV1 among neighboring blocks of CP1, and the neighboring blocks of CP1 may include an upper right neighboring block of the current block and an upper neighboring block adjacent to the left side of the upper right neighboring block.

[0325] Alternatively, for example, the candidate may include an SbTMVP candidate, and the weight index information about the SbTMVP candidate may be derived based on the weight index information about the left neighboring block of the current block. That is, the weight index information about the SbTMVP candidate may be derived based on the weight index information about the left neighboring block.

[0326] Alternatively, for example, the candidate may include an SbTMVP candidate, and the weight index information about the SbTMVP candidate may be derived as 0. That is, the weight index information about the SbTMVP candidate may be derived as 0. Here, the weight index information equal to 0 may indicate that the weights of the respective reference directions (i.e., the L0 prediction direction and the L1 prediction direction in the bidirectional prediction) are the same.

[0327] Alternatively, for example, the candidate may include an SbTMVP candidate, and the weight index information may be derived based on the weight index information about the center block in the collocated block. That is, the weight index information about the SbTMVP candidate may be derived based on the weight index information about the center block in the collocated block. Here, the collocated block may include a block in the same position as the current block in a reference picture different from the current picture in which the current block is located, and the center block may include the lower right subblock of the four subblocks in the center area of ​​the collocated block.

[0328] Alternatively, for example, the candidate may include an SbTMVP candidate, and the weight index information may be derived based on the weight index information about each subblock in the collocated block. That is, the weight index information about the SbTMVP candidate may be derived based on the weight index information about each subblock in the collocated block.

[0329] Despite Figure 14 Although not illustrated in the example, the decoding device may derive residual samples based on the residual-related information included in the image information. Furthermore, the decoding device may generate reconstructed samples based on the predicted samples and the residual samples. A reconstructed block and a reconstructed picture may be derived based on the reconstructed samples.

[0330] For example, a decoding device can obtain video / image information including all or part of the above-mentioned multiple pieces of information (or syntax elements) by decoding a bitstream or coding information. In addition, the bitstream or coding information can be stored in a computer-readable storage medium and can cause the above-mentioned decoding method to be executed.

[0331] Although the method has been described based on a flowchart that lists steps or blocks in sequence in the above embodiments, the steps of this document are not limited to a specific order, and specific steps may be performed in different steps or in a different order or simultaneously with respect to the above steps. In addition, it will be understood by those skilled in the art that the steps in the flowchart are not exclusive and that another step may be included therein or one or more steps in the flowchart may be deleted without affecting the scope of the present disclosure.

[0332] The above-mentioned method according to the present disclosure may be in the form of software, and the encoding device and / or decoding device according to the present disclosure may be included in an apparatus for performing image processing (e.g., TV, computer, smart phone, set-top box, display device, etc.).

[0333] When the embodiments of the present disclosure are implemented with software, the above-mentioned methods can be implemented with modules (processing or functions) that perform the above-mentioned functions. The modules can be stored in a memory and executed by a processor. The memory can be installed inside or outside the processor and can be connected to the processor via various well-known devices. The processor may include an application-specific integrated circuit (ASIC), other chipsets, logic circuits and / or data processing devices. The memory may include a read-only memory (ROM), a random access memory (RAM), a flash memory, a memory card, a storage medium and / or other storage devices. In other words, according to the embodiments of the present disclosure, it can be implemented and executed on a processor, a microprocessor, a controller or a chip. For example, the functional units illustrated in the corresponding figures can be implemented and executed on a computer, a processor, a microprocessor, a controller or a chip. In this case, information about the implementation (for example, information about instructions) or the algorithm can be stored in a digital storage medium.

[0334] In addition, the decoding device and encoding device of the embodiment of the present document can be included in a multimedia broadcast transceiver, a mobile communication terminal, a home theater video device, a digital theater video device, a surveillance camera, a video chat device, a real-time communication device such as video communication, a mobile streaming device, a storage medium, a portable camera, a video on demand (VoD) service provider, an over-the-top (OTT) video device, an Internet streaming service provider, a 3D video device, a virtual reality (VR) device, an augmented reality (AR) device, an image phone video device, a vehicle terminal (e.g., a vehicle (including an autonomous vehicle) terminal, an aircraft terminal, or a ship terminal) and a medical video device; and can be used to process image signals or data. For example, the OTT video device may include a game console, a Blueray player, a networked TV, a home theater system, a smartphone, a tablet PC, and a digital video recorder (DVR).

[0335] In addition, the processing method of the embodiment of the application of this document can be generated in the form of a program executed by a computer and can be stored in a computer-readable recording medium. The multimedia data with a data structure according to the embodiment of this document can also be stored in a computer-readable recording medium. The computer-readable recording medium includes all kinds of storage devices and distributed storage devices stored with computer-readable data. The computer-readable recording medium may include, for example, a Blu-ray disc (BD), a universal serial bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. The computer-readable recording medium also includes a medium implemented in the form of a carrier wave (for example, transmission on the Internet). In addition, the bit stream generated by the encoding method can be stored in a computer-readable recording medium, or can be transmitted through a wired or wireless communication network.

[0336] In addition, the embodiments of this document can be implemented as a computer program product based on a program code, and the program code can be executed on a computer according to the embodiments of this document. The program code can be stored on a computer-readable carrier.

[0337] Figure 16 is a diagram illustrating an example of a content streaming system to which an embodiment of the present disclosure is applied.

[0338] Reference Figure 16 The content streaming transmission system to which the embodiments of the present disclosure are applied may mainly include an encoding server, a streaming transmission server, a network server, a media storage, a user device, and a multimedia input device.

[0339] The encoding server is used to compress content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data, generate a bitstream, and transmit it to the streaming server. As another example, if the multimedia input device such as smartphones, cameras, and camcorders directly generates the bitstream, the encoding server can be omitted.

[0340] The bitstream may be generated by the encoding method or the bitstream generation method to which the embodiments of this document are applied, and the streaming server may temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0341] The streaming server transmits multimedia data to user devices via a network server based on user requests. The network server serves as a tool for notifying users of available services. When a user requests a desired service, the network server transfers the request to the streaming server, which then transmits the multimedia data to the user. In this regard, the content streaming system may include a separate control server, and in this case, the control server is used to control commands and responses between the various devices in the content streaming system.

[0342] The streaming server may receive content from a media storage device and / or an encoding server. For example, when receiving content from an encoding server, the content may be received in real time. In this case, the streaming server may store the bitstream for a predetermined period of time to smoothly provide a streaming service.

[0343] For example, user devices may include mobile phones, smart phones, laptop computers, digital broadcast terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation, tablet PCs, tablet PCs, ultrabooks, wearable devices (e.g., watch-type terminals (smart watches), glasses-type terminals (smart glasses), head-mounted displays (HMDs)), digital TVs, desktop computers, digital signage, etc.

[0344] Each server in the content streaming system may be operated as a distributed server, and in this case, data received by each server may be processed in a distributed manner.

[0345] The claims in this specification may be combined in various ways. For example, the technical features in the method claims of this specification may be combined to be implemented or performed in a device, and the technical features in the device claims may be combined to be implemented or performed in a method. Furthermore, the technical features in method and device claims may be combined to be implemented or performed in a device. Furthermore, the technical features in method and device claims may be combined to be implemented or performed in a method.

Claims

1. An image decoding method performed by a decoding device, the image decoding method comprising the following steps: receiving image information including inter-frame prediction mode information via a bitstream; generating a merge candidate list for the current block based on the inter prediction mode information; deriving motion information about the current block based on a candidate selected from among the candidates in the merge candidate list; generating an L0 prediction sample and an L1 prediction sample of the current block based on the motion information; as well as generating a prediction sample of the current block based on the L0 prediction sample, the L1 prediction sample, and a weight index of the current block, wherein the weight index is derived based on a weight index for the selected candidate, The candidate includes a constructed affine merge candidate, and the constructed affine merge candidate includes a control point motion vector CPMV, Wherein, based on the case where the constructed affine merge candidate includes CPMV0 for control point 0, ie, CP0, the weight index for the constructed affine merge candidate is set equal to the weight index for the CP0, wherein the CP0 is associated with the upper left corner of the current block, and Wherein, based on the case where the constructed affine merge candidate does not include the CPMV0 for the CP0, the weight index for the constructed affine merge candidate is set equal to the weight index for CP1, wherein the CP1 is related to the upper right corner of the current block.

2. An image encoding method performed by an encoding device, the image encoding method comprising the following steps: determining an inter prediction mode of a current block and generating inter prediction mode information indicating the inter prediction mode; generating a merge candidate list for the current block based on the inter prediction mode; generating selection information indicating one candidate among the candidates included in the merge candidate list; as well as encoding the image information including the inter-frame prediction mode information and the selection information, The candidate includes a constructed affine merge candidate, and the constructed affine merge candidate includes a control point motion vector CPMV, Wherein, based on the case where the constructed affine merge candidate includes CPMV0 for control point 0, ie, CP0, the weight index for the constructed affine merge candidate is set equal to the weight index for the CP0, wherein the CP0 is associated with the upper left corner of the current block, and In which, based on the case where the constructed affine merge candidate does not include the CPMV0 for the CP0, the weight index for the constructed affine merge candidate is set equal to the weight index for control point 1, i.e., CP1, wherein the CP1 is related to the upper right corner of the current block.

3. A method for transmitting image data, the method comprising the following steps: obtaining a bitstream for the image, wherein the bitstream is generated based on the following steps: determining an inter-frame prediction mode for a current block and generating inter-frame prediction mode information indicating the inter-frame prediction mode; generating a merge candidate list for the current block based on the inter-frame prediction mode; generating selection information indicating one candidate among the candidates included in the merge candidate list; and encoding image information including the inter-frame prediction mode information and the selection information, The candidate includes a constructed affine merge candidate, and the constructed affine merge candidate includes a control point motion vector CPMV, Wherein, based on the case where the constructed affine merge candidate includes CPMV0 for control point 0, ie, CP0, the weight index for the constructed affine merge candidate is set equal to the weight index for the CP0, wherein the CP0 is associated with the upper left corner of the current block, and In which, based on the case where the constructed affine merge candidate does not include the CPMV0 for the CP0, the weight index for the constructed affine merge candidate is set equal to the weight index for control point 1, i.e., CP1, wherein the CP1 is related to the upper right corner of the current block.

Images