Image decoding method, image encoding method, and image data transmission method

Abstract

The invention provides an image decoding method, an image encoding method, and an image data transmission method. According to an embodiment of the present document, when an inter prediction type of a current block indicates bidirectional prediction, weight index information of candidates in a merge candidate list or a sub-block merge candidate list may be derived, and thus coding efficiency may be increased.

Description

[0001] This application is a divisional application of the original invention patent application with application number 202080059109.6 (International application number: PCT / KR2020 / 007943, application date: June 19, 2020, invention name: Image decoding method and device for generating prediction samples by applying a determined prediction mode). Technical Field

[0002] The present disclosure relates to an image decoding method and an apparatus thereof for generating a prediction sample by applying a determined prediction mode. Background Art

[0003] Recently, there has been an increasing demand for high-resolution, high-quality images / videos, such as 4K or 8K ultra-high-definition (UHD) images / videos, in various fields. As image / video resolution or quality becomes higher, relatively more information or bits are transmitted compared to conventional image / video data. Therefore, if image / video data is transmitted via a medium such as an existing wired / wireless broadband line or stored in a conventional storage medium, the cost of transmission and storage is likely to increase.

[0004] In addition, there is growing interest and demand for virtual reality (VR) and artificial reality (AR) content and immersive media such as holograms; and there is also growing broadcasting of images / videos that exhibit image / video characteristics that are different from actual images / videos (e.g., game images / videos).

[0005] Therefore, highly efficient image / video compression technology is required to effectively compress and transmit, store, or play high-resolution, high-quality images / videos exhibiting various characteristics as described above. Summary of the Invention

[0006] Technical issues

[0007] The present disclosure provides a method and apparatus for improving image coding efficiency.

[0008] The present disclosure also provides a method and apparatus for deriving prediction samples based on a default merge mode when no merge mode is ultimately selected.

[0009] The present disclosure also provides a method and apparatus for deriving a prediction sample by applying a normal merge mode as a default merge mode.

[0010] Technical Solution

[0011] In one aspect, an image decoding method performed by a decoding device is provided. The method includes: acquiring image information including inter-frame prediction mode information and residual information through a bitstream; generating residual samples based on the residual information; generating prediction samples of a current block by applying a prediction mode determined based on the inter-frame prediction mode information; and generating reconstructed samples based on the prediction samples and the residual samples, wherein the inter-frame prediction mode information includes a general merge flag indicating whether a merge mode is available for the current block, a merge mode (MMVD mode), a merge sub-block mode, a combined inter-picture merge and intra-picture prediction (CIIP mode), and a partitioning mode (partitioning mode) that performs prediction by dividing the current block into two partitions and is available for the current block based on the general merge flag are not available, a normal merge mode is applied, the inter-frame prediction mode information includes merge index information indicating one of the merge candidates included in the merge candidate list generated by applying the normal merge mode, and generating the prediction samples using the merge index information.

[0012] In another aspect, an image encoding method performed by an encoding device is provided. The method includes: determining an inter-frame prediction mode of a current block and generating inter-frame prediction mode information indicating the inter-frame prediction mode; generating prediction samples of the current block based on the determined prediction mode; generating residual information based on residual samples of the current block; and encoding image information including the inter-frame prediction mode information and the residual information, wherein the inter-frame prediction mode information includes a general merge flag indicating whether a merge mode is available for the current block, and applying the general merge mode based on the unavailability of a merge mode with motion vector difference (MMVD) mode, a merge sub-block mode, a combined inter-picture merge and intra-picture prediction (CIIP) mode, and a partition mode for performing prediction by partitioning the current block into two partitions.

[0013] The inter prediction mode information includes merge index information indicating one of merge candidates included in a merge candidate list generated by applying the normal merge mode.

[0014] In another aspect, a computer-readable storage medium storing encoding information for enabling an image decoding device to perform an image decoding method is provided, wherein the image decoding method includes: obtaining image information including inter-frame prediction mode information and residual information through a bit stream; generating residual samples based on the residual information; generating prediction samples of a current block by applying a prediction mode determined based on the inter-frame prediction mode information; and generating reconstructed samples based on the prediction samples and the residual samples, wherein the inter-frame prediction mode information includes a general merge flag indicating whether a merge mode is available for the current block, a merge mode MMVD mode, a merge sub-block mode, a combined inter-picture merge and intra-picture prediction CIIP mode, and a partition mode for performing prediction by dividing the current block into two partitions are not available, and a conventional merge mode is applied.

[0015] The inter prediction mode information includes merge index information indicating one candidate among merge candidates included in a merge candidate list generated by applying the normal merge mode, and a prediction sample is generated using the merge index information.

[0016] Technical Effects

[0017] According to the present disclosure, the overall image / video compression efficiency can be improved.

[0018] According to the present disclosure, when the merge mode is not finally selected, inter prediction can be efficiently performed by applying the default merge mode.

[0019] According to the present disclosure, when the merge mode is not finally selected, the normal merge mode is applied and motion information is derived based on the candidate indicated by the merge index information, thereby efficiently performing inter prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 An example of a video / image encoding system to which an embodiment of the present disclosure is applied is schematically shown.

[0021] Figure 2 is a diagram schematically illustrating a configuration of a video / image encoding device to which embodiments of this document can be applied.

[0022] Figure 3 is a diagram schematically illustrating a configuration of a video / image decoding device to which embodiments of this document can be applied.

[0023] Figure 4 is a diagram illustrating an example of a video / image encoding method based on inter-frame prediction.

[0024] Figure 5 is a diagram schematically illustrating an inter-frame predictor in an encoding device.

[0025] Figure 6 is a diagram illustrating an example of a video / image decoding method based on inter-frame prediction.

[0026] Figure 7 is a diagram schematically illustrating an inter-frame predictor in a decoding device.

[0027] Figure 8 This is a diagram illustrating the merge mode in inter-frame prediction.

[0028] Figure 9 This is a diagram illustrating the merge mode with motion vector difference (MMVD) mode in inter-frame prediction.

[0029] Figure 10a and Figure 10b CPMV for affine motion prediction is exemplarily shown.

[0030] Figure 11 A case where the affine MVF is determined in units of sub-blocks is exemplarily shown.

[0031] Figure 12 This is a diagram illustrating the affine merge mode or subblock merge mode in inter-frame prediction.

[0032] Figure 13 This is a diagram illustrating positions of candidates in the affine merge mode or the subblock merge mode.

[0033] Figure 14 This is a diagram illustrating SbTMVP in inter-frame prediction.

[0034] Figure 15 This is a diagram illustrating a combined inter-picture merging and intra-picture prediction (CIIP) mode in inter-frame prediction.

[0035] Figure 16 This is a diagram illustrating the partitioning mode in inter-frame prediction.

[0036] Figure 17 and Figure 18 An example of a video / image encoding method and related components according to an embodiment of the present disclosure is schematically shown.

[0037] Figure 19 and Figure 20 An example of an image / video decoding method and related components according to an embodiment of the present disclosure is schematically shown.

[0038] Figure 21 An example of a content streaming system to which the embodiments disclosed in the present disclosure can be applied is shown. DETAILED DESCRIPTION

[0039] The present disclosure can be modified differently in various forms and can have various embodiments, and its specific embodiments will be illustrated in the drawings and described in detail. However, these embodiments are not intended to limit the present disclosure. The terms used in the following description are only used to describe specific embodiments and are not intended to limit the technical spirit of the present disclosure. Singular expressions include plural expressions as long as it is clearly interpreted differently. Terms such as "including" and "having" in this specification are intended to indicate the presence of features, quantities, steps, operations, elements, components, or combinations thereof used in the following description, and therefore it should be understood that the possibility of the presence or addition of one or more different features, quantities, steps, operations, elements, components, or combinations thereof is not excluded.

[0040] In addition, for the purpose of illustrating different specific functions, the elements of the drawings described in this disclosure are drawn independently, and do not mean that the elements are implemented by independent hardware or independent software. For example, two or more elements in the elements can be combined to form a single element, or an element can be separated into multiple elements. Embodiments in which elements are combined and / or separated fall within the scope of this disclosure.

[0041] This document relates to video / image coding. For example, the methods / implementations disclosed in this document may be applied to methods disclosed in the Versatile Video Coding (VVC) standard. Furthermore, the methods / implementations disclosed in this document may be applied to methods disclosed in the Essential Video Coding (EVC) standard, the AO Media Video 1 (AV1) standard, the second-generation Audio Video Coding standard (AVS2), or next-generation video / image coding standards (e.g., H.267 or H.268).

[0042] This document presents various embodiments related to video / image coding, and unless otherwise stated, the embodiments can be combined with each other.

[0043] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Hereinafter, the same reference numerals may be used for the same components in the accompanying drawings, and repeated descriptions of the same components may be omitted.

[0044] Figure 1 An example of a video / image encoding system to which the embodiments of the present disclosure can be applied is illustrated.

[0045] Reference Figure 1 The video / image coding system may include a first device (source device) and a second device (receiver device). The source device may send the coded video / image information or data to the receive device in the form of a file or stream transmission via a digital storage medium or a network.

[0046] The source device may include a video source, an encoding device, and a transmitter. The receiving device may include a receiver, a decoding device, and a renderer. The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display, and the display may be configured as a separate device or an external component.

[0047] The video source can obtain the video / image through a process of capturing, synthesizing, or generating the video / image. The video source may include a video / image capture device and / or a video / image generation device. For example, the video / image capture device may include one or more cameras, a video / image archive including previously captured videos / images, etc. For example, the video / image generation device may include a computer, a tablet computer, and a smartphone, and may (electronically) generate the video / image. For example, a virtual video / image may be generated by a computer, etc. In this case, the video / image capture process may be replaced by a process of generating relevant data.

[0048] An encoding device encodes input video / images. For compression and coding efficiency, the encoding device performs a series of processes such as prediction, transformation, and quantization. The encoded data (encoded video / image information) is output as a bitstream.

[0049] The transmitter can transmit the encoded image / image information or data, output as a bitstream, in the form of a file or stream to a receiver of a receiving device via a digital storage medium or network. Digital storage media can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter can include components for generating a media file in a predetermined file format and can also include components for transmission via a broadcast / communication network. The receiver can receive / extract the bitstream and transmit the received bitstream to a decoding device.

[0050] The decoding device may decode a video / image by performing a series of processes such as dequantization, inverse transformation, and prediction corresponding to the operations of the encoding device.

[0051] The renderer may render the decoded video / image, and the rendered video / image may be displayed on a display.

[0052] The present disclosure relates to video / image coding. For example, the methods / implementations disclosed in the present disclosure can be applied to methods disclosed in the Versatile Video Coding (VVC) standard, the Essential Video Coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the second-generation Audio Video Coding standard (AVS2), or next-generation video / image coding standards (e.g., H.267 or H.268, etc.).

[0053] This document proposes various embodiments of video / image coding, and unless otherwise mentioned, the above embodiments may also be performed in combination with each other.

[0054] In this document, video may refer to a series of images over time. A picture generally refers to a unit that represents an image at a specific time frame, and a slice / tile refers to a unit that constitutes a part of a picture from a coding perspective. A slice / tile may include one or more coding tree units (CTUs). A picture may be composed of one or more slices / tiles.

[0055] A tile is a rectangular area of ​​CTUs within a specific tile column and a specific tile row in a picture. A tile column is a rectangular area of ​​CTUs whose height is equal to the height of the picture and whose width is specified by the syntax elements in the picture parameter set. A tile row is a rectangular area of ​​CTUs whose height is specified by the syntax elements in the picture parameter set and whose width is equal to the width of the picture. Tile scan is a specific sequential ordering of the CTUs that partition a picture, where the CTUs are ordered consecutively in a CTU raster scan within a tile, and the tiles in a picture are ordered consecutively in a raster scan of the tiles of the picture. A slice may include multiple complete tiles of a picture that may be included in one NAL unit or multiple consecutive CTU rows in a tile. In this document, tile group and slice may be used interchangeably. For example, in this document, a tile group / tile group header may be referred to as a slice / slice header.

[0056] In addition, a picture can be divided into two or more sub-pictures. A sub-picture can be a rectangular area of ​​one or more slices within a picture.

[0057] A pixel or a picture element (pel) may refer to the smallest unit constituting a picture (or image). In addition, "sample" may be used as a term corresponding to a pixel. A sample may generally represent a pixel or a pixel value, and may represent only a pixel / pixel value of a luminance component, or only a pixel / pixel value of a chrominance component.

[0058] A unit may represent a basic unit of image processing. A unit may include at least one of a specific area of ​​a picture and information related to the area. A unit may include a luminance block and two chrominance (e.g., cb, cr) blocks. In some cases, units may be used interchangeably with terms such as blocks or areas. In general, an M×N block may include M columns and N rows of samples (or sample arrays) or a set (or array) of transform coefficients. Alternatively, a sample may refer to a pixel value in the spatial domain, and when such a pixel value is transformed into the frequency domain, it may refer to a transform coefficient in the frequency domain.

[0059] In this document, "A or B (A or B)" can mean "only A", "only B", or "both A and B". In other words, "A or B (A or B)" in this document can be interpreted as "A and / or B (A and / or B)". For example, in this document, "A, B, or C (A, B, or C)" means "only A", "only B", "only C", or "any combination of A, B, and C".

[0060] As used in this document, a slash ( / ) or a comma can mean "and / or." For example, "A / B" can mean "A and / or B." Thus, "A / B" can mean "only A," "only B," or "both A and B." For example, "A, B, C" can mean "A, B, or C."

[0061] In this document, “at least one of A and B” may mean “only A,” “only B,” or “both A and B.” Furthermore, in this document, the expression “at least one of A or B” or “at least one of A and / or B” may be equally interpreted as “at least one of A and B.”

[0062] In addition, in this document, "at least one of A, B, and C" means "only A", "only B", "only C", or "any combination of A, B, and C". In addition, "at least one of A, B, or C" or "at least one of A, B, and / or C" may mean "at least one of A, B, and C".

[0063] In addition, brackets used in this document may indicate "for example." Specifically, when "prediction (intra-frame prediction)" is indicated, it may be referred to as "intra-frame prediction" as an example of "prediction." In other words, "prediction" in this document is not limited to "intra-frame prediction," and "intra-frame prediction" may be provided as an example of "prediction." In addition, even when "prediction (i.e., intra-frame prediction)" is indicated, it may be referred to as "intra-frame prediction" as an example of "prediction."

[0064] Technical features described individually in one drawing in this document can be implemented individually or simultaneously.

[0065] Figure 2 This figure schematically illustrates the configuration of a video / image encoding device to which embodiments of this document can be applied. Hereinafter, the so-called encoding device may include an image encoding device and / or a video encoding device. Furthermore, the so-called image encoding method / device may include a video encoding method / device. Alternatively, the so-called video encoding method / device may include an image encoding method / device.

[0066] Reference Figure 2, the encoding device 200 includes an image divider 210, a predictor 220, a residual processor 230 and an entropy encoder 240, an adder 250, a filter 260 and a memory 270. The predictor 220 may include an inter-frame predictor 221 and an intra-frame predictor 222. The residual processor 230 may include a transformer 232, a quantizer 233, a dequantizer 234 and an inverse transformer 235. The residual processor 230 may also include a subtractor 231. The adder 250 may be referred to as a reconstructor or a reconstructed block generator. According to an embodiment, the image divider 210, the predictor 220, the residual processor 230, the entropy encoder 240, the adder 250 and the filter 260 may be configured by at least one hardware component (e.g., an encoder chipset or processor). In addition, the memory 270 may include a decoded picture buffer (DPB) or may be configured by a digital storage medium. The hardware components may also include the memory 270 as an internal / external component.

[0067] The image divider 210 can divide the input image (or picture or frame) input to the encoding device 200 into one or more processors. For example, the processor may be referred to as a coding unit (CU). In this case, the coding unit may be recursively divided from a coding tree unit (CTU) or a maximum coding unit (LCU) based on a quadtree binary tree ternary tree (QTBTTT) structure. For example, a coding unit may be divided into multiple coding units of greater depth based on a quadtree structure, a binary tree structure, and / or a ternary structure. In this case, for example, the quadtree structure may be applied first, and the binary tree structure and / or ternary structure may be applied later. Alternatively, the binary tree structure may be applied first. The encoding process according to the present disclosure may be performed based on the final coding unit that is no longer divided. In this case, based on image characteristics, coding efficiency, etc., the maximum coding unit may be used as the final coding unit, or, if necessary, the coding unit may be recursively divided into coding units of greater depth, and the coding unit of the optimal size may be used as the final coding unit. Here, the encoding process may include prediction, transformation, and reconstruction processes (described later). As another example, the processor may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may be split or divided from the final coding unit. The prediction unit may be a unit for sample prediction, and the transform unit may be a unit for deriving transform coefficients and / or a unit for deriving a residual signal from the transform coefficients.

[0068] In some cases, the term "unit" is used interchangeably with terms such as "block" or "region." In general, an M×N block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or pixel value, and can represent only the pixel / pixel value of the luma component or only the pixel / pixel value of the chroma component. A sample can be used as a term corresponding to a picture (or image) of a pixel or pixel.

[0069] The encoding device 200 can subtract the prediction signal (prediction block, prediction sample array) output from the inter-frame predictor 221 or the intra-frame predictor 222 from the input image signal (original block, original sample array) to generate a residual signal (residual block, residual sample array), and the generated residual signal is sent to the transformer 232. In this case, as illustrated, the unit in the encoding device 200 for subtracting the prediction signal (prediction block, prediction sample array) from the input image signal (original block, original sample array) can be referred to as a subtractor 231. The predictor can perform prediction on a processing target block (hereinafter referred to as a current block) and generate a prediction block including prediction samples of the current block. The predictor can determine whether to apply intra-frame prediction or inter-frame prediction in units of the current block or CU. The predictor can generate various information related to prediction, such as prediction mode information, and send the generated information to the entropy encoder 240, as described below when describing each prediction mode. The information related to the prediction can be encoded by the entropy encoder 240 and output in the form of a bitstream.

[0070] The intra-frame predictor 222 can predict the current block with reference to samples in the current picture. Depending on the prediction mode, the referenced samples may be located near the current block or may be spaced apart. In intra-frame prediction, the prediction mode may include multiple non-directional modes and multiple directional modes. For example, the non-directional mode may include a DC mode and a planar mode. For example, depending on the level of detail of the prediction direction, the directional mode may include 33 directional prediction modes or 65 directional prediction modes. However, this is merely an example, and more or fewer directional prediction modes may be used depending on the settings. The intra-frame predictor 222 may use the prediction mode applied to the neighboring blocks to determine the prediction mode applied to the current block.

[0071] The inter-frame predictor 221 can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector in a reference picture. To reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. Motion information can include a motion vector and a reference picture index. Motion information can also include information about the inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter-frame prediction, neighboring blocks can include spatially neighboring blocks in the current picture and temporally neighboring blocks in a reference picture. The reference picture containing the reference block and the reference picture containing the temporally neighboring block can be the same or different. Temporally neighboring blocks can be referred to as collocated reference blocks, collocated CUs (colCUs), etc., and the reference picture containing temporally neighboring blocks can be referred to as collocated pictures (colPics). For example, the inter-frame predictor 221 can configure a motion information candidate list based on neighboring blocks and generate information indicating which candidate is used to derive the motion vector and / or reference picture index for the current block. Inter-frame prediction can be performed based on various prediction modes. For example, in the case of skip mode and merge mode, the inter-frame predictor 221 can use the motion information of the adjacent block as the motion information of the current block. In skip mode, unlike merge mode, a residual signal may not be transmitted. In the case of motion vector prediction (MVP) mode, the motion vector of the adjacent block can be used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0072] The predictor 220 can generate a prediction signal based on various prediction methods described below. For example, the predictor can apply intra prediction or inter prediction to the prediction of a block, and can apply intra prediction and inter prediction at the same time. This can be referred to as a combination of inter and intra prediction (CIIP). In addition, the predictor can predict the block based on an intra-block copy (IBC) prediction mode or based on a palette mode. The IBC prediction mode or palette mode can be used for image / video coding of content such as games, for example, screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in terms of deriving reference blocks within the current picture. That is, IBC can use at least one of the inter prediction techniques described in this document. The palette mode can be regarded as an example of intra coding or intra prediction. When the palette mode is applied, the sample values ​​in the picture can be signaled based on information about the palette table and the palette index.

[0073] The prediction signal generated by the predictor (including the inter-frame predictor 221 and / or the intra-frame predictor 222) can be used to generate a reconstructed signal or can be used to generate a residual signal. The transformer 232 can generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen-Loeve transform (KLT), a graph-based transform (GBT), or a conditional nonlinear transform (CNT). Here, GBT refers to a transform obtained from a curve graph when the relationship information between pixels is represented by a curve graph. CNT refers to a transform obtained based on a prediction signal generated using all previously reconstructed pixels. In addition, the transform process can be applied to blocks of pixels of the same size as a square, or can be applied to blocks of variable size rather than square.

[0074] The quantizer 233 quantizes the transform coefficients and transmits them to the entropy encoder 240. The entropy encoder 240 encodes the quantized signal (information about the quantized transform coefficients) and outputs the encoded signal as a bitstream. The information about the quantized transform coefficients can be referred to as residual information. The quantizer 233 can rearrange the quantized transform coefficients in block form into a one-dimensional vector form based on the coefficient scanning order, and can generate information about the transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. The entropy encoder 240 can perform various encoding methods such as (for example) exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). The entropy encoder 240 can encode information necessary for video / image reconstruction (for example, syntax element values, etc.) in addition to the quantized transform coefficients, together or separately. The encoded information (for example, the encoded video / image information) can be transmitted or stored in the form of a bitstream in units of network abstraction layer (NAL) units. The video / image information may also include information about various parameter sets such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may also include conventional constraint information. In this document, information and / or syntax elements sent / signaled from the encoding device to the decoding device may be included in the video / image information. The video / image information may be encoded through the above-mentioned encoding process and included in the bitstream. The bitstream may be sent over a network or may be stored in a digital storage medium. Here, the network may include a broadcast network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. A sending unit (not shown) and / or a storage unit (not shown) for sending or storing a signal output from the entropy encoder 240 may be configured as an internal / external element of the encoding device 200, or the sending unit may be included in the entropy encoder 240.

[0075] The quantized transform coefficients output from the quantizer 233 can be used to generate a prediction signal. For example, the residual signal (residual block or residual sample) can be reconstructed by applying dequantization and inverse transform to the quantized transform coefficients using the dequantizer 234 and the inverse transformer 235. The adder 250 can add the reconstructed residual signal to the prediction signal output from the inter-frame predictor 221 or the intra-frame predictor 222 to generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array). When there is no residual of the processing target block, such as when skip mode is applied, the prediction block can be used as a reconstructed block. The adder 250 can be referred to as a recovery unit or a recovery block generator. The generated reconstructed signal can be used for intra-frame prediction of the next processing target block in the current picture, and can be used for inter-frame prediction of the next picture after filtering as described below.

[0076] Furthermore, luma mapping with chroma scaling (LMCS) may be applied during the picture encoding and / or reconstruction process.

[0077] The filter 260 can improve the subjective / objective image quality by applying filtering to the reconstructed signal. For example, the filter 260 can generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture, and store the modified reconstructed picture in the memory 270, specifically, in the DPB of the memory 270. Various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filter 260 can generate various types of information related to filtering and transmit the generated information to the entropy encoder 240, as described later when each filtering method is described. The information related to filtering can be encoded by the entropy encoder 240 and output in the form of a bitstream.

[0078] The modified reconstructed picture transmitted to the memory 270 may be used as a reference picture in the inter predictor 221. When inter prediction is applied by the encoding apparatus, prediction mismatch between the encoding apparatus 200 and the decoding apparatus may be avoided, and encoding efficiency may be improved.

[0079] The DPB of the memory 270 can store the corrected reconstructed picture for use as a reference picture in the inter-frame predictor 221. The memory 270 can store the motion information of the block from which the motion information in the current picture is derived (or encoded) and / or the motion information of the block in the reconstructed picture. The stored motion information can be transmitted to the inter-frame predictor 221 to be used as the motion information of the spatially adjacent blocks or the motion information of the temporally adjacent blocks. The memory 270 can store the reconstructed samples of the reconstructed blocks in the current picture and can transmit the reconstructed samples to the intra-frame predictor 222.

[0080] In addition, in this document, at least one of quantization / dequantization and / or transform / inverse transform may be omitted. When quantization / dequantization is omitted, the quantized transform coefficient may be referred to as a transform coefficient. When transform / inverse transform is omitted, the transform coefficient may be referred to as a coefficient or a residual coefficient, or may still be referred to as a transform coefficient for consistency of description.

[0081] In addition, in this document, quantized transform coefficients and transform coefficients may be referred to as transform coefficients and scaled transform coefficients, respectively. In this case, the residual information may include information about the transform coefficients, and the information about the transform coefficients may be signaled through residual coding syntax. The transform coefficients may be derived based on the residual information (or information about the transform coefficients), and the scaled transform coefficients may be derived by inverse transforming (scaling) the transform coefficients. Residual samples may be derived based on the inverse transform (transform) of the scaled transform coefficients. This may also be applied / expressed in other parts of this document.

[0082] Figure 3 is a diagram schematically illustrating the configuration of a video / image decoding device to which the disclosure of this document can be applied.

[0083] Reference Figure 3 , the decoding device 300 may include and be configured with an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an intra-frame predictor 331 and an inter-frame predictor 332. The residual processor 320 may include a dequantizer 321 and an inverse transformer 322. According to an embodiment, the entropy decoder 310, the residual processor 320, the predictor 330, the adder 340, and the filter 350 described above may be configured by one or more hardware components (e.g., a decoder chipset or processor). In addition, the memory 360 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware components may also include the memory 360 as an internal / external component.

[0084] When a bit stream including video / image information is input, the decoding device 300 may generate a Figure 2 The image is reconstructed by processing the video / image information in the encoding device illustrated in . For example, the decoding device 300 can derive the unit / block based on the block segmentation related information obtained from the bit stream. The decoding device 300 can perform decoding using the processing unit applied to the encoding device. Therefore, the processing unit for decoding can be, for example, a coding unit, and the coding unit can be divided from the coding tree unit or the maximum coding unit according to a quadtree structure, a binary tree structure and / or a ternary tree structure. One or more transform units can be derived from the coding unit. In addition, the reconstructed image signal decoded and output by the decoding device 300 can be reproduced by a reproduction device.

[0085] The decoding device 300 may receive the data from the Figure 2 The received signal is output by the encoding device, and the entropy decoder 310 can decode the received signal. For example, the entropy decoder 310 can parse the bitstream to derive information necessary for image reconstruction (or picture reconstruction) (e.g., video / image information). The video / image information may also include information about various parameter sets such as the Adaptation Parameter Set (APS), Picture Parameter Set (PPS), Sequence Parameter Set (SPS), or Video Parameter Set (VPS). In addition, the video / image information may also include general constraint information. The decoding device can also decode the picture based on the information about the parameter sets and / or general constraint information. The signaled / received information and / or syntax elements subsequently described in this document can be decoded and obtained from the bitstream through a decoding process. For example, the entropy decoder 310 decodes the information in the bitstream based on a coding method such as Exponential Golomb coding, Context-Adaptive Variable Length Coding (CAVLC), or Context-Adaptive Binary Arithmetic Coding (CABAC), and outputs syntax elements necessary for image reconstruction and quantized values ​​of the transform coefficients for the residual. More specifically, the CABAC entropy decoding method can receive a bin corresponding to each syntax element in the bitstream, determine the context model by using the decoding target syntax element information, the decoding information of the decoding target block, or the information of the symbol / bin decoded in the previous level, and perform arithmetic decoding on the bin by predicting the probability of the bin appearing according to the determined context model, and generate a symbol corresponding to the value of each syntax element. In this case, the CABAC entropy decoding method can update the context model by using the information of the decoded symbol / bin of the context model for the next symbol / bin after determining the context model. The information related to the prediction among the information decoded by the entropy decoder 310 can be provided to the predictor (inter-frame predictor 332 and intra-frame predictor 331), and the residual value (i.e., quantized transform coefficients and related parameter information) that has been entropy decoded in the entropy decoder 310 can be input to the residual processor 320.

[0086] The dequantizer 321 may dequantize the quantized transform coefficients and output the transform coefficients. The dequantizer 321 may rearrange the quantized transform coefficients in a two-dimensional block format. In this case, the rearrangement may be performed based on the coefficient scanning order performed in the encoding device. The dequantizer 321 may dequantize the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain the transform coefficients.

[0087] The inverse transformer 322 inversely transforms the transform coefficients to obtain a residual signal (residual block, residual sample array).

[0088] The predictor may perform prediction on the current block and generate a prediction block including prediction samples of the current block. The predictor may determine whether to apply intra prediction or inter prediction to the current block based on the information on prediction output from the entropy decoder 310 and may determine a specific intra / inter prediction mode.

[0089] The predictor 330 can generate a prediction signal based on various prediction methods to be described later. For example, the predictor can apply intra prediction or inter prediction to the prediction of a block, and can apply intra prediction and inter prediction at the same time. This can be referred to as a combination of inter and intra prediction (CIIP). In addition, the predictor can predict the block based on an intra-block copy (IBC) prediction mode or based on a palette mode. The IBC prediction mode or palette mode can be used for image / video coding of content such as games, for example, screen content coding (SCC). IBC can basically perform prediction within the current picture, but can be performed similarly to inter prediction in terms of deriving reference blocks within the current picture. That is, IBC can use at least one of the inter prediction techniques described in this document. The palette mode can be considered an example of intra coding or intra prediction. When the palette mode is applied, information about the palette table and palette index can be included in the video / image information and signaled.

[0090] The intra-frame predictor 331 can predict the current block by referencing samples in the current picture. Depending on the prediction mode, the referenced samples may be located near the current block or may be located separately from the current block. In intra-frame prediction, the prediction modes may include multiple non-directional modes and multiple directional modes. The intra-frame predictor 331 may determine the prediction mode to be applied to the current block by using the prediction modes applied to neighboring blocks.

[0091] The inter-frame predictor 332 can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can also include information about the inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter-frame prediction, the neighboring blocks can include spatial neighboring blocks in the current picture and temporal neighboring blocks in the reference picture. For example, the inter-frame predictor 332 can construct a motion information candidate list based on the neighboring blocks and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter-frame prediction can be performed based on various prediction modes, and the prediction information can include information indicating the inter-frame prediction mode for the current block.

[0092] The adder 340 can generate a reconstructed signal (reconstructed picture, reconstructed block, or reconstructed sample array) by adding the obtained residual signal to the prediction signal (prediction block or prediction sample array) output from the predictor (including the inter-frame predictor 332 and / or the intra-frame predictor 331). If the residual of the target block is not processed (such as when skip mode is applied), the prediction block can be used as the reconstructed block.

[0093] The adder 340 may be referred to as a reconstructor or a reconstructed block generator. The generated reconstructed signal may be used for intra-frame prediction of the next block to be processed in the current picture, and as described later, may also be output through filtering or may also be used for inter-frame prediction of the next picture.

[0094] In addition, luminance mapping with chroma scaling (LMCS) can also be applied to the picture decoding process.

[0095] The filter 350 can improve the subjective / objective image quality by applying filtering to the reconstructed signal. For example, the filter 350 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image, and store the modified reconstructed image in the memory 360, specifically, in the DPB of the memory 360. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0096] The (modified) reconstructed picture stored in the DPB of the memory 360 can be used as a reference picture in the inter-frame predictor 332. The memory 360 can store the motion information of the block from which the motion information in the current picture is derived (or decoded) and / or the motion information of the block in the reconstructed picture. The stored motion information can be transmitted to the inter-frame predictor 332 to be used as the motion information of the spatial neighboring block or the motion information of the temporal neighboring block. The memory 360 can store the reconstructed samples of the reconstructed block in the current picture and transmit the reconstructed samples to the intra-frame predictor 331.

[0097] In the present disclosure, the embodiments described in the filter 260 , the inter predictor 221 , and the intra predictor 222 of the encoding apparatus 200 may be applied equally or correspondingly to the filter 350 , the inter predictor 332 , and the intra predictor 331 .

[0098] In addition, as described above, when performing video encoding, prediction is performed to improve compression efficiency. Accordingly, a prediction block including prediction samples for a current block as a block to be encoded (i.e., an encoding target block) can be generated. Here, the prediction block includes prediction samples in the spatial domain (or pixel domain). The prediction block is derived in the same manner in the encoding device and the decoding device, and the encoding device can signal information about the residual between the original block and the prediction block (residual information) instead of the original sample value of the original block to the decoding device, thereby improving image coding efficiency. The decoding device can derive a residual block including residual samples based on the residual information, add the residual block and the prediction block to generate a reconstructed block including reconstructed samples, and generate a reconstructed picture including the reconstructed block.

[0099] Residual information can be generated through a transformation and quantization process. For example, the encoding device can derive a residual block between the original block and the prediction block, perform a transformation process on the residual samples (residual sample array) included in the residual block to derive a transformation coefficient, perform a quantization process on the transformation coefficient to derive a quantized transformation coefficient, and signal the relevant residual information to the decoding device (via a bitstream). Here, the residual information may include value information, position information, transformation technology, transformation kernel, quantization parameter, etc. of the quantized transformation coefficient. The decoding device can perform a dequantization / inverse transformation process based on the residual information and derive residual samples (or residual blocks). The decoding device can generate a reconstructed picture based on the prediction block and the residual block. In addition, in order to provide a reference for inter-frame prediction of subsequent pictures, the encoding device can also dequantize / inverse transform the quantized transformation coefficient to derive a residual block and generate a reconstructed picture based on this.

[0100] In addition, various inter-frame prediction modes can be used for the prediction of the current block within the picture. For example, various modes such as merge mode, skip mode, motion vector prediction (MVP) mode, affine mode, sub-block merge mode, merge with MVD (MMVD) mode, etc. can be used. Decoder-side motion vector refinement (DMVR) mode, adaptive motion vector resolution (AMVR) mode, bidirectional prediction with CU-level weight (BCW), bidirectional optical flow (BDOF), etc. can be used additionally or alternatively as auxiliary modes. Affine mode can be referred to as affine motion prediction mode. MVP mode can be referred to as advanced motion vector prediction (AMVP) mode. In the present disclosure, some modes and / or motion information candidates derived from some modes can be included as one of the motion information-related candidates of other modes. For example, an HMVP candidate can be added as a merge candidate for merge / skip mode, or can be added as an MVP candidate for MVP mode.

[0101] Inter-frame prediction mode information indicating the inter-frame prediction mode of the current block can be signaled from the encoding device to the decoding device. The inter-frame prediction mode information can be included in the bitstream and received at the decoding device. The inter-frame prediction mode information may include index information indicating one of a plurality of candidate modes. In addition, the inter-frame prediction mode can be indicated by hierarchical signaling of flag information. In this case, the inter-frame prediction mode information may include one or more flags. For example, whether the skip mode is applied can be indicated by signaling a skip flag; for when the skip mode is not applied, whether the merge mode is applied can be indicated by signaling a merge flag; and when the merge mode is not applied, the application of the MVP mode can be indicated or a flag for further segmentation can also be signaled. The affine mode can be signaled as an independent mode, or can be signaled as a mode that depends on the merge mode, the MVP mode, etc. For example, the affine mode can include an affine merge mode and an affine MVP mode.

[0102] In addition, information indicating whether the above-mentioned list0 (L0) prediction, list1 (L1) prediction, or bidirectional prediction is used in the current block (current coding unit) can be signaled to the current block. This information can be referred to as motion prediction direction information, inter-frame prediction direction information, or inter-frame prediction indication information, and can be constructed / encoded / signaled in the form of, for example, an inter_pred_idc syntax element. That is, the inter_pred_idc syntax element can indicate whether the above-mentioned list0 (L0) prediction, list1 (L1) prediction, or bidirectional prediction is used for the current block (current coding unit). In the present disclosure, for ease of description, the inter-frame prediction type (L0 prediction, L1 prediction, or BI prediction) indicated by the inter_pred_idc syntax element can be expressed as a motion prediction direction. L0 prediction can be represented by pred_L0; L1 prediction can be represented by pred_L1; and bidirectional prediction can be represented by pred_BI. For example, the following prediction types can be indicated according to the value of the inter_pred_idc syntax element.

[0103] As described above, a picture may include one or more slices. A slice may have one of slice types including an intra (I) slice, a predicted (P) slice, and a bidirectional predicted (B) slice. The slice type may be indicated based on slice type information. For blocks in an I slice, inter prediction is not used for prediction, and only intra prediction may be used. Of course, even in this case, the original sample values ​​may be encoded and signaled without prediction. For blocks in a P slice, intra prediction or inter prediction may be used, and when inter prediction is used, only unidirectional prediction may be used. In addition, for blocks in a B slice, intra prediction or inter prediction may be used, and when inter prediction is used, up to a maximum bidirectional prediction may be used.

[0104] L0 and L1 may include reference pictures that are encoded / decoded before the current picture. For example, L0 may include reference pictures that are before and / or after the current picture in POC order, and L1 may include reference pictures that are after and / or before the current picture in POC order. In this case, a lower reference picture index relative to a reference picture that is earlier than the current picture in POC order may be assigned to L0, and a lower reference picture index relative to a reference picture that is later than the current picture in POC order may be assigned to L1. In the case of B slices, bidirectional prediction may be applied, and in this case, unidirectional bidirectional prediction may be applied, or bidirectional bidirectional prediction may be applied. Bidirectional bidirectional prediction may be referred to as true bidirectional prediction.

[0105] For example, information about the inter prediction mode of the current block can be encoded and signaled at the CU (CU syntax) level, or can be implicitly determined according to conditions. In this case, some modes can be explicitly signaled, while other modes can be implicitly derived.

[0106] For example, the CU syntax may carry information about (inter) prediction mode, etc. The CU syntax may be as shown in Table 1 below.

[0107] [Table 1]

[0108]

[0109]

[0110]

[0111]

[0112]

[0113] In Table 1, cu_skip_flag may indicate whether skip mode is applied to the current block (CU).

[0114] pred_mode_flag equal to 0 can specify that the current coding unit is encoded in inter-frame prediction mode. pred_mode_flag equal to 1 can specify that the current coding unit is encoded in intra-frame prediction mode.

[0115] pred_mode_IBC_flag is equal to 1 to specify that the current coding unit is encoded in IBC prediction mode. pred_mode_IBC_flag is equal to 0 to specify that the current coding unit is not encoded in IBC prediction mode.

[0116] pcm_flag[x0][y0] equal to 1 can specify that the pcm_sample() syntax structure is present and the transform_tree() syntax structure is not present in the coding unit including the luma coding block at position (x0, y0). pcm_flag[x0][y0] equal to 0 can specify that the pcm_sample() syntax structure is not present. That is, pcm_flag can indicate whether the pulse code modulation (PCM) mode is applied to the current block. If PCM mode is applied to the current block, prediction, transform, quantization, etc. are not applied, and the values ​​of the original samples in the current block can be encoded and signaled.

[0117] intra_mip_flag[x0][y0] equal to 1 can specify that the intra prediction type for luma samples is matrix-based intra prediction (MIP). intra_mip_flag[x0][y0] equal to 0 can specify that the intra prediction type for luma samples is not matrix-based intra prediction. In other words, intra_mip_flag can indicate whether the MIP prediction mode (type) is applied to (luma samples) of the current block.

[0118] intra_chroma_pred_mode[x0][y0] may specify the intra prediction mode for chroma samples in the current block.

[0119] general_merge_flag[x0][y0] may specify that inter prediction parameters of the current coding unit are inferred from adjacent inter-prediction partitions. That is, general_merge_flag may indicate that general merging is available, and when the value of general_merge_flag is 1, conventional merge mode, MMVD mode, and subblock merge mode may be available. For example, when the value of general_merge_flag is 1, merge data syntax may be parsed from coded video / image information (or bitstream), and the merge data syntax may be configured / encoded to include information as shown in Table 2 below.

[0120] [Table 2]

[0121]

[0122]

[0123] In Table 2, regular_merge_flag[x0][y0] equal to 1 may specify that the regular merge mode is used to generate the inter prediction parameters of the current coding unit. That is, regular_merge_flag may indicate whether the merge mode (regular merge mode) is applied to the current block.

[0124] mmvd_merge_flag[x0][y0] equal to 1 can specify that the merge mode with motion vector difference is used to generate inter-frame prediction parameters of the current block. In other words, mmvd_merge_flag indicates whether MMVD is applied to the current block.

[0125] mmvd_cand_flag[x0][y0] may specify whether the first (0) candidate or the second (1) candidate in the merge candidate list is used with the motion vector difference derived from mmvd_distance_idx[x0][y0] and mmvd_direction_idx[x0][y0].

[0126] mmvd_distance_idx[x0][y0] may specify an index used to derive MmvdDistance[x0][y0].

[0127] mmvd_direction_idx[x0][y0] may specify an index used to derive MmvdSign[x0][y0].

[0128] merge_block_flag[x0][y0] may specify subblock-based inter prediction parameters for the current block. That is, merge_subblock_flag may indicate whether subblock merge mode (or affine merge mode) is applied to the current block.

[0129] merge_block_idx[x0][y0] may specify a merge candidate index of a sub-block based merge candidate list.

[0130] ciip_flag[x0][y0] may specify whether combined inter-picture merging and intra-picture prediction (CIIP) is applied to the current coding unit.

[0131] merge_triangle_idx0[x0][y0] may specify the first merge candidate index of a triangle-shaped based motion compensation candidate list.

[0132] merge_triangle_idx1[x0][y0] may specify a second merge candidate index of a triangle-shaped based motion compensation candidate list.

[0133] merge_idx[x0][y0] may specify a merge candidate index of the merge candidate list.

[0134] In addition, referring back to the CU syntax, mvp_l0_flag[x0][y0] may specify the motion vector predictor index of list 0. That is, when MVP mode is applied, mvp_l0_flag may indicate the candidate from MVP candidate list 0 selected for MVP derivation of the current block.

[0135] ref_idx_l1[x0][y0] has the same semantics as ref_idx_l0, where l0 and list0 can be replaced by l1 and list1 respectively.

[0136] inter_pred_idc[x0][y0] can specify whether list0, list1 or bidirectional prediction is used for the current coding unit.

[0137] sym_mvd_flag[x0][y0] equal to 1 may specify the absence of the syntax elements ref_idx_l0[x0][y0] and ref_idx_l1[x0][y0] and the mvd_coding(x0, y0, reflist, cIdx) syntax structure for RefList equal to 1. In other words, sym_mvd_flag indicates whether symmetric MVD is used in mvd coding.

[0138] ref_idx_l0[x0][y0] may specify the list 0 reference picture index of the current block.

[0139] ref_idx_l1[x0][y0] has the same semantics as ref_idx_l0, where l0, L0, and list0 are replaced by l1, L1, and list1, respectively.

[0140] inter_affine_flag[x0][y0] equal to 1 can specify that affine model-based motion compensation is used to generate prediction samples for the current block when decoding P or B slices.

[0141] cu_affine_type_flag[x0][y0] equal to 1 specifies that for the current coding unit, when decoding P or B slices, motion compensation based on a 6-parameter affine model is used to generate prediction samples for the current coding unit. cu_affine_type_flag[x0][y0] equal to 0 specifies that motion compensation based on a 4-parameter affine model is used to generate prediction samples for the current block.

[0142] amvr_flag[x0][y0] may specify the resolution of the motion vector difference. Array indices x0, y0 specify the position (x0, y0) of the top-left luma sample of the coding block under consideration relative to the top-left luma sample of the picture. amvr_flag[x0][y0] being 0 specifies that the resolution of the motion vector difference is 1 / 4 of the luma sample. amvr_flag[x0][y0] being 1 specifies that the resolution of the motion vector difference is also specified by amvr_precision_flag[x0][y0].

[0143] amvr_precision_flag[x0][y0] equal to 0 specifies that the resolution of motion vector differences is one whole luma sample if inter_affine_flag[x0][y0] is 0, and 1 / 16 of a luma sample otherwise. amvr_precision_flag[x0][y0] equal to 1 specifies that the resolution of motion vector differences is four luma samples if inter_affine_flag[x0][y0] is 0, and one whole luma sample otherwise.

[0144] bcw_idx[x0][y0] may specify a weight index for bidirectional prediction with CU weight.

[0145] Figure 4 An example of a video / image encoding method based on inter-frame prediction is shown, and Figure 5 is a diagram schematically illustrating an example of an inter-frame predictor in an encoding device. Figure 5 The inter-frame predictor in the encoding device can be combined with the above Figure 2 The inter-frame predictor 221 of the encoding device 200 is applied identically or correspondingly.

[0146] Reference Figure 4 and Figure 5 , the encoding device performs inter prediction on the current block (S400). The encoding device may derive an inter prediction mode and motion information of the current block and generate a prediction sample of the current block. Here, the processes for determining the inter prediction mode, deriving motion information, and generating prediction samples may be performed simultaneously, or one process may be performed before performing the other process.

[0147] For example, the inter-frame predictor 221 of the encoding device may include a prediction mode determiner 221_1, a motion information deriver 221_2, and a prediction sample deriver 221_3. The prediction mode determiner 221_1 may determine the prediction mode of the current block, the motion information deriver 221_2 may derive the motion information of the current block, and the prediction sample deriver 221_3 may derive the prediction sample of the current block. For example, the inter-frame predictor 221 of the encoding device may search for a block similar to the current block within a predetermined area (search area) of a reference picture through motion estimation, and may derive a reference block that has the smallest difference from the current block or is a predetermined reference or less. Based on this, a reference picture index indicating the reference picture in which the reference block is located may be derived, and a motion vector may be derived based on the position difference between the reference block and the current block. The encoding device may determine a mode to be applied to the current block from among various prediction modes. The encoding device may compare the rate-distortion (RD) costs under various prediction modes and determine the optimal prediction mode for the current block.

[0148] For example, when skip mode or merge mode is applied to the current block, the encoding device may construct a merge candidate list to be described below and derive a reference block whose difference from the current block is minimal or is a predetermined reference or less among the reference blocks indicated by the merge candidates included in the merge candidate list. In this case, a merge candidate associated with the derived reference block may be selected, and merge index information indicating the selected merge candidate may be generated and signaled to the decoding device. The motion information of the selected merge candidate may be used to derive the motion information of the current block.

[0149] As another example, when the (A)MVP mode is applied to the current block, the encoding device constructs the (A)MVP candidate list to be described below, and uses the motion vector of the MVP candidate selected from the motion vector predictor (MVP) candidates included in the (A)MVP candidate list as the MVP of the current block. In this case, for example, the motion vector indicating the reference block derived by the above-mentioned motion estimation can be used as the motion vector of the current block, and the MVP candidate with the smallest difference from the motion vector of the current block among the MVP candidates can be the selected MVP candidate. A motion vector difference (MVD) as the difference obtained by subtracting the MVP from the motion vector of the current block can be derived. In this case, information about the MVD can be signaled to the decoding device. In addition, when the (A)MVP mode is applied, the value of the reference picture index can be constructed as reference image index information and separately signaled to the decoding device.

[0150] The encoding apparatus may derive residual samples based on the predicted samples (S410). The encoding apparatus may derive residual samples by comparing original samples of the current block with the predicted samples.

[0151] The encoding device encodes the image information including prediction information and residual information (S420). The encoding device can output the encoded image information in the form of a bitstream. The prediction information is information related to the prediction process and may include prediction mode information (e.g., a skip flag, a merge flag, or a mode index, etc.) and motion information. The information about the motion information may include candidate selection information (e.g., a merge index) as information for deriving a motion vector. In addition, the information about the motion information may include information indicating whether L0 prediction, L1 prediction, or bidirectional prediction is applied. The residual information is information about the residual sample. The residual information may include information about the quantized transform coefficient for the residual sample.

[0152] The output bitstream may be stored in a (digital) storage medium and transmitted to the decoding device, or may be transmitted to the decoding device over a network.

[0153] In addition, as described above, the encoding device can generate a reconstructed picture (including reconstructed samples and reconstructed blocks) based on the reference samples and the residual samples. This is because the encoding device can derive the same prediction result as the prediction result performed by the decoding device, and accordingly, the encoding efficiency can be improved. Therefore, the encoding device can store the reconstructed picture (or reconstructed sample, reconstructed block) in a memory and use the reconstructed picture as a reference picture for inter-frame prediction. As described above, an in-loop filtering process can also be applied to the reconstructed picture.

[0154] Figure 6 An example of a video / image decoding method based on inter-frame prediction is shown, and Figure 7 Schematically illustrates an example of an inter-frame predictor in a decoding device. Figure 7 The inter-frame predictor in the decoding device can be combined with the above Figure 3 The same or corresponding inter-frame predictor 332 of the decoding device 300 is applied.

[0155] Reference Figure 6 and Figure 7 , the decoding device may perform an operation corresponding to the operation performed by the encoding device. The decoding device may perform prediction on the current block based on the received prediction information and derive a prediction sample.

[0156] Specifically, the decoding apparatus may determine a prediction mode of the current block based on the received prediction information (S600).The decoding apparatus may determine which inter prediction mode to apply to the current block based on prediction mode information in the prediction information.

[0157] Inter prediction mode candidates may include skip mode, merge mode and / or (A)MVP mode, or may include various inter prediction modes to be described later.

[0158] The decoding device derives the motion information of the current block based on the determined inter-frame prediction mode (S610). For example, when skip mode or merge mode is applied to the current block, the decoding device can configure a merge candidate list and select a merge candidate from the merge candidates included in the merge candidate list. The selection can be performed based on the above-mentioned selection information (merge index). The motion information of the selected merge candidate can be used to derive the motion information of the current block. The motion information of the selected merge candidate can be used as the motion information of the current block.

[0159] As another example, when the (A)MVP mode is applied to the current block, the decoding device may construct an (A)MVP candidate list and use the motion vector of the motion vector predictor (MVP) candidate selected from the motion vector predictor (MVP) candidates included in the (A)MVP candidate list as the MVP of the current block. The selection may be performed based on the above-mentioned selection information (MVP flag or MVP index). In this case, the MVD of the current block may be derived based on the information about the MVD, and the motion vector of the current block may be derived based on the MVP and MVD of the current block. In addition, the reference picture index of the current block may be derived based on the reference picture index information. The picture indicated by the reference picture index in the reference picture list of the current block may be derived as the reference picture referenced by the inter-frame prediction of the current block.

[0160] In addition, as will be described below, the motion information of the current block can be derived without configuring a candidate list. In this case, the motion information of the current block can be derived according to the process disclosed in the prediction mode to be described later. In this case, the configuration of the candidate list as described above can be omitted.

[0161] The decoding device may generate prediction samples for the current block based on the motion information of the current block (S620). In this case, a reference picture may be derived based on a reference picture index of the current block, and the prediction samples of the current block may be derived using samples of the reference block indicated by the motion vector of the current block on the reference picture. In this case, in some cases, a prediction sample filtering process may also be performed on all or some of the prediction samples of the current block.

[0162] For example, the inter-frame predictor 332 of the decoding device may include a prediction mode determiner 332_1, a motion information deriver 332_2 and a prediction sample deriver 332_3, and the prediction mode determiner 332_1 can determine the prediction mode of the current block based on the received prediction mode information, the motion information deriver 332_2 can derive the motion information (motion vector and / or reference picture index, etc.) of the current block based on the received information about the motion information, and the prediction sample deriver 332_3 can derive the prediction sample of the current block.

[0163] The decoding device generates residual samples of the current block based on the received residual information (S630). The decoding device can generate reconstructed samples of the current block based on the predicted samples and the residual samples, and can generate a reconstructed picture based on the samples (S640). Thereafter, as described above, an in-loop filtering process can also be applied to the reconstructed picture.

[0164] As described above, the inter-frame prediction process may include the steps of determining an inter-frame prediction mode, deriving motion information according to the determined prediction mode, and performing prediction (generating prediction samples) based on the derived motion information. The inter-frame prediction process may be performed by the encoding device and the decoding device as described above.

[0165] In addition, when deriving the motion information of the current block, motion information candidates are derived based on spatially adjacent blocks and temporally adjacent blocks, and based on the derived motion information candidates, a motion information candidate for the current block can be selected. In this case, the selected motion information candidate can be used as the motion information of the current block.

[0166] Figure 8 This is a diagram illustrating the merge mode in inter-frame prediction.

[0167] When the merge mode is applied, the motion information of the current prediction block is not directly sent, but the motion information of the adjacent prediction blocks is used to derive the motion information of the current prediction block. Therefore, the motion information of the current prediction block can be indicated by sending flag information indicating the use of the merge mode and a merge index indicating which nearby prediction block to use. The merge mode can be referred to as the normal merge mode.

[0168] In order to perform the merge mode, the encoding device needs to search for a merge candidate block for deriving motion information about the current prediction block. For example, up to five merge candidate blocks can be used, but the embodiments of the present disclosure are not limited to this. In addition, the maximum number of merge candidate blocks can be sent in the slice header or the tile group header, but the embodiments of the present disclosure are not limited to this. After finding the merge candidate blocks, the encoding device can generate a merge candidate list and can select the merge candidate block with the minimum cost among the merge candidate blocks as the final merge candidate block.

[0169] The present disclosure may provide various implementations of merging candidate blocks constituting a merging candidate list.

[0170] For example, the merge candidate list can use five merge candidate blocks. For example, four spatial merge candidates and one temporal merge candidate can be used. As a specific example, in the case of spatial merge candidates, Figure 4The blocks illustrated in may be used as spatial merging candidates. Hereinafter, a spatial merging candidate or a spatial MVP candidate to be described later may be referred to as an SMVP, and a temporal merging candidate or a temporal MVP candidate to be described later may be referred to as a TMVP.

[0171] For example, the merge candidate list of the current block may be constructed based on the following process.

[0172] The encoding device (encoding device / decoding device) may insert the spatial merge candidates derived by searching the current spatial adjacent blocks into the merge candidate list. For example, the spatial adjacent blocks may include the lower left adjacent block, the left adjacent block (A1), the upper right adjacent block, the upper adjacent block, and the upper left adjacent block of the current block. However, this is an example, and in addition to the above-mentioned spatial adjacent blocks, additional adjacent blocks such as the right adjacent block, the lower adjacent block, and the lower right adjacent block may be further used as spatial adjacent blocks. The encoding device may detect available blocks by searching the spatial adjacent blocks based on priority, and may derive the motion information of the detected blocks as spatial merge candidates. For example, the encoding device or the decoding device may be configured to search in the order of A1->B1->B0->A0->B2. Figure 8 The five blocks illustrated in , and the merge candidate list can be configured by indexing the available candidates in sequence.

[0173] The encoding device may search for temporally adjacent blocks of the current block and insert the derived temporal merge candidates into the merge candidate list. The temporally adjacent blocks may be at a reference picture that is a different picture from the current picture in which the current block is located. The reference picture in which the temporally adjacent blocks are located may be referred to as a collocated picture or collocated pictures. The temporally adjacent blocks may be searched for in the collocated picture in the order of the bottom-right adjacent block and the bottom-right center block of the co-located block of the current block. In addition, when motion data compression is applied, specific motion information may be stored as representative motion information in each predetermined storage unit in the collocated picture. In this case, it is not necessary to store the motion information about all blocks in the predetermined storage unit, and accordingly, a motion data compression effect can be obtained. In this case, the predetermined storage unit may be predetermined as a unit of, for example, 16×16 samples or 8×8 samples, or the size information about the predetermined storage unit may be signaled from the encoding device to the decoding device. When motion data compression is applied, the motion information about the temporally adjacent blocks may be replaced with the representative motion information on the predetermined storage unit where the temporally adjacent blocks are located. That is, in this case, from the perspective of implementation, instead of the prediction block at the coordinates of the temporally adjacent blocks, the motion information of the prediction block covering the arithmetic left-shift position after arithmetic right-shift by a certain value based on the coordinates (top-left sample position) of the temporally adjacent blocks may be used to derive the temporal merge candidates. For example, when the predetermined storage unit is a 2n×2n sample unit, if the coordinates of the temporally adjacent block are (xTnb, yTnb), the motion information of the prediction block at the corrected position ((xTnb>>n)<<n), (yTnb>>n)<<n)) may be used for the temporal merge candidates. Specifically, when the predetermined storage unit is a 16×16 sample unit, if the coordinates of the temporally adjacent block are (xTnb, yTnb), the motion information of the prediction block at the corrected position ((xTnb>>4)<<4), (yTnb>>4)<<4)) may be used for the temporal merge candidates. Alternatively, when the predetermined storage unit is an 8×8 sample unit, if the coordinates of the temporally adjacent block are (xTnb, yTnb), the motion information of the prediction block at the corrected position (xTnb>>3)<<3), (yTnb>>3)<<3)) may be used for the temporal merge candidates.

[0174] The encoding device may check whether the number of current merge candidates is less than the maximum number of merge candidates. The maximum number of merge candidates may be predefined or signaled from the encoding device to the decoding device. For example, the encoding device may generate information about the maximum number of merge candidates, encode it, and send the information to the decoder in the form of a bitstream. When the maximum number of merge candidates is full, the subsequent candidate addition process may not be performed.

[0175] As a result of the check, when the current number of merge candidates is less than the maximum number of merge candidates, the encoding device may insert an additional merge candidate into the merge candidate list. For example, the additional merge candidate may include at least one of a history-based merge candidate, a pairwise average merge candidate, an ATMVP, a combined bidirectional prediction merge candidate (when the slice / tile group type of the current slice / tile group is type B), and / or a zero vector merge candidate, which will be described later.

[0176] As a result of the check, when the current number of merge candidates is not less than the maximum number of merge candidates, the encoding device may terminate the construction of the merge candidate list. In this case, the encoding device may select the optimal merge candidate from the merge candidates constituting the merge candidate list based on the rate-distortion (RD) cost, and signal selection information (e.g., a merge index) indicating the selected merge candidate to the decoding device. The decoding device may select the optimal merge candidate based on the merge candidate list and the selection information.

[0177] As described above, the motion information about the selected merge candidate may be used as the motion information about the current block, and the prediction samples of the current block may be derived based on the motion information about the current block. The encoding device may derive residual samples of the current block based on the prediction samples, and may signal residual information about the residual samples to the decoding device. As described above, the decoding device may generate reconstructed samples based on the residual samples derived based on the residual information and the prediction samples, and may generate a reconstructed picture based on the residual samples.

[0178] When skip mode is applied, motion information about the current block can be derived in the same manner as when merge mode is applied. However, when skip mode is applied, the residual signal of the corresponding block is omitted, so the prediction sample can be directly used as the reconstructed sample. For example, when the value of the cu_skip_flag syntax element is 1, skip mode can be applied.

[0179] Figure 9 This is a diagram illustrating the merge mode with motion vector difference (MMVD) mode in inter-frame prediction.

[0180] The MMVD mode is a method of applying motion vector difference (MVD) to a merge mode in which motion information derived to generate a prediction sample of a current block is directly used.

[0181] For example, an MMVD flag (e.g., mmvd_flag) indicating whether MMVD for the current block (i.e., current CU) is used may be signaled, and MMVD may be performed based on this MMVD flag. When MMVD is applied to the current block (e.g., when mmvd_flag is 1), additional information about the MMVD may be signaled.

[0182] Here, additional information about MMVD may include a merge candidate flag (e.g., mmvd_cand_flag) indicating whether the first candidate or the second candidate is used with MVD in the merge candidate list, a distance index (e.g., mmvd_distance_idx) for indicating the motion magnitude, and a direction index (e.g., mmvd_direction_idx) for indicating the motion direction.

[0183] In MMVD mode, two candidates located in the first entry and the second entry among the candidates in the merge candidate list (i.e., the first candidate or the second candidate) can be used, and one of the two candidates (i.e., the first candidate or the second candidate) can be used as the base MV. For example, a merge candidate flag (e.g., mmvd_cand_flag) can be signaled to indicate either of the two candidates in the merge candidate list (i.e., the first candidate or the second candidate).

[0184] In addition, the distance index (eg, mmvd_distance_idx) may indicate motion magnitude information and may indicate a predetermined offset from a starting point. Figure 9 , an offset may be added to the horizontal component or the vertical component of the starting motion vector. The relationship between the distance index and the predetermined offset may be as shown in Table 3 below.

[0185] [Table 3]

[0186]

[0187] Referring to Table 3, the distance of the MVD (e.g., MmvdDistance) may be determined according to the value of the distance index (e.g., mmvd_distance_idx), and the distance of the MVD (e.g., MmvdDistance) may be derived using integer sample precision or fractional sample precision based on the value of slice_fpel_mmvd_enabled_flag. For example, slice_fpel_mmvd_enabled_flag being equal to 1 may indicate that the distance of the MVD is derived using integer samples in the current slice, and slice_fpel_mmvd_enabled_flag being equal to 0 may indicate that the distance of the MVD is derived using fractional samples in the current slice.

[0188] In addition, the direction index (e.g., mmvd_direction_idx) indicates the direction of the MVD relative to the starting point and can indicate four directions, as shown in Table 4 below. In this case, the direction of the MVD can indicate the sign of the MVD. The relationship between the direction index and the MVD code can be expressed as shown in Table 4 below.

[0189] [Table 4]

[0190] mmvd_direction_idx[x0][y0] MmvdSign[x0][y0][0] MmvdSign[x0][y0][1] 0 +1 0 1 -1 0 2 0 +1 3 0 -1

[0191] Referring to Table 4, the sign of MVD (eg, MmvdSign) may be determined according to the value of the direction index (eg, mmvd_direction_idx), and may be derived for L0 and L1 reference pictures.

[0192] Based on the distance index (eg, mmvd_distance_idx) and the direction index (eg, mmvd_direction_idx) described above, the offset of the MVD may be calculated as shown in Equation 1 below.

[0193] [Formula 1]

[0194] MmvdOffset[x0][y0][0]=(MmvdDistance[x0][y0]<<2)*MmvdSign[x0][y0][0]

[0195] MmvdOffset[x0][y0][1]=MmvdDistance[x0][y0]<<2)*MmvdSign[x0][y0][1]

[0196] That is, in MMVD mode, a merge candidate indicated by a merge candidate flag (e.g., mmvd_cand_flag) is selected from among the merge candidates in the merge candidate list derived based on the neighboring blocks, and the selected merge candidate can be used as a base candidate (e.g., MVP). In addition, the motion information (i.e., motion vector) of the current block can be derived by adding the MVD derived using the distance index (e.g., mmvd_distance_idx) and the direction index (e.g., mmvd_direction_idx) based on the base candidate.

[0197] Figure 10a and Figure 10b CPMV for affine motion prediction is shown.

[0198] Conventionally, only one motion vector can be used to represent the motion of a coding block. In other words, a translational motion model is used. However, while this method can represent optimal motion per block, it is not actually optimal motion for each sample. If the optimal motion vector can be determined per sample, coding efficiency can be increased. For this purpose, an affine motion model can be used. The affine motion prediction method for coding using an affine motion model can be as follows.

[0199] The affine motion prediction method can use two, three, or four motion vectors to represent the motion vector per sample of the block. For example, the affine motion model can represent four types of motion. An affine motion model that represents three types of motion (translation, scaling, and rotation) among the motions that the affine motion model can represent can be called a similarity (or simplified) affine motion model. However, the affine motion model is not limited to the above motion models.

[0200] Affine motion prediction can use two or more control point motion vectors (CPMVs) to determine the motion vector of a sample position included in a block. In this case, the set of motion vectors can be called an affine motion vector field (MVF).

[0201] For example, Figure 10a A case where two CPMVs are used, which may be referred to as a 4-parameter affine model, may be shown. In this case, for example, a motion vector at a sample position (x, y) may be determined as, for example, Equation 2.

[0202] [Formula 2]

[0203]

[0204] For example, Figure 10b A case where three CPMVs are used, which may be referred to as a 6-parameter affine model, may be shown. In this case, for example, a motion vector at a sample position (x, y) may be determined as, for example, Equation 3.

[0205] [Formula 3]

[0206]

[0207] In Equations 2 and 3, {v x ,v y} can represent the motion vector at position (x, y). In addition, {v 0x ,v 0y} may indicate the CPMV of the control point (CP) at the upper left corner of the coding block, and {v 1x ,v 1y} can indicate the CPMV of the CP at the upper right corner, {v 2x ,v 2y} may indicate the CPMV of the CP at the lower left corner position. In addition, W may indicate the width of the current block, and H may indicate the height of the current block.

[0208] Figure 11 The case where the affine MVF is determined in units of sub-blocks is exemplified.

[0209] In the encoding / decoding process, the affine MVF can be determined in units of samples or predefined sub-blocks. For example, when the affine MVF is determined in units of samples, a motion vector can be obtained based on each sample value. Alternatively, for example, when the affine MVF is determined in units of sub-blocks, the motion vector of the corresponding block can be obtained based on the sample value at the center of the sub-block (i.e., the lower right side of the center, i.e., the lower right sample among the four center samples). In other words, in affine motion prediction, the motion vector of the current block can be derived in units of samples or sub-blocks.

[0210] exist Figure 11 In the case of , the affine MVF is determined in units of 4×4 sub-blocks, but the size of the sub-block can be modified differently.

[0211] That is, when affine prediction is available, the three motion models applicable to the current block may include a translation motion model, a 4-parameter affine motion model, and a 6-parameter affine motion model. The translation motion model may represent a model using existing block unit motion vectors, the 4-parameter affine motion model may represent a model using two CPMVs, and the 6-parameter affine motion model may represent a model using three CPMVs.

[0212] Furthermore, affine motion prediction may include an affine MVP (or affine inter) mode or an affine merge mode.

[0213] Figure 12 This is a diagram illustrating the affine merge mode or subblock merge mode in inter-frame prediction.

[0214] For example, in affine merge mode, the CPMV can be determined based on the affine motion model of the neighboring blocks encoded using affine motion prediction. For example, neighboring blocks encoded as affine motion predictions in search order can be used in affine merge mode. That is, when at least one of the neighboring blocks is encoded using affine motion prediction, the current block can be encoded in affine merge mode. Here, the refined merge mode may be referred to as AF_MERGE.

[0215] When the affine merge mode is applied, the CPMV of the neighboring block can be used to derive the CPMV of the current block. In this case, the CPMV of the neighboring block can be used as the CPMV of the current block as is, or the CPMV of the neighboring block can be modified based on the size of the neighboring block and the size of the current block and used as the CPMV of the current block.

[0216] On the other hand, in the case of an affine merge mode in which a motion vector (MV) is derived in units of sub-blocks, this may be referred to as a sub-block merge mode, and the sub-block merge mode may be indicated based on a sub-block merge flag (or merge_subblock_flag syntax element). Alternatively, when the value of the merge_subblock_flag syntax element is 1, it may be indicated that the sub-block merge mode is applied. In this case, the affine merge candidate list to be described later may be referred to as a sub-block merge candidate list. In this case, the sub-block merge candidate list may also include a candidate derived by the SbTMVP to be described later. In this case, the candidate derived by the SbTMVP may be used as a candidate of index 0 of the sub-block merge candidate list. In other words, the candidate derived from the SbTMVP may be before the inherited affine candidate or the constructed affine candidate to be described later in the sub-block merge candidate list.

[0217] When the affine merge mode is applied, an affine merge candidate list may be constructed to derive the CPMV of the current block. For example, the affine merge candidate list may include at least one of the following candidates. 1) Inherited affine merge candidate. 2) Constructed affine merge candidate. 3) Zero motion vector candidate (or zero vector). Here, the inherited affine merge candidate is a candidate derived based on the CPMV of the adjacent block when the adjacent block is encoded in the affine mode, the constructed affine merge candidate is a candidate derived by constructing the CPMV based on the MV of the adjacent block of the corresponding CP in units of each CPMV, and the zero motion vector candidate may indicate a candidate consisting of a CPMV whose value is 0.

[0218] For example, the affine merge candidate list can be constructed as follows.

[0219] There can be up to two inherited affine candidates, and the inherited affine candidates can be derived from the affine motion models of the neighboring blocks. The neighboring blocks can include a left neighboring block and an upper neighboring block. The candidate blocks can be as follows Figure 8 The scan order of the left predictor may be A1->A0, and the scan order of the upper predictor may be B1->B0->B2. Only one inherited candidate may be selected from each of the left predictor and the upper predictor. A pruning check may not be performed between two inherited candidates.

[0220] When identifying adjacent affine blocks, the control point motion vectors of the checked blocks can be used to derive CPMVP candidates in the affine merge list of the current block. Here, the adjacent affine blocks can refer to blocks encoded in affine prediction mode among the adjacent blocks of the current block. For example, referring to Figure 12When encoding the lower left neighboring block A in affine prediction mode, the motion vectors v2, v3, and v4 of the upper left, upper right, and lower left corners of the neighboring block A can be obtained. When encoding the neighboring block A using a 4-parameter affine motion model, two CPMVs of the current block can be calculated based on v2 and v3. When encoding the neighboring block A using a 6-parameter affine motion model, three CPMVs of the current block can be calculated based on v2, v3, and v4.

[0221] Figure 13 This is a diagram illustrating positions of candidates in the affine merge mode or the subblock merge mode.

[0222] Affine candidates constructed in affine merge mode or sub-block merge mode can represent candidates constructed by combining translational motion information around each control point. Motion information about a control point can be derived from the specified spatial and temporal surroundings. CPMVk (k=0, 1, 2, 3) can represent the kth control point.

[0223] Reference Figure 13 For CPMV0, blocks can be checked in the order B2->B3->A2, and the motion vector of the first available block can be used. For CPMV1, blocks can be checked according to the order B1->B0, and for CPMV2, blocks can be checked according to the order A1->A0. If available, the temporal motion vector predictor (TMVP) can be used with CPMV3.

[0224] After obtaining the motion vectors of the four control points, an affine merge candidate can be generated based on the obtained motion information. The combination of the control point motion vectors can be any one of {CPMV0, CPMV1, CPMV2}, {CPMV0, CPMV1, CPMV3}, {CPMV0, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV3}, {CPMV0, CPMV1}, and {CPMV0, CPMV2}.

[0225] A combination of three CPMVs can constitute a 6-parameter affine merge candidate, and a combination of two CPMVs can constitute a 4-parameter affine merge candidate. To avoid motion scaling processing, relevant combinations of control point motion vectors can be discarded if the reference indices of the control points are different.

[0226] Figure 14 This is a diagram illustrating SbTMVP in inter-frame prediction.

[0227] Sub-block based temporal motion vector prediction (SbTMVP) can also be called advanced temporal motion vector prediction (ATMVP). SbTMVP can use the motion field in the collocated picture to improve the motion vector prediction and merge mode of the CU in the current picture. Here, the collocated picture can be called a col picture.

[0228] For example, SbTMVP can predict motion at the block (or sub-CU) level. In addition, SbTMVP can apply motion shifting before obtaining temporal motion information from the collocated picture. Here, the motion shifting can be obtained from the motion vector of one of the spatial neighboring blocks of the current block.

[0229] SbTMVP can predict the motion vector of a sub-block (or sub-CU) in the current block (or CU) according to two steps.

[0230] In the first step, you can Figure 4 The spatial neighboring blocks are tested in the order A1, B1, B0, and A0 in

[0045] . The first spatial neighboring block having a motion vector that uses the collocated picture as its reference picture may be checked, and that motion vector may be selected as the motion shift to be applied. When no such motion is detected from the spatial neighboring blocks, the motion shift may be set to (0, 0).

[0231] In the second step, the motion shift identified in the first step can be applied to obtain sub-block level motion information (motion vector and reference index) from the collocated picture. For example, the motion shift can be added to the coordinates of the current block. For example, the motion shift can be set to Figure 8 In this case, for each sub-block, the motion information of the sub-block can be derived using the motion information of the corresponding block in the collocated picture. Temporal motion scaling can be applied to align the reference picture of the temporal motion vector with the reference picture of the current block.

[0232] A sub-block based merge list including a combination of SbTVMP candidates and affine merge candidates can be used for signaling of affine merge mode. Here, affine merge mode can be referred to as sub-block based merge mode. Depending on the flag included in the sequence parameter set (SPS), SbTVMP mode can be available or unavailable. When SbTMVP mode is available, the SbTMVP predictor can be added as the first entry of the sub-block based merge candidate list, and the affine merge candidate can follow. The maximum allowable size of affine merge candidates can be 5.

[0233] The size of the sub-CU (or sub-block) used in SbTMVP can be fixed to 8×8, and as in affine merge mode, SbTMVP mode can be applied only to blocks with both width and height of 8 or greater. The encoding logic of the additional SbTMVP merge candidate can be the same as the encoding logic of other merge candidates. That is, for each CU in a P or B slice, an RD check using an additional rate-distortion (RD) cost can be performed to determine whether to use the SbTMVP candidate.

[0234] Figure 15 This is a diagram illustrating a combined inter-picture merging and intra-picture prediction (CIIP) mode in inter-frame prediction.

[0235] CIIP can be applied to the current CU. For example, when the CU is encoded in merge mode, the CU includes at least 64 luma samples (i.e., when the product of the CU width and the CU height is 64 or greater), and the CU width and the CU height are both less than 128 luma samples, then an additional flag (e.g., ciip_flag) can be signaled to indicate whether CIIP mode is applied to the current CU.

[0236] In CIIP prediction, inter prediction signals and intra prediction signals can be combined. In CIIP mode, the inter prediction signal P_inter can be derived using the same inter prediction process applied to the regular merge mode. The intra prediction signal P_intra can be derived according to the intra prediction process with planar mode.

[0237] The intra prediction signal and the inter prediction signal may be combined using weighted averaging and may be expressed in the following Equation 4. Figure 15 The weights are calculated based on the coding modes of the upper neighboring block and the left neighboring block shown.

[0238] [Formula 4]

[0239] P CIIP =((4-wt)*P inter +wt*P intra+ 2)>>2

[0240] In Equation 4, when the upper neighboring block is available and intra-coded, isIntratop may be set to 1, otherwise isIntratop may be set to 0. If the left neighboring block is available and intra-coded, isIntraLeft may be set to 1, otherwise isIntraLeft may be set to 0. When (isIntraLeft+isIntraLeft) is 2, wt may be set to 3, and when (isIntraLeft+isIntraLeft) is 1, wt may be set to 2. Otherwise, wt may be set to 1.

[0241] Figure 16 This is a diagram illustrating the partitioning mode in inter-frame prediction.

[0242] Reference Figure 16 When a partitioning mode is applied, a CU may be equally divided into two triangular partitions using a diagonal partition or an anti-diagonal partition in the opposite direction. However, this is only an example of a partitioning mode, and a CU may be equally or unevenly divided into partitions of various shapes.

[0243] For each partition of a CU, only unidirectional prediction may be allowed. That is, each partition may have one motion vector and one reference index. The unidirectional prediction constraint ensures that only two motion compensated predictions are required for each CU, which is similar to bidirectional prediction.

[0244] When the split mode is applied, a flag indicating the split direction (diagonal or anti-diagonal) and two merge indices (for each partition) may be additionally signaled.

[0245] After predicting each partition, a blending process with adaptive weights can be used to adjust the sample values ​​based on boundary lines along diagonal or anti-diagonal lines.

[0246] In addition, when the merge mode or skip mode is applied, motion information can be derived based on the normal merge mode, MMVD mode (merge mode with motion vector difference), merge sub-block mode, CIIP mode (combined inter-picture merging and intra-picture prediction mode), or motion information can be derived using the partition mode to generate the prediction samples as described above. Each mode can be enabled or disabled by an on / off flag in the sequence parameter set (SPS). If the on / off flag for a specific mode is disabled in the SPS, the syntax clearly sent for the prediction mode in units of CU or PU may not be signaled.

[0247] Table 5 below relates to the process of deriving merge mode or skip mode from the conventional merge_data syntax. In Table 5 below, CUMergeTriangleFlag[x0][y0] may correspond to Figure 12

[0066] The on / off flag for the split mode described in

[0067] , and merge_triangle_split_dir[x0][y0] may indicate the split direction (diagonal or anti-diagonal) when the split mode is applied. In addition, merge_triangle_idx0[x0][y0] and merge_triangle_idx1[x0][y0] may indicate two merge indexes for each partition when the split mode is applied.

[0248] [Table 5]

[0249]

[0250]

[0251] In addition, each prediction mode including normal merge mode, MMVD mode, merge subblock mode, CIIP mode and partition mode can be enabled or disabled from the sequence parameter set (SPS) as shown in Table 6 below. In Table 6 below, sps_triangle_enabled_flag may correspond to enabling or disabling the above-mentioned Figure 12 Flags for the segmentation mode described in .

[0252] [Table 6]

[0253]

[0254]

[0255]

[0256]

[0257] The merge_data syntax of Table 5 can be parsed or derived according to the flag of the SPS of Table 6 and the conditions under which each prediction mode can be used. All cases according to the flag of the SPS and the conditions under which each prediction mode can be used are shown in Tables 7 and 8. Table 7 shows the number of cases where the current block is in merge mode, and Table 8 shows the number of cases where the current block is in skip mode. In the following Tables 7 and 8, conventional may correspond to conventional merge mode, mmvd may correspond to triangle, or TRI may correspond to the above referenced Figure 12 The segmentation pattern described.

[0258] [Table 7]

[0259]

[0260] [Table 8]

[0261]

[0262] As an example of the cases mentioned in Tables 7 and 8, the case where the current block is 4×16 and skip mode is described. When the merge subblock mode, MMVD mode, CIIP mode, and split mode are all enabled in the SPS, if regular_merge_flag[x0][y0], mmvd_flag[x0][y0], and merge_subblock_flag[x0][y0] are all 0 in the merge_data syntax, the motion information of the current block should be derived in the split mode. However, even if the split mode is enabled from the on / off flag in the SPS, it can be used as a prediction mode only when the conditions of the following Table 9 are additionally met. In the following Table 9, MergeTriangleFlag[x0][y0] may correspond to the on / off flag for the split mode, and sps_triangle_enabled_flag may correspond to a flag from the SPS that enables or disables the split mode.

[0263] [Table 9]

[0264]

[0265] Referring to Table 9 above, if the current slice is a P slice, the decoder may not be able to decode any bitstreams because the prediction samples cannot be generated using the partitioning mode. Thus, to address the problem that occurs in exceptional cases where decoding is not performed because the final prediction mode cannot be selected based on each on / off flag of the SPS and the merge data syntax, the present disclosure proposes a default merge mode. The default merge mode can be predefined in various ways or can be derived through additional syntax signaling.

[0266] In an embodiment, the normal merge mode may be applied to the current block based on a situation where the MMVD mode, the merge sub-block mode, the CIIP mode, and the partition mode for performing prediction by dividing the current block into two partitions are all unavailable. That is, when the merge mode cannot be selected for the current block, the normal merge mode may be applied as the default merge mode.

[0267] For example, although the value of the general merge flag indicating whether the merge mode is available for the current block is 1, if the merge mode cannot be selected for the current block in the end, the general merge mode may be applied as the default merge mode.

[0268] In this case, the prediction sample of the current block may be generated using the merge index information indicating one of the merge candidates included in the merge candidate list generated by applying the normal merge mode. For example, the motion information of the current block may be derived based on the indicated merge index information, and the prediction sample may be generated based on the derived motion information.

[0269] Therefore, the merge data syntax can be shown in Table 10 below.

[0270] [Table 10]

[0271]

[0272] Referring to Table 10 and Table 6, based on the case where the MMVD mode is unavailable, the flag sps_mmvd_enabled_flag from the SPS for enabling or disabling the MMVD mode may be 0 or the first flag (mmvd_merge_flag[x0][y0]) indicating whether the MMVD mode is applied may be 0.

[0273] In addition, based on the case where the merge subblock mode is unavailable, the flag sps_affine_enabled_flag from the SPS for enabling or disabling the merge subblock mode may be 0 or the second flag (merge_subblock_flag[x0][y0]) indicating whether the merge subblock mode is applied may be 0.

[0274] In addition, based on the case where the CIIP mode is unavailable, the flag sps_ciip_enabled_flag from the SPS for enabling or disabling the CIIP mode may be 0 or the third flag (ciip_flag[x0][y0]) indicating whether the CIIP mode is applied may be 0.

[0275] In addition, based on the case where the split mode is not available, the flag sps_triangle_enabled_flag from the SPS for enabling or disabling the split mode may be 0 or the fourth flag (MergeTriangleFlag[x0][y0]) indicating whether the split mode is applied may be 0.

[0276] In addition, for example, based on the case where the value of the general merge flag indicating whether the merge mode is applicable to the current block is 1, the values ​​of the first flag mmvd_merge_flag[x0][y0] indicating whether the MMVD mode is applied, the second flag merge_subblock_flag[x0][y0] indicating whether the merge subblock mode is applied, and the third flag ciip_flag[x0][y0] indicating whether the CIIP mode is applied can be signaled.

[0277] Furthermore, for example, based on a case in which the split mode is disabled based on the flag sps_triangle_enabled_flag, the fourth flag (MergeTriangleFlag[x0][y0]) indicating whether the split mode is applied may be set to 0.

[0278] In another embodiment, if the normal merge mode, the MMVD mode, the merge sub-block mode, the CIIP mode, and the partition mode for performing prediction by dividing the current block into two partitions are unavailable, the normal merge mode may be applied to the current block. In other words, when a merge mode cannot be selected for the current block, the normal merge mode may be applied as the default merge mode.

[0279] For example, based on a general merge flag indicating whether the merge mode is available for the current block, in case the merge mode cannot be finally selected for the current block although the merge mode is available for the current block, the normal merge mode may be applied as the default merge mode.

[0280] For example, based on the case where the MMVD mode is unavailable, the flag sps_mmvd_enabled_flag from the SPS for enabling or disabling the MMVD mode may be 0 or the first flag (mmvd_merge_flag[x0][y0]) indicating whether the MMVD mode is applied may be 0.

[0281] In addition, based on the case where the merge subblock mode is unavailable, the flag sps_affine_enabled_flag from the SPS for enabling or disabling the merge subblock mode may be 0 or the second flag (merge_subblock_flag[x0][y0]) indicating whether the merge subblock mode is applied may be 0.

[0282] In addition, based on the case where the CIIP mode is unavailable, the flag sps_ciip_enabled_flag from the SPS for enabling or disabling the CIIP mode may be 0 or the third flag (ciip_flag[x0][y0]) indicating whether the CIIP mode is applied may be 0.

[0283] In addition, based on the case where the split mode is not available, the flag sps_triangle_enabled_flag from the SPS for enabling or disabling the split mode may be 0 or the fourth flag (MergeTriangleFlag[x0][y0]) indicating whether the split mode is applied may be 0.

[0284] In addition, based on the case where the regular merge mode is not available, the fifth flag (regular_merge_flag[x0][y0]) indicating whether the regular merge mode is applied may be 0. That is, even when the value of the fifth flag is 0, the regular merge mode may be applied to the current block based on the case where the MMVD mode, the merge subblock mode, the CIIP mode, and the partition mode are not available.

[0285] In this case, motion information of the current block may be derived based on a first candidate among merge candidates included in a merge candidate list of the current block, and a prediction sample may be generated based on the derived motion information.

[0286] In another embodiment, the regular merge mode may be applied to the current block based on the unavailability of the regular merge mode, the MMVD mode, the merge sub-block mode, the CIIP mode, and the partition mode for performing prediction by dividing the current block into two partitions. That is, when the merge mode cannot be ultimately selected for the current block, the regular merge mode may be applied as the default merge mode.

[0287] For example, in a case where a value of a general merge flag indicating whether the merge mode is available for the current block is 1 but the merge mode is ultimately not selected for the current block, the normal merge mode may be applied as the default merge mode.

[0288] For example, based on the case where the MMVD mode is unavailable, the flag sps_mmvd_enabled_flag from the SPS for enabling or disabling the MMVD mode may be 0 or the first flag (mmvd_merge_flag[x0][y0]) indicating whether the MMVD mode is applied may be 0.

[0289] In addition, based on the case where the merge subblock mode is unavailable, the flag sps_affine_enabled_flag from the SPS for enabling or disabling the merge subblock mode may be 0 or the second flag (merge_subblock_flag[x0][y0]) indicating whether the merge subblock mode is applied may be 0.

[0290] In addition, based on the case where the CIIP mode is unavailable, the flag sps_ciip_enabled_flag from the SPS for enabling or disabling the CIIP mode may be 0 or the third flag (ciip_flag[x0][y0]) indicating whether the CIIP mode is applied may be 0.

[0291] In addition, based on the case where the split mode is not available, the flag sps_triangle_enabled_flag from the SPS for enabling or disabling the split mode may be 0 or the fourth flag (MergeTriangleFlag[x0][y0]) indicating whether the split mode is applied may be 0.

[0292] In addition, based on the case where the regular merge mode is not available, the fifth flag (regular_merge_flag[x0][y0]) indicating whether the regular merge mode is applied may be 0. That is, even when the value of the fifth flag is 0, the regular merge mode may be applied to the current block based on the case where the MMVD mode, the merge subblock mode, the CIIP mode, and the partition mode are not available.

[0293] In this case, the (0, 0) motion vector can be derived as the motion information of the current block, and the prediction sample of the current block can be generated based on the (0, 0) motion information. For the (0, 0) motion vector, prediction can be performed with reference to the 0th reference picture of the L0 reference list. However, when the 0th reference picture of the L0 reference list (RefPicList[0][0]) does not exist, prediction can be performed by referring to the 0th reference picture of the L1 reference list (RefPicList[1][0]).

[0294] Figure 17 and Figure 18 An example of a video / image encoding method and related components according to an embodiment of the present disclosure is schematically shown.

[0295] Figure 17 The method disclosed in Figure 2 or Figure 18 Specifically, for example, Figure 17 Steps S1700 to S1710 can be performed by Figure 18 The predictor 220 of the encoding device 200 performs, and Figure 17 Step S1720 can be performed by Figure 18 The residual processor 230 of the encoding device is executed. Figure 17 Step S1730 can be performed by Figure 18 The encoding is performed by the entropy encoder 240 of the encoding device. Figure 17 The method disclosed in may include the embodiments described above in the present disclosure.

[0296] Reference Figure 17, the encoding device may determine an inter prediction mode of the current block and generate inter prediction mode information indicating the inter prediction mode (S1700). For example, the encoding device may determine at least one of a normal merge mode, a skip mode, a motion vector prediction (MVP) mode, a merge mode with a motion vector difference (MMVD), a merge subblock mode, a CIIP mode (combined inter-picture merging and intra-picture prediction mode), and a partition mode for performing prediction by dividing the current block into two partitions as the inter prediction mode to be applied to the current block and generate the inter prediction mode information indicating the inter prediction mode.

[0297] The encoding device may generate a prediction sample of the current block based on the determined prediction mode (S1710).For example, the encoding device may generate a merge candidate list according to the determined inter prediction mode.

[0298] For example, candidates may be inserted into the merge candidate list until the number of candidates in the merge candidate list is the maximum number of candidates. Here, the candidate may indicate a candidate or candidate block for deriving motion information (or motion vector) of the current block. For example, the candidate block may be derived by searching adjacent blocks of the current block. For example, the adjacent blocks may include spatial adjacent blocks and / or temporal adjacent blocks of the current block, and spatial adjacent blocks may be preferentially searched (spatial merging) to derive the candidate, and the derived candidate may be inserted into the merge candidate list. For example, when the number of candidates in the merge candidate list (even after the candidate is inserted) is less than the maximum number of candidates in the merge candidate list, additional candidates may be inserted. For example, the additional candidates may include at least one of a history-based merge candidate, a pairwise average merge candidate, an ATMVP, and a combined bidirectional prediction merge candidate (when the slice / tile group type of the current slice / tile group is type B), and / or a zero vector merge candidate.

[0299] As described above, the merge candidate list may include at least some of spatial merge candidates, temporal merge candidates, pair candidates, or zero vector candidates, and one of these candidates may be selected for inter prediction of the current block.

[0300] For example, the selection information may include index information indicating one of the merge candidates included in the merge candidate list. For example, the selection information may be referred to as merge index information.

[0301] For example, the encoding device may generate a prediction sample of the current block based on the candidate indicated by the merge index information. Alternatively, for example, the encoding device may derive motion information based on the candidate indicated by the merge index information, and may generate a prediction sample of the current block based on the motion information.

[0302] Furthermore, according to an embodiment, the inter prediction mode information includes a general merge flag indicating whether the merge mode is available for the current block, and based on the general merge flag, the merge mode is available for the current block.

[0303] At this time, according to an embodiment, the conventional merge mode may be applied based on that the MMVD mode (merge mode with motion vector difference), the merge sub-block mode, the CIIP mode (combined inter-picture merging and intra-picture prediction mode), and the partitioning mode for performing prediction by dividing the current block into partitions are unavailable.

[0304] For example, the inter-frame prediction mode information may include merge index information indicating one of the merge candidates included in the merge candidate list generated by applying the normal merge mode, and the prediction sample may be generated using the merge index information. That is, the motion information of the current block may be derived based on the candidate indicated by the merge index information, and the prediction sample of the current block may be generated based on the derived motion information.

[0305] For example, the inter prediction mode information may include a first flag indicating whether the MMVD mode is applied, a second flag indicating whether the merge subblock mode is applied, and a third flag indicating whether the CIIP mode is applied.

[0306] For example, based on a case where the MMVD mode, the merging subblock mode, the CIIP mode, and the partitioning mode are unavailable, the values ​​of the first flag, the second flag, and the third flag may all be 0.

[0307] Furthermore, for example, based on a case where the value of the general merge flag is 1, the merge mode may be available for the current block, and based on a case where the general merge flag is 1, the values ​​of the first flag, the second flag, and the third flag may be signaled.

[0308] For example, a flag for enabling or disabling the segmentation mode may be included in a sequence parameter set (SPS) of image information, and based on the case of disabling the segmentation mode, the value of a fourth flag indicating whether the segmentation mode is applied may be set to 0.

[0309] In addition, the inter prediction mode information may further include a fifth flag indicating whether the normal merge mode is applied. Even when the value of the fifth flag is 0, the normal merge mode may be applied to the current block based on the case where the MMVD mode, the merge sub-block mode, the CIIP mode, and the partition mode are not available.

[0310] In this case, the motion information of the current block may be derived based on the first merge candidate among the merge candidates included in the merge candidate list of the current block. In addition, the prediction sample may be generated based on the motion information of the current block derived based on the first merge candidate.

[0311] Alternatively, in this case, motion information of the current block may be derived based on the (0, 0) motion vector, and a prediction sample may be generated based on the motion information of the current block derived based on the (0, 0) motion vector.

[0312] The encoding device may generate residual information based on the residual samples of the current block (S1720). For example, the encoding device may derive the residual samples based on the predicted samples and the original samples. For example, the encoding device may generate residual information indicating quantized transform coefficients of the residual samples. The residual information may be generated using various encoding methods such as Exponential Golomb, CAVLC, CABAC, etc.

[0313] The encoding device may encode image information including inter-frame prediction mode information (S1730). For example, the image information may be referred to as video information. According to the embodiments of the present disclosure described above, the image information may include various information. For example, the image information may include at least some of the prediction-related information or the residual-related information. For example, the prediction-related information may include at least some of the inter-frame prediction mode information, the selection information, and the inter-frame prediction type information. For example, the encoding device may encode all or part of the image information including the aforementioned information (or syntax elements) to generate a bitstream or encoding information. Alternatively, the encoding device may output the information in the form of a bitstream. In addition, the bitstream or encoding information may be sent to the decoding device via a network or a storage medium.

[0314] Alternatively, although Figure 17 Although not shown in the figure, for example, the encoding device may generate reconstructed samples based on the residual samples and the prediction samples. A reconstructed block and a reconstructed picture may be derived based on the reconstructed samples. Alternatively, for example, the encoding device may encode image information including residual information or prediction-related information.

[0315] For example, the encoding device may generate a bitstream or encoding information by encoding all or part of the image information including the aforementioned information (or syntax elements). Alternatively, the encoding device may output the information in the form of a bitstream. In addition, the bitstream or encoding information may be transmitted to the decoding device via a network or storage medium. Alternatively, the bitstream or encoding information may be stored in a computer-readable storage medium, and the bitstream or encoding information may be generated by the aforementioned image encoding method.

[0316] Figure 19 and Figure 20 An example of a video / image decoding method and related components according to an embodiment of the present disclosure is schematically shown.

[0317] Figure 19 The method disclosed in Figure 3 or Figure 20 The decoding device shown is executed. Specifically, for example, Figure 19 Step S1900 in Figure 20 The entropy decoder 310 of the decoding device 300 in the embodiment of the present invention performs the following steps: Figure 19 Step S1910 in the Figure 20 The residual processor 320 of the decoding device 300 is executed. In addition, Figure 19 Step S1920 can be performed by Figure 20 The predictor 330 of the decoding device 300 in the embodiment of the present invention performs Figure 19 Step S1930 can be performed by Figure 20 The adder 340 of the decoding device 300 is executed. Figure 19 The method disclosed in may include the embodiments described above in the present disclosure.

[0318] Reference Figure 19 , the decoding device may receive image information including inter-frame prediction mode information and residual information through a bitstream (S1900). For example, the image information may be referred to as video information. The image information may include various information according to the aforementioned embodiments of the present disclosure. For example, the image information may include at least a portion of prediction-related information or residual-related information.

[0319] For example, the prediction-related information may include inter-frame prediction mode information or inter-frame prediction type information. For example, the inter-frame prediction mode information may include information indicating at least some of the various inter-frame prediction modes. For example, various modes such as normal merge mode, skip mode, MVP (motion vector prediction) mode, MMVD mode (merge mode with motion vector difference), merge sub-block mode, CIIP mode (combined inter-picture merging and intra-picture prediction mode), and partition mode for performing prediction by dividing the current block into two partitions may be used. For example, the inter-frame prediction type information may include an inter_pred_idc syntax element. Alternatively, the inter-frame prediction type information may include information indicating any one of L0 prediction, L1 prediction, and bidirectional prediction.

[0320] The decoding apparatus may generate residual samples based on the residual information (S1910). The decoding apparatus may derive quantized transform coefficients based on the residual information, and may derive residual samples based on an inverse transform process for the transform coefficients.

[0321] The decoding device may generate a prediction sample of the current block by applying the prediction mode determined based on the inter-frame prediction mode information (S1910). For example, the decoding device may generate a merge candidate list based on the inter-frame prediction mode information as the inter-frame prediction mode of the current block, using the inter-frame prediction mode determined among the normal merge mode, skip mode, MVP mode, MMVD mode, merge sub-block mode, CIIP mode, and partition mode in which prediction is performed by dividing the current block into two partitions.

[0322] For example, candidates may be inserted into the merge candidate list until the number of candidates in the merge candidate list is the maximum number of candidates. Here, the candidate may indicate a candidate or candidate block for deriving motion information (or motion vector) of the current block. For example, the candidate block may be derived by searching adjacent blocks of the current block. For example, the adjacent blocks may include spatial adjacent blocks and / or temporal adjacent blocks of the current block, and spatial adjacent blocks may be preferentially searched (spatial merging) to derive the candidate, and the derived candidate may be inserted into the merge candidate list. For example, when the number of candidates in the merge candidate list (even after the candidate is inserted) is less than the maximum number of candidates in the merge candidate list, additional candidates may be inserted. For example, the additional candidates may include at least one of a history-based merge candidate, a pairwise average merge candidate, an ATMVP, and a combined bidirectional prediction merge candidate (when the slice / tile group type of the current slice / tile group is type B), and / or a zero vector merge candidate.

[0323] As described above, the merge candidate list may include at least some of spatial merge candidates, temporal merge candidates, pair candidates, or zero vector candidates, and one of these candidates may be selected for inter prediction of the current block.

[0324] For example, the selection information may include index information indicating one of the merge candidates included in the merge candidate list. For example, the selection information may be referred to as merge index information.

[0325] For example, the decoding device may generate a prediction sample of the current block based on the candidate indicated by the merge index information. Alternatively, for example, the decoding device may derive motion information based on the candidate indicated by the merge index information, and may generate a prediction sample of the current block based on the motion information.

[0326] In addition, according to an embodiment, the inter prediction mode information may include a general merge flag indicating whether the merge mode is available for the current block, and the merge mode may be available for the current block based on the general merge flag.

[0327] Here, based on the unavailability of the merge mode with motion vector difference (MMVD) mode, the merge subblock mode, the combined inter-picture merging and intra-picture prediction (CIIP) mode, and the partition mode for performing prediction by dividing the current block into two partitions, the normal merge mode can be applied.

[0328] For example, the inter-frame prediction mode information may include merge index information indicating one of the merge candidates included in the merge candidate list generated by applying the normal merge mode, and the prediction sample may be generated using the merge index information. That is, the motion information of the current block may be derived based on the candidate indicated by the merge index information, and the prediction sample of the current block may be generated based on the derived motion information.

[0329] For example, the inter prediction mode information may include a first flag indicating whether the MMVD mode is applied, a second flag indicating whether the merge subblock mode is applied, and a third flag indicating whether the CIIP mode is applied.

[0330] For example, based on a case where the MMVD mode, the merging subblock mode, the CIIP mode, and the partitioning mode are unavailable, the values ​​of the first flag, the second flag, and the third flag may all be 0.

[0331] Furthermore, for example, based on a case where the value of the general merge flag is 1, the merge mode may be enabled for the current block, and based on a case where the value of the general merge flag is 1, the values ​​of the first flag, the second flag, and the third flag may be signaled.

[0332] For example, a flag for enabling or disabling the segmentation mode may be included in a sequence parameter set (SPS) of image information, and based on the case of disabling the segmentation mode, the value of a fourth flag indicating whether the segmentation mode is applied may be set to 0.

[0333] In addition, the inter prediction mode information may further include a fifth flag indicating whether the normal merge mode is applied. Even when the value of the fifth flag is 0, the normal merge mode may be applied to the current block based on the case where the MMVD mode, the merge sub-block mode, the CIIP mode, and the partition mode are not available.

[0334] In this case, the motion information of the current block may be derived based on the first merge candidate among the merge candidates included in the merge candidate list of the current block. In addition, the prediction sample may be generated based on the motion information of the current block derived based on the first merge candidate.

[0335] Alternatively, in this case, motion information of the current block may be derived based on the (0, 0) motion vector, and a prediction sample may be generated based on the motion information of the current block derived based on the (0, 0) motion vector.

[0336] The decoding device may generate a reconstructed sample based on the prediction sample (S1930). For example, the decoding device may generate a reconstructed sample based on the prediction sample and the residual sample, and may derive a reconstructed block and a reconstructed picture based on the reconstructed sample.

[0337] For example, a decoding device can obtain video / image information including all or part of the above-mentioned multiple pieces of information (or syntax elements) by decoding a bitstream or coding information. In addition, the bitstream or coding information can be stored in a computer-readable storage medium and can cause the above-mentioned decoding method to be executed.

[0338] Although the method has been described based on a flowchart that lists steps or blocks in sequence in the above embodiments, the steps of this document are not limited to a specific order, and specific steps may be performed in different steps or in a different order or simultaneously with respect to the above steps. In addition, it will be understood by those skilled in the art that the steps in the flowchart are not exclusive and that another step may be included therein or one or more steps in the flowchart may be deleted without affecting the scope of the present disclosure.

[0339] The above-mentioned method according to the present disclosure may be in the form of software, and the encoding device and / or decoding device according to the present disclosure may be included in an apparatus for performing image processing (e.g., TV, computer, smart phone, set-top box, display device, etc.).

[0340] When the embodiments of the present disclosure are implemented with software, the above-mentioned methods can be implemented with modules (processing or functions) that perform the above-mentioned functions. The modules can be stored in a memory and executed by a processor. The memory can be installed inside or outside the processor and can be connected to the processor via various well-known devices. The processor may include an application-specific integrated circuit (ASIC), other chipsets, logic circuits and / or data processing devices. The memory may include a read-only memory (ROM), a random access memory (RAM), a flash memory, a memory card, a storage medium and / or other storage devices. In other words, according to the embodiments of the present disclosure, it can be implemented and executed on a processor, a microprocessor, a controller or a chip. For example, the functional units illustrated in the corresponding figures can be implemented and executed on a computer, a processor, a microprocessor, a controller or a chip. In this case, information about the implementation (for example, information about instructions) or the algorithm can be stored in a digital storage medium.

[0341] In addition, the decoding device and the encoding device to which the embodiments of this document are applied may be included in a multimedia broadcast transceiver, a mobile communication terminal, a home theater video device, a digital theater video device, a surveillance camera, a video chat device, a real-time communication device such as video communication, a mobile streaming device, a storage medium, a portable camera, a video on demand (VoD) service provider, an over-the-top (OTT) video device, an Internet streaming service provider, a 3D video device, a virtual reality (VR) device, an augmented reality (AR) device, an image phone video device, an in-vehicle terminal (e.g., an in-vehicle (including an autonomous vehicle) terminal, an airplane terminal, or a ship terminal), and a medical video device; and may be used to process image signals or data. For example, an OTT video device may include a game console, a Blueray player, an Internet-connected TV, a home theater system, a smartphone, a tablet PC, and a digital video recorder (DVR).

[0342] In addition, the processing method of the embodiment of the present document can be generated in the form of a program executed by a computer and can be stored in a computer-readable recording medium. The multimedia data with a data structure according to the embodiment of the present document can also be stored in a computer-readable recording medium. The computer-readable recording medium includes all kinds of storage devices and distributed storage devices storing computer-readable data. The computer-readable recording medium may include, for example, a Blu-ray disc (BD), a universal serial bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. The computer-readable recording medium also includes a medium implemented in the form of a carrier wave (e.g., transmission on the Internet). In addition, the bit stream generated by the encoding method can be stored in a computer-readable recording medium or can be transmitted through a wired or wireless communication network.

[0343] In addition, the embodiments of this document can be implemented as a computer program product based on a program code, and the program code can be executed on a computer according to the embodiments of this document. The program code can be stored on a computer-readable carrier.

[0344] Figure 21 represents an example of a content streaming system to which embodiments of this document may be applied.

[0345] Reference Figure 21 A content streaming system to which embodiments of this document are applied may generally include an encoding server, a streaming server, a network server, a media storage, a user device, and a multimedia input device.

[0346] The encoding server is used to compress content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data, generate a bitstream, and transmit it to the streaming server. As another example, if the multimedia input device such as smartphones, cameras, and camcorders directly generates the bitstream, the encoding server can be omitted.

[0347] The bitstream may be generated by an encoding method or a bitstream generation method to which the embodiments of this document are applied, and the streaming server may temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0348] The streaming server transmits multimedia data to user devices via a network server based on user requests. The network server serves as a tool for notifying users of available services. When a user requests a desired service, the network server transfers the request to the streaming server, which then transmits the multimedia data to the user. In this regard, the content streaming system may include a separate control server, which in this case is used to control commands and responses between the various devices in the content streaming system.

[0349] The streaming server may receive content from a media storage device and / or an encoding server. For example, when receiving content from an encoding server, the content may be received in real time. In this case, the streaming server may store the bitstream for a predetermined period of time to smoothly provide a streaming service.

[0350] For example, user devices may include mobile phones, smart phones, laptop computers, digital broadcast terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation, tablet PCs, tablet PCs, ultrabooks, wearable devices (e.g., watch-type terminals (smart watches), glasses-type terminals (smart glasses), head-mounted displays (HMDs)), digital TVs, desktop computers, digital signage, etc.

[0351] Each server in the content streaming system may be operated as a distributed server, and in this case, data received by each server may be processed in a distributed manner.

[0352] The claims in this specification may be combined in various ways. For example, the technical features in the method claims of this specification may be combined to be implemented or performed in a device, and the technical features in the device claims may be combined to be implemented or performed in a method. Furthermore, the technical features in method and device claims may be combined to be implemented or performed in a device. Furthermore, the technical features in method and device claims may be combined to be implemented or performed in a method.

Claims

1. An image decoding method performed by a decoding device, the image decoding method comprising the following steps: Acquire image information including inter-frame prediction mode information and residual information through a bit stream; generating residual samples based on the residual information; generating a prediction sample of the current block by applying a prediction mode determined based on the inter-frame prediction mode information; as well as generating a reconstructed sample based on the predicted sample and the residual sample, The inter-frame prediction mode information includes a first flag related to whether a sub-block merge mode is applied, a second flag related to whether a normal merge mode is applied, and a third flag related to whether a merge mode with a motion vector difference (MMVD) is applied. The parsing order of the second flag is before the parsing order of the third flag. wherein a CIIP mode based on combined inter-picture merging and intra-picture prediction is not enabled, a partition prediction mode for performing prediction by dividing the current block into two partitions is not enabled, a value of the first flag is equal to 0, and a value of the third flag is equal to 0, the inter-frame prediction mode information includes merge index information indicating one of the merge candidates included in the merge candidate list generated for the regular merge mode, wherein the predicted sample is generated based on the merge index information, and Wherein, based on the value of the general merge flag being 1 and the MMVD being not enabled, the normal merge mode is applied to the current block.

2. An image encoding method performed by an encoding device, the image encoding method comprising the following steps: determining an inter prediction mode of a current block and generating inter prediction mode information indicating the inter prediction mode; Generating a prediction sample of the current block based on the determined prediction mode; generating residual information based on residual samples for the current block; as well as encoding the image information including the inter-frame prediction mode information and the residual information, The inter-frame prediction mode information includes a first flag related to whether a sub-block merge mode is applied, a second flag related to whether a normal merge mode is applied, and a third flag related to whether a merge mode with a motion vector difference (MMVD) is applied. The parsing order of the second flag is before the parsing order of the third flag. wherein a CIIP mode based on combined inter-picture merging and intra-picture prediction is not enabled, a partition prediction mode for performing prediction by dividing the current block into two partitions is not enabled, a value of the first flag is equal to 0 and a value of the third flag is equal to 0, the inter-prediction mode information includes merge index information indicating one of the merge candidates included in the merge candidate list generated for the normal merge mode, and Wherein, based on the value of the general merge flag being 1 and the MMVD being not enabled, the normal merge mode is applied to the current block.

3. A method for sending image data, the sending method comprising the following steps: Obtaining a bitstream for the image, wherein the bitstream is generated based on the following steps: determining an inter-frame prediction mode of a current block and generating inter-frame prediction mode information indicating the inter-frame prediction mode, generating a prediction sample of the current block based on the determined prediction mode, generating residual information based on a residual sample of the current block, and encoding image information including the inter-frame prediction mode information and the residual information; and sending said data comprising said bit stream, The inter-frame prediction mode information includes a first flag related to whether a sub-block merge mode is applied, a second flag related to whether a normal merge mode is applied, and a third flag related to whether a merge mode with a motion vector difference (MMVD) is applied. The parsing order of the second flag is before the parsing order of the third flag. wherein a CIIP mode based on combined inter-picture merging and intra-picture prediction is not enabled, a partition prediction mode for performing prediction by dividing the current block into two partitions is not enabled, a value of the first flag is equal to 0 and a value of the third flag is equal to 0, the inter-prediction mode information includes merge index information indicating one of the merge candidates included in the merge candidate list generated for the normal merge mode, and Wherein, based on the value of the general merge flag being 1 and the MMVD being not enabled, the normal merge mode is applied to the current block.

Images