Image decoding apparatus, image encoding apparatus, and image data transmission apparatus

Abstract

The invention provides an image decoding apparatus, an image encoding apparatus, and an image data transmission apparatus. According to an embodiment of the present document, when an inter prediction type of a current block indicates bidirectional prediction, weight index information of candidates in a merge candidate list or a sub-block merge candidate list may be derived, and thus coding efficiency may be increased.

Description

[0001] This application is a divisional application of the invention patent application with the original application number 202080059109.6 (International application number: PCT / KR2020 / 007943, application date: June 19, 2020, invention name: image decoding method and device for generating prediction samples by applying a determined prediction mode). Technical Field

[0002] The present disclosure relates to an image decoding method and an apparatus thereof for generating a prediction sample by applying a determined prediction mode. Background Art

[0003] Recently, there is an increasing demand for high-resolution, high-quality images / videos such as 4K or 8K ultra-high definition (UHD) images / videos in various fields. As the image / video resolution or quality becomes higher, relatively more information or bits are transmitted compared to conventional image / video data. Therefore, if the image / video data is transmitted via a medium such as an existing wired / wireless broadband line or is stored in a conventional storage medium, the cost of transmission and storage is easily increased.

[0004] In addition, there is growing interest and demand for virtual reality (VR) and artificial reality (AR) content and immersive media such as holograms; and there is also growing broadcasting of images / videos that exhibit image / video characteristics that are different from actual images / videos (e.g., game images / videos).

[0005] Therefore, highly efficient image / video compression technology is required to effectively compress and transmit, store, or play high-resolution, high-quality images / videos showing various characteristics as described above. Summary of the invention

[0006] Technical issues

[0007] The present disclosure provides a method and an apparatus for improving image coding efficiency.

[0008] The present disclosure also provides a method and apparatus for deriving a prediction sample based on a default merge mode when the merge mode is not ultimately selected.

[0009] The present disclosure also provides a method and apparatus for deriving a prediction sample by applying a normal merge mode as a default merge mode.

[0010] Technical Solution

[0011] In one aspect, an image decoding method performed by a decoding device is provided. The method includes: acquiring image information including inter-frame prediction mode information and residual information through a bitstream; generating residual samples based on the residual information; generating prediction samples of a current block by applying a prediction mode determined based on the inter-frame prediction mode information; and generating reconstructed samples based on the prediction samples and the residual samples, wherein the inter-frame prediction mode information includes a general merge flag indicating whether a merge mode is available for the current block, a merge mode MMVD mode, a merge sub-block mode, a combined inter-picture merge and intra-picture prediction CIIP mode, and a partitioning mode (partitioning mode) for performing prediction by dividing the current block into two partitions based on the merge mode are not available, a regular merge mode is applied, the inter-frame prediction mode information includes merge index information indicating one of the merge candidates included in the merge candidate list generated by applying the regular merge mode, and the prediction sample is generated using the merge index information.

[0012] In another aspect, an image encoding method performed by an encoding device is provided. The method includes: determining an inter-frame prediction mode of a current block and generating inter-frame prediction mode information indicating the inter-frame prediction mode; generating a prediction sample of the current block based on the determined prediction mode; generating residual information based on the residual sample of the current block; and encoding image information including the inter-frame prediction mode information and the residual information, wherein the inter-frame prediction mode information includes a general merge flag indicating whether a merge mode is available for the current block, and applying a conventional merge mode based on the fact that a merge mode with a motion vector difference (MMVD) mode, a merge sub-block mode, a combined inter-picture merge and intra-picture prediction (CIIP) mode, and a partition mode for performing prediction by dividing the current block into two partitions are not available.

[0013] The inter prediction mode information includes merge index information indicating one of merge candidates included in a merge candidate list generated by applying the normal merge mode.

[0014] In another aspect, a computer-readable storage medium is provided for storing encoding information that enables an image decoding device to perform an image decoding method, wherein the image decoding method includes: obtaining image information including inter-frame prediction mode information and residual information through a bit stream; generating residual samples based on the residual information; generating prediction samples of a current block by applying a prediction mode determined based on the inter-frame prediction mode information; and generating reconstructed samples based on the prediction samples and the residual samples, wherein the inter-frame prediction mode information includes a general merge flag indicating whether a merge mode is available for the current block, a merge mode MMVD mode, a merge sub-block mode, a combined inter-picture merge and intra-picture prediction CIIP mode, and a partition mode for performing prediction by dividing the current block into two partitions are not available based on the merge mode, and a conventional merge mode is applied.

[0015] The inter prediction mode information includes merge index information indicating one candidate among merge candidates included in a merge candidate list generated by applying the normal merge mode, and a prediction sample is generated using the merge index information.

[0016] Technical Effects

[0017] According to the present disclosure, the overall image / video compression efficiency can be improved.

[0018] According to the present disclosure, when a merge mode is not finally selected, inter prediction may be efficiently performed by applying a default merge mode.

[0019] According to the present disclosure, when the merge mode is not finally selected, the normal merge mode is applied and motion information is derived based on the candidate indicated by the merge index information, thereby efficiently performing inter prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 An example of a video / image encoding system to which an embodiment of the present disclosure is applied is schematically shown.

[0021] Figure 2 is a diagram schematically illustrating a configuration of a video / image encoding device to which an embodiment of the present document can be applied.

[0022] Figure 3 is a diagram schematically illustrating a configuration of a video / image decoding device to which an embodiment of the present document can be applied.

[0023] Figure 4 is a diagram illustrating an example of a video / image encoding method based on inter-frame prediction.

[0024] Figure 5 is a diagram schematically illustrating an inter-frame predictor in an encoding device.

[0025] Figure 6 is a diagram illustrating an example of a video / image decoding method based on inter-frame prediction.

[0026] Figure 7 is a diagram schematically illustrating an inter-frame predictor in a decoding device.

[0027] Figure 8 This is a diagram illustrating the merge mode in inter-frame prediction.

[0028] Fig. 9 is a diagram illustrating a merge mode with motion vector difference (MMVD) mode in inter prediction.

[0029] Fig.10a and Fig.10b CPMV for affine motion prediction is shown as an example.

[0030] Fig.11 A case where the affine MVF is determined in units of sub-blocks is exemplarily shown.

[0031] Fig.12 The diagram illustrates the affine merge mode or the subblock merge mode in inter prediction.

[0032] Fig.13 is a diagram illustrating positions of candidates in the affine merge mode or the subblock merge mode.

[0033] Fig.14 This is a diagram illustrating SbTMVP in inter-frame prediction.

[0034] Fig.15 is a diagram illustrating a combined inter-picture merging and intra-picture prediction (CIIP) mode in inter-frame prediction.

[0035] Fig.16 This is a diagram illustrating a partition mode in inter-frame prediction.

[0036] Fig.17 and Fig.18 An example of a video / image encoding method and related components according to an embodiment of the present disclosure is schematically shown.

[0037] Fig.19 and Fig. 20 An example of an image / video decoding method and related components according to an embodiment of the present disclosure is schematically shown.

[0038] Fig.21 An example of a content streaming system to which the embodiments disclosed in the present disclosure can be applied is shown. DETAILED DESCRIPTION

[0039] The present disclosure can be modified differently in various forms, and can have various embodiments, and its specific embodiments will be illustrated in the drawings and described in detail. However, these embodiments are not intended to limit the present disclosure. The terms used in the following description are only used to describe specific embodiments, and are not intended to limit the technical spirit of the present disclosure. The expression of the singular includes the expression of the plural number, as long as it is clearly interpreted differently. In this specification, terms such as "including" and "having" are intended to indicate that the features, quantities, steps, operations, elements, components, or combinations thereof used in the following description exist, and it should be understood that the possibility of the presence or addition of one or more different features, quantities, steps, operations, elements, components, or combinations thereof is not excluded.

[0040] In addition, for the convenience of illustrating different specific functions, the elements of the drawings described in the present disclosure are drawn independently, and it is not meant that the elements are implemented by independent hardware or independent software. For example, two or more elements in the elements can be combined to form a single element, or an element can be separated into multiple elements. Embodiments in which elements are combined and / or separated belong to the scope of the present disclosure.

[0041] This document relates to video / image coding. For example, the methods / implementations disclosed in this document may be applied to methods disclosed in the Versatile Video Coding (VVC) standard. In addition, the methods / implementations disclosed in this document may be applied to methods disclosed in the Basic Video Coding (EVC) standard, the AO Media Video 1 (AV1) standard, the second generation audio video coding standard (AVS2), or the next generation video / image coding standard (e.g., H.267 or H.268, etc.).

[0042] This document presents various embodiments related to video / image coding, and unless otherwise stated, the embodiments can be combined with each other.

[0043] Hereinafter, embodiments of the present document will be described with reference to the accompanying drawings. Hereinafter, the same reference numerals may be used for the same components in the accompanying drawings, and repeated description of the same components may be omitted.

[0044] Figure 1 An example of a video / image encoding system to which an embodiment of the present disclosure can be applied is illustrated.

[0045] Reference Figure 1 The video / image coding system may include a first device (source device) and a second device (receiving device). The source device may send the coded video / image information or data to the receiving device in the form of a file or stream transmission via a digital storage medium or a network.

[0046] The source device may include a video source, an encoding device, and a transmitter. The receiving device may include a receiver, a decoding device, and a renderer. The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display, and the display may be configured as a separate device or an external component.

[0047] The video source may acquire the video / image by capturing, synthesizing or generating a video / image process. The video source may include a video / image capturing device, and / or a video / image generating device. For example, the video / image capturing device may include one or more cameras, a video / image archive including previously captured videos / images, etc. For example, the video / image generating device may include a computer, a tablet computer, and a smart phone, and may (electronically) generate the video / image. For example, a virtual video / image may be generated by a computer, etc. In this case, the video / image capturing process may be replaced by a process that generates relevant data.

[0048] The encoding device can encode the input video / image. For compression and encoding efficiency, the encoding device can perform a series of processes such as prediction, transformation and quantization. The encoded data (encoded video / image information) can be output in the form of a bit stream.

[0049] The transmitter may transmit the encoded image / image information or data output in the form of a bit stream to a receiver of a receiving device in the form of a file or stream through a digital storage medium or a network. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter may include an element for generating a media file in a predetermined file format, and may include an element for transmitting through a broadcast / communication network. The receiver may receive / extract a bit stream and transmit the received bit stream to a decoding device.

[0050] The decoding device may decode a video / image by performing a series of processes such as dequantization, inverse transformation, and prediction corresponding to the operations of the encoding device.

[0051] The renderer may render the decoded video / image. The rendered video / image may be displayed through a display.

[0052] The present disclosure relates to video / image coding. For example, the methods / implementations disclosed in the present disclosure may be applied to methods disclosed in the Versatile Video Coding (VVC) standard, the Essential Video Coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the second generation audio video coding standard (AVS2), or the next generation video / image coding standard (e.g., H.267 or H.268, etc.).

[0053] This document proposes various embodiments of video / image coding, and unless otherwise mentioned, the above embodiments may also be performed in combination with each other.

[0054] In this document, video may refer to a series of images over time. A picture generally refers to a unit that represents an image at a specific time frame, and a slice / tile refers to a unit that constitutes a part of a picture from a coding perspective. A slice / tile may include one or more coding tree units (CTUs). A picture may consist of one or more slices / tiles.

[0055] A tile is a rectangular area of ​​a CTU within a specific tile column and a specific tile row in a picture. A tile column is a rectangular area of ​​a CTU whose height is equal to the height of the picture and whose width is specified by a syntax element in a picture parameter set. A tile row is a rectangular area of ​​a CTU whose height is specified by a syntax element in a picture parameter set and whose width is equal to the width of the picture. Tile scanning is a specific sequential ordering of the CTUs of a partitioned picture, where the CTUs are continuously ordered in a CTU raster scan in tiles, and the tiles in a picture are continuously ordered in a raster scan of the tiles of the picture. A slice may include multiple complete tiles of a picture that may be included in one NAL unit or multiple consecutive CTU rows in a tile. In this document, tile groups and slices may be used interchangeably. For example, in this document, a tile group / tile group header may be referred to as a slice / slice header.

[0056] In addition, a picture can be divided into two or more sub-pictures. A sub-picture can be a rectangular area of ​​one or more slices within a picture.

[0057] A pixel or a pel may refer to the smallest unit constituting a picture (or image). In addition, a "sample" may be used as a term corresponding to a pixel. A sample may generally represent a pixel or a value of a pixel, and may represent only a pixel / pixel value of a luminance component, or only a pixel / pixel value of a chrominance component.

[0058] A unit may represent a basic unit of image processing. A unit may include at least one of a specific area of ​​a picture and information related to the area. A unit may include a luminance block and two chrominance (e.g., cb, cr) blocks. In some cases, a unit may be used interchangeably with terms such as a block or an area. In general, an M×N block may include M columns and N rows of samples (or sample arrays) or a set (or array) of transform coefficients. Alternatively, a sample may refer to a pixel value in a spatial domain, and when such a pixel value is transformed to a frequency domain, it may refer to a transform coefficient in a frequency domain.

[0059] In this document, "A or B (A or B)" may mean "only A", "only B", or "both A and B". In other words, "A or B (A or B)" in this document may be interpreted as "A and / or B (A and / or B)". For example, in this document, "A, B or C (A, B or C)" means "only A", "only B", "only C", or "any combination of A, B, and C".

[0060] As used in this document, a slash ( / ) or a comma may mean "and / or". For example, "A / B" may mean "A and / or B". Thus, "A / B" may mean "only A", "only B", or "both A and B". For example, "A, B, C" may mean "A, B, or C".

[0061] In this document, "at least one of A and B" may mean "only A", "only B", or "both A and B". In addition, in this document, the expression "at least one of A or B" or "at least one of A and / or B" may be equally interpreted as "at least one of A and B".

[0062] Furthermore, in this document, “at least one of A, B, and C” means “only A”, “only B”, “only C”, or “any combination of A, B, and C”. Furthermore, “at least one of A, B, or C” or “at least one of A, B and / or C” may mean “at least one of A, B, and C”.

[0063] In addition, brackets used in this document may indicate "for example". Specifically, when "prediction (intra-frame prediction)" is indicated, it may be referred to as "intra-frame prediction" as an example of "prediction". In other words, "prediction" in this document is not limited to "intra-frame prediction", and "intra-frame prediction" may be proposed as an example of "prediction". In addition, even when "prediction (i.e., intra-frame prediction)" is indicated, it may be referred to as "intra-frame prediction" as an example of "prediction".

[0064] Technical features described individually in one drawing in this document may be implemented individually or simultaneously.

[0065] Figure 2 Schematically illustrates a configuration of a video / image encoding device to which an embodiment of the present document can be applied. Hereinafter, the so-called encoding device may include an image encoding device and / or a video encoding device. In addition, the so-called image encoding method / device may include a video encoding method / device. Alternatively, the so-called video encoding method / device may include an image encoding method / device.

[0066] Reference Figure 2, the encoding device 200 includes an image divider 210, a predictor 220, a residual processor 230 and an entropy encoder 240, an adder 250, a filter 260 and a memory 270. The predictor 220 may include an inter-frame predictor 221 and an intra-frame predictor 222. The residual processor 230 may include a transformer 232, a quantizer 233, a dequantizer 234 and an inverse transformer 235. The residual processor 230 may also include a subtractor 231. The adder 250 may be referred to as a reconstructor or a reconstructed block generator. According to an embodiment, the image divider 210, the predictor 220, the residual processor 230, the entropy encoder 240, the adder 250 and the filter 260 may be configured by at least one hardware component (e.g., an encoder chipset or processor). In addition, the memory 270 may include a decoded picture buffer (DPB), or may be configured by a digital storage medium. The hardware component may also include the memory 270 as an internal / external component.

[0067] The image divider 210 may divide the input image (or picture or frame) input to the encoding device 200 into one or more processors. For example, the processor may be referred to as a coding unit (CU). In this case, the coding unit may be recursively divided from a coding tree unit (CTU) or a maximum coding unit (LCU) according to a quadtree binary tree ternary tree (QTBTTT) structure. For example, one coding unit may be divided into a plurality of coding units with a deeper depth based on a quadtree structure, a binary tree structure, and / or a ternary structure. In this case, for example, a quadtree structure may be applied first, and a binary tree structure and / or a ternary structure may be applied later. Alternatively, a binary tree structure may be applied first. The encoding process according to the present disclosure may be performed based on a final coding unit that is no longer divided. In this case, the maximum coding unit may be used as the final coding unit based on coding efficiency, etc. according to image characteristics, or if necessary, the coding unit may be recursively divided into coding units with a deeper depth and a coding unit with an optimal size may be used as the final coding unit. Here, the encoding process may include a process of prediction, transformation, and reconstruction (to be described later). As another example, the processor may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may be split or divided from the above-mentioned final coding unit. The prediction unit may be a unit for sample prediction, and the transform unit may be a unit for deriving a transform coefficient and / or a unit for deriving a residual signal from the transform coefficient.

[0068] In some cases, a unit may be used interchangeably with terms such as a block or region. In general, an M×N block may represent a set of samples or transform coefficients consisting of M columns and N rows. A sample may generally represent a pixel or a pixel value, and may represent only a pixel / pixel value of a luminance component or only a pixel / pixel value of a chrominance component. A sample may be used as a term corresponding to one picture (or image) of a pixel or a pixel element.

[0069] The encoding device 200 can subtract the prediction signal (prediction block, prediction sample array) output from the inter predictor 221 or the intra predictor 222 from the input image signal (original block, original sample array) to generate a residual signal (residual block, residual sample array), and the generated residual signal is sent to the transformer 232. In this case, as illustrated, the unit for subtracting the prediction signal (prediction block, prediction sample array) from the input image signal (original block, original sample array) in the encoding device 200 can be referred to as a subtractor 231. The predictor can perform prediction on the processing target block (hereinafter, referred to as the current block) and generate a prediction block including the prediction sample of the current block. The predictor can determine whether to apply intra prediction or inter prediction in units of the current block or CU. The predictor can generate various information about prediction such as prediction mode information, and send the generated information to the entropy encoder 240, as described below when describing each prediction mode. The information about the prediction can be encoded by the entropy encoder 240 and output in the form of a bitstream.

[0070] The intra-frame predictor 222 may predict the current block with reference to samples in the current picture. Depending on the prediction mode, the referenced samples may be located near the current block or may be spaced apart. In intra-frame prediction, the prediction mode may include multiple non-directional modes and multiple directional modes. For example, the non-directional mode may include a DC mode and a plane mode. For example, depending on the level of detail of the prediction direction, the directional mode may include 33 directional prediction modes or 65 directional prediction modes. However, this is only an example, and more or fewer directional prediction modes may be used depending on the settings. The intra-frame predictor 222 may use the prediction mode applied to the adjacent blocks to determine the prediction mode applied to the current block.

[0071] The inter-frame predictor 221 may derive a prediction block of the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. Here, in order to reduce the amount of motion information sent in the inter-frame prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the adjacent blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may also include inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter-frame prediction, the adjacent blocks may include spatial adjacent blocks present in the current picture and temporal adjacent blocks present in the reference picture. The reference picture including the reference block and the reference picture including the temporal adjacent block may be the same or different. The temporal adjacent block may be referred to as a collocated reference block, a collocated CU (colCU), etc., and the reference picture including the temporal adjacent block may be referred to as a collocated picture (colPic). For example, the inter-frame predictor 221 may configure a motion information candidate list based on the adjacent blocks and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter-frame prediction may be performed based on various prediction modes. For example, in the case of skip mode and merge mode, the inter-frame predictor 221 may use the motion information of the neighboring blocks as the motion information of the current block. In skip mode, unlike merge mode, a residual signal may not be sent. In the case of motion vector prediction (MVP) mode, the motion vector of the neighboring blocks may be used as a motion vector predictor, and the motion vector of the current block may be indicated by signaling a motion vector difference.

[0072] The predictor 220 may generate a prediction signal based on various prediction methods described below. For example, the predictor may apply intra prediction or inter prediction to the prediction of a block, and may apply intra prediction and inter prediction at the same time. This may be referred to as a combination of inter and intra prediction (CIIP). In addition, the predictor may predict a block based on an intra-block copy (IBC) prediction mode or based on a palette mode. The IBC prediction mode or the palette mode may be used for image / video encoding of content such as games, for example, screen content coding (SCC). IBC basically performs prediction within the current picture, but may be performed similarly to inter prediction in terms of deriving a reference block within the current picture. That is, IBC may use at least one of the inter prediction techniques described in this document. The palette mode may be considered an example of intra coding or intra prediction. When the palette mode is applied, the sample values ​​in the picture may be signaled based on information about the palette table and the palette index.

[0073] The prediction signal generated by the predictor (including the inter-frame predictor 221 and / or the intra-frame predictor 222) can be used to generate a reconstruction signal or can be used to generate a residual signal. The transformer 232 can generate a transform coefficient by applying a transform technique to the residual signal. For example, the transform technique may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen-Loeve transform (KLT), a graph-based transform (GBT), or a conditional nonlinear transform (CNT). Here, GBT refers to a transform obtained from a curve graph when the relationship information between pixels is represented by a curve graph. CNT refers to a transform obtained based on a prediction signal generated using all previously reconstructed pixels. In addition, the transform process can be applied to a block of pixels of the same size as a square, or can be applied to a block of variable size rather than a square.

[0074] The quantizer 233 quantizes the transform coefficient and sends it to the entropy encoder 240, and the entropy encoder 240 encodes the quantized signal (information about the quantized transform coefficient) and outputs the encoded signal as a bitstream. Information about the quantized transform coefficient may be referred to as residual information. The quantizer 233 may rearrange the quantized transform coefficients in block form into a one-dimensional vector form based on the coefficient scanning order, and may generate information about the transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. The entropy encoder 240 may perform various encoding methods such as (for example) exponential Golomb, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC). The entropy encoder 240 may encode information necessary for video / image reconstruction (for example, the value of a syntax element, etc.) other than the quantized transform coefficients together or separately. The encoded information (for example, the encoded video / image information) may be transmitted or stored in units of network abstraction layer (NAL) units in the form of a bitstream. The video / image information may also include information about various parameter sets such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may also include conventional constraint information. In this document, information and / or syntax elements sent / signaled to a decoding device from an encoding device may be included in the video / image information. The video / image information may be encoded and included in a bitstream through the above-mentioned encoding process. The bitstream may be sent through a network, or may be stored in a digital storage medium. Here, the network may include a broadcast network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. A sending unit (not shown) and / or a storage unit (not shown) for sending or storing a signal output from the entropy encoder 240 may be configured as an internal / external element of the encoding device 200, or the sending unit may be included in the entropy encoder 240.

[0075] The quantized transform coefficients output from the quantizer 233 can be used to generate a prediction signal. For example, the residual signal (residual block or residual sample) can be reconstructed by applying dequantization and inverse transformation to the quantized transform coefficients using a dequantizer 234 and an inverse transformer 235. The adder 250 can add the reconstructed residual signal to the prediction signal output from the inter-frame predictor 221 or the intra-frame predictor 222 to generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array). When there is no residual of the processing target block, such as when a skip mode is applied, the prediction block can be used as a reconstructed block. The adder 250 can be referred to as a recovery unit or a recovery block generator. The generated reconstructed signal can be used for intra-frame prediction of the next processing target block in the current picture, and can be used for inter-frame prediction of the next picture after filtering as described below.

[0076] Furthermore, luma mapping with chroma scaling (LMCS) may be applied during the picture encoding and / or reconstruction process.

[0077] The filter 260 can improve the subjective / objective image quality by applying filtering to the reconstructed signal. For example, the filter 260 can generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture, and store the modified reconstructed picture in the memory 270, specifically, in the DPB of the memory 270. Various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filter 260 can generate various types of information related to filtering, and transmit the generated information to the entropy encoder 240, as described later when each filtering method is described. The information related to filtering can be encoded by the entropy encoder 240 and output in the form of a bit stream.

[0078] The modified reconstructed picture transmitted to the memory 270 may be used as a reference picture in the inter predictor 221. When inter prediction is applied by the encoding device, prediction mismatch between the encoding device 200 and the decoding device may be avoided, and encoding efficiency may be improved.

[0079] The DPB of the memory 270 may store the modified reconstructed picture for use as a reference picture in the inter-frame predictor 221. The memory 270 may store the motion information of the block from which the motion information in the current picture is derived (or encoded) and / or the motion information of the block in the reconstructed picture. The stored motion information may be transmitted to the inter-frame predictor 221 to be used as the motion information of the spatial neighboring block or the motion information of the temporal neighboring block. The memory 270 may store the reconstructed samples of the reconstructed block in the current picture and may transmit the reconstructed samples to the intra-frame predictor 222.

[0080] In addition, in this document, at least one of quantization / dequantization and / or transform / inverse transform may be omitted. When quantization / dequantization is omitted, the quantized transform coefficient may be referred to as a transform coefficient. When transform / inverse transform is omitted, the transform coefficient may be referred to as a coefficient or a residual coefficient, or may still be referred to as a transform coefficient for consistency of expression.

[0081] In addition, in this document, the quantized transform coefficient and the transform coefficient may be referred to as a transform coefficient and a scaled transform coefficient, respectively. In this case, the residual information may include information about the transform coefficient, and the information about the transform coefficient may be signaled by a residual coding syntax. The transform coefficient may be derived based on the residual information (or information about the transform coefficient), and the scaled transform coefficient may be derived by inverse transforming (scaling) the transform coefficient. The residual sample may be derived based on the inverse transform (transform) of the scaled transform coefficient. This may also be applied / expressed in other parts of this document.

[0082] Figure 3 is a diagram schematically illustrating a configuration of a video / image decoding device to which the disclosure of this document can be applied.

[0083] Reference Figure 3 , the decoding device 300 may include and be configured with an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an intra-frame predictor 331 and an inter-frame predictor 332. The residual processor 320 may include a dequantizer 321 and an inverse transformer 322. According to an embodiment, the entropy decoder 310, the residual processor 320, the predictor 330, the adder 340, and the filter 350 described above may be configured by one or more hardware components (e.g., a decoder chipset or a processor). In addition, the memory 360 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may also include a memory 360 as an internal / external component.

[0084] When a bit stream including video / image information is input, the decoding device 300 may respond to the Figure 2 The image is reconstructed by processing video / image information in the encoding device illustrated in . For example, the decoding device 300 can derive the unit / block based on the block segmentation related information obtained from the bit stream. The decoding device 300 can perform decoding using a processing unit applied to the encoding device. Therefore, the processing unit for decoding can be, for example, a coding unit, and the coding unit can be divided from the coding tree unit or the maximum coding unit according to a quadtree structure, a binary tree structure and / or a ternary tree structure. One or more transform units can be derived from the coding unit. In addition, the reconstructed image signal decoded and output by the decoding device 300 can be reproduced by a reproduction device.

[0085] The decoding device 300 may receive the Figure 2 The signal output by the encoding device of the video image processing apparatus 310 can be decoded by the entropy decoder 310. For example, the entropy decoder 310 can parse the bitstream to derive information (e.g., video / image information) necessary for image reconstruction (or picture reconstruction). The video / image information may also include information about various parameter sets such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may also include conventional constraint information. The decoding device may also decode the picture based on the information about the parameter set and / or the conventional constraint information. The signaled / received information and / or syntax elements subsequently described in this document may be decoded by a decoding process and obtained from the bitstream. For example, the entropy decoder 310 decodes the information in the bitstream based on a coding method such as exponential Golomb coding, context adaptive variable length coding (CAVLC), or context adaptive binary arithmetic coding (CABAC), and outputs syntax elements required for image reconstruction and quantized values ​​of transform coefficients for the residual. More specifically, the CABAC entropy decoding method can receive a bin corresponding to each syntax element in a bitstream, determine a context model by using information of a decoded target syntax element, information of a decoded target block, or information of a symbol / bin decoded in a previous stage, and perform arithmetic decoding on the bin by predicting the probability of the occurrence of the bin according to the determined context model, and generate a symbol corresponding to the value of each syntax element. In this case, the CABAC entropy decoding method can update the context model by using information of a decoded symbol / bin of a context model for the next symbol / bin after determining the context model. Information related to prediction among the information decoded by the entropy decoder 310 can be provided to a predictor (inter-frame predictor 332 and intra-frame predictor 331), and residual values ​​(i.e., quantized transform coefficients and related parameter information) that have been entropy decoded in the entropy decoder 310 can be input to a residual processor 320.

[0086] The dequantizer 321 may dequantize the quantized transform coefficients and output the transform coefficients. The dequantizer 321 may rearrange the quantized transform coefficients in a two-dimensional block form. In this case, the rearrangement may be performed based on the coefficient scanning order performed in the encoding device. The dequantizer 321 may perform dequantization on the quantized transform coefficients using a quantization parameter (e.g., quantization step size information) and obtain the transform coefficients.

[0087] The inverse transformer 322 inversely transforms the transform coefficients to obtain a residual signal (residual block, residual sample array).

[0088] The predictor may perform prediction on the current block and generate a prediction block including prediction samples of the current block. The predictor may determine whether to apply intra prediction or inter prediction to the current block based on information on prediction output from the entropy decoder 310 and may determine a specific intra / inter prediction mode.

[0089] The predictor 330 may generate a prediction signal based on various prediction methods to be described later. For example, the predictor may apply intra prediction or inter prediction to the prediction of a block, and may apply intra prediction and inter prediction at the same time. This may be referred to as a combination of inter and intra prediction (CIIP). In addition, the predictor may predict a block based on an intra-block copy (IBC) prediction mode or based on a palette mode. The IBC prediction mode or the palette mode may be used for image / video encoding of content such as games, for example, screen content coding (SCC). IBC may basically perform prediction within the current picture, but may be performed similarly to inter prediction in terms of deriving a reference block within the current picture. That is, IBC may use at least one of the inter prediction techniques described in this document. The palette mode may be considered an example of intra coding or intra prediction. When the palette mode is applied, information about the palette table and the palette index may be included in the video / image information and signaled.

[0090] The intra-frame predictor 331 can predict the current block by referring to samples in the current picture. Depending on the prediction mode, the referenced sample can be located near the current block, or its location can be separated from the current block. In intra-frame prediction, the prediction mode can include multiple non-directional modes and multiple directional modes. The intra-frame predictor 331 can determine the prediction mode to be applied to the current block by using the prediction mode applied to the neighboring block.

[0091] The inter-frame predictor 332 may derive a prediction block of the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. In this case, in order to reduce the amount of motion information sent in the inter-frame prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the adjacent blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may also include information about the inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter-frame prediction, the adjacent blocks may include spatial adjacent blocks present in the current picture and temporal adjacent blocks present in the reference picture. For example, the inter-frame predictor 332 may construct a motion information candidate list based on adjacent blocks, and derive a motion vector and / or a reference picture index of the current block based on the received candidate selection information. Inter-frame prediction may be performed based on various prediction modes, and information about the prediction may include information indicating a mode of inter-frame prediction for the current block.

[0092] The adder 340 can generate a reconstructed signal (reconstructed picture, reconstructed block or reconstructed sample array) by adding the obtained residual signal to the prediction signal (prediction block or prediction sample array) output from the predictor (including the inter-frame predictor 332 and / or the intra-frame predictor 331). If the residual of the target block is not processed (such as when the skip mode is applied), the prediction block can be used as the reconstructed block.

[0093] The adder 340 may be referred to as a reconstructor or a reconstructed block generator. The generated reconstructed signal may be used for intra prediction of the next block to be processed in the current picture, and as described later, may also be output through filtering or may also be used for inter prediction of the next picture.

[0094] In addition, luminance mapping with chroma scaling (LMCS) can also be applied to the picture decoding process.

[0095] The filter 350 can improve the subjective / objective image quality by applying filtering to the reconstructed signal. For example, the filter 350 can generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture, and store the modified reconstructed picture in the memory 360, specifically, in the DPB of the memory 360. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0096] The (modified) reconstructed picture stored in the DPB of the memory 360 may be used as a reference picture in the inter-frame predictor 332. The memory 360 may store the motion information of the block from which the motion information in the current picture is derived (or decoded) and / or the motion information of the block in the reconstructed picture. The stored motion information may be transmitted to the inter-frame predictor 332 so as to be used as the motion information of the spatial neighboring block or the motion information of the temporal neighboring block. The memory 360 may store the reconstructed samples of the reconstructed block in the current picture and transmit the reconstructed samples to the intra-frame predictor 331.

[0097] In the present disclosure, the embodiments described in the filter 260 , the inter predictor 221 , and the intra predictor 222 of the encoding apparatus 200 may be applied equally or correspondingly to the filter 350 , the inter predictor 332 , and the intra predictor 331 .

[0098] In addition, as described above, when performing video encoding, prediction is performed to improve compression efficiency. Accordingly, a prediction block including prediction samples for a current block as a block to be encoded (i.e., an encoding target block) can be generated. Here, the prediction block includes prediction samples in a spatial domain (or a pixel domain). The prediction block is derived in the same manner in the encoding device and the decoding device, and the encoding device can signal information about the residual between the original block and the prediction block (residual information) rather than the original sample value of the original block to the decoding device, thereby improving the image coding efficiency. The decoding device can derive a residual block including residual samples based on the residual information, add the residual block and the prediction block to generate a reconstructed block including reconstructed samples, and generate a reconstructed picture including the reconstructed block.

[0099] Residual information can be generated through a transformation and quantization process. For example, the encoding device can derive a residual block between the original block and the prediction block, perform a transformation process on the residual samples (residual sample array) included in the residual block to derive a transformation coefficient, perform a quantization process on the transformation coefficient to derive a quantized transformation coefficient, and signal the relevant residual information to the decoding device (through a bitstream). Here, the residual information may include value information, position information, transformation technology, transformation kernel, quantization parameter, etc. of the quantized transformation coefficient. The decoding device can perform a dequantization / inverse transformation process based on the residual information and derive residual samples (or residual blocks). The decoding device can generate a reconstructed picture based on the prediction block and the residual block. In addition, in order to provide a reference for inter-frame prediction of subsequent pictures, the encoding device can also dequantize / inverse transform the quantized transformation coefficient to derive a residual block and generate a reconstructed picture based on this.

[0100] In addition, various inter-frame prediction modes can be used for prediction of the current block within the picture. For example, various modes such as merge mode, skip mode, motion vector prediction (MVP) mode, affine mode, sub-block merge mode, merge with MVD (MMVD) mode, etc. can be used. Decoder-side motion vector refinement (DMVR) mode, adaptive motion vector resolution (AMVR) mode, bidirectional prediction with CU-level weight (BCW), bidirectional optical flow (BDOF), etc. can be used additionally or alternatively as auxiliary modes. Affine mode can be referred to as affine motion prediction mode. MVP mode can be referred to as advanced motion vector prediction (AMVP) mode. In the present disclosure, some modes and / or motion information candidates derived from some modes can be included as one of the motion information related candidates of other modes. For example, an HMVP candidate can be added as a merge candidate for a merge / skip mode, or can be added as an MVP candidate for an MVP mode.

[0101] Inter-frame prediction mode information indicating the inter-frame prediction mode of the current block may be signaled from the encoding device to the decoding device. The inter-frame prediction mode information may be included in the bitstream and received at the decoding device. The inter-frame prediction mode information may include index information indicating one of a plurality of candidate modes. In addition, the inter-frame prediction mode may be indicated by hierarchical signaling of flag information. In this case, the inter-frame prediction mode information may include one or more flags. For example, whether the skip mode is applied may be indicated by signaling a skip flag; for the skip mode not being applied, whether the merge mode is applied may be indicated by signaling a merge flag; and when the merge mode is not applied, the application of the MVP mode may be indicated or a flag for further segmentation may also be signaled. The affine mode may be signaled as an independent mode, or may be signaled as a mode depending on the merge mode, the MVP mode, and the like. For example, the affine mode may include an affine merge mode and an affine MVP mode.

[0102] In addition, information indicating whether the above-mentioned list0 (L0) prediction, list1 (L1) prediction, or bidirectional prediction is used in the current block (current coding unit) may be signaled to the current block. This information may be referred to as motion prediction direction information, inter-frame prediction direction information, or inter-frame prediction indication information, and may be constructed / encoded / signaled in the form of, for example, an inter_pred_idc syntax element. That is, the inter_pred_idc syntax element may indicate whether the above-mentioned list0 (L0) prediction, list1 (L1) prediction, or bidirectional prediction is used for the current block (current coding unit). In the present disclosure, for ease of description, the inter-frame prediction type (L0 prediction, L1 prediction, or BI prediction) indicated by the inter_pred_idc syntax element may be represented as a motion prediction direction. L0 prediction may be represented by pred_L0; L1 prediction may be represented by pred_L1; and bidirectional prediction may be represented by pred_BI. For example, the following prediction types may be indicated according to the value of the inter_pred_idc syntax element.

[0103] As described above, a picture may include one or more slices. A slice may have one of slice types including an intra (I) slice, a predicted (P) slice, and a bidirectional predicted (B) slice. The slice type may be indicated based on slice type information. For blocks in an I slice, inter prediction is not used for prediction, but only intra prediction may be used. Of course, even in this case, the original sample values ​​may be encoded and signaled without prediction. For blocks in a P slice, intra prediction or inter prediction may be used, and when inter prediction is used, only unidirectional prediction may be used. In addition, for blocks in a B slice, intra prediction or inter prediction may be used, and when inter prediction is used, up to a maximum bidirectional prediction may be used.

[0104] L0 and L1 may include reference pictures that are encoded / decoded before the current picture. For example, L0 may include reference pictures that are before and / or after the current picture in POC order, and L1 may include reference pictures that are after and / or before the current picture in POC order. In this case, a lower reference picture index relative to a reference picture that is earlier than the current picture in POC order may be assigned to L0, and a lower reference picture index relative to a reference picture that is later than the current picture in POC order may be assigned to L1. In the case of a B slice, bidirectional prediction may be applied, and in this case, unidirectional bidirectional prediction may be applied, or bidirectional bidirectional prediction may be applied. Bidirectional bidirectional prediction may be referred to as true bidirectional prediction.

[0105] For example, information about the inter prediction mode of the current block may be encoded and signaled at the CU (CU syntax) level, etc., or may be implicitly determined according to conditions. In this case, some modes may be explicitly signaled, while other modes may be implicitly derived.

[0106] For example, the CU syntax may carry information about (inter) prediction mode, etc. The CU syntax may be as shown in Table 1 below.

[0107] [Table 1]

[0108]

[0109]

[0110]

[0111]

[0112]

[0113] In Table 1, cu_skip_flag may indicate whether skip mode is applied to the current block (CU).

[0114] pred_mode_flag equal to 0 can specify that the current coding unit is encoded in inter-frame prediction mode. pred_mode_flag equal to 1 can specify that the current coding unit is encoded in intra-frame prediction mode.

[0115] pred_mode_IBC_flag equal to 1 can specify that the current coding unit is encoded in IBC prediction mode. pred_mode_IBC_flag equal to 0 can specify that the current coding unit is not encoded in IBC prediction mode.

[0116] pcm_flag[x0][y0] equal to 1 can specify that the pcm_sample() syntax structure exists and the transform_tree() syntax structure does not exist in the coding unit including the luma coding block at position (x0, y0). pcm_flag[x0][y0] equal to 0 can specify that the pcm_sample() syntax structure does not exist. That is, pcm_flag can indicate whether the pulse code modulation (PCM) mode is applied to the current block. If the PCM mode is applied to the current block, prediction, transformation, quantization, etc. are not applied, and the value of the original sample in the current block can be encoded and signaled.

[0117] intra_mip_flag[x0][y0] equal to 1 can specify that the intra prediction type of luma samples is matrix-based intra prediction (MIP). intra_mip_flag[x0][y0] equal to 0 can specify that the intra prediction type of luma samples is not matrix-based intra prediction. That is, intra_mip_flag can indicate whether the MIP prediction mode (type) is applied to (luma samples) of the current block.

[0118] intra_chroma_pred_mode[x0][y0] may specify the intra prediction mode for chroma samples in the current block.

[0119] general_merge_flag[x0][y0] may specify that inter prediction parameters of the current coding unit are inferred from adjacent inter-prediction partitions. That is, general_merge_flag may indicate that general merging is available, and when the value of general_merge_flag is 1, conventional merge mode, MMVD mode, and merge subblock mode may be available. For example, when the value of general_merge_flag is 1, merge data syntax may be parsed from coded video / image information (or bitstream), and the merge data syntax may be configured / encoded to include information as shown in Table 2 below.

[0120] [Table 2]

[0121]

[0122]

[0123] In Table 2, regular_merge_flag[x0][y0] equal to 1 may specify that the regular merge mode is used to generate the inter prediction parameters of the current coding unit. That is, regular_merge_flag may indicate whether the merge mode (regular merge mode) is applied to the current block.

[0124] mmvd_merge_flag[x0][y0] equal to 1 can specify that the merge mode with motion vector difference is used to generate the inter prediction parameters of the current block. In other words, mmvd_merge_flag indicates whether MMVD is applied to the current block.

[0125] mmvd_cand_flag[x0][y0] may specify whether the first (0) candidate or the second (1) candidate in the merge candidate list is used with the motion vector difference derived from mmvd_distance_idx[x0][y0] and mmvd_direction_idx[x0][y0].

[0126] mmvd_distance_idx[x0][y0] may specify the index used to derive MmvdDistance[x0][y0].

[0127] mmvd_direction_idx[x0][y0] may specify an index used to derive MmvdSign[x0][y0].

[0128] merge_block_flag[x0][y0] may specify subblock-based inter prediction parameters for the current block. That is, merge_subblock_flag may indicate whether a subblock merge mode (or affine merge mode) is applied to the current block.

[0129] merge_block_idx[x0][y0] may specify a merge candidate index of a sub-block based merge candidate list.

[0130] ciip_flag[x0][y0] may specify whether combined inter-picture merging and intra-picture prediction (CIIP) is applied to the current coding unit.

[0131] merge_triangle_idx0[x0][y0] may specify the first merge candidate index of a triangle-shaped based motion compensation candidate list.

[0132] merge_triangle_idx1[x0][y0] may specify a second merge candidate index of a triangle-shaped based motion compensation candidate list.

[0133] merge_idx[x0][y0] may specify a merge candidate index of the merge candidate list.

[0134] In addition, referring back to the CU syntax, mvp_l0_flag[x0][y0] may specify the motion vector predictor index of list 0. That is, when the MVP mode is applied, mvp_l0_flag may indicate a candidate from the MVP candidate list 0 selected for MVP derivation of the current block.

[0135] ref_idx_l1[x0][y0] has the same semantics as ref_idx_l0, where l0 and list0 can be replaced by l1 and list1 respectively.

[0136] inter_pred_idc[x0][y0] can specify whether list0, list1 or bidirectional prediction is used for the current coding unit.

[0137] sym_mvd_flag[x0][y0] equal to 1 may specify that the syntax elements ref_idx_l0[x0][y0] and ref_idx_l1[x0][y0] are absent and the mvd_coding(x0, y0, reflist, cIdx) syntax structure for RefList equal to 1. That is, sym_mvd_flag indicates whether symmetric MVD is used in mvd coding.

[0138] ref_idx_l0[x0][y0] may specify the list 0 reference picture index of the current block.

[0139] ref_idx_l1[x0][y0] has the same semantics as ref_idx_l0, where l0, L0, and list0 are replaced by l1, L1, and list1, respectively.

[0140] inter_affine_flag[x0][y0] equal to 1 can specify that affine model-based motion compensation is used to generate prediction samples of the current block when decoding P or B slices.

[0141] cu_affine_type_flag[x0][y0] equal to 1 can specify that for the current coding unit, when decoding a P or B slice, motion compensation based on a 6-parameter affine model is used to generate prediction samples for the current coding unit. cu_affine_type_flag[x0][y0] equal to 0 can specify that motion compensation based on a 4-parameter affine model is used to generate prediction samples for the current block.

[0142] amvr_flag[x0][y0] may specify the resolution of the motion vector difference. The array indices x0, y0 specify the position (x0, y0) of the top-left luma sample of the coding block under consideration relative to the top-left luma sample of the picture. amvr_flag[x0][y0] equal to 0 may specify that the resolution of the motion vector difference is 1 / 4 of the luma sample. amvr_flag[x0][y0] equal to 1 may specify that the resolution of the motion vector difference is also specified by amvr_precision_flag[x0][y0].

[0143] amvr_precision_flag[x0][y0] equal to 0 specifies that the resolution of motion vector differences is one whole luma sample if inter_affine_flag[x0][y0] is 0, otherwise it is 1 / 16 of a luma sample. amvr_precision_flag[x0][y0] equal to 1 specifies that the resolution of motion vector differences is four luma samples if inter_affine_flag[x0][y0] is 0, otherwise it is one whole luma sample.

[0144] bcw_idx[x0][y0] may specify a weight index for bidirectional prediction with CU weight.

[0145] Figure 4 An example of a video / image encoding method based on inter-frame prediction is shown, and Figure 5 is a diagram schematically illustrating an example of an inter-frame predictor in an encoding device. Figure 5 The inter-frame predictor in the encoding device can be combined with the above Figure 2 The inter-frame predictor 221 of the encoding device 200 is applied identically or correspondingly.

[0146] Reference Figure 4 and Figure 5 , the encoding device performs inter prediction on the current block (S400). The encoding device may derive an inter prediction mode and motion information of the current block, and generate a prediction sample of the current block. Here, the processes for determining the inter prediction mode, deriving motion information, and generating prediction samples may be performed simultaneously, or one process may be performed before performing another process.

[0147] For example, the inter-frame predictor 221 of the encoding device may include a prediction mode determiner 221_1, a motion information deriver 221_2, and a prediction sample deriver 221_3, and the prediction mode determiner 221_1 may determine the prediction mode of the current block, the motion information deriver 221_2 may derive the motion information of the current block, and the prediction sample deriver 221_3 may derive the prediction sample of the current block. For example, the inter-frame predictor 221 of the encoding device may search for a block close to the current block in a predetermined area (search area) of the reference picture through motion estimation, and may derive a reference block whose difference with the current block is the smallest or is a predetermined reference or less. Based on this, a reference picture index indicating the reference picture where the reference block is located may be derived, and a motion vector may be derived based on the position difference between the reference block and the current block. The encoding device may determine a mode applied to the current block among various prediction modes. The encoding device may compare the rate-distortion RD costs under various prediction modes and determine the best prediction mode for the current block.

[0148] For example, when a skip mode or a merge mode is applied to the current block, the encoding device may construct a merge candidate list to be described below, and derive a reference block whose difference with the current block is the smallest or is a predetermined reference or less among the reference blocks indicated by the merge candidates included in the merge candidate list. In this case, a merge candidate associated with the derived reference block may be selected, and merge index information indicating the selected merge candidate may be generated and signaled to the decoding device. The motion information of the selected merge candidate may be used to derive the motion information of the current block.

[0149] As another example, when the (A)MVP mode is applied to the current block, the encoding device constructs the (A)MVP candidate list to be described below, and uses the motion vector of the MVP candidate selected from among the motion vector predictor (MVP) candidates included in the (A)MVP candidate list as the MVP of the current block. In this case, for example, the motion vector indicating the reference block derived by the above-mentioned motion estimation can be used as the motion vector of the current block, and the MVP candidate having the motion vector with the smallest difference from the motion vector of the current block among the MVP candidates can be the selected MVP candidate. A motion vector difference (MVD) as a difference obtained by subtracting the MVP from the motion vector of the current block can be derived. In this case, information about the MVD can be signaled to the decoding device. In addition, when the (A)MVP mode is applied, the value of the reference picture index can be constructed as reference image index information and signaled separately to the decoding device.

[0150] The encoding apparatus may induce residual samples based on the prediction samples (S410). The encoding apparatus may induce residual samples by comparing original samples of the current block with the prediction samples.

[0151] The encoding device encodes the image information including prediction information and residual information (S420). The encoding device can output the encoded image information in the form of a bitstream. The prediction information is information related to the prediction process and may include prediction mode information (e.g., a skip flag, a merge flag, or a mode index, etc.) and motion information. The information about the motion information may include candidate selection information (e.g., a merge index) as information for deriving a motion vector. In addition, the information about the motion information may include information indicating whether L0 prediction, L1 prediction, or bidirectional prediction is applied. The residual information is information about the residual sample. The residual information may include information about the quantized transform coefficient for the residual sample.

[0152] The output bitstream may be stored in a (digital) storage medium and transmitted to the decoding device, or may be transmitted to the decoding device over a network.

[0153] In addition, as described above, the encoding device can generate a reconstructed picture (including a reconstructed sample and a reconstructed block) based on the reference sample and the residual sample. This is because the encoding device can derive the same prediction result as the prediction result performed by the decoding device, and accordingly, the encoding efficiency can be improved. Therefore, the encoding device can store the reconstructed picture (or reconstructed sample, reconstructed block) in a memory and use the reconstructed picture as a reference picture for inter-frame prediction. As described above, an in-loop filtering process can also be applied to the reconstructed picture.

[0154] Figure 6 An example of a video / image decoding method based on inter-frame prediction is shown, and Figure 7 is a diagram schematically illustrating an example of an inter-frame predictor in a decoding device. Figure 7 The inter-frame predictor in the decoding device can be combined with the above Figure 3 The inter-frame predictor 332 of the decoding device 300 is applied identically or correspondingly.

[0155] Reference Figure 6 and Figure 7 The decoding device may perform an operation corresponding to the operation performed by the encoding device. The decoding device may perform prediction on the current block based on the received prediction information and derive a prediction sample.

[0156] Specifically, the decoding device may determine a prediction mode of the current block based on the received prediction information (S600). The decoding device may determine which inter prediction mode to apply to the current block based on prediction mode information in the prediction information.

[0157] The inter prediction mode candidates may include a skip mode, a merge mode, and / or an (A)MVP mode, or may include various inter prediction modes to be described later.

[0158] The decoding device derives the motion information of the current block based on the determined inter-frame prediction mode (S610). For example, when the skip mode or merge mode is applied to the current block, the decoding device may configure a merge candidate list and select a merge candidate from the merge candidates included in the merge candidate list. The selection may be performed based on the above-mentioned selection information (merge index). The motion information of the selected merge candidate may be used to derive the motion information of the current block. The motion information of the selected merge candidate may be used as the motion information of the current block.

[0159] As another example, when the (A)MVP mode is applied to the current block, the decoding device may construct an (A)MVP candidate list, and use the motion vector of the motion vector predictor (MVP) candidate selected from the motion vector predictor (MVP) candidates included in the (A)MVP candidate list as the MVP of the current block. The selection may be performed based on the above-mentioned selection information (MVP flag or MVP index). In this case, the MVD of the current block may be derived based on the information about the MVD, and the motion vector of the current block may be derived based on the MVP and MVD of the current block. In addition, the reference picture index of the current block may be derived based on the reference picture index information. The picture indicated by the reference picture index in the reference picture list of the current block may be derived as the reference picture referenced by the inter-frame prediction of the current block.

[0160] In addition, as will be described below, the motion information of the current block can be derived without configuring the candidate list. In this case, the motion information of the current block can be derived according to the process disclosed in the prediction mode to be described later. In this case, the configuration of the candidate list as described above can be omitted.

[0161] The decoding device may generate a prediction sample of the current block based on the motion information of the current block (S620). In this case, the reference picture may be derived based on the reference picture index of the current block, and the prediction sample of the current block may be derived using the sample of the reference block indicated by the motion vector of the current block on the reference picture. In this case, in some cases, a prediction sample filtering process may also be performed on all or some of the prediction samples of the current block.

[0162] For example, the inter-frame predictor 332 of the decoding device may include a prediction mode determiner 332_1, a motion information deriver 332_2 and a prediction sample deriver 332_3, and the prediction mode determiner 332_1 can determine the prediction mode of the current block based on the received prediction mode information, the motion information deriver 332_2 can derive the motion information (motion vector and / or reference picture index, etc.) of the current block based on the received information about the motion information, and the prediction sample deriver 332_3 can derive the prediction sample of the current block.

[0163] The decoding device generates a residual sample of the current block based on the received residual information (S630). The decoding device can generate a reconstructed sample of the current block based on the predicted sample and the residual sample, and can generate a reconstructed picture based on this (S640). Thereafter, as described above, an in-loop filtering process can also be applied to the reconstructed picture.

[0164] As described above, the inter-frame prediction process may include a step of determining an inter-frame prediction mode, a step of deriving motion information according to the determined prediction mode, and a step of performing prediction (generation of prediction samples) based on the derived motion information. The inter-frame prediction process may be performed by the encoding device and the decoding device as described above.

[0165] In addition, when deriving the motion information of the current block, motion information candidates are derived based on spatial neighboring blocks and temporal neighboring blocks, and based on the derived motion information candidates, the motion information candidate of the current block can be selected. In this case, the selected motion information candidate can be used as the motion information of the current block.

[0166] Figure 8 This is a diagram illustrating the merge mode in inter-frame prediction.

[0167] When the merge mode is applied, the motion information of the current prediction block is not sent directly, but the motion information of the adjacent prediction block is used to derive the motion information of the current prediction block. Therefore, the motion information of the current prediction block can be indicated by sending flag information indicating the use of the merge mode and a merge index indicating which nearby prediction block is used. The merge mode can be called a normal merge mode.

[0168] In order to perform the merge mode, the encoding device needs to search for a merge candidate block for deriving motion information about the current prediction block. For example, up to five merge candidate blocks can be used, but the embodiments of the present disclosure are not limited to this. In addition, the maximum number of merge candidate blocks can be sent in a slice header or a tile group header, but the embodiments of the present disclosure are not limited to this. After finding the merge candidate block, the encoding device can generate a merge candidate list, and can select the merge candidate block with the smallest cost among the merge candidate blocks as the final merge candidate block.

[0169] The present disclosure may provide various implementations of merging candidate blocks constituting a merging candidate list.

[0170] For example, the merge candidate list may use five merge candidate blocks. For example, four spatial merge candidates and one temporal merge candidate may be used. As a specific example, in the case of spatial merge candidates, Figure 4The block illustrated in may be used as a spatial merge candidate. Hereinafter, a spatial merge candidate or a spatial MVP candidate to be described later may be referred to as an SMVP, and a temporal merge candidate or a temporal MVP candidate to be described later may be referred to as a TMVP.

[0171] For example, the merge candidate list of the current block may be constructed based on the following process.

[0172] The encoding device (encoding device / decoding device) may insert the spatial merge candidate derived by searching the current spatial neighboring blocks into the merge candidate list. For example, the spatial neighboring blocks may include the lower left corner neighboring block of the current block, the left neighboring block (A 1 ), an upper right neighboring block, an upper neighboring block, and an upper left neighboring block. However, this is an example, and in addition to the above-mentioned spatial neighboring blocks, additional neighboring blocks such as a right neighboring block, a lower neighboring block, and a lower right neighboring block may be further used as spatial neighboring blocks. The encoding device may detect an available block by searching the spatial neighboring blocks based on priority, and may derive motion information of the detected block as a spatial merge candidate. For example, the encoding device or the decoding device may be configured to perform A 1 ->B 1 ->B 0 ->A 0 ->B 2 Sequential search Figure 8 The five blocks illustrated in FIG. 1 and the merge candidate list can be configured by indexing the available candidates in sequence.

[0173] The encoding device may search for temporally adjacent blocks of the current block and insert the derived temporal merge candidates into the merge candidate list. The temporally adjacent blocks may be at a reference picture that is a different picture from the current picture in which the current block is located. The reference picture in which the temporally adjacent blocks are located may be referred to as a collocated picture or a collocation picture. The temporally adjacent blocks may be searched for on the collocated picture in the order of the bottom-right adjacent block and the bottom-right center block of the co-located block of the current block. Further, when motion data compression is applied, specific motion information may be stored as representative motion information in each predetermined storage unit in the collocated picture. In this case, it is not necessary to store the motion information about all blocks in the predetermined storage unit, and thus, a motion data compression effect can be obtained. In this case, the predetermined storage unit may be predetermined as a unit of, for example, 16×16 samples or 8×8 samples, or the size information about the predetermined storage unit may be signaled from the encoding device to the decoding device. When motion data compression is applied, the motion information about the temporally adjacent blocks may be replaced with the representative motion information on the predetermined storage unit where the temporally adjacent blocks are located. That is, in this case, from the perspective of implementation, instead of the prediction block at the coordinates of the temporally adjacent blocks, the motion information of the prediction block covering the arithmetic left shift position after arithmetic right shift of a certain value based on the coordinates (top-left sample position) of the temporally adjacent blocks may be used to derive the temporal merge candidates. For example, when the predetermined storage unit is a 2n×2n sample unit, if the coordinates of the temporally adjacent block are (xTnb, yTnb), the motion information of the prediction block at the corrected position ((xTnb>>n)<<n), (yTnb>>n)<<n)) may be used for the temporal merge candidates. Specifically, when the predetermined storage unit is a 16×16 sample unit, if the coordinates of the temporally adjacent block are (xTnb, yTnb), the motion information of the prediction block at the corrected position ((xTnb>>4)<<4), (yTnb>>4)<<4)) may be used for the temporal merge candidates. Alternatively, when the predetermined storage unit is an 8×8 sample unit, if the coordinates of the temporally adjacent block are (xTnb, yTnb), the motion information of the prediction block at the corrected position ((xTnb>>3)<<3), (yTnb>>3)<<3)) may be used for the temporal merge candidates.

[0174] The encoding device may check whether the number of current merge candidates is less than the maximum number of merge candidates. The maximum number of merge candidates may be predefined or signaled from the encoding device to the decoding device. For example, the encoding device may generate and encode information about the maximum number of merge candidates and send the information to the decoder in the form of a bitstream. When the maximum number of merge candidates is full, the subsequent candidate addition process may not be performed.

[0175] As a result of the check, when the current number of merge candidates is less than the maximum number of merge candidates, the encoding device may insert an additional merge candidate into the merge candidate list. For example, the additional merge candidate may include at least one of a history-based merge candidate, a pairwise average merge candidate, an ATMVP, a combined bidirectional prediction merge candidate (when the slice / tile group type of the current slice / tile group is type B), and / or a zero vector merge candidate to be described later.

[0176] As a result of the check, when the current number of merge candidates is not less than the maximum number of merge candidates, the encoding device may terminate the construction of the merge candidate list. In this case, the encoding device may select the optimal merge candidate from the merge candidates constituting the merge candidate list based on the rate-distortion (RD) cost, and signal the selection information (e.g., merge index) indicating the selected merge candidate to the decoding device. The decoding device may select the optimal merge candidate based on the merge candidate list and the selection information.

[0177] As described above, the motion information about the selected merge candidate may be used as the motion information about the current block, and the prediction sample of the current block may be derived based on the motion information about the current block. The encoding device may derive the residual sample of the current block based on the prediction sample, and may signal the residual information about the residual sample to the decoding device. As described above, the decoding device may generate a reconstructed sample based on the residual sample derived based on the residual information and the prediction sample, and may generate a reconstructed picture based on this.

[0178] When skip mode is applied, motion information about the current block may be derived in the same manner as when merge mode is applied. However, when skip mode is applied, the residual signal of the corresponding block is omitted, and thus the prediction sample may be directly used as the reconstructed sample. For example, when the value of the cu_skip_flag syntax element is 1, skip mode may be applied.

[0179] Fig. 9 is a diagram illustrating a merge mode with motion vector difference (MMVD) mode in inter prediction.

[0180] The MMVD mode is a method of applying motion vector difference (MVD) to a merge mode in which motion information of a prediction sample derived to generate a current block is directly used.

[0181] For example, an MMVD flag (e.g., mmvd_flag) indicating whether to use MMVD for the current block (i.e., current CU) may be signaled, and MMVD may be performed based on this MMVD flag. When MMVD is applied to the current block (e.g., when mmvd_flag is 1), additional information about MMVD may be signaled.

[0182] Here, additional information about MMVD may include a merge candidate flag (e.g., mmvd_cand_flag) indicating whether the first candidate or the second candidate is used with MVD in the merge candidate list, a distance index (e.g., mmvd_distance_idx) for indicating a motion magnitude, and a direction index (e.g., mmvd_direction_idx) for indicating a motion direction.

[0183] In MMVD mode, two candidates (i.e., the first candidate or the second candidate) located in the first entry and the second entry among the candidates in the merge candidate list may be used, and one of the two candidates (i.e., the first candidate or the second candidate) may be used as the base MV. For example, a merge candidate flag (e.g., mmvd_cand_flag) may be signaled to indicate either one of the two candidates (i.e., the first candidate or the second candidate) in the merge candidate list.

[0184] In addition, the distance index (eg, mmvd_distance_idx) may indicate motion magnitude information and may indicate a predetermined offset from the starting point. Fig. 9 , an offset may be added to the horizontal component or the vertical component of the starting motion vector. The relationship between the distance index and the predetermined offset may be as shown in Table 3 below.

[0185] [Table 3]

[0186]

[0187] Referring to Table 3, the distance of the MVD (e.g., MmvdDistance) may be determined according to the value of the distance index (e.g., mmvd_distance_idx), and the distance of the MVD (e.g., MmvdDistance) may be derived using integer sample precision or fractional sample precision based on the value of slice_fpel_mmvd_enabled_flag. For example, slice_fpel_mmvd_enabled_flag equal to 1 may indicate that the distance of the MVD is derived using integer samples in the current slice, and slice_fpel_mmvd_enabled_flag equal to 0 may indicate that the distance of the MVD is derived using fractional samples in the current slice.

[0188] In addition, the direction index (e.g., mmvd_direction_idx) indicates the direction of the MVD relative to the starting point, and may indicate four directions, as shown in Table 4 below. In this case, the direction of the MVD may indicate the sign of the MVD. The relationship between the direction index and the MVD code may be expressed as shown in Table 4 below.

[0189] [Table 4]

[0190] mmvd_direction_idx[x0][y0] MmvdSign[x0][y0][0] MmvdSign[x0][y0][1] 0 +1 0 1 -1 0 2 0 +1 3 0 -1

[0191] Referring to Table 4, the sign of MVD (eg, MmvdSign) may be determined according to the value of the direction index (eg, mmvd_direction_idx), and the sign of MVD (eg, MmvdSign) may be derived for the L0 reference picture and the L1 reference picture.

[0192] Based on the distance index (eg, mmvd_distance_idx) and the direction index (eg, mmvd_direction_idx) described above, the offset of the MVD may be calculated as shown in Equation 1 below.

[0193] [Formula 1]

[0194] MmvdOffset[x0][y0][0]=(MmvdDistance[x0][y0]<<2)*MmvdSign[x0][y0][0]

[0195] MmvdOffset[x0][y0][1]=(MmvdDistance[x0][y0]<<2)*MmvdSign[x0][y0][1]

[0196] That is, in MMVD mode, a merge candidate indicated by a merge candidate flag (e.g., mmvd_cand_flag) is selected from the merge candidates of the merge candidate list derived based on the neighboring blocks, and the selected merge candidate can be used as a base candidate (e.g., MVP). In addition, the motion information (i.e., motion vector) of the current block can be derived by adding the MVD derived using the distance index (e.g., mmvd_distance_idx) and the direction index (e.g., mmvd_direction_idx) based on the base candidate.

[0197] Fig.10a and Fig.10b CPMV for affine motion prediction is shown.

[0198] Conventionally, only one motion vector can be used to represent the motion of a coding block. That is, a translation motion model is used. However, although this method can represent the optimal motion in units of blocks, it is not actually the optimal motion for each sample, and if the optimal motion vector can be determined in units of samples, the coding efficiency can be increased. For this purpose, an affine motion model can be used. The affine motion prediction method for encoding using an affine motion model can be as follows.

[0199] The affine motion prediction method can use two, three or four motion vectors to represent the motion vector in units of each sample of the block. For example, the affine motion model can represent four types of motion. The affine motion model that represents three types of motion (translation, scaling and rotation) among the motions that the affine motion model can represent can be called a similarity (or simplified) affine motion model. However, the affine motion model is not limited to the above motion models.

[0200] Affine motion prediction may use two or more control point motion vectors (CPMVs) to determine a motion vector of a sample position included in a block. In this case, a set of motion vectors may be referred to as an affine motion vector field (MVF).

[0201] For example, Fig.10a A case where two CPMVs are used can be shown, which can be referred to as a 4-parameter affine model. In this case, for example, the motion vector at the sample position (x, y) can be determined as, for example, Equation 2.

[0202] [Formula 2]

[0203]

[0204] For example, Fig.10b A case of using three CPMVs, which may be referred to as a 6-parameter affine model, may be shown. In this case, for example, a motion vector at a sample position (x, y) may be determined as, for example, Equation 3.

[0205] [Formula 3]

[0206]

[0207] In Equation 2 and Equation 3, {v x ,v y} can represent the motion vector at position (x, y). In addition, {v 0x ,v 0y} may indicate the CPMV of the control point (CP) at the upper left corner of the coding block, and {v 1x ,v 1y} can indicate the CPMV of the CP at the upper right corner, {v 2x ,v 2y} may indicate the CPMV of the CP at the lower left corner position. In addition, W may indicate the width of the current block, and H may indicate the height of the current block.

[0208] Fig.11 The case where the affine MVF is determined in units of sub-blocks is exemplarily illustrated.

[0209] In the encoding / decoding process, the affine MVF can be determined in units of samples or predefined sub-blocks. For example, when the affine MVF is determined in units of samples, a motion vector can be obtained based on each sample value. Alternatively, for example, when the affine MVF is determined in units of sub-blocks, the motion vector of the corresponding block can be obtained based on the sample value of the center of the sub-block (i.e., the lower right side of the center, i.e., the lower right sample among the four samples in the center). That is, in affine motion prediction, the motion vector of the current block can be derived in units of samples or sub-blocks.

[0210] exist Fig.11 In the case of , the affine MVF is determined in units of 4×4 sub-blocks, but the size of the sub-block may be modified differently.

[0211] That is, when affine prediction is available, the three motion models applicable to the current block may include a translation motion model, a 4-parameter affine motion model, and a 6-parameter affine motion model. The translation motion model may represent a model using an existing block unit motion vector, the 4-parameter affine motion model may represent a model using two CPMVs, and the 6-parameter affine motion model may represent a model using three CPMVs.

[0212] Furthermore, affine motion prediction may include an affine MVP (or affine inter) mode or an affine merge mode.

[0213] Fig.12 The diagram illustrates the affine merge mode or the subblock merge mode in inter prediction.

[0214] For example, in the affine merge mode, the CPMV may be determined based on the affine motion model of the neighboring blocks encoded by affine motion prediction. For example, the neighboring blocks encoded as affine motion prediction in search order may be used for the affine merge mode. That is, when at least one of the neighboring blocks is encoded in the affine motion prediction, the current block may be encoded in the affine merge mode. Here, the refined merge mode may be referred to as AF_MERGE.

[0215] When the affine merge mode is applied, the CPMV of the neighboring block can be used to derive the CPMV of the current block. In this case, the CPMV of the neighboring block can be used as the CPMV of the current block as it is, and the CPMV of the neighboring block can be modified based on the size of the neighboring block and the size of the current block and used as the CPMV of the current block.

[0216] On the other hand, in the case of an affine merge mode in which a motion vector (MV) is derived in units of sub-blocks, this may be referred to as a sub-block merge mode, and the sub-block merge mode may be indicated based on a sub-block merge flag (or merge_subblock_flag syntax element). Alternatively, when the value of the merge_subblock_flag syntax element is 1, it may be indicated that the sub-block merge mode is applied. In this case, the affine merge candidate list to be described later may be referred to as a sub-block merge candidate list. In this case, the sub-block merge candidate list may also include a candidate derived by the SbTMVP to be described later. In this case, the candidate derived by the SbTMVP may be used as a candidate of index 0 of the sub-block merge candidate list. In other words, the candidate derived from the SbTMVP may be before the inherited affine candidate or the constructed affine candidate to be described later in the sub-block merge candidate list.

[0217] When the affine merge mode is applied, an affine merge candidate list can be constructed to derive the CPMV of the current block. For example, the affine merge candidate list may include at least one of the following candidates. 1) Inherited affine merge candidate. 2) Constructed affine merge candidate. 3) Zero motion vector candidate (or zero vector). Here, the inherited affine merge candidate is a candidate derived based on the CPMV of the adjacent block when the adjacent block is encoded in the affine mode, the constructed affine merge candidate is a candidate derived by constructing the CPMV based on the MV of the adjacent block of the corresponding CP in units of each CPMV, and the zero motion vector candidate may indicate a candidate consisting of a CPMV whose value is 0.

[0218] For example, the affine merge candidate list can be constructed as follows.

[0219] There may be up to two inherited affine candidates, and the inherited affine candidates may be derived from the affine motion models of neighboring blocks. The neighboring blocks may include a left neighbor and an upper neighbor. The candidate blocks may be as follows: Figure 8 The scanning order of the left predictor can be A 1 ->A 0 , and the scanning order of the upper predictor can be B 1 ->B 0 ->B 2 Only one inherited candidate may be selected from each of the left and above predictors. No pruning check may be performed between two inherited candidates.

[0220] When the neighboring affine blocks are identified, the control point motion vectors of the checked blocks can be used to derive the CPMVP candidates in the affine merge list of the current block. Here, the neighboring affine blocks can indicate blocks encoded in the affine prediction mode among the neighboring blocks of the current block. For example, referring to Fig.12 , when the lower left neighboring block A is encoded in affine prediction mode, the motion vectors v2, v3 and v4 of the upper left corner, upper right corner and lower left corner of the neighboring block A can be obtained. When the neighboring block A is encoded with a 4-parameter affine motion model, two CPMVs of the current block can be calculated based on v2 and v3. When the neighboring block A is encoded with a 6-parameter affine motion model, three CPMVs of the current block can be calculated based on v2, v3 and v4.

[0221] Fig.13 The diagram illustrates positions of candidates in the affine merge mode or the sub-block merge mode.

[0222] Affine candidates constructed in affine merge mode or sub-block merge mode may represent candidates constructed by combining translational motion information around each control point. Motion information about a control point may be derived from a specified spatial perimeter and temporal perimeter. CPMVk (k=0, 1, 2, 3) may represent the kth control point.

[0223] Reference Fig.13 , for CPMV0, blocks may be checked in the order B2->B3->A2, and the motion vector of the first available block may be used. For CPMV1, blocks may be checked in the order B1->B0, and for CPMV2, blocks may be checked in the order A1->A0. If available, the Temporal Motion Vector Predictor (TMVP) may be used with CPMV3.

[0224] After obtaining the motion vectors of the four control points, an affine merge candidate can be generated based on the obtained motion information. The combination of the control point motion vectors can be any one of {CPMV0, CPMV1, CPMV2}, {CPMV0, CPMV1, CPMV3}, {CPMV0, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV3}, {CPMV0, CPMV1} and {CPMV0, CPMV2}.

[0225] A combination of three CPMVs may constitute a 6-parameter affine merge candidate, and a combination of two CPMVs may constitute a 4-parameter affine merge candidate. To avoid motion scaling processing, relevant combinations of control point motion vectors may be discarded if the reference indexes of the control points are different.

[0226] Fig.14 This is a diagram illustrating SbTMVP in inter-frame prediction.

[0227] Sub-block based temporal motion vector prediction (SbTMVP) may also be referred to as advanced temporal motion vector prediction (ATMVP). SbTMVP may use the motion field in a collocated picture to improve the motion vector prediction and merge mode of the CU in the current picture. Here, the collocated picture may be referred to as a col picture.

[0228] For example, SbTMVP can predict motion at the sub-block (or sub-CU) level. In addition, SbTMVP can apply motion shifting before obtaining temporal motion information from the collocated picture. Here, the motion shifting can be obtained from the motion vector of one of the spatial neighboring blocks of the current block.

[0229] SbTMVP can predict the motion vector of a sub-block (or sub-CU) in the current block (or CU) according to two steps.

[0230] In the first step, we can Figure 4 The order in A 1 , B 1 , B 0 and A 0 To test the spatial neighboring blocks. The first spatial neighboring block that has a motion vector that uses the collocated picture as its reference picture can be checked, and this motion vector can be selected as the motion shift to be applied. When no such motion is detected from the spatial neighboring blocks, the motion shift can be set to (0, 0).

[0231] In the second step, the motion shift identified in the first step can be applied to obtain sub-block level motion information (motion vector and reference index) from the collocated picture. For example, the motion shift can be added to the coordinates of the current block. For example, the motion shift can be set to Figure 8 A 1 In this case, for each sub-block, the motion information of the sub-block can be derived using the motion information of the corresponding block in the collocated picture. Temporal motion scaling can be applied to align the reference picture of the temporal motion vector with the reference picture of the current block.

[0232] A sub-block based merge list including a combination of both SbTVMP candidates and affine merge candidates can be used for signaling of affine merge mode. Here, affine merge mode may be referred to as sub-block based merge mode. Depending on the flag included in the sequence parameter set (SPS), SbTVMP mode may be available or unavailable. When SbTMVP mode is available, the SbTMVP predictor may be added as the first entry in the sub-block based merge candidate list, and the affine merge candidate may follow. The maximum allowable size of affine merge candidates may be 5.

[0233] The size of the sub-CU (or sub-block) used in SbTMVP can be fixed to 8×8, and as in affine merge mode, SbTMVP mode can be applied only to blocks with both width and height of 8 or more. The encoding logic of the additional SbTMVP merge candidate can be the same as the encoding logic of other merge candidates. That is, for each CU in a P or B slice, an RD check using an additional rate-distortion (RD) cost can be performed to determine whether to use the SbTMVP candidate.

[0234] Fig.15 is a diagram illustrating a combined inter-picture merging and intra-picture prediction (CIIP) mode in inter-frame prediction.

[0235] CIIP can be applied to the current CU. For example, in the case where the CU is encoded in merge mode, the CU includes at least 64 luma samples (i.e., when the product of the CU width and the CU height is 64 or greater), and the CU width and the CU height are both less than 128 luma samples, then an additional flag (e.g., ciip_flag) can be signaled to indicate whether the CIIP mode is applied to the current CU.

[0236] In CIIP prediction, the inter prediction signal and the intra prediction signal may be combined. In CIIP mode, the inter prediction signal P_inter may be derived using the same inter prediction process applied to the regular merge mode. The intra prediction signal P_intra may be derived according to the intra prediction process with the planar mode.

[0237] The intra prediction signal and the inter prediction signal may be combined using weighted averaging, and may be expressed in the following Equation 4. Fig.15 The weights are calculated based on the coding modes of the upper neighboring blocks and the left neighboring blocks shown.

[0238] [Formula 4]

[0239] P CIIP =((4-wt)*P inter +wt*P intra +2)>>2

[0240] In Formula 4, when the upper neighboring block is available and intra-coded, isIntratop may be set to 1, otherwise isIntratop may be set to 0. If the left neighboring block is available and intra-coded, isIntraLeft may be set to 1, otherwise isIntraLeft may be set to 0. When (isIntraLeft+isIntraLeft) is 2, wt may be set to 3, and when (isIntraLeft+isIntraLeft) is 1, wt may be set to 2. Otherwise, wt may be set to 1.

[0241] Fig.16 This is a diagram illustrating a partition mode in inter-frame prediction.

[0242] Reference Fig.16 , when a partition mode is applied, a CU may be equally divided into two triangular partitions using a diagonal partition or an anti-diagonal partition in the opposite direction. However, this is only an example of a partition mode, and a CU may be equally or unevenly divided into partitions having various shapes.

[0243] For each partition of a CU, only unidirectional prediction may be allowed. That is, each partition may have one motion vector and one reference index. The unidirectional prediction constraint is to ensure that only two motion compensation predictions are required for each CU, which is similar to bidirectional prediction.

[0244] When the split mode is applied, a flag indicating the split direction (diagonal or anti-diagonal) and two merge indexes (for each partition) may be additionally signaled.

[0245] After predicting each partition, a hybrid process with adaptive weights can be used to adjust sample values ​​based on boundary lines along diagonal or anti-diagonal lines.

[0246] In addition, when the merge mode or skip mode is applied, motion information may be derived based on the normal merge mode, the MMVD mode (merge mode with motion vector difference), the merge sub-block mode, the CIIP mode (combined inter-picture merge and intra-picture prediction mode), or the partition mode may be used to derive motion information to generate prediction samples as described above. Each mode may be enabled or disabled by an on / off flag in a sequence parameter set (SPS). If the on / off flag for a particular mode is disabled in the SPS, the syntax clearly sent for the prediction mode in units of CU or PU may not be signaled.

[0247] Table 5 below relates to the process of deriving merge mode or skip mode from the conventional merge_data syntax. In Table 5 below, CUMergeTriangleFlag[x0][y0] may correspond to Fig.12 The on / off flag for the partition mode described in , and merge_triangle_split_dir[x0][y0] can indicate the partition direction (diagonal direction or anti-diagonal direction) when the partition mode is applied. In addition, merge_triangle_idx0[x0][y0] and merge_triangle_idx1[x0][y0] can indicate two merge indexes for each partition when the partition mode is applied.

[0248] [Table 5]

[0249]

[0250]

[0251] In addition, each prediction mode including normal merge mode, MMVD mode, merge sub-block mode, CIIP mode, and partition mode can be enabled or disabled from a sequence parameter set (SPS), as shown in Table 6 below. In Table 6 below, sps_triangle_enabled_flag may correspond to enabling or disabling the above in SPS Fig.12 Flags for the segmentation mode described in .

[0252] [Table 6]

[0253]

[0254]

[0255]

[0256]

[0257] The merge_data syntax of Table 5 may be parsed or derived according to the flag of the SPS of Table 6 and the condition under which each prediction mode may be used. All cases according to the flag of the SPS and the condition under which each prediction mode may be used are shown in Tables 7 and 8. Table 7 shows the number of cases in which the current block is in merge mode, and Table 8 shows the number of cases in which the current block is in skip mode. In the following Tables 7 and 8, conventional may correspond to conventional merge mode, mmvd may correspond to triangle, or TRI may correspond to the above referenced Fig.12 The segmentation pattern described.

[0258] [Table 7]

[0259]

[0260] [Table 8]

[0261]

[0262] As an example of the cases mentioned in Tables 7 and 8, the case where the current block is 4×16 and skip mode is described. When the merge subblock mode, MMVD mode, CIIP mode, and split mode are all enabled in the SPS, if regular_merge_flag[x0][y0], mmvd_flag[x0][y0], and merge_subblock_flag[x0][y0] are all 0 in the merge_data syntax, the motion information of the current block should be derived in the split mode. However, even if the split mode is enabled from the on / off flag in the SPS, it can be used as a prediction mode only when the conditions of the following Table 9 are additionally met. In the following Table 9, MergeTriangleFlag[x0][y0] may correspond to the on / off flag for the split mode, and sps_triangle_enabled_flag may correspond to a flag from the SPS that enables or disables the split mode.

[0263] [Table 9]

[0264]

[0265] Referring to Table 9 above, if the current slice is a P slice, the decoder may not be able to decode any bitstreams because the prediction samples cannot be generated by the partitioning mode. In this way, in order to solve the problem that occurs in the exceptional case where the final prediction mode cannot be selected according to each on / off flag of the SPS and the merge data syntax and the decoding is not performed, in the present disclosure, a default merge mode is proposed. The default merge mode can be predefined in various ways, or can be derived by additional syntax signaling.

[0266] In an embodiment, the conventional merge mode may be applied to the current block based on a situation in which the MMVD mode, the merge sub-block mode, the CIIP mode, and the partition mode for performing prediction by dividing the current block into two partitions are all unavailable. That is, when the merge mode cannot be selected for the current block in the end, the conventional merge mode may be applied as the default merge mode.

[0267] For example, although a value of a general merge flag indicating whether a merge mode is available for a current block is 1, if a merge mode cannot be finally selected for the current block, the general merge mode may be applied as a default merge mode.

[0268] In this case, the prediction sample of the current block can be generated using the merge index information indicating one of the merge candidates included in the merge candidate list generated by applying the normal merge mode. For example, the motion information of the current block can be derived based on the indicated merge index information, and the prediction sample can be generated based on the derived motion information.

[0269] Therefore, the merge data syntax can be shown in Table 10 below.

[0270] [Table 10]

[0271]

[0272] Referring to Table 10 and Table 6, based on the case where the MMVD mode is unavailable, the flag sps_mmvd_enabled_flag from the SPS for enabling or disabling the MMVD mode may be 0 or the first flag (mmvd_merge_flag[x0][y0]) indicating whether the MMVD mode is applied may be 0.

[0273] In addition, based on the situation where the merge subblock mode is unavailable, a flag sps_affine_enabled_flag from the SPS for enabling or disabling the merge subblock mode may be 0 or a second flag (merge_subblock_flag[x0][y0]) indicating whether the merge subblock mode is applied may be 0.

[0274] In addition, based on the case where the CIIP mode is not available, the flag sps_ciip_enabled_flag from the SPS for enabling or disabling the CIIP mode may be 0 or the third flag (ciip_flag[x0][y0]) indicating whether the CIIP mode is applied may be 0.

[0275] In addition, based on the situation where the split mode is not available, the flag sps_triangle_enabled_flag from the SPS for enabling or disabling the split mode may be 0 or the fourth flag (MergeTriangleFlag[x0][y0]) indicating whether the split mode is applied may be 0.

[0276] In addition, for example, based on the case where the value of the general merge flag indicating whether the merge mode is applicable to the current block is 1, the values ​​of the first flag mmvd_merge_flag[x0][y0] indicating whether the MMVD mode is applied, the second flag merge_subblock_flag[x0][y0] indicating whether the merge subblock mode is applied, and the third flag ciip_flag[x0][y0] indicating whether the CIIP mode is applied can be signaled.

[0277] Furthermore, for example, based on a case in which the split mode is disabled based on the flag sps_triangle_enabled_flag, the fourth flag (MergeTriangleFlag[x0][y0]) indicating whether the split mode is applied may be set to 0.

[0278] In another embodiment, based on the unavailability of the conventional merge mode, the MMVD mode, the merge sub-block mode, the CIIP mode, and the partition mode for performing prediction by dividing the current block into two partitions, the conventional merge mode may be applied to the current block. That is, when the merge mode cannot be selected for the current block in the end, the conventional merge mode may be applied as the default merge mode.

[0279] For example, based on a general merge flag indicating whether the merge mode is available for the current block, in a case where the merge mode cannot be finally selected for the current block although the merge mode is available for the current block, the normal merge mode may be applied as the default merge mode.

[0280] For example, based on the case where the MMVD mode is not available, the flag sps_mmvd_enabled_flag from the SPS for enabling or disabling the MMVD mode may be 0 or the first flag (mmvd_merge_flag[x0][y0]) indicating whether the MMVD mode is applied may be 0.

[0281] In addition, based on the situation where the merge subblock mode is unavailable, a flag sps_affine_enabled_flag from the SPS for enabling or disabling the merge subblock mode may be 0 or a second flag (merge_subblock_flag[x0][y0]) indicating whether the merge subblock mode is applied may be 0.

[0282] In addition, based on the case where the CIIP mode is not available, the flag sps_ciip_enabled_flag from the SPS for enabling or disabling the CIIP mode may be 0 or the third flag (ciip_flag[x0][y0]) indicating whether the CIIP mode is applied may be 0.

[0283] In addition, based on the situation where the split mode is not available, the flag sps_triangle_enabled_flag from the SPS for enabling or disabling the split mode may be 0 or the fourth flag (MergeTriangleFlag[x0][y0]) indicating whether the split mode is applied may be 0.

[0284] In addition, based on the case where the regular merge mode is not available, the fifth flag (regular_merge_flag[x0][y0]) indicating whether the regular merge mode is applied may be 0. That is, even when the value of the fifth flag is 0, the regular merge mode may be applied to the current block based on the case where the MMVD mode, the merge subblock mode, the CIIP mode, and the partition mode are not available.

[0285] In this case, motion information of the current block may be derived based on a first candidate among merge candidates included in a merge candidate list of the current block, and a prediction sample may be generated based on the derived motion information.

[0286] In another embodiment, the conventional merge mode may be applied to the current block based on the fact that the conventional merge mode, the MMVD mode, the merge sub-block mode, the CIIP mode, and the partition mode for performing prediction by dividing the current block into two partitions are unavailable. That is, when the merge mode cannot be selected for the current block in the end, the conventional merge mode may be applied as the default merge mode.

[0287] For example, in a case where a value of a general merge flag indicating whether a merge mode is available for a current block is 1 but a merge mode is not finally selected for the current block, the normal merge mode may be applied as a default merge mode.

[0288] For example, based on the case where the MMVD mode is not available, the flag sps_mmvd_enabled_flag from the SPS for enabling or disabling the MMVD mode may be 0 or the first flag (mmvd_merge_flag[x0][y0]) indicating whether the MMVD mode is applied may be 0.

[0289] In addition, based on the situation where the merge subblock mode is unavailable, a flag sps_affine_enabled_flag from the SPS for enabling or disabling the merge subblock mode may be 0 or a second flag (merge_subblock_flag[x0][y0]) indicating whether the merge subblock mode is applied may be 0.

[0290] In addition, based on the case where the CIIP mode is not available, the flag sps_ciip_enabled_flag from the SPS for enabling or disabling the CIIP mode may be 0 or the third flag (ciip_flag[x0][y0]) indicating whether the CIIP mode is applied may be 0.

[0291] In addition, based on the situation where the split mode is not available, the flag sps_triangle_enabled_flag from the SPS for enabling or disabling the split mode may be 0 or the fourth flag (MergeTriangleFlag[x0][y0]) indicating whether the split mode is applied may be 0.

[0292] In addition, based on the case where the regular merge mode is not available, the fifth flag (regular_merge_flag[x0][y0]) indicating whether the regular merge mode is applied may be 0. That is, even when the value of the fifth flag is 0, the regular merge mode may be applied to the current block based on the case where the MMVD mode, the merge subblock mode, the CIIP mode, and the partition mode are not available.

[0293] In this case, the (0, 0) motion vector can be derived as the motion information of the current block, and the prediction sample of the current block can be generated based on the (0, 0) motion information. For the (0, 0) motion vector, prediction can be performed with reference to the 0th reference picture of the L0 reference list. However, when the 0th reference picture of the L0 reference list (RefPicList[0][0]) does not exist, prediction can be performed by referring to the 0th reference picture of the L1 reference list (RefPicList[1][0]).

[0294] Fig.17 and Fig.18 An example of a video / image encoding method and related components according to an embodiment of the present disclosure is schematically shown.

[0295] Fig.17 The method disclosed in Figure 2 or Fig.18 Specifically, for example, Fig.17 Steps S1700 to S1710 may be performed by Fig.18 The predictor 220 of the encoding device 200 performs, and Fig.17 Step S1720 may be performed by Fig.18 The residual processor 230 of the encoding device is executed. Fig.17 Step S1730 may be performed by Fig.18 The encoding is performed by the entropy encoder 240 of the encoding device. Fig.17 The method disclosed in may include the embodiments described above in the present disclosure.

[0296] Reference Fig.17, the encoding device may determine an inter prediction mode of the current block and generate inter prediction mode information indicating the inter prediction mode (S1700). For example, the encoding device may determine at least one of a normal merge mode, a skip mode, a motion vector prediction (MVP) mode, a merge mode with a motion vector difference (MMVD), a merge subblock mode, a CIIP mode (combined inter-picture merging and intra-picture prediction mode), and a partition mode in which prediction is performed by dividing the current block into two partitions as the inter prediction mode to be applied to the current block and generate the inter prediction mode information indicating the inter prediction mode.

[0297] The encoding device may generate a prediction sample of the current block based on the determined prediction mode (S1710). For example, the encoding device may generate a merge candidate list according to the determined inter prediction mode.

[0298] For example, a candidate may be inserted into the merge candidate list until the number of candidates in the merge candidate list is the maximum number of candidates. Here, the candidate may indicate a candidate or candidate block for deriving motion information (or motion vector) of the current block. For example, the candidate block may be derived by searching the neighboring blocks of the current block. For example, the neighboring blocks may include spatial neighboring blocks and / or temporal neighboring blocks of the current block, and the spatial neighboring blocks may be preferentially searched (spatial merging) to derive the candidate, and the derived candidate may be inserted into the merge candidate list. For example, when the number of candidates in the merge candidate list (even after the candidate is inserted) is less than the maximum number of candidates in the merge candidate list, an additional candidate may be inserted. For example, the additional candidate may include at least one of a history-based merge candidate, a pairwise average merge candidate, an ATMVP, and a combined bidirectional prediction merge candidate (when the slice / tile group type of the current slice / tile group is type B) and / or a zero vector merge candidate.

[0299] As described above, the merge candidate list may include at least some of spatial merge candidates, temporal merge candidates, pair candidates, or zero vector candidates, and one of these candidates may be selected for inter prediction of the current block.

[0300] For example, the selection information may include index information indicating one of the merge candidates included in the merge candidate list. For example, the selection information may be referred to as merge index information.

[0301] For example, the encoding device may generate a prediction sample of the current block based on the candidate indicated by the merge index information. Alternatively, for example, the encoding device may derive motion information based on the candidate indicated by the merge index information, and may generate a prediction sample of the current block based on the motion information.

[0302] Furthermore, according to an embodiment, the inter prediction mode information includes a general merge flag indicating whether the merge mode is available for the current block, and based on the general merge flag, the merge mode is available for the current block.

[0303] At this time, according to an embodiment, the conventional merge mode can be applied because the MMVD mode (merge mode with motion vector difference), the merge sub-block mode, the CIIP mode (combined inter-picture merging and intra-picture prediction mode), and the partition mode for performing prediction by dividing the current block into partitions are unavailable.

[0304] For example, the inter-prediction mode information may include merge index information indicating one of the merge candidates included in the merge candidate list generated by applying the normal merge mode, and the prediction sample may be generated using the merge index information. That is, the motion information of the current block may be derived based on the candidate indicated by the merge index information, and the prediction sample of the current block may be generated based on the derived motion information.

[0305] For example, the inter prediction mode information may include a first flag indicating whether the MMVD mode is applied, a second flag indicating whether the merge subblock mode is applied, and a third flag indicating whether the CIIP mode is applied.

[0306] For example, based on a case where the MMVD mode, the merging sub-block mode, the CIIP mode, and the split mode are unavailable, the values ​​of the first flag, the second flag, and the third flag may all be zero.

[0307] Furthermore, for example, based on a case where the value of the general merge flag is 1, the merge mode may be available for the current block, and based on a case where the general merge flag is 1, the values ​​of the first flag, the second flag, and the third flag may be signaled.

[0308] For example, a flag for enabling or disabling the segmentation mode may be included in a sequence parameter set (SPS) of the image information, and based on the case of disabling the segmentation mode, the value of the fourth flag indicating whether the segmentation mode is applied may be set to 0.

[0309] In addition, the inter prediction mode information may further include a fifth flag indicating whether the normal merge mode is applied. Even when the value of the fifth flag is 0, the normal merge mode may be applied to the current block based on the case where the MMVD mode, the merge subblock mode, the CIIP mode, and the partition mode are unavailable.

[0310] In this case, the motion information of the current block may be derived based on the first merge candidate among the merge candidates included in the merge candidate list of the current block. In addition, the prediction sample may be generated based on the motion information of the current block derived based on the first merge candidate.

[0311] Alternatively, in this case, motion information of the current block may be derived based on the (0, 0) motion vector, and a prediction sample may be generated based on the motion information of the current block derived based on the (0, 0) motion vector.

[0312] The encoding device may generate residual information based on the residual sample of the current block (S1720). For example, the encoding device may derive the residual sample based on the predicted sample and the original sample. For example, the encoding device may generate residual information indicating the quantized transform coefficient of the residual sample. The residual information may be generated by various encoding methods such as exponential Golomb, CAVLC, CABAC, etc.

[0313] The encoding device may encode image information including inter-frame prediction mode information (S1730). For example, the image information may be referred to as video information. According to the embodiments of the present disclosure described above, the image information may include various information. For example, the image information may include at least some of the prediction-related information or the residual-related information. For example, the prediction-related information may include at least some of the inter-frame prediction mode information, the selection information, and the inter-frame prediction type information. For example, the encoding device may encode all or part of the image information including the aforementioned information (or syntax elements) to generate a bitstream or encoding information. Alternatively, the encoding device may output the information in the form of a bitstream. In addition, the bitstream or encoding information may be sent to a decoding device via a network or a storage medium.

[0314] Alternatively, although Fig.17 Although not shown in the figure, for example, the encoding device may generate a reconstructed sample based on the residual sample and the prediction sample. A reconstructed block and a reconstructed picture may be derived based on the reconstructed sample. Alternatively, for example, the encoding device may encode image information including residual information or prediction-related information.

[0315] For example, the encoding device may generate a bitstream or coding information by encoding all or part of the image information including the aforementioned information (or syntax elements). Alternatively, the encoding device may output information in the form of a bitstream. In addition, the bitstream or coding information may be sent to a decoding device via a network or a storage medium. Alternatively, the bitstream or coding information may be stored in a computer-readable storage medium, and the bitstream or coding information may be generated by the aforementioned image coding method.

[0316] Fig.19 and Fig. 20 An example of a video / image decoding method and related components according to an embodiment of the present disclosure is schematically shown.

[0317] Fig.19 The method disclosed in can be Figure 3 or Fig. 20 The decoding device shown is executed. Specifically, for example, Fig.19 Step S1900 in Fig. 20 The entropy decoder 310 of the decoding device 300 in the embodiment of the present invention is executed, and Fig.19 Step S1910 in Fig. 20 The residual processor 320 of the decoding device 300 is executed. In addition, Fig.19 Step S1920 may be performed by Fig. 20 The predictor 330 of the decoding device 300 in FIG. 1 is executed, and Fig.19 Step S1930 may be performed by Fig. 20 The adder 340 of the decoding device 300 is executed. Fig.19 The method disclosed in may include the embodiments described above in the present disclosure.

[0318] Reference Fig.19 , the decoding device may receive image information including inter-frame prediction mode information and residual information through a bit stream (S1900). For example, the image information may be referred to as video information. The image information may include various information according to the aforementioned embodiments of the present disclosure. For example, the image information may include at least a portion of prediction-related information or residual-related information.

[0319] For example, the prediction related information may include inter-frame prediction mode information or inter-frame prediction type information. For example, the inter-frame prediction mode information may include information indicating at least some of the various inter-frame prediction modes. For example, various modes such as a conventional merge mode, a skip mode, an MVP (motion vector prediction) mode, an MMVD mode (merge mode with motion vector difference), a merged sub-block mode, a CIIP mode (combined inter-picture merge and intra-picture prediction mode), and a partition mode in which prediction is performed by dividing the current block into two partitions may be used. For example, the inter-frame prediction type information may include an inter_pred_idc syntax element. Alternatively, the inter-frame prediction type information may include information indicating any one of L0 prediction, L1 prediction, and bidirectional prediction.

[0320] The decoding apparatus may generate residual samples based on the residual information (S1910). The decoding apparatus may derive quantized transform coefficients based on the residual information, and may derive residual samples based on an inverse transform process for the transform coefficients.

[0321] The decoding device may generate a prediction sample of the current block by applying the prediction mode determined based on the inter-prediction mode information (S1910). For example, the decoding device may generate a merge candidate list based on the inter-prediction mode information as the inter-prediction mode of the current block, according to the inter-prediction mode determined among the normal merge mode, the skip mode, the MVP mode, the MMVD mode, the merge sub-block mode, the CIIP mode, and the partition mode in which the prediction is performed by dividing the current block into two partitions.

[0322] For example, a candidate may be inserted into the merge candidate list until the number of candidates in the merge candidate list is the maximum number of candidates. Here, the candidate may indicate a candidate or candidate block for deriving motion information (or motion vector) of the current block. For example, the candidate block may be derived by searching the neighboring blocks of the current block. For example, the neighboring blocks may include spatial neighboring blocks and / or temporal neighboring blocks of the current block, and the spatial neighboring blocks may be preferentially searched (spatial merging) to derive the candidate, and the derived candidate may be inserted into the merge candidate list. For example, when the number of candidates in the merge candidate list (even after the candidate is inserted) is less than the maximum number of candidates in the merge candidate list, an additional candidate may be inserted. For example, the additional candidate may include at least one of a history-based merge candidate, a pairwise average merge candidate, an ATMVP, and a combined bidirectional prediction merge candidate (when the slice / tile group type of the current slice / tile group is type B) and / or a zero vector merge candidate.

[0323] As described above, the merge candidate list may include at least some of spatial merge candidates, temporal merge candidates, pair candidates, or zero vector candidates, and one of these candidates may be selected for inter prediction of the current block.

[0324] For example, the selection information may include index information indicating one of the merge candidates included in the merge candidate list. For example, the selection information may be referred to as merge index information.

[0325] For example, the decoding device may generate a prediction sample of the current block based on the candidate indicated by the merge index information. Alternatively, for example, the decoding device may derive motion information based on the candidate indicated by the merge index information, and may generate a prediction sample of the current block based on the motion information.

[0326] In addition, according to an embodiment, the inter prediction mode information may include a general merge flag indicating whether the merge mode is available for the current block, and the merge mode may be available for the current block based on the general merge flag.

[0327] Here, based on the fact that the merge mode with motion vector difference (MMVD) mode, the merge subblock mode, the combined inter-picture merge and intra-picture prediction (CIIP) mode, and the partition mode for performing prediction by dividing the current block into two partitions are unavailable, the normal merge mode may be applied.

[0328] For example, the inter-prediction mode information may include merge index information indicating one of the merge candidates included in the merge candidate list generated by applying the normal merge mode, and the prediction sample may be generated using the merge index information. That is, the motion information of the current block may be derived based on the candidate indicated by the merge index information, and the prediction sample of the current block may be generated based on the derived motion information.

[0329] For example, the inter prediction mode information may include a first flag indicating whether the MMVD mode is applied, a second flag indicating whether the merge subblock mode is applied, and a third flag indicating whether the CIIP mode is applied.

[0330] For example, based on a case where the MMVD mode, the merging sub-block mode, the CIIP mode, and the split mode are unavailable, the values ​​of the first flag, the second flag, and the third flag may all be zero.

[0331] Furthermore, for example, based on a case where the value of the general merge flag is 1, the merge mode may be enabled for the current block, and based on a case where the value of the general merge flag is 1, the values ​​of the first flag, the second flag, and the third flag may be signaled.

[0332] For example, a flag for enabling or disabling the segmentation mode may be included in a sequence parameter set (SPS) of the image information, and based on the case of disabling the segmentation mode, the value of the fourth flag indicating whether the segmentation mode is applied may be set to 0.

[0333] In addition, the inter prediction mode information may further include a fifth flag indicating whether the normal merge mode is applied. Even when the value of the fifth flag is 0, the normal merge mode may be applied to the current block based on the case where the MMVD mode, the merge subblock mode, the CIIP mode, and the partition mode are unavailable.

[0334] In this case, the motion information of the current block may be derived based on the first merge candidate among the merge candidates included in the merge candidate list of the current block. In addition, the prediction sample may be generated based on the motion information of the current block derived based on the first merge candidate.

[0335] Alternatively, in this case, motion information of the current block may be derived based on the (0, 0) motion vector, and a prediction sample may be generated based on the motion information of the current block derived based on the (0, 0) motion vector.

[0336] The decoding device may generate a reconstructed sample based on the prediction sample (S1930). For example, the decoding device may generate a reconstructed sample based on the prediction sample and the residual sample, and may derive a reconstructed block and a reconstructed picture based on the reconstructed sample.

[0337] For example, the decoding device can obtain video / image information including all or part of the above-mentioned multiple pieces of information (or syntax elements) by decoding the bit stream or coding information. In addition, the bit stream or coding information can be stored in a computer-readable storage medium, and can cause the above-mentioned decoding method to be executed.

[0338] Although the method has been described based on a flowchart that lists steps or blocks in sequence in the above-mentioned embodiments, the steps of this document are not limited to a specific order, and specific steps may be performed in different steps or in a different order or simultaneously relative to the above-mentioned steps. In addition, it will be understood by a person of ordinary skill in the art that the steps in the flowchart are not exclusive, and another step may be included therein, or one or more steps in the flowchart may be deleted without affecting the scope of the present disclosure.

[0339] The above-mentioned method according to the present disclosure may be in the form of software, and the encoding device and / or decoding device according to the present disclosure may be included in a device for performing image processing (e.g., TV, computer, smart phone, set-top box, display device, etc.).

[0340] When the embodiments of the present disclosure are implemented by software, the above-mentioned methods can be implemented by modules (processing or functions) that perform the above-mentioned functions. The module can be stored in a memory and executed by a processor. The memory can be installed inside or outside the processor and can be connected to the processor via various well-known devices. The processor may include an application-specific integrated circuit (ASIC), other chipsets, logic circuits and / or data processing devices. The memory may include a read-only memory (ROM), a random access memory (RAM), a flash memory, a memory card, a storage medium and / or other storage devices. In other words, according to the embodiments of the present disclosure, it can be implemented and executed on a processor, a microprocessor, a controller or a chip. For example, the functional units illustrated in the corresponding figures can be implemented and executed on a computer, a processor, a microprocessor, a controller or a chip. In this case, information about the implementation (e.g., information about instructions) or an algorithm can be stored in a digital storage medium.

[0341] In addition, the decoding device and the encoding device applying the embodiments of this document can be included in a multimedia broadcast transceiver, a mobile communication terminal, a home theater video device, a digital theater video device, a surveillance camera, a video chat device, a real-time communication device such as video communication, a mobile streaming device, a storage medium, a portable camera, a video on demand (VoD) service provider, an over-the-top (OTT) video device, an Internet streaming service provider, a 3D video device, a virtual reality (VR) device, an augmented reality (AR) device, an image phone video device, a vehicle terminal (e.g., a vehicle (including an autonomous vehicle) terminal, an aircraft terminal or a ship terminal) and a medical video device; and can be used to process image signals or data. For example, an OTT video device may include a game console, a Blueray player, a networked TV, a home theater system, a smart phone, a tablet PC, and a digital video recorder (DVR).

[0342] In addition, the processing method of the embodiment of the present document can be generated in the form of a program executed by a computer and can be stored in a computer-readable recording medium. The multimedia data with a data structure according to the embodiment of the present document can also be stored in a computer-readable recording medium. The computer-readable recording medium includes all kinds of storage devices and distributed storage devices storing computer-readable data. The computer-readable recording medium may include, for example, a Blu-ray disc (BD), a universal serial bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. The computer-readable recording medium also includes a medium implemented in the form of a carrier wave (e.g., transmission on the Internet). In addition, the bit stream generated by the encoding method can be stored in a computer-readable recording medium, or can be transmitted through a wired or wireless communication network.

[0343] In addition, the embodiments of the present document can be implemented as a computer program product based on a program code, and the program code can be executed on a computer according to the embodiments of the present document. The program code can be stored on a computer readable carrier.

[0344] Fig.21 An example of a content streaming system to which embodiments of this document may be applied is shown.

[0345] Reference Fig.21 , a content streaming system to which embodiments of this document are applied may generally include an encoding server, a streaming server, a network server, a media storage, a user device, and a multimedia input device.

[0346] The encoding server is used to compress the content input from the multimedia input device such as a smart phone, a camera, a camcorder, etc. into digital data, generate a bit stream, and transmit it to the streaming server. As another example, in the case where the multimedia input device such as a smart phone, a camera, a camcorder, etc. directly generates a bit stream, the encoding server can be omitted.

[0347] The bitstream may be generated by an encoding method or a bitstream generation method to which the embodiments of this document are applied, and the streaming server may temporarily store the bitstream in the process of sending or receiving the bitstream.

[0348] The streaming server transmits multimedia data to the user device through the network server based on the user's request, and the network server acts as a tool to inform the user of what services exist. When the user requests the service the user wants, the network server transfers the request to the streaming server, and the streaming server transmits the multimedia data to the user. In this regard, the content streaming system may include a separate control server, and in this case, the control server is used to control the command / response between the various devices in the content streaming system.

[0349] The streaming server may receive content from a media storage device and / or an encoding server. For example, when receiving content from an encoding server, the content may be received in real time. In this case, the streaming server may store the bitstream for a predetermined period of time to smoothly provide a streaming service.

[0350] For example, user devices may include mobile phones, smart phones, laptop computers, digital broadcast terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation, tablet PCs, tablet PCs, ultrabooks, wearable devices (e.g., watch-type terminals (smart watches), glasses-type terminals (smart glasses), head-mounted displays (HMDs)), digital TVs, desktop computers, digital signage, etc.

[0351] Each server in the content streaming system may be operated as a distributed server, and in this case, data received by each server may be processed in a distributed manner.

[0352] The claims in this specification can be combined in various ways. For example, the technical features in the method claims of this specification can be combined to be implemented or performed in a device, and the technical features in the device claims can be combined to be implemented or performed in a method. In addition, the technical features in the method claims and the device claims can be combined to be implemented or performed in a device. In addition, the technical features in the method claims and the device claims can be combined to be implemented or performed in a method.

Claims

1. A decoding device for image decoding, the decoding device comprising: Memory; as well as at least one processor connected to the memory, the at least one processor being configured to: Acquire image information including inter-frame prediction mode information and residual information through a bit stream; generating residual samples based on the residual information; generating a prediction sample of the current block by applying a prediction mode determined based on the inter-frame prediction mode information; and generating a reconstructed sample based on the predicted sample and the residual sample, The inter-frame prediction mode information includes a first flag related to whether a sub-block merge mode is applied, a second flag related to whether a normal merge mode is applied, and a third flag related to whether a merge mode with a motion vector difference (MMVD) is applied. The parsing order of the second flag is before the parsing order of the third flag. wherein a CIIP mode based on combined inter-picture merging and intra-picture prediction is not enabled, a partition prediction mode for performing prediction by dividing the current block into two partitions is not enabled, a value of the first flag is equal to 0, and a value of the third flag is equal to 0, the inter-frame prediction mode information includes merge index information indicating one of the merge candidates included in the merge candidate list generated for the regular merge mode, wherein the predicted sample is generated based on the merge index information, and Wherein, based on the value of the general merge flag being 1 and the MMVD being not enabled, the normal merge mode is applied to the current block.

2. A coding device for image coding, the coding device comprising: Memory; as well as at least one processor connected to the memory, the at least one processor being configured to: determining an inter prediction mode of a current block and generating inter prediction mode information indicating the inter prediction mode; Generating a prediction sample of the current block based on the determined prediction mode; generating residual information based on residual samples for the current block; and encoding the image information including the inter-frame prediction mode information and the residual information, The inter-frame prediction mode information includes a first flag related to whether a sub-block merge mode is applied, a second flag related to whether a normal merge mode is applied, and a third flag related to whether a merge mode with a motion vector difference (MMVD) is applied. The parsing order of the second flag is before the parsing order of the third flag. wherein a CIIP mode based on combined inter-picture merging and intra-picture prediction is not enabled, a partition prediction mode for performing prediction by dividing the current block into two partitions is not enabled, a value of the first flag is equal to 0 and a value of the third flag is equal to 0, the inter-prediction mode information includes merge index information indicating one of the merge candidates included in the merge candidate list generated for the normal merge mode, and Wherein, based on the value of the general merge flag being 1 and the MMVD being not enabled, the normal merge mode is applied to the current block.

3. A device for transmitting data for an image, the device comprising: at least one processor configured to obtain a bitstream for the image, wherein the bitstream is generated based on the following operations: determining an inter-frame prediction mode of a current block and generating inter-frame prediction mode information indicating the inter-frame prediction mode, generating a prediction sample of the current block based on the determined prediction mode, generating residual information based on a residual sample of the current block, and encoding image information including the inter-frame prediction mode information and the residual information; and a transmitter configured to transmit the data including the bit stream, The inter-frame prediction mode information includes a first flag related to whether a sub-block merge mode is applied, a second flag related to whether a normal merge mode is applied, and a third flag related to whether a merge mode with a motion vector difference (MMVD) is applied. The parsing order of the second flag is before the parsing order of the third flag. wherein a CIIP mode based on combined inter-picture merging and intra-picture prediction is not enabled, a partition prediction mode for performing prediction by dividing the current block into two partitions is not enabled, a value of the first flag is equal to 0 and a value of the third flag is equal to 0, the inter-prediction mode information includes merge index information indicating one of the merge candidates included in the merge candidate list generated for the normal merge mode, and Wherein, based on the value of the general merge flag being 1 and the MMVD being not enabled, the normal merge mode is applied to the current block.

Images