Methods for decoding and encoding images, and methods for transmitting image data.

Abstract

The invention relates to a method for decoding images performed by a decoder-side device as described in the invention, comprising the following steps: generating a motion vector for the current block; determining whether a motion vector refinement is applicable to the current block; generating a refined motion vector by applying decoder-side motion vector refinement (DMVR) to the current block if it is determined that a motion vector refinement is applicable to the current block; generating predictive patterns for the current block on the basis of the refined motion vector; and generating reconstructed patterns for the current block on the basis of the predictive patterns, wherein the step of determining whether a refinement is applicable contains a determination of whether a refinement is applicable on the basis of at least one component of the size of the current block and the double prediction weighting information for the current block.The invention also reveals a method for encoding images and a method for transmitting data to images.

Description

DMVR-based inter-prediction method and device

[0001] This document relates to video coding technology, and more specifically, to a method and device for performing inter prediction based on Decoder-side Motion Vector Refinement (DMVR).

[0002] Demand for high-resolution, high-quality video, such as 4K or 8K Ultra High Definition (UHD) video, is growing across a variety of fields. As video data becomes higher-resolution and higher-quality, the amount of information or bits transmitted increases relative to conventional video data. Therefore, when transmitting video data using existing media, such as wired or wireless broadband lines, or storing video data using existing storage media, transmission and storage costs increase.

[0003] In addition, interest in and demand for immersive media such as VR (Virtual Reality), AR (Artificial Reality) content and holograms have been increasing recently, and broadcasting of images / videos with different image characteristics from reality images, such as game images, is increasing.

[0004] Accordingly, a highly efficient image / video compression technology is required to effectively compress, transmit, store, and play high-resolution, high-quality image / video information having various characteristics as described above.

[0005] The technical task of this document is to provide a method and device for improving video coding efficiency.

[0006] Another technical challenge of this paper is to provide an efficient inter prediction method and device.

[0007] Another technical challenge of this paper is to provide a method and device for performing inter prediction based on Decoder-side Motion Vector Refinement (DMVR).

[0008] Another technical challenge of this paper is to provide a method and device for performing inter prediction based on BDOF (Bi-directional optical flow).

[0009] Another technical challenge of this document is to provide a method and device for improving prediction performance by providing conditions for determining whether to apply DMVR to improve image coding efficiency and / or conditions for determining whether to apply BDOF.

[0010] According to one embodiment of the present document, a video decoding method performed by a decoding device is provided. The method includes the steps of: deriving a motion vector of a current block; determining whether to apply refinement to the motion vector of the current block; applying Decoder-side Motion Vector Refinement (DMVR) to the motion vector to derive a refined motion vector when it is determined to apply refinement to the motion vector of the current block; deriving prediction samples for the current block based on the refined motion vector; and generating reconstruction samples for the current block based on the prediction samples, wherein the step of determining whether to apply refinement determines whether to apply the refinement based on at least one of a size of the current block or bi-prediction weight index information of the current block.

[0011] According to another embodiment of the present document, a video encoding method performed by an encoding device is provided. The method includes the steps of: deriving a motion vector of a current block; determining whether to apply refinement to the motion vector of the current block; applying Decoder-side Motion Vector Refinement (DMVR) to the motion vector to derive a refined motion vector when it is determined to apply refinement to the motion vector of the current block; deriving prediction samples for the current block based on the refined motion vector; deriving residual samples based on the prediction samples; generating residual information based on the residual samples; and encoding image information including the residual information. The step of determining whether to apply refinement is characterized in that whether to apply refinement is determined based on at least one of a size of the current block or bi-prediction weight index information of the current block.

[0012] According to this document, the overall image / video compression efficiency can be improved.

[0013] According to this paper, computational complexity can be reduced and overall coding efficiency can be improved through efficient inter prediction.

[0014] According to this paper, by proposing various application conditions when applying DMVR and / or BDOF to refine motion information in the motion compensation process, efficiency in terms of complexity and performance can be improved.

[0015] Figure 1 schematically illustrates an example of a video / image coding system that can be applied to embodiments of the present document.

[0016] FIG. 2 is a drawing schematically illustrating the configuration of a video / image encoding device that can be applied to embodiments of this document.

[0017] FIG. 3 is a drawing schematically illustrating the configuration of a video / image decoding device that can be applied to embodiments of the present document.

[0018] FIG. 4 is a diagram illustrating one embodiment of a process for performing DMVR (Decoder-side Motion Vector Refinement) in true pair prediction.

[0019] Figure 5 is an example showing a method of performing a decoding process by checking the application conditions of DMVR and BDOF.

[0020] Figures 6 and 7 are other examples showing a method of performing a decoding process by checking the application conditions of DMVR and BDOF.

[0021] FIG. 8 is a flowchart schematically illustrating an encoding method that can be performed by an encoding device according to one embodiment of the present document.

[0022] FIG. 9 is a flowchart schematically illustrating a decoding method that may be performed by a decoding device according to one embodiment of a document.

[0023] FIG. 10 illustrates an example of a content streaming system to which the embodiments disclosed in this document can be applied.

[0024] This document may have various modifications and embodiments, and thus specific embodiments are illustrated and described in detail in the drawings. However, this is not intended to limit this document to specific embodiments. The terminology used herein is only used to describe specific embodiments and is not intended to limit the technical idea of ​​this document. The singular expression includes plural expressions unless the context clearly indicates otherwise. It should be understood that the terms "comprises" or "has" in this specification specify the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0025] Meanwhile, each component in the drawings described in this document is depicted independently for the convenience of explaining their distinct functions. This does not imply that each component is implemented using separate hardware or software. For example, two or more components may be combined to form a single component, or a single component may be divided into multiple components. Embodiments in which each component is integrated and / or separated are also included within the scope of this document, as long as they do not deviate from the essence of this document.

[0026] Hereinafter, with reference to the attached drawings, a preferred embodiment of the present document will be described in more detail. Hereinafter, identical components in the drawings will be designated by the same reference numerals, and redundant descriptions of identical components may be omitted.

[0027] Figure 1 schematically illustrates an example of a video / image coding system that can be applied to embodiments of the present document.

[0028] Referring to FIG. 1, a video / image coding system may include a first device (source device) and a second device (receiving device). The source device may transmit encoded video / image information or data to the receiving device via a digital storage medium or a network in the form of a file or streaming.

[0029] The source device may include a video source, an encoding device, and a transmitter. The receiving device may include a receiver, a decoding device, and a renderer. The encoding device may be referred to as a video / video encoding device, and the decoding device may be referred to as a video / video decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, and the display unit may be configured as a separate device or an external component.

[0030] A video source may obtain video / images through a process of capturing, synthesizing, or generating video / images. The video source may include a video / image capture device and / or a video / image generation device. A video / image capture device may include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. A video / image generation device may include, for example, a computer, a tablet, a smartphone, etc., and may (electronically) generate video / images. For example, a virtual video / image may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process of generating related data.

[0031] An encoding device can encode input video / images. The encoding device can perform a series of procedures, such as prediction, transformation, and quantization, to improve compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0032] The transmission unit can transmit encoded video / image information or data output in the form of a bitstream to the receiving unit of a receiving device via a digital storage medium or network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit can include an element for generating a media file via a predetermined file format and an element for transmission via a broadcasting / communication network. The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0033] The decoding device can decode the video / image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoding device.

[0034] The renderer can render decoded video / images. The rendered video / images can be displayed through the display unit.

[0035] This document relates to video / image coding. For example, the methods / embodiments disclosed in this document can be applied to methods disclosed in the VVC (versatile video coding) standard, the EVC (essential video coding) standard, the AV1 (AOMedia Video 1) standard, the AVS2 (2nd generation of audio video coding standard), or the next generation video / image coding standard (e.g., H.267 or H.268).

[0036] This document presents various embodiments of video / image coding, and unless otherwise stated, the embodiments may be performed in combination with each other.

[0037] In this document, a video may refer to a set of images over time. A picture generally refers to a unit representing one image at a specific time point, and a slice / tile is a unit that constitutes a part of a picture in coding. A slice / tile may include one or more CTUs (coding tree units). A picture may be composed of one or more slices / tiles. A picture may be composed of one or more tile groups. A tile group may include one or more tiles. A brick may represent a rectangular region of CTU rows within a tile in a picture. A tile may be partitioned into multiple bricks, each of which consisting of one or more CTU rows within the tile. A tile that is not partitioned into multiple bricks may also be referred to as a brick.A brick scan may represent a specific sequential ordering of CTUs partitioning a picture in which the CTUs are ordered consecutively in a CTU raster scan in a brick, bricks within a tile are ordered consecutively in a raster scan of the bricks of the tile, and tiles in a picture are ordered consecutively in a raster scan of the tiles of the picture. A tile is a rectangular region of CTUs within a particular tile column and a particular tile row in a picture. The tile column is a rectangular region of CTUs having a height equal to the height of the picture and a width specified by syntax elements in the picture parameter set.The tile row is a rectangular region of CTUs having a height specified by syntax elements in the picture parameter set and a width equal to the width of the picture. A tile scan may represent a specific sequential ordering of CTUs partitioning a picture, wherein the CTUs are ordered consecutively in a CTU raster scan in a tile whereas tiles in a picture are ordered consecutively in a raster scan of the tiles of the picture. A slice includes an integer number of bricks of a picture that may be exclusively contained in a single NAL unit.A slice may consist of either a number of complete tiles or only a consecutive sequence of complete bricks of one tile. In this document, the terms tile group and slice may be used interchangeably. For example, in this document, the terms tile group / tile group header may be referred to as slice / slice header.

[0038] A pixel or pel can mean the smallest unit that constitutes a picture (or image). Additionally, the term "sample" can be used as a counterpart to a pixel. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luma component, or only the pixel / pixel value of the chroma component. Alternatively, a sample can mean a pixel value in the spatial domain, or when such a pixel value is converted to the frequency domain, it can mean a transform coefficient in the frequency domain.

[0039] A unit may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. One unit may include one luma block and two chroma (e.g., cb, cr) blocks. In some cases, the term "unit" may be used interchangeably with terms such as "block" or "area." In general, an MxN block may include a set (or array) of samples (or sample array) or transform coefficients consisting of M columns and N rows.

[0040] In this document, " / " and "," are interpreted as "and / or." For example, "A / B" is interpreted as "A and / or B," and "A, B" is interpreted as "A and / or B." Additionally, "A / B / C" means "at least one of A, B, and / or C." Also, "A, B, C" means "at least one of A, B, and / or C." (In this document, the terms " / " and "," should be interpreted to indicate "and / or." For instance, the expression "A / B" may mean "A and / or B." Further, "A, B" may mean "A and / or B." Further, "A / B / C" may mean "at least one of A, B, and / or C." Also, "A / B / C" may mean "at least one of A, B, and / or C.")

[0041] Additionally, in this document, "or" is interpreted as "and / or." For example, "A or B" can mean 1) only "A," 2) only "B," or 3) both "A and B." In other words, "or" in this document can mean "additionally or alternatively." (Furthermore, in the document, the term "or" should be interpreted to indicate "and / or." For instance, the expression "A or B" may comprise 1) only A, 2) only B, and / or 3) both A and B. In other words, the term "or" in this document should be interpreted to indicate "additionally or alternatively.")

[0042] FIG. 2 is a diagram schematically illustrating the configuration of a video / image encoding device that can be applied to the embodiments of this document. Hereinafter, the term "video encoding device" may include an image encoding device.

[0043] Referring to FIG. 2, the encoding device (200) may be configured to include an image partitioner (210), a prediction unit (predictor) 220, a residual processor (residual processor) 230, an entropy encoder (entropy encoder) 240, an adder (adder) 250, a filter (filter) 260, and a memory (memory) 270. The prediction unit (220) may include an inter prediction unit (221) and an intra prediction unit (222). The residual processor (230) may include a transformer (transformer) 232, a quantizer (quantizer) 233, a dequantizer (dequantizer) 234, and an inverse transformer (inverse transformer) 235. The residual processing unit (230) may further include a subtractor (231). The addition unit (250) may be called a reconstructor or a recontructed block generator. The image segmentation unit (210), the prediction unit (220), the residual processing unit (230), the entropy encoding unit (240), the addition unit (250), and the filtering unit (260) described above may be configured by one or more hardware components (e.g., an encoder chipset or a processor) according to an embodiment. In addition, the memory (270) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (270) as an internal / external component.

[0044] The image segmentation unit (210) can segment an input image (or picture, frame) input to the encoding device (200) into one or more processing units. For example, the processing unit may be called a coding unit (CU). In this case, the coding unit may be recursively segmented from a coding tree unit (CTU) or a largest coding unit (LCU) according to a QTBTTT (Quad-tree binary-tree ternary-tree) structure. For example, one coding unit may be segmented into a plurality of coding units of deeper depth based on a quad-tree structure, a binary tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure may be applied first, and the binary tree structure and / or the ternary structure may be applied later. Alternatively, the binary tree structure may be applied first. The coding procedure according to the present document may be performed based on the final coding unit that is no longer segmented. In this case, based on coding efficiency according to image characteristics, etc., the maximum coding unit can be used as the final coding unit, or, if necessary, the coding unit can be recursively divided into coding units of lower depths, and the coding unit of the optimal size can be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration described below. As another example, the processing unit may further include a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transformation unit may each be divided or partitioned from the final coding unit described above.The above prediction unit may be a unit of sample prediction, and the above transformation unit may be a unit for deriving a transformation coefficient and / or a unit for deriving a residual signal from a transformation coefficient.

[0045] The term "unit" may be used interchangeably with terms such as "block" or "area" depending on the case. In general, an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luminance component, or only the pixel / pixel value of the chroma component. A sample can be used as a term corresponding to a pixel or pel in a picture (or image).

[0046] The encoding device (200) can generate a residual signal (residual block, residual sample array) by subtracting a prediction signal (predicted block, prediction sample array) output from an inter prediction unit (221) or an intra prediction unit (222) from an input video signal (original block, original sample array), and the generated residual signal is transmitted to a conversion unit (232). In this case, as illustrated, a unit that subtracts a prediction signal (prediction block, prediction sample array) from an input video signal (original block, original sample array) within the encoder (200) may be called a subtraction unit (231). The prediction unit can perform prediction on a block to be processed (hereinafter, referred to as a current block) and generate a predicted block including prediction samples for the current block. The prediction unit can determine whether intra prediction or inter prediction is applied on a current block or CU basis. The prediction unit can generate various information regarding prediction, such as prediction mode information, as described later in the description of each prediction mode, and transmit the information to the entropy encoding unit (240). The information regarding prediction can be encoded in the entropy encoding unit (240) and output in the form of a bitstream.

[0047] The intra prediction unit (222) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it depending on the prediction mode. In intra prediction, the prediction modes may include multiple non-directional modes and multiple directional modes. The non-directional modes may include, for example, a DC mode and a planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the degree of detail in the prediction direction. However, this is merely an example, and a greater or lesser number of directional prediction modes may be used depending on the settings. The intra prediction unit (222) may also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0048] The inter prediction unit (221) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring block can include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The above temporal neighboring blocks may be called collocated reference blocks, collocated CUs (colCUs), etc., and the reference pictures including the temporal neighboring blocks may be called collocated pictures (colPic). For example, the inter prediction unit (221) may construct a motion information candidate list based on the neighboring blocks, and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, and for example, in the case of skip mode and merge mode, the inter prediction unit (221) may use the motion information of the neighboring blocks as the motion information of the current block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of the surrounding blocks is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0049] The prediction unit (220) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction for prediction of a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document. Palette mode can be viewed as an example of intra coding or intra prediction. When palette mode is applied, sample values ​​within a picture can be signaled based on information about the palette table and palette index.

[0050] The prediction signal generated through the above prediction unit (including the inter prediction unit (221) and / or the intra prediction unit (222)) can be used to generate a restoration signal or to generate a residual signal. The transformation unit (232) can generate transform coefficients by applying a transformation technique to the residual signal. For example, the transformation technique may be a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a KLT ( ), GBT (Graph-Based Transform), or CNT (Conditionally Non-linear Transform). Here, GBT refers to a transformation obtained from a graph when the relationship information between pixels is expressed as a graph. CNT refers to a transformation obtained based on generating a prediction signal using all previously reconstructed pixels. In addition, the transformation process can be applied to a square pixel block with the same size, or to a block with a variable size other than a square.

[0051] The quantization unit (233) quantizes the transform coefficients and transmits them to the entropy encoding unit (240), and the entropy encoding unit (240) can encode the quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients may be called residual information. The quantization unit (233) can rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on a coefficient scan order, and can also generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. The entropy encoding unit (240) can perform various encoding methods, such as, for example, exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc. The entropy encoding unit (240) may encode, together or separately, information necessary for video / image restoration (e.g., values ​​of syntax elements, etc.) in addition to the quantized transform coefficients. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in units of NAL (network abstraction layer) units. The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. In this document, information and / or syntax elements transmitted / signaled from an encoding device to a decoding device may be included in the video / image information. The video / image information may be encoded through the above-described encoding procedure and included in the bitstream.The above bitstream may be transmitted through a network or stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The signal output from the entropy encoding unit (240) may be configured as an internal / external element of the encoding device (200) by a transmitting unit (not shown) and / or a storing unit (not shown), or the transmitting unit may be included in the entropy encoding unit (240).

[0052] The quantized transform coefficients output from the quantization unit (233) can be used to generate a prediction signal. For example, by applying inverse quantization and inverse transformation to the quantized transform coefficients through the inverse quantization unit (234) and the inverse transform unit (235), a residual signal (residual block or residual samples) can be reconstructed. The addition unit (155) can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter prediction unit (221) or the intra prediction unit (222). When there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as a reconstructed block. The addition unit (250) may be called a reconstructor or a reconstructed block generation unit. The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, and can also be used for inter prediction of the next picture after filtering as described below.

[0053] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.

[0054] The filtering unit (260) can improve subjective / objective picture quality by applying filtering to the restoration signal. For example, the filtering unit (260) can apply various filtering methods to the restoration picture to generate a modified restoration picture, and store the modified restoration picture in the memory (270), specifically, in the DPB of the memory (270). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit (260) can generate various information regarding filtering and transmit the information to the entropy encoding unit (240), as described later in the description of each filtering method. The information regarding filtering may be encoded by the entropy encoding unit (240) and output in the form of a bitstream.

[0055] The modified restored picture transmitted to the memory (270) can be used as a reference picture in the inter prediction unit (221). Through this, when inter prediction is applied, the encoding device can avoid prediction mismatch between the encoding device (100) and the decoding device, and can also improve encoding efficiency.

[0056] The memory (270) DPB can store the modified restored picture to be used as a reference picture in the inter prediction unit (221). The memory (270) can store motion information of a block from which motion information is derived (or encoded) within the current picture and / or motion information of blocks within a picture that has already been restored. The stored motion information can be transferred to the inter prediction unit (221) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (270) can store restored samples of restored blocks within the current picture and transfer them to the intra prediction unit (222).

[0057] FIG. 3 is a drawing schematically illustrating the configuration of a video / image decoding device that can be applied to embodiments of the present document.

[0058] Referring to FIG. 3, the decoding device (300) may be configured to include an entropy decoder (310), a residual processor (320), a predictor (330), an adder (340), a filter (350), and a memory (360). The predictor (330) may include an inter-prediction unit (331) and an intra-prediction unit (332). The residual processor (320) may include a dequantizer (321) and an inverse transformer (321). The entropy decoding unit (310), residual processing unit (320), prediction unit (330), addition unit (340), and filtering unit (350) described above may be configured by a single hardware component (e.g., decoder chipset or processor) depending on the embodiment. In addition, the memory (360) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (360) as an internal / external component.

[0059] When a bitstream including video / image information is input, the decoding device (300) can restore the image corresponding to the process in which the video / image information is processed in the encoding device of FIG. 2. For example, the decoding device (300) can derive units / blocks based on block division-related information obtained from the bitstream. The decoding device (300) can perform decoding using a processing unit applied in the encoding device. Therefore, the processing unit of decoding may be, for example, a coding unit, and the coding unit may be divided from a coding tree unit or a maximum coding unit according to a quad tree structure, a binary tree structure, and / or a ternary tree structure. One or more transform units may be derived from the coding unit. Then, the restored image signal decoded and output through the decoding device (300) can be reproduced through a reproduction device.

[0060] The decoding device (300) can receive a signal output from the encoding device of FIG. 2 in the form of a bitstream, and the received signal can be decoded through the entropy decoding unit (310). For example, the entropy decoding unit (310) can parse the bitstream to derive information (e.g., video / image information) necessary for image restoration (or picture restoration). The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The decoding device can decode the picture further based on the information on the parameter set and / or the general constraint information. The signaling / received information and / or syntax elements described later in this document can be decoded and obtained from the bitstream through the decoding procedure. For example, the entropy decoding unit (310) can decode information in a bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the values ​​of syntax elements required for image restoration and the quantized values ​​of transform coefficients for residuals. More specifically, the CABAC entropy decoding method receives a bin corresponding to each syntax element in the bitstream, determines a context model using information of a syntax element to be decoded and decoding information of surrounding and decoding target blocks or information of symbols / bins decoded in the previous step, and predicts the occurrence probability of a bin according to the determined context model to perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element.At this time, the CABAC entropy decoding method can update the context model using the information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Information regarding prediction among the information decoded by the entropy decoding unit (310) is provided to the prediction unit (inter prediction unit (332) and intra prediction unit (331)), and residual values ​​on which entropy decoding is performed by the entropy decoding unit (310), i.e., quantized transform coefficients and related parameter information, can be input to the residual processing unit (320). The residual processing unit (320) can derive a residual signal (residual block, residual samples, residual sample array). In addition, information regarding filtering among the information decoded by the entropy decoding unit (310) can be provided to the filtering unit (350). Meanwhile, a receiving unit (not shown) that receives a signal output from an encoding device may be further configured as an internal / external element of the decoding device (300), or the receiving unit may be a component of an entropy decoding unit (310). Meanwhile, the decoding device according to the present document may be called a video / video / picture decoding device, and the decoding device may be divided into an information decoder (video / video / picture information decoder) and a sample decoder (video / video / picture sample decoder). The information decoder may include the entropy decoding unit (310), and the sample decoder may include at least one of the inverse quantization unit (321), the inverse transformation unit (322), the addition unit (340), the filtering unit (350), the memory (360), the inter prediction unit (332), and the intra prediction unit (331).

[0061] The inverse quantization unit (321) can inverse quantize the quantized transform coefficients and output the transform coefficients. The inverse quantization unit (321) can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scanning order performed in the encoding device. The inverse quantization unit (321) can perform inverse quantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain transform coefficients.

[0062] In the inverse transform unit (322), the transform coefficients are inversely transformed to obtain a residual signal (residual block, residual sample array).

[0063] The prediction unit can perform a prediction on the current block and generate a predicted block containing prediction samples for the current block. Based on the information regarding the prediction output from the entropy decoding unit (310), the prediction unit can determine whether intra-prediction or inter-prediction is applied to the current block, and can determine a specific intra / inter-prediction mode.

[0064] The prediction unit (320) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction for prediction of a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document. Palette mode can be viewed as an example of intra coding or intra prediction. When palette mode is applied, information about the palette table and palette index may be signaled and included in the video / image information.

[0065] The intra prediction unit (331) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the prediction mode. In intra prediction, the prediction modes may include multiple non-directional modes and multiple directional modes. The intra prediction unit (331) can also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0066] The inter prediction unit (332) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks can include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (332) can construct a motion information candidate list based on the neighboring blocks, and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter prediction can be performed based on various prediction modes, and the information about the prediction can include information indicating the mode of inter prediction for the current block.

[0067] The addition unit (340) can generate a restoration signal (restored picture, restoration block, restoration sample array) by adding the acquired residual signal to the prediction signal (predicted block, prediction sample array) output from the prediction unit (including the inter-prediction unit (332) and / or intra-prediction unit (331)). When there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as the restoration block.

[0068] The addition unit (340) may be referred to as a restoration unit or restoration block generation unit. The generated restoration signal may be used for intra prediction of the next processing target block within the current picture, may be output after filtering as described below, or may be used for inter prediction of the next picture.

[0069] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0070] The filtering unit (350) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (350) can apply various filtering methods to the restored picture to generate a modified restored picture, and transmit the modified restored picture to the memory (360), specifically, to the DPB of the memory (360). The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0071] The (modified) reconstructed picture stored in the DPB of the memory (360) can be used as a reference picture in the inter prediction unit (332). The memory (360) can store motion information of a block from which motion information is derived (or decoded) within the current picture and / or motion information of blocks within a picture that has already been reconstructed. The stored motion information can be transferred to the inter prediction unit (332) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (360) can store reconstructed samples of reconstructed blocks within the current picture and transfer them to the intra prediction unit (331).

[0072] In this specification, the embodiments described in the filtering unit (260), the inter prediction unit (221), and the intra prediction unit (222) of the encoding device (200) can be applied to the filtering unit (350), the inter prediction unit (332), and the intra prediction unit (331) of the decoding device (300) in the same or corresponding manner, respectively.

[0073] As described above, prediction is performed to increase compression efficiency when performing video coding. Through this, a predicted block including prediction samples for a current block, which is a coding target block, can be generated. Here, the predicted block includes prediction samples in a spatial domain (or pixel domain). The predicted block is derived identically from an encoding device and a decoding device, and the encoding device can increase video coding efficiency by signaling information (residual information) about the residual between the original block and the predicted block, rather than the original sample value of the original block itself, to a decoding device. The decoding device can derive a residual block including residual samples based on the residual information, and generate a reconstructed block including reconstructed samples by combining the residual block and the predicted block, and can generate a reconstructed picture including the reconstructed blocks.

[0074] The residual information may be generated through a transformation and quantization procedure. For example, the encoding device may derive a residual block between the original block and the predicted block, perform a transformation procedure on residual samples (a residual sample array) included in the residual block to derive transform coefficients, and perform a quantization procedure on the transform coefficients to derive quantized transform coefficients, thereby signaling the related residual information to a decoding device (via a bitstream). Here, the residual information may include information such as value information, position information, a transformation technique, a transformation kernel, and quantization parameters of the quantized transform coefficients. The decoding device may perform an inverse quantization / inverse transformation procedure based on the residual information to derive residual samples (or residual blocks). The decoding device may generate a reconstructed picture based on the predicted block and the residual block. The encoding device can also inversely quantize / inversely transform the quantized transform coefficients to derive a residual block for reference in inter prediction of a subsequent picture, and generate a restored picture based on the residual block.

[0075] Meanwhile, as described above, intra-prediction or inter-prediction can be applied when performing prediction for the current block. The following describes the case where inter-prediction is applied to the current block.

[0076] The prediction unit (more specifically, the inter prediction unit) of the encoding / decoding device can perform inter prediction on a block-by-block basis to derive prediction samples. Inter prediction can refer to a prediction derived in a manner dependent on data elements (e.g., sample values, motion information, etc.) of pictures other than the current picture. When inter prediction is applied to the current block, a predicted block (prediction sample array) for the current block can be derived based on a reference block (reference sample array) specified by a motion vector on a reference picture pointed to by a reference picture index. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information of the current block can be predicted on a block, sub-block, or sample basis based on the correlation of the motion information between the surrounding blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on the inter prediction type (L0 prediction, L1 prediction, Bi prediction, etc.). When inter prediction is applied, the neighboring blocks may include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The temporal neighboring blocks may be called collocated reference blocks, collocated CUs (colCUs), etc., and the reference picture including the temporal neighboring blocks may be called collocated pictures (colPic).For example, a list of motion information candidates can be constructed based on the neighboring blocks of the current block, and flag or index information can be signaled to indicate which candidate is selected (used) to derive the motion vector and / or reference picture index of the current block. Inter prediction can be performed based on various prediction modes, for example, in the case of skip mode and merge mode, the motion information of the current block can be the same as the motion information of the selected neighboring block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted. In the case of motion vector prediction (MVP) mode, the motion vector of the selected neighboring block can be used as a motion vector predictor, and the motion vector difference can be signaled. In this case, the motion vector of the current block can be derived using the sum of the motion vector predictor and the motion vector difference.

[0077] The above motion information may include L0 motion information and / or L1 motion information depending on the inter prediction type (L0 prediction, L1 prediction, Bi prediction, etc.). A motion vector in the L0 direction may be called an L0 motion vector or MVL0, and a motion vector in the L1 direction may be called an L1 motion vector or MVL1. Prediction based on an L0 motion vector may be called L0 prediction, prediction based on an L1 motion vector may be called L1 prediction, and prediction based on both an L0 motion vector and an L1 motion vector may be called bi-prediction. Here, the L0 motion vector may indicate a motion vector associated with a reference picture list L0 (L0), and the L1 motion vector may indicate a motion vector associated with a reference picture list L1 (L1). The reference picture list L0 may include pictures that are earlier in output order than the current picture as reference pictures, and the reference picture list L1 may include pictures that are later in output order than the current picture. Previous pictures may be called forward (reference) pictures, and subsequent pictures may be called backward (reference) pictures. The reference picture list L0 may further include subsequent pictures in output order than the current picture as reference pictures. In this case, the previous pictures may be indexed first and subsequent pictures may be indexed next within the reference picture list L0. The reference picture list L1 may further include previous pictures in output order than the current picture as reference pictures. In this case, the subsequent pictures may be indexed first and subsequent pictures may be indexed next within the reference picture list 1. Here, the output order may correspond to the picture order count (POC) order.

[0078] In addition, when applying inter prediction to the current block, various inter prediction modes can be used. For example, various modes such as merge mode, skip mode, MVP (motion vector prediction) mode, affine mode, and HMVP (historical motion vector prediction) mode can be used. Decoder side motion vector refinement (DMVR) mode, adaptive motion vector resolution (AMVR) mode, and bi-directional optical flow (BDOF) can be further used as auxiliary modes. Affine mode may also be called affine motion prediction mode. MVP mode may also be called advanced motion vector prediction (AMVP) mode. In this document, motion information candidates derived by some modes and / or some modes may be included as one of the motion information-related candidates of other modes.

[0079] Prediction mode information indicating the inter prediction mode of the current block may be signaled from the encoding device to the decoding device. At this time, the prediction mode information may be included in the bitstream and received by the decoding device. The prediction mode information may include index information indicating one of a plurality of candidate modes. Alternatively, the inter prediction mode may be indicated through hierarchical signaling of flag information. In this case, the prediction mode information may include one or more flags. For example, a skip flag may be signaled to indicate whether skip mode is applied, a merge flag may be signaled to indicate whether merge mode is applied if skip mode is not applied, and if merge mode is not applied, an MVP mode may be indicated to be applied, or a flag may be further signaled for additional distinction. The affine mode may be signaled as an independent mode, or as a mode dependent on the merge mode or the MVP mode. For example, the affine mode may include an affine merge mode and an affine MVP mode.

[0080] Additionally, when applying inter prediction to the current block, the motion information of the current block can be utilized. The encoding device can derive optimal motion information for the current block through a motion estimation procedure. For example, the encoding device can search for similar reference blocks with high correlation in fractional pixel units within a predetermined search range within the reference picture using the original block within the original picture for the current block, thereby deriving motion information. The similarity between blocks can be derived based on the difference in phase-based sample values. For example, the similarity between blocks can be calculated based on the sum of absolute differences (SAD) between the current block (or a template of the current block) and the reference block (or a template of the reference block). In this case, the motion information can be derived based on the reference block with the smallest SAD within the search range. The derived motion information can be signaled to the decoding device in various ways based on the inter prediction mode.

[0081] As described above, a predicted block for a current block can be derived based on motion information derived according to the inter prediction mode. The predicted block can include prediction samples (prediction sample array) of the current block. If the motion vector (MV) of the current block indicates a fractional sample unit, an interpolation procedure can be performed, through which prediction samples of the current block can be derived based on reference samples of the fractional sample unit within a reference picture. If affine inter prediction is applied to the current block, prediction samples can be generated based on a sample / subblock unit MV. If bi-prediction is applied, prediction samples derived based on L0 prediction (i.e., prediction using a reference picture in a reference picture list L0 and MVL0) and L1 prediction (i.e., prediction using a reference picture in a reference picture list L1 and MVL1) can be used as prediction samples of the current block through a weighted sum or weighted average (according to phase). When bi-prediction is applied, if the reference picture used for L0 prediction and the reference picture used for L1 prediction are located in different temporal directions with respect to the current picture (i.e., if it is bi-prediction and bidirectional prediction), it can be called true bi-prediction.

[0082] As described above, restoration samples and restoration pictures can be generated based on the derived prediction samples, and then procedures such as in-loop filtering can be performed.

[0083] Meanwhile, skip mode and / or merge mode have limitations in motion prediction because they predict the motion of the current block based on the motion vectors of surrounding blocks without MVD (Motion Vector Difference). To improve the limitations of skip mode and / or merge mode, motion vectors can be refined by applying Decoder-side Motion Vector Refinement (DMVR) mode, Bi-directional optical flow (BDOF) mode, etc. DMVR and BDOF mode can be used when true bi-prediction is applied to the current block.

[0084] FIG. 4 is a diagram illustrating one embodiment of a process for performing DMVR (Decoder-side Motion Vector Refinement) in true pair prediction.

[0085] DMVR is a method for performing motion prediction by refining motion information from surrounding blocks at the decoder side. When DMVR is applied, the decoder can derive refined motion information through cost comparison based on a template generated using motion information from surrounding blocks in merge / skip mode. This can increase the precision of motion prediction and enhance compression performance without requiring additional signaling information.

[0086] In this document, for convenience of explanation, the description focuses on the decoding device, but the DMVR according to the embodiment of this document can be performed in the same manner in the encoding device.

[0087] Referring to FIG. 4, the decoding device can derive prediction blocks (i.e., reference blocks) identified by initial motion vectors (or motion information) (e.g., MV0 and MV1) in the list0 and list1 directions, and weight the derived prediction blocks (e.g., average) to generate a template (or bilateral template) (step 1). Here, the initial motion vectors (MV0 and MV1) can represent motion vectors derived using motion information of surrounding blocks in the merge / skip mode.

[0088] And, the decoding device can derive motion vectors (e.g., MV0' and MV1') that minimize the difference between the sample region of the template and the reference picture through a template matching operation (step 2). Here, the sample region represents a surrounding region of the initial prediction block within the reference picture, and the sample region may be referred to as a surrounding region, a reference region, a search region, a search range, a search space, etc. The template matching operation may include an operation of calculating a cost measurement value between the sample region of the template and the reference picture. For example, the cost measurement may be performed using SAD (sum of absolute differences). As an example, a normalized SAD may be used as the cost function. In this case, the matching cost may be given as SAD(T - mean(T), 2 * P[x] - 2 * mean(P[x])). Here, T represents the template, and P[x] represents a block within the search region. And, the motion vector that produces the minimum template cost for each of the two reference pictures can be considered as an updated motion vector (replacing the initial motion vector). As illustrated in Fig. 4, the decoding device can generate the final bidirectional prediction result (i.e., the final bidirectional prediction block) using the updated motion vectors MV0' and MV1'. In one embodiment, multi-iteration for deriving the updated (or new) motion vectors can be used to obtain the final bidirectional prediction result.

[0089] In one embodiment, the decoding device may invoke the DMVR process to improve the accuracy of the initial motion compensation prediction (i.e., motion compensation prediction via the conventional merge / skip mode). For example, the decoding device may perform the DMVR process when the prediction mode of the current block is merge mode or skip mode, and bidirectional bi-prediction is applied to the current block, in which the bidirectional reference pictures are in opposite directions relative to the current picture in display order.

[0090] As described above, when true bi-prediction is applied to the current block, BDOF can be used to refine the bi-prediction signal. BDOF (Bi-directional optical flow) can be used to calculate improved motion information and generate prediction samples based on it when bi-prediction is applied to the current block. For example, BDOF can be applied at the 4x4 sub-block level. That is, BDOF can be performed in units of 4x4 sub-blocks within the current block. Alternatively, BDOF can be applied only to the luma component. Alternatively, BDOF can be applied only to the chroma component, or to the luma component and the chroma component.

[0091] The BDOF mode, as its name suggests, is based on the concept of optical flow, which assumes that the object's motion is smooth. For each 4x4 sub-block, motion refinement (v) is performed by minimizing the difference between the L0 and L1 prediction samples. x , v y ) can be calculated. And motion refinement can be used to adjust the biprediction sample values ​​in a 4x4 sub-block.

[0092] The above-described DMVR and BDOF are techniques for performing prediction by refining motion information when applying true biprediction (where true biprediction refers to a case where motion is predicted / compensated from a reference picture in a different direction based on the picture of the current block). They can be seen to be refinement techniques with a similar concept in that they assume that the movement of objects within the picture occurs at a constant speed and in a constant direction. However, when true biprediction is performed, since the conditions for applying DMVR and the conditions for applying BDOF are different, a process of repeatedly performing condition checks multiple times must be performed for each technique. Therefore, this paper proposes a method to improve efficiency in terms of decoder complexity and performance by improving the process of performing condition checks when determining the prediction mode to be applied to the current block.

[0093] Table 1 below shows the conditions for applying DMVR to existing true pair predictions. DMVR can be applied only when all of the conditions listed below are satisfied.

[0094]

[0095] Referring to Table 1 above, 1) Whether to apply DMVR can be determined based on flag information (e.g., sps_dmvr_enabled_flag) signaled in the SPS (Sequence Parameter Set) syntax. Here, the flag information (e.g., sps_dmvr_enabled_flag) can indicate whether true biprediction-based DMVR is enabled. For example, if sps_dmvr_enabled_flag is 1 (i.e., true biprediction-based DMVR is enabled), it can be determined that the DMVR availability condition is satisfied.

[0096] 2) Whether to apply DMVR can be determined based on flag information (e.g., merge_flag) indicating whether inter prediction is performed using merge mode / skip mode. For example, if merge_flag is 1 (i.e., inter prediction is performed using merge mode / skip mode), the condition for applying merge mode / skip mode can be determined to be satisfied.

[0097] 3) Whether to apply DMVR can be determined based on flag information (e.g., mmvd_flag) indicating whether inter prediction is performed using MMVD (merge mode with motion vector difference) mode. For example, if mmvd_flag is 0 (i.e., MMVD mode is not used), the condition for applying MMVD mode can be determined to be satisfied.

[0098] 4) Whether to apply DMVR can be determined based on whether bidirectional prediction (bidirectional prediction) is used. Here, bidirectional prediction can refer to inter prediction performed based on reference pictures existing in different directions relative to the current picture. For example, if predFlagL0[0][0] = 1 and predFlagL0[1][1] = 1, bidirectional prediction can be determined to be applied, and the condition for bidirectional prediction can be determined to be satisfied.

[0099] 5) It is true that it is a pairwise prediction, and it is possible to determine whether to apply DMVR based on whether the distances between the current picture and the bidirectional reference pictures are the same. That is, it is possible to determine whether the distances between the current picture and the LO reference picture (i.e., the reference picture in the reference picture list L0) and the distances between the current picture and the L1 reference picture (i.e., the reference picture in the reference picture list L1) are the same. For example, if DiffPicOrderCnt(currPic, RefPicList0 [refIdxL0]) - DiffPicOrderCnt(currPic, RefPicList1 [refIdxL1]) = 0, it is determined that it is a true pairwise prediction, and the distances between the current picture and the bidirectional reference pictures are the same, so it can be determined that the condition of whether the bidirectional reference picture distances are the same is satisfied.

[0100] 6) Whether to apply DMVR can be determined based on whether the current block's length (Height) exceeds a threshold. For example, if the current block's length is 8 or greater, the current block size (length) condition can be determined to be satisfied.

[0101] 7) Whether to apply DMVR can be determined based on whether the current block size is larger than a threshold. For example, if the current block size (height x width) is 64 or greater, the current block size (length x width) condition can be determined to be satisfied.

[0102] The decoding device can determine whether to apply DMVR based on whether conditions 1) to 7) of Table 1 above are satisfied. That is, the decoding device can perform true pair prediction by applying DMVR when all conditions 1) to 7) of Table 1 above are satisfied, and does not apply DMVR when even one of the conditions of Table 1 above is not satisfied.

[0103] Table 2 below shows the conditions for applying BDOF to existing true pair predictions. BDOF can be applied only when all of the conditions listed below are satisfied.

[0104]

[0105] Referring to Table 2 above, 1) Whether to apply BDOF can be determined based on flag information (e.g., sps_bdof_enabled_flag) signaled in the SPS (Sequence Parameter Set) syntax. Here, the flag information (e.g., sps_bdof_enabled_flag) can indicate whether true pair prediction-based BDOF is enabled. For example, if sps_bdof_enabled_flag is 1 (i.e., true pair prediction-based BDOF is enabled), it can be determined that the BDOF availability condition is satisfied.

[0106] 2) Whether to apply BDOF can be determined based on whether bidirectional prediction is used. Here, bidirectional prediction can refer to inter-prediction performed based on reference pictures located in different directions relative to the current picture. For example, if both predFlagL0 and predFlagL1 are 1, bidirectional prediction can be determined to be applied, satisfying the bidirectional prediction condition.

[0107] 3) Whether to apply BDOF can be determined based on whether it is true bi-prediction. That is, it can be determined whether the LO reference picture (i.e., the reference picture in the reference picture list L0) and the L1 reference picture (i.e., the reference picture in the reference picture list L1) are located in different temporal directions with respect to the current picture. For example, if DiffPicOrderCnt( currPic, refPicList0[refIdxL0] ) * DiffPicOrderCnt( currPic, refPicList1[refIdxL1] ) is less than 0, it can be determined that the bi-directional reference pictures are located in different directions with respect to the current picture, and thus the true bi-prediction condition can be determined to be satisfied.

[0108] 4) Whether BDOF is applied can be determined based on whether affine mode is used. Here, whether affine mode is used can be determined by deriving MotionModelIdc. For example, if the derived MotionModelIdc is 0, it can be determined that affine mode is not used, and in this case, the condition for whether affine mode is applied can be determined to be satisfied.

[0109] 5) Whether to apply BDOF can be determined based on flag information (e.g., merge_subblock_flag) indicating whether inter prediction in merge mode is performed on a subblock basis. For example, if merge_subblock_flag is 0 (i.e., merge mode is not applied on a subblock basis), it can be determined that the condition for subblock-based merge mode is satisfied.

[0110] 6) Whether to apply BDOF can be determined based on whether GBi exists. Here, whether GBi exists can be determined based on GBi index information (e.g., GbiIdx). For example, if GbiIdx is 0 (i.e., GbiIdx is the default), the GBi condition can be determined to be satisfied.

[0111] 7) Whether to apply BDOF can be determined based on whether the current block is a luma block containing a luma component. For example, if the index indicating whether it is a luma block (e.g., cIdx) is 0 (i.e., if it is a luma block), it can be determined that the condition for whether it is a luma block is satisfied.

[0112] The decoding device can determine whether to apply BDOF based on whether conditions 1) to 7) of Table 2 above are satisfied. That is, the decoding device can perform true pair prediction by applying BDOF when all conditions 1) to 7) of Table 2 above are satisfied, and does not apply BDOF when even one of the conditions of Table 2 above is not satisfied.

[0113] As described above, some of the conditions for applying DMVR and BDOF are the same, while others are similar or different. Conventional methods require condition checks for each technique even when conditions are the same, increasing the complexity of performing paired prediction. Therefore, this paper proposes efficient conditions for applying DMVR and BDOF in paired prediction.

[0114] Since Merge / Skip mode has relatively low motion accuracy compared to AMVP mode, refining motion information using the DMVR method is effective in terms of performance. However, unlike DMVR, BDOF mode is applied not only in Merge / Skip mode but also in AMVP mode. When BDOF is applied in AMVP mode, the complexity of performing BDOF may increase compared to the performance. Therefore, in this embodiment, we propose a method to apply BDOF in Merge / Skip mode as well, similar to DMVR.

[0115] In this case, as an example proposed in this document, the application conditions of BDOF may include the conditions presented in Table 3 below.

[0116]

[0117] Referring to Table 3 above, whether to apply BDOF can be determined based on flag information (e.g., merge_flag) indicating whether inter prediction is performed using merge mode / skip mode. For example, if merge_flag is 1 (i.e., inter prediction is performed using merge mode / skip mode), it can be determined that the condition for applying merge mode / skip mode is satisfied. Therefore, similar to DMVR, BDOF can also be applied when in merge / skip mode.

[0118] That is, in this embodiment, it is possible to determine whether to apply BDOF based on the conditions applied in the case of the merge / skip mode, the conditions applied in the case of BDOF being available, the conditions applied in the case of bidirectional prediction, the conditions applied in the case of true bidirectional prediction, the conditions applied in the case of non-affine prediction, the conditions applied in the case of non-subblock-based merge mode, the conditions applied in the case of the GBi index being default, and the conditions applied in the case of a luma block.

[0119] Accordingly, the decoding device can determine whether all the conditions listed in Table 3 above are satisfied, and if all conditions are satisfied, it can perform true pair prediction by applying BDOF. If even one of the conditions listed in Table 3 above is not satisfied, the decoding device may not apply BDOF. These conditions in Table 3 above can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0120] The method for performing true pair prediction based on the conditions listed in Table 3 above can be applied independently to DMVR and BDOF, or can be applied with the same conditions for DMVR and BDOF.

[0121] Meanwhile, if the conditions listed in this embodiment are identical to the conditions described in Tables 1 and 2 above, the specific operations and meanings are applied identically, so a detailed description of each condition will be omitted. Furthermore, any overlapping content will be omitted in the embodiments described below.

[0122] Encoding / decoding devices can be configured using a variety of hardware, and their performance-to-complexity ratio preferences may vary. Therefore, this embodiment proposes a method for refining motion information by applying DMVR not only in merge / skip mode but also in AMVP mode.

[0123] In this case, as an example of the proposed embodiment in this document, the application conditions of DMVR may include the conditions presented in Table 4 below.

[0124]

[0125] Referring to Table 4 above, the process of determining whether to apply DMVR based on flag information (e.g., merge_flag) indicating whether inter prediction is performed using merge mode / skip mode can be omitted. In this way, by omitting the condition for applying merge mode / skip mode, DMVR can be applied not only in merge mode / skip mode but also in AMVP mode.

[0126] According to Table 4 above, whether to apply DMVR can be determined based on the conditions applied when DMVR is available, the conditions applied when MMVD mode is not used, the conditions applied when bidirectional prediction is used, the conditions applied when the distance between the current picture and the bidirectional reference picture is the same as true bidirectional prediction, the conditions applied when the length of the current block is 8 or more, and the conditions applied when the size (length*width) of the current block is 64 or more.

[0127] That is, the decoding device determines whether all conditions listed in Table 4 above are satisfied, and if all conditions are satisfied, it can apply DMVR to perform true pair prediction. If even one of the conditions listed in Table 4 above is not satisfied, the decoding device may not apply DMVR. These conditions in Table 4 above can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0128] The method for performing true pair prediction based on the conditions listed in Table 4 above can be applied independently to DMVR and BDOF, or can be applied with the same conditions for DMVR and BDOF.

[0129] Meanwhile, as one embodiment of this document, both DMVR and BDOF can be applied to the normal merge mode. That is, DMVR and BDOF can be applied in any mode other than ATMVP (advanced temporal motion vector prediction), affine mode, or CPR. In this case, the application conditions for DMVR may include the conditions presented in Table 5 below.

[0130]

[0131] Referring to Table 5 above, DMVR can be applied only in the case of general merge mode by determining whether the conditions that apply when affine mode is not used (e.g., when MotionModelIdc is 0) and the conditions that apply when not in subblock-based merge mode (e.g., when merge_subblock_flag is 0) are satisfied.

[0132] In addition, in this embodiment, it is possible to determine whether to apply DMVR based on the conditions for whether the affine mode is used, the conditions for whether the subblock-based merge mode is used, the conditions for whether DMVR is available, the conditions for whether merge mode / skip mode is used, the conditions for whether MMVD mode is not used, the conditions for whether bidirectional prediction is used, the conditions for whether the current picture and the bidirectional reference picture are true bidirectional predictions with the same distance, the conditions for whether the current block has a length of 8 or more, and the conditions for whether the current block has a size (length*width) of 64 or more.

[0133] That is, the decoding device determines whether all conditions listed in Table 5 above are satisfied, and if all conditions are satisfied, it can apply DMVR to perform true pair prediction. If even one of the conditions listed in Table 5 above is not satisfied, the decoding device may not apply DMVR. These conditions in Table 5 above can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0134] Among the application conditions in Table 5 above, the condition for whether to apply subblock-based merge mode (e.g., merge_subblock_flag) contains conditions that overlap with existing DMVR application conditions. Therefore, in one embodiment of this document, conditions that overlap with the condition for whether to apply subblock-based merge mode (e.g., merge_subblock_flag) can be removed. In this case, the condition can be removed as suggested in Table 6 below.

[0135]

[0136] Referring to Table 6 above, the subblock-based merge mode can be applied when the size of the current block is 8X8 or larger. Therefore, since the condition for applying the subblock-based merge mode (e.g., merge_subblock_flag = 0) includes a condition related to the size of the current block, the conditions related to the size of the current block (e.g., CbHeight, CbHeight*CbWidth) can be excluded from the application conditions of the existing DMVR. For example, the condition for whether the length of the current block is 8 or larger and the condition for whether the length*width of the current block is 64 or larger can be omitted, and the remaining conditions listed in Table 6 can be used to determine whether DMVR is applied.

[0137] That is, the decoding device determines whether all conditions listed in Table 6 above are satisfied, and if all conditions are satisfied, it can perform true pair prediction by applying DMVR. If even one of the conditions listed in Table 6 above is not satisfied, the decoding device may not apply DMVR. These conditions in Table 6 above can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0138] The method of performing true pair prediction based on the conditions listed in Table 5 or Table 6 above can be applied independently to DMVR and BDOF, or can be applied with the same conditions for DMVR and BDOF.

[0139] In one embodiment of this document, in order to increase the accuracy of motion vectors at low complexity, refinement techniques such as DMVR and BDOF may not be applied when the block size is small. In the existing method, the refinement technique is applied when the current block is a block larger than or equal to 8X8, but in the case of DMVR, when the current block size is large, the refinement is applied by dividing it into 16X16 units, so DMVR may not be applied to blocks smaller than 16X16. In this case, the application conditions of DMVR may include the conditions presented in Table 7 below.

[0140]

[0141] Referring to Table 7 above, by changing the conditions related to the size of the current block (e.g., CbHeight, CbWidth), DMVR can be not applied to blocks smaller than 16X16. For example, a condition that applies when the length of the current block (e.g., CbHeight) is 16 or more and a condition that applies when the width of the current block (e.g., CbWidth) is 16 or more can be used. When these application conditions related to the size of the current block are satisfied (i.e., when the size of the current block is 16X16 or more), DMVR can be applied, and when the application conditions related to the size of the current block are not satisfied (i.e., when the size of the current block is smaller than 16X16), DMVR can be not applied.

[0142] Additionally, in this embodiment, it is possible to determine whether to apply DMVR based on the remaining application conditions listed in Table 7 above, along with conditions related to the size of the current block (e.g., CbHeight, CbWidth).

[0143] That is, the decoding device determines whether all conditions listed in Table 7 above are satisfied, and if all conditions are satisfied, it can apply DMVR to perform true pair prediction. If even one of the conditions listed in Table 7 above is not satisfied, the decoding device may not apply DMVR. These conditions in Table 7 above can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0144] In one embodiment of this document, if the current block is smaller than 16X16, BDOF may not be applied, as well as DMVR. In this case, the application conditions of BDOF may include the conditions presented in Table 8 below.

[0145]

[0146] Referring to Table 8 above, by changing the conditions related to the size of the current block (e.g., CbHeight, CbWidth), BDOF can be not applied to blocks smaller than 16X16. For example, a condition that applies when the length of the current block (e.g., CbHeight) is 16 or more and a condition that applies when the width of the current block (e.g., CbWidth) is 16 or more can be used. When these application conditions related to the size of the current block are satisfied (i.e., when the size of the current block is 16X16 or more), BDOF can be applied, and when the application conditions related to the size of the current block are not satisfied (i.e., when the size of the current block is smaller than 16X16), BDOF can be not applied.

[0147] Additionally, in this embodiment, it is possible to determine whether to apply BDOF based on the remaining application conditions listed in Table 8 above, along with conditions related to the size of the current block (e.g., CbHeight, CbWidth).

[0148] That is, the decoding device determines whether all the conditions listed in Table 8 above are satisfied, and if all the conditions are satisfied, it can perform true pair prediction by applying BDOF. If even one of the conditions listed in Table 8 above is not satisfied, the decoding device may not apply BDOF. These conditions in Table 8 above can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0149] The method for performing true pair prediction based on the conditions listed in Table 7 or Table 8 above can be applied independently to DMVR and BDOF, or can be applied with the same conditions for DMVR and BDOF.

[0150] As described above, DMVR applies only when the distances between the current picture and the bidirectional reference pictures are equal, whereas BDOF always applies when true bidirectional prediction is achieved, even when the distances between the current picture and the bidirectional reference pictures differ. Therefore, this paper proposes a method to unify the conditions related to the bidirectional reference picture distances and apply them to DMVR and BDOF to improve coding efficiency.

[0151] As an example of this document, the application conditions of BDOF may include the conditions presented in Table 9 below.

[0152]

[0153] Referring to Table 9 above, by changing the condition related to the reference picture distance among the application conditions of BDOF (e.g., DiffPicOrderCnt), the corresponding condition can be applied equally to DMVR and BDOF. For example, by determining whether DiffPicOrderCnt(currPic, RefPicList0 [refIdxL0]) - DiffPicOrderCnt(currPic, RefPicList1 [refIdxL1]) is 0, it can be determined whether the distance between the current picture and the LO reference picture (i.e., the reference picture in the reference picture list L0) and the distance between the current picture and the L1 reference picture (i.e., the reference picture in the reference picture list L1) are mutually identical. In other words, BDOF can be applied only when the distances between the current picture and the bidirectional reference pictures are mutually identical. In this way, since the condition that the true pair prediction and the bidirectional reference picture distances are the same is added, the scope of application of BDOF is limited, so the decoding complexity can be reduced.

[0154] Additionally, in this embodiment, it is possible to determine whether to apply BDOF based on the remaining application conditions listed in Table 9 above, together with conditions related to the reference picture distance (e.g., DiffPicOrderCnt).

[0155] That is, the decoding device determines whether all the conditions listed in Table 9 above are satisfied, and if all the conditions are satisfied, it can perform true pair prediction by applying BDOF. If even one of the conditions listed in Table 9 above is not satisfied, the decoding device may not apply BDOF. These conditions in Table 9 above can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0156] As one embodiment of this document, the application conditions of DMVR may include the conditions presented in Table 10 below.

[0157]

[0158] Referring to Table 10 above, by changing the conditions related to the reference picture distance among the application conditions of DMVR (e.g., DiffPicOrderCnt), the corresponding conditions can be applied equally to DMVR and BDOF. For example, by determining whether DiffPicOrderCnt( currPic, refPicList0[ refIdxL0 ] ) * DiffPicOrderCnt( currPic, refPicList1[ refIdxL1 ] ) is less than 0, it can be determined whether the bidirectional reference pictures (i.e., LO reference picture and L1 reference picture) are true bidirectional predictions located in different directions based on the current picture. In other words, even if the distances between the current picture and the bidirectional reference pictures are not the same, DMVR can always be applied if it is a true bidirectional prediction. In this way, since the true pair prediction condition is applied, the motion vector derived by considering the decoding complexity can be used without scaling even when the reference picture distances in both directions are different.

[0159] Additionally, in this embodiment, it is possible to determine whether to apply DMVR based on the remaining application conditions listed in Table 10, along with conditions related to the reference picture distance (e.g., DiffPicOrderCnt).

[0160] That is, the decoding device determines whether all conditions listed in Table 10 above are satisfied, and if all conditions are satisfied, it can apply DMVR to perform true pair prediction. If even one of the conditions listed in Table 10 above is not satisfied, the decoding device may not apply DMVR. These conditions in Table 10 above can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0161] The method for performing true pair prediction based on the conditions listed in Table 9 or Table 10 above can be applied independently to DMVR and BDOF, or can be applied with the same conditions for DMVR and BDOF.

[0162] Meanwhile, each reference block may experience motion compensation using a weighted sum due to changes in light, etc. In this case, since this phenomenon can be identified using GBi or LIC (local illumination compensation), the application conditions of DMVR and BDOF can be determined by considering GBi or LIC conditions.

[0163] As an example, this document proposes a method for determining whether to apply DMVR by considering GBi and LIC conditions. In this case, the DMVR application conditions may include the conditions presented in Table 11 below.

[0164]

[0165] Referring to Table 11 above, whether DMVR is applied can be determined by adding a GBi condition (e.g., GbiIdx) and a LIC condition (e.g., LICFlag). Here, GBi can represent generalized bi-prediction that can apply different weights to L0 prediction and L1 prediction, and can be represented using GbiIdx, for example. GbiIdx can exist in the case of bi-prediction and can represent a bi-prediction weight index. For example, GbiIdx can be derived from neighboring blocks in the case of merge mode, or can be signaled from the encoding device to the decoding device via the GbiIdx syntax element (e.g., gbi_idx) in the case of MVP mode. For example, GbiIdx can indicate a weight w applied to L1 prediction, in which case a weight of (1-w) can be applied to L0 prediction. As another example, GbiIdx may indicate the weight w applied to the L0 prediction, in which case the L1 prediction may be weighted by (1-w).

[0166] According to Table 11 above, if GbiIdx is 0 (i.e., if GbiIdx is default), it can be determined that the GBi condition is satisfied, and if LICFlag is 0 (i.e., if LIC exists), it can be determined that the LIC condition is satisfied.

[0167] Additionally, in this embodiment, it is possible to determine whether to apply DMVR based on the remaining application conditions listed in Table 11 above, together with the GBi condition (e.g., GbiIdx) and the LIC condition (e.g., LICFlag).

[0168] That is, the decoding device determines whether all conditions listed in Table 11 above are satisfied, and if all conditions are satisfied, it can apply DMVR to perform true pair prediction. If even one of the conditions listed in Table 11 above is not satisfied, the decoding device may not apply DMVR. These conditions in Table 11 above can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0169] As an example, this document proposes a method for determining whether to apply BDOF by considering GBi and LIC conditions. In this case, the BDOF application conditions may include the conditions presented in Table 12 below.

[0170]

[0171] According to Table 12 above, whether BDOF is applied can be determined by adding a LIC condition (e.g., LICFlag) together with the existing GBi condition (e.g., GbiIdx). For example, if GbiIdx is 0 (i.e., GbiIdx is the default), the GBi condition is determined to be satisfied, and if LICFlag is 0 (i.e., LIC exists), the LIC condition is determined to be satisfied.

[0172] Therefore, in this embodiment, it is possible to determine whether to apply BDOF based on the remaining application conditions listed in Table 12 above, together with the GBi condition (e.g., GbiIdx) and the LIC condition (e.g., LICFlag).

[0173] That is, the decoding device determines whether all the conditions listed in Table 12 above are satisfied, and if all the conditions are satisfied, it can perform true pair prediction by applying BDOF. If even one of the conditions listed in Table 12 above is not satisfied, the decoding device may not apply BDOF. These conditions in Table 12 above can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0174] The method for performing true pair prediction based on the conditions listed in Table 11 or Table 12 above can be applied independently to DMVR and BDOF, or can be applied with the same conditions for DMVR and BDOF.

[0175] Meanwhile, DMVR and BDOF derive motion information through a refinement process at the decoder, which leads to decoding complexity issues. Therefore, this paper proposes a method to reduce decoding complexity by using a merge index to determine whether to apply DMVR and BDOF. Since both DMVR and BDOF perform motion vector refinement within a limited range, the effect of the refinement may be reduced if the motion vector is inaccurate. Therefore, this paper proposes a method that can be applied only when the value indicated by the merge index is small, considering the efficiency of the refinement.

[0176] Here, the merge index may be a syntax element signaled from the encoding device to the decoding device. For example, the encoding / decoding device may construct a merge candidate list based on the surrounding blocks of the current block when the merge mode / skip mode is applied to the current block. At this time, the encoding device may select an optimal merge candidate among the merge candidates included in the merge candidate list based on the RD (rate-distortion) cost, and signal merge index information indicating the selected merge candidate to the decoding device. The decoding device may select a merge candidate applied to the current block based on the merge candidate list and the merge index information.

[0177] In one embodiment of this document, a method for determining whether to apply DMVR using a merge index may include the conditions presented in Table 13 below.

[0178]

[0179] Referring to Table 13 above, the application of DMVR can be determined by adding a merge index condition (e.g., merge_idx). For example, if the merge index (e.g., merge_idx) is less than 2, the merge index condition can be determined to be satisfied. Here, the merge index value (threshold) is set to 2, but this is only an example, and the value can be changed depending on coding efficiency.

[0180] Therefore, in this embodiment, it is possible to determine whether to apply DMVR based on the remaining application conditions listed in Table 13 above, together with the merge index condition (e.g., merge_idx).

[0181] That is, the decoding device determines whether all conditions listed in Table 13 above are satisfied, and if all conditions are satisfied, it can apply DMVR to perform true pair prediction. If even one of the conditions listed in Table 13 above is not satisfied, the decoding device may not apply DMVR. These conditions in Table 13 above can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0182] As one embodiment of this document, a method for determining whether to apply BDOF using a merge index may include the conditions presented in Table 14 below.

[0183]

[0184] Referring to Table 14 above, whether to apply BDOF can be determined by adding a merge mode / skip mode condition (e.g., merge_flag) and a merge index condition (e.g., merge_idx). For example, if merge_flag is 1 (i.e., when performing inter prediction using merge mode / skip mode) and merge_idx is not 2 or greater, the condition for applying BDOF can be determined to be satisfied only when the value of the merge index is small. In other words, if merge_flag is 1 (i.e., when performing inter prediction using merge mode / skip mode) and merge_idx is less than 2, the merge index condition is determined to be satisfied and BDOF can be applied. Here, the merge index value (threshold) is set to 2, but this is only an example and the value may be changed depending on coding efficiency.

[0185] That is, in this embodiment, it is possible to determine whether BDOF is applied based on the remaining application conditions listed in Table 14 above, along with the merge mode / skip mode condition (e.g., merge_flag) and the merge index condition (e.g., merge_idx).

[0186] That is, the decoding device determines whether all the conditions listed in Table 14 above are satisfied, and if all the conditions are satisfied, it can perform true pair prediction by applying BDOF. If even one of the conditions listed in Table 14 above is not satisfied, the decoding device may not apply BDOF. These conditions in Table 14 above can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0187] The method of performing true pair prediction based on the conditions listed in Table 13 or Table 14 above can be applied independently to DMVR and BDOF, or can be applied with the same conditions for DMVR and BDOF.

[0188] Meanwhile, in Merge / Skip mode, motion information can be refined through MMVD. However, this increases decoder complexity, so DMVR is not performed when MMVD is applied. However, when applied without considering MMVD, DMVR can be applied without MMVD conditions to improve performance. In this case, according to one embodiment of this document, the DMVR application conditions may include the conditions presented in Table 15 below.

[0189]

[0190] Referring to Table 15 above, the condition for applying MMVD mode (e.g., mmvd_flag) can be excluded from the existing DMVR application conditions. That is, the process of determining whether mmvd_flag is 0 (i.e., not using MMVD mode) can be omitted, and whether DMVR is applied can be determined based on the conditions listed in Table 15 above.

[0191] That is, the decoding device determines whether all conditions listed in Table 15 above (except for the condition for applying the MMVD mode (e.g., mmvd_flag)) are satisfied, and if all conditions are satisfied, it can apply DMVR to perform true pair prediction. If even one of the conditions listed in Table 15 above is not satisfied, the decoding device may not apply DMVR. These conditions in Table 15 above can also be applied to the encoding device, and can be performed in the encoding device in a corresponding manner as in the decoding device.

[0192] Furthermore, unlike the embodiment of Table 15 described above, when refining motion information through MMVD in Merge / Skip mode, decoder complexity can be considered when determining whether to perform BDOF. Therefore, this document proposes a method for determining whether to apply BDOF by considering MMVD conditions. In this case, according to one embodiment of this document, the application conditions for BDOF may include the conditions presented in Table 16 below.

[0193]

[0194] Referring to Table 16 above, by adding a condition for whether MMVD mode is applied (e.g., mmvd_flag), BDOF can be omitted when refining motion information through MMVD. For example, if mmvd_flag is 0 (i.e., MMVD mode is not used), the condition for whether MMVD mode is applied can be determined to be satisfied.

[0195] Therefore, in this embodiment, it is possible to determine whether BDOF is applied based on the remaining application conditions listed in Table 16 above, along with the condition for whether MMVD mode is applied (e.g., mmvd_flag).

[0196] That is, the decoding device determines whether all conditions listed in Table 16 above are satisfied, and if all conditions are satisfied, it can perform true pair prediction by applying BDOF. If even one of the conditions listed in Table 16 above is not satisfied, the decoding device may not apply BDOF. These conditions in Table 16 above can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0197] The method of performing true pair prediction based on the conditions listed in Table 15 or Table 16 above can be applied independently to DMVR and BDOF, or can be applied with the same conditions for DMVR and BDOF.

[0198] Meanwhile, Adaptive Motion Vector Resolution (AMVR) technology can be applied in AMVP mode. However, when the motion vector resolution is large, such as integer-pel rounding or 4-sample rounding, this technology may not be suitable for BDOF, which performs refinement within a limited area. Therefore, this document proposes a method for determining the performance of BDOF based on AMVR conditions. In this case, according to one embodiment of this document, the application conditions of BDOF may include the conditions presented in Table 17 below.

[0199]

[0200] Referring to Table 17 above, it is possible to determine whether to apply BDOF by adding an AMVR condition (e.g., amvr_flag). Here, amvr_flag may be information indicating the resolution of the motion vector difference (MVD). For example, if amvr_flag is 0, it may indicate that the resolution of the MVD is derived in units of 1 / 4 samples (quarter-luma-sample), and if amvr_flag is not 0, it may indicate that it is derived in units of integer samples (integer-luma-sample) or 4 samples (four-luma-sample). Alternatively, the opposite may be determined. Depending on the embodiment, as presented in Table 17 above, a condition may be set such that BDOF is applied when amvr_flag is not 0. In other words, it may be restricted so that BDOF is not applied when amvr_flag is 0.

[0201] Therefore, in this embodiment, it is possible to determine whether BDOF is applied based on the remaining conditions listed in Table 17 above, together with the AMVR condition (e.g., amvr_flag).

[0202] That is, the decoding device determines whether all the conditions listed in Table 17 above are satisfied, and if all the conditions are satisfied, it can perform true pair prediction by applying BDOF. If even one of the conditions listed in Table 17 above is not satisfied, the decoding device may not apply BDOF. These conditions in Table 17 above can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0203] Meanwhile, DMVR and BDOF, as described above, can be signaled in the Sequence Parameter Set (SPS) syntax, respectively. Table 18 below shows examples of syntax elements that indicate whether DMVR and BDOF are available, signaled via the SPS syntax.

[0204]

[0205] Referring to Table 18 above, the sps_dmvr_enabled_flag can be signaled in the SPS syntax, and based on this syntax element, it can indicate whether true biprediction-based DMVR is enabled. For example, when sps_dmvr_enabled_flag is 1, it indicates that true biprediction-based DMVR is enabled, and when sps_dmvr_enabled_flag is 0, it indicates that true biprediction-based DMVR is not enabled.

[0206] Additionally, the SPS syntax can signal sps_bdof_enabled_flag, and based on this syntax element, it can indicate whether true biprediction-based BDOF is enabled. For example, if sps_bdof_enabled_flag is 1, it indicates that true biprediction-based BDOF is enabled, and if sps_bdof_enabled_flag is 0, it indicates that true biprediction-based BDOF is not enabled.

[0207] As shown in Table 18 above, the application conditions of DMVR and BDOF can be checked using a syntax element indicating whether DMVR is available (e.g., sps_dmvr_enabled_flag) and a syntax element indicating whether BDOF is available (e.g., sps_bdof_enabled_flag).

[0208] Figure 5 is an example showing a method of performing a decoding process by checking the application conditions of DMVR and BDOF.

[0209] The method of FIG. 5 can be applied when using a syntax element indicating whether DMVR is available (e.g., sps_dmvr_enabled_flag) and a syntax element indicating whether BDOF is available (e.g., sps_bdof_enabled_flag) as in Table 18 above.

[0210] Referring to FIG. 5, the decoding device can derive motion information (e.g., motion vector, reference picture index, etc.) for the current block (S500).

[0211] The decoding device can check the application conditions of DMVR (S510). At this time, the application conditions of DMVR can be checked based on a syntax element (e.g., sps_dmvr_enabled_flag) indicating whether DMVR is available. For example, the application conditions of DMVR can be checked if DMVR is available (e.g., sps_dmvr_enabled_flag is 1).

[0212] The decoding device can determine whether to apply the DMVR process based on whether the application conditions of DMVR are satisfied (S520).

[0213] If all DMVR application conditions are satisfied, the decoding device can apply the DMVR process to derive refined motion information (S530). If one or more of the DMVR application conditions are not satisfied, the decoding device may not apply the DMVR process.

[0214] The decoding device can derive prediction samples of the current block based on refined motion information derived when DMVR is applied or (unrefined) motion information derived when DMVR is not applied (S540).

[0215] In addition, the decoding device can check the application conditions of BDOF (S550). At this time, the application conditions of BDOF can be checked based on a syntax element (e.g., sps_bdof_enabled_flag) indicating whether BDOF is available. For example, the application conditions of BDOF can be checked when BDOF is available (e.g., sps_bdof_enabled_flag is 1).

[0216] If all of the BDOF application conditions are satisfied, the decoding device may perform a refinement on the prediction samples by applying the BDOF process (S570). If one or more of the BDOF application conditions are not satisfied, the decoding device may not apply the BDOF process.

[0217] The decoding device can derive residual samples for the current block (S580), and derive restored samples based on the residual samples and the refined prediction samples derived when the BDOF is applied or the (unrefined) prediction samples derived when the BDOF is not applied (S590).

[0218] This document proposes various embodiments that can improve coding efficiency and reduce complexity by harmonizing the application conditions of DMVR and BDOF as described above. When checking the application conditions of DMVR and BDOF according to the embodiments of this document and applying them to the decoding process, each condition can be checked and applied separately, but to improve coding efficiency, the application conditions can also be checked at once. In other words, this document proposes a method that can check the application conditions of DMVR and BDOF at once by integrating them.

[0219] In one embodiment of this document, the process of checking the application condition of DMVR / BDOF can be performed by signaling information (e.g., sps_refinement_enabled_flag) indicating whether refinement is applied in a decoding device in the SPS (Sequence Parameter Set) syntax. Table 19 below shows an example of a syntax element (e.g., sps_refinement_enabled_flag) indicating whether refinement is applied in a decoding device signaled through the SPS syntax.

[0220]

[0221] Referring to Table 19 above, sps_refinement_enabled_flag can be signaled in the SPS syntax, and based on this syntax element, it can be indicated whether refinement application is possible in the decoding device. For example, when sps_refinement_enabled_flag exists (i.e., sps_refinement_enabled_flag is true), it can be determined that refinement application is possible in the decoding device. In this case, the decoding device can obtain the sps_dmvr_enabled_flag syntax element indicating whether DMVR is enabled and the sps_bdof_enabled_flag syntax element indicating whether BDOF is enabled, and determine the application conditions of DMVR and BDOF.

[0222] Figures 6 and 7 are other examples showing a method of performing a decoding process by checking the application conditions of DMVR and BDOF.

[0223] The method of FIG. 6 and the method of FIG. 7 can be applied when a syntax element (e.g., sps_refinement_enabled_flag) indicating whether refinement application is possible in a decoding device is used, as in Table 19 above. In addition, descriptions of the overlapping contents of FIG. 6 and FIG. 7 with those of FIG. 5 described above are omitted.

[0224] Referring to FIG. 6, it can be seen that, compared to the process of FIG. 5, the application conditions of DMVR and BDOF are not checked separately, but are checked at once in the previous stage. In one embodiment, in step S610 of FIG. 6, the decoding device can check the application conditions of refinement. At this time, the checking of the application conditions of refinement can be performed based on sps_refinement_enabled_flag as in Table 19. For example, when sps_refinement_enabled_flag is 1, the sps_dmvr_enabled_flag syntax element indicating whether DMVR is enabled or the sps_bdof_enabled_flag syntax element indicating whether BDOF is enabled can be obtained, and the application conditions of DMVR and BDOF can be checked based on the sps_bdof_enabled_flag syntax element.

[0225] In addition, referring to FIG. 7, compared to the process of FIG. 5, instead of checking the application conditions of DMVR and BDOF individually, they can be checked at once in the previous step, and a simple check process (application conditions of BDOF) can be performed for other conditions. In one embodiment, in step S710 of FIG. 7, the decoding device can check the refinement application conditions. Thereafter, in step S750 of FIG. 7, the decoding device can additionally perform a simple check for the application conditions of BDOF that have different conditions from the refinement application conditions.

[0226] The methods listed in Tables 3 to 17 according to the embodiments of the present document described above can be applied in combination. That is, by checking the refinement application conditions, it is possible to determine whether DMVR and BDOF are applied, and conditions such as those in Table 20 below can be applied.

[0227]

[0228] Referring to Table 20 above, conditions applicable when DMVR or BDOF is applied in case of bidirectional prediction, conditions applicable when the distance between the current picture and the bidirectional reference picture is the same as true bidirectional prediction, conditions applicable when MMVD mode is not used, conditions applicable when not affine prediction, conditions applicable when not subblock-based merge mode, and conditions applicable when the GBi index is default can be used. In other words, whether DMVR or BDOF is applied can be determined based on whether the above conditions are satisfied.

[0229] Additionally, as described above, conditions may be added, such as determining whether it is merge mode for DMVR or whether it is a luma block for BDOF.

[0230] The application conditions listed in Table 20 above are only examples, and it is obvious that various conditions listed in the above-described embodiments (the embodiments of Tables 3 to 17 above) can be used in combination.

[0231] FIG. 8 is a flowchart schematically illustrating an encoding method that can be performed by an encoding device according to one embodiment of the present document.

[0232] The method disclosed in FIG. 8 can be performed by the encoding device (200) disclosed in FIG. 2. Specifically, steps S800 to S830 of FIG. 8 can be performed by the prediction unit (220) and the inter prediction unit (221) disclosed in FIG. 2, steps S840 to S850 of FIG. 8 can be performed by the residual processing unit (230) disclosed in FIG. 2, and step S860 of FIG. 8 can be performed by the entropy encoding unit (240) disclosed in FIG. 2. In addition, the method disclosed in FIG. 8 can include the embodiments described above in this document. Therefore, in FIG. 8, a detailed description of the overlapping contents with the embodiments described above will be omitted or simplified.

[0233] Referring to FIG. 8, the encoding device can derive a motion vector of the current block (S800).

[0234] In one embodiment, when performing inter prediction on a current block, the encoding device can derive motion information (motion vector, reference picture index, etc.) of the current block. For example, the encoding device can search for a block similar to the current block within a certain area (search area) of reference pictures through motion estimation, and derive a reference block whose difference from the current block is minimal or below a certain standard. Based on this, the encoding device can derive a reference picture index indicating the reference picture where the reference block is located, and derive a motion vector based on the positional difference between the reference block and the current block.

[0235] Additionally, the encoding device can determine an inter-prediction mode to be applied to the current block among various prediction modes. The encoding device can compare the RD costs for various prediction modes and determine the optimal prediction mode for the current block.

[0236] For example, when skip mode or merge mode is applied to the current block, the encoding device can construct a merge candidate list. At this time, among the reference blocks pointed to by the merge candidates included in the merge candidate list, a reference block having a difference from the current block that is minimal or below a certain standard can be derived. In this case, a merge candidate associated with the derived reference block is selected, and merge index information pointing to the selected merge candidate can be generated and signaled to the decoding device. Motion information of the current block can be derived using the motion information of the selected merge candidate.

[0237] As another example, when the (A)MVP mode is applied to the current block, the encoding device may configure an (A)MVP candidate list, and use the motion vector of an MVP candidate selected from among the MVP candidates included in the (A)MVP candidate list as the MVP of the current block. For example, a motion vector pointing to a reference block derived by the above-described motion estimation may be used as the motion vector of the current block. In this case, an MVP candidate having a motion vector with a smallest difference from the motion vector of the current block derived by the motion estimation may be selected from the MVP candidates. In addition, the encoding device may derive a motion vector difference (MVD), which is the difference obtained by subtracting the motion vector of the selected MVP candidate from the motion vector of the current block. In this case, information about the MVD may be signaled to the decoding device. In addition, when the (A)MVP mode is applied, the value of the reference picture index may be configured with reference picture index information and signaled separately to the decoding device.

[0238] The encoding device can determine whether to apply refinement to the motion vector of the current block (S810).

[0239] That is, the encoding device can determine whether to apply refinement by considering coding efficiency, complexity, prediction performance, etc. At this time, the encoding device can determine whether to apply refinement to the current block by using the various application conditions described in Tables 1 to 20 above.

[0240] In one embodiment, the encoding device may determine whether to apply refinement based on at least one of the size of the current block or the bi-prediction weight index information of the current block. For example, if the size of the current block is 16X16 or larger, it may be determined to apply DMVR to the motion vector of the current block. Additionally, if the value of the bi-prediction weight index information of the current block is 0, it may be determined to apply DMVR to the motion vector of the current block.

[0241] Here, the bi-prediction weight index information represents the above-described GbiIdx. This can be used as information indicating the weight factor applied to L0 prediction and L1 prediction in generalized bi-prediction where different weights can be applied to L0 prediction and L1 prediction. For example, when the merge mode is applied to the current block, the bi-prediction weight index information of the current block may be the same as the bi-prediction weight index information of the neighboring blocks of the current block indicated by the merge index.

[0242] Additionally, the encoding device can determine whether to apply DMVR-based refinement based on at least one of DMVR enabled flag information (e.g., sps_dmvr_enabled_flag) indicating whether DMVR included in the bitstream is enabled, merge flag information (e.g., merge_flag) indicating whether merge mode is applied to the current block, or MMVD (merge mode with motion vector difference) flag information (e.g., mmvd_flag) indicating whether MMVD is applied to the current block. For example, the encoding device can determine to apply DMVR to the motion vector of the current block when the value of the DMVR enabled flag information is 1, when the value of the merge flag information is 1, or when the value of the MMVD flag information is 0.

[0243] Additionally, the encoding device can determine whether to apply DMVR to the motion vector of the current block based on whether bi-prediction performed based on the L0 reference picture and the L1 reference picture is applied to the current block, or whether the distance between the current picture and the L0 reference picture and the distance between the current picture and the L1 reference picture are equal. For example, if the bi-prediction based on the L0 reference picture and the L1 reference picture is applied to the current block (e.g., if predFlagL0[0][0]=1 and predFlagL0[1][1]=1), or if the distances from the L0 reference picture and the L1 reference picture are equal to each other based on the current picture (e.g., if DiffPicOrderCnt(currPic, RefPicList0 [refIdxL0]) - DiffPicOrderCnt(currPic, RefPicList1 [ refIdxL1 ] ) = 0), the encoding device may decide to apply DMVR to the motion vector of the current block.

[0244] Additionally, as another example, the encoding device can determine whether to apply DMVR to the motion vector of the current block based on whether bi-prediction performed based on the L0 reference picture and the L1 reference picture is applied to the current block, whether the L0 reference picture and the L1 reference picture are located in different directions in time with respect to the current picture, or whether the affine mode is applied to the current block. For example, if the bi-prediction based on the L0 reference picture and the L1 reference picture is applied to the current block (e.g., when predFlagL0=1 and predFlagL1=1), if the L0 reference picture and the L1 reference picture are located in different directions in time with respect to the current picture (e.g., when DiffPicOrderCnt( currPic, refPicList0[refIdxL0] ) * DiffPicOrderCnt( currPic, refPicList1[refIdxL1] ) is less than 0), or if the affine mode is not applied to the current block (e.g., when MotionModelIdc = 0), the encoding device may decide to apply DMVR to the motion vector of the current block.

[0245] In determining whether to apply DMVR-based refinement here, the conditions listed above are only examples, and the conditions in Tables 1 to 20 described above may be used in various combinations.

[0246] If the encoding device decides to apply refinement to the motion vector of the current block, it can apply DMVR to the motion vector to derive a refined motion vector (S820).

[0247] At this time, the encoding device can apply the DMVR process of FIG. 4 described above. The process of deriving a refined motion vector by applying DMVR has been described in detail in FIG. 4, so a description thereof will be omitted here.

[0248] The encoding device can derive prediction samples for the current block based on the refined motion vector (S830).

[0249] In some embodiments, the encoding device may apply refinement to the prediction samples of the current block. In this case, the encoding device may apply BDOF-based refinement to the prediction samples, taking into account coding efficiency, complexity, prediction performance, etc.

[0250] In one embodiment, the encoding device may determine whether to apply BDOF to prediction samples based on at least one of the size of the current block and bi-prediction weight index information of the current block. For example, if the size of the current block is 16X16 or larger, it may be determined to apply BDOF to the prediction samples. Additionally, if the value of the bi-prediction weight index information of the current block is 0, it may be determined to apply BDOF to the prediction samples.

[0251] Additionally, the encoding device can determine whether to apply BDOF-based refinement based on BDOF enabled flag information (e.g., sps_bdof_enabled_flag) indicating whether BDOF included in the bitstream is enabled, or subblock-based merge flag information (e.g., merge_subblock_flag) indicating whether subblock-based merge mode is applied to the current block. For example, the encoding device can determine to apply BDOF to prediction samples when the value of the BDOF enabled flag information is 1, or when the value of the subblock-based merge flag information is 0.

[0252] Additionally, the encoding device can determine whether to apply BDOF to the prediction samples based on whether bi-prediction performed based on the L0 reference picture and the L1 reference picture is applied to the current block, whether the L0 reference picture and the L1 reference picture are located in different directions in time with respect to the current picture, or whether the affine mode is applied to the current block. For example, if bi-prediction based on an L0 reference picture and an L1 reference picture is applied to the current block (e.g., when predFlagL0=1 and predFlagL1=1), if the L0 reference picture and the L1 reference picture are located in different directions in time with respect to the current picture (e.g., when DiffPicOrderCnt( currPic, refPicList0[refIdxL0] ) * DiffPicOrderCnt( currPic, refPicList1[refIdxL1] ) is less than 0), or if no affine mode is applied to the current block (e.g., when MotionModelIdc = 0), the encoding device may decide to apply BDOF to the prediction samples.

[0253] Additionally, as another example, the encoding device can determine whether to apply BDOF to prediction samples based on whether bi-prediction performed based on an L0 reference picture and an L1 reference picture is applied to the current block, or whether the distance between the current picture and the L0 reference picture and the distance between the current picture and the L1 reference picture are equal. For example, if the bi-prediction based on the L0 reference picture and the L1 reference picture is applied to the current block (e.g., if predFlagL0[0][0]=1 and predFlagL0[1][1]=1), or if the distances to the L0 reference picture and the distances to the L1 reference picture are equal to each other based on the current picture (e.g., if DiffPicOrderCnt(currPic, RefPicList0 [refIdxL0]) - DiffPicOrderCnt(currPic, RefPicList1 [refIdxL1]) = 0), the encoding device may decide to apply BDOF to the prediction samples.

[0254] In determining whether to apply BDOF-based refinement here, the conditions listed above are only examples, and the conditions in Tables 1 to 20 described above can be used in various combinations.

[0255] If the encoding device decides to apply BDOF to the prediction samples, it can apply BDOF to the prediction samples to derive refined prediction samples.

[0256] The encoding device can derive residual samples based on prediction samples (S840) and generate residual information based on the residual samples (S850).

[0257] That is, the encoding device can derive residual samples based on the original samples for the current block and the predicted samples for the current block. In addition, the encoding device can generate information about the residual samples.

[0258] The encoding device can encode image information including residual information (S860).

[0259] That is, the encoding device can encode information about residual samples, output it as a bitstream, and transmit it to a decoding device via a network or storage medium.

[0260] Additionally, the encoding device can encode the information derived from steps S800 to S830 described above and output it as a bitstream. For example, DMVR availability flag information, merge flag information, MMVD flag information, BDOF availability flag information, sub-block-based merge flag information, etc. can be encoded and included in the video information, and such encoded video information can be signaled to the decoding device.

[0261] FIG. 9 is a flowchart schematically illustrating a decoding method that may be performed by a decoding device according to one embodiment of a document.

[0262] The method disclosed in FIG. 9 can be performed by the decoding device (300) disclosed in FIG. 3. Specifically, steps S900 to S930 of FIG. 9 can be performed by the prediction unit (330) and the inter prediction unit (332) disclosed in FIG. 3, and step S940 of FIG. 9 can be performed by the addition unit (340) disclosed in FIG. 3. In addition, the method disclosed in FIG. 9 can include the embodiments described above in this document. Therefore, in FIG. 9, a detailed description of the overlapping contents with the embodiments described above will be omitted or simplified.

[0263] Referring to FIG. 9, the decoding device can derive a motion vector of the current block (S900).

[0264] In one embodiment, the decoding device may determine a prediction mode for the current block based on prediction information signaled from the encoding device. Furthermore, the decoding device may derive motion information (motion vectors, reference picture indices, etc.) for the current block based on the prediction mode. The prediction mode may include skip mode, merge mode, (A)MVP mode, etc.

[0265] For example, when skip mode or merge mode is applied to the current block, the decoding device can configure a merge candidate list and select one merge candidate from among the merge candidates included in the merge candidate list. At this time, one merge candidate can be selected from the merge candidate list based on the merge index information described above. The decoding device can derive motion information of the current block using motion information of the selected merge candidate. That is, motion information of a merge candidate selected by the merge index from among the merge candidates included in the merge candidate list can be used as motion information of the current block.

[0266] As another example, when the (A)MVP mode is applied to the current block, the decoding device may construct an (A)MVP candidate list, and use the motion vector of an MVP candidate selected from among the MVP candidates included in the (A)MVP candidate list as the MVP of the current block. The selection of the MVP candidate may be performed based on MVP flag information (mvp flag or mvp index). In addition, the decoding device may derive the MVD of the current block based on information about the MVD signaled from the encoding device. In this case, the decoding device may derive the motion vector of the current block based on the MVP and MVD of the current block. In addition, the decoding device may derive the reference picture index of the current block based on reference picture index information signaled from the encoding device. In this case, the picture indicated by the reference picture index in the reference picture list for the current block may be derived as a reference picture referenced for inter prediction of the current block.

[0267] The decoding device can determine whether to apply refinement to the motion vector of the current block (S910).

[0268] In one embodiment, the decoding device may determine whether to apply refinement based on at least one of the size of the current block or the bi-prediction weight index information of the current block. For example, if the size of the current block is 16X16 or larger, it may be determined to apply DMVR to the motion vector of the current block. Additionally, if the value of the bi-prediction weight index information of the current block is 0, it may be determined to apply DMVR to the motion vector of the current block.

[0269] Here, the bi-prediction weight index information represents the above-described GbiIdx. This can be used as information indicating the weight factor applied to L0 prediction and L1 prediction in generalized bi-prediction where different weights can be applied to L0 prediction and L1 prediction. For example, when the merge mode is applied to the current block, the bi-prediction weight index information of the current block may be the same as the bi-prediction weight index information of the neighboring blocks of the current block indicated by the merge index.

[0270] In addition, the decoding device can obtain at least one of DMVR enabled flag information (e.g., sps_dmvr_enabled_flag) indicating whether DMVR is enabled from the bitstream, merge flag information (e.g., merge_flag) indicating whether merge mode is applied to the current block, or MMVD (merge mode with motion vector difference) flag information (e.g., mmvd_flag) indicating whether MMVD is applied to the current block, and can determine whether to apply DMVR-based refinement based on the information obtained from the bitstream. For example, the decoding device can determine to apply DMVR to the motion vector of the current block when the value of the DMVR enabled flag information is 1, when the value of the merge flag information is 1, or when the value of the MMVD flag information is 0.

[0271] Additionally, the decoding device can determine whether to apply DMVR to the motion vector of the current block based on whether bi-prediction performed based on the L0 reference picture and the L1 reference picture is applied to the current block, or whether the distance between the current picture and the L0 reference picture and the distance between the current picture and the L1 reference picture are equal. For example, if the bi-prediction based on the L0 reference picture and the L1 reference picture is applied to the current block (e.g., if predFlagL0[0][0]=1 and predFlagL0[1][1]=1), or if the distances to the L0 reference picture and the distances to the L1 reference picture are equal to each other based on the current picture (e.g., if DiffPicOrderCnt(currPic, RefPicList0 [refIdxL0]) - DiffPicOrderCnt(currPic, RefPicList1 [ refIdxL1 ] ) = 0), the decoding device may decide to apply DMVR to the motion vector of the current block.

[0272] Additionally, as another example, the decoding device can determine whether to apply DMVR to the motion vector of the current block based on whether bi-prediction performed based on the L0 reference picture and the L1 reference picture is applied to the current block, whether the L0 reference picture and the L1 reference picture are located in different directions in time with respect to the current picture, or whether the affine mode is applied to the current block. For example, if the bi-prediction based on the L0 reference picture and the L1 reference picture is applied to the current block (e.g., when predFlagL0=1 and predFlagL1=1), if the L0 reference picture and the L1 reference picture are located in different directions in time with respect to the current picture (e.g., when DiffPicOrderCnt( currPic, refPicList0[refIdxL0] ) * DiffPicOrderCnt( currPic, refPicList1[refIdxL1] ) is less than 0), or if the affine mode is not applied to the current block (e.g., when MotionModelIdc = 0), the decoding device may decide to apply DMVR to the motion vector of the current block.

[0273] In determining whether to apply DMVR-based refinement here, the conditions listed above are only examples, and the conditions in Tables 1 to 20 described above may be used in various combinations.

[0274] If the decoding device decides to apply refinement to the motion vector of the current block, it can apply DMVR to the motion vector to derive a refined motion vector (S920).

[0275] At this time, the decoding device can apply the DMVR process of FIG. 4 described above. The process of deriving a refined motion vector by applying DMVR has been described in detail in FIG. 4, so a description thereof will be omitted here.

[0276] The decoding device can derive prediction samples for the current block based on the refined motion vector (S930).

[0277] In some embodiments, the decoding device may apply refinement to the prediction samples of the current block.

[0278] In one embodiment, the decoding device may determine whether to apply BDOF to the prediction samples based on at least one of the size of the current block and bi-prediction weight index information of the current block. For example, if the size of the current block is 16X16 or larger, it may be determined to apply BDOF to the prediction samples. Additionally, if the value of the bi-prediction weight index information of the current block is 0, it may be determined to apply BDOF to the prediction samples.

[0279] Additionally, the decoding device can obtain at least one of BDOF enabled flag information (e.g., sps_bdof_enabled_flag) indicating whether BDOF is enabled from the bitstream, or subblock-based merge flag information (e.g., merge_subblock_flag) indicating whether subblock-based merge mode is applied to the current block, and can determine whether to apply BDOF-based refinement based on the information obtained from the bitstream. For example, the decoding device can determine to apply BDOF to prediction samples when the value of the BDOF enabled flag information is 1, or when the value of the subblock-based merge flag information is 0.

[0280] Additionally, the decoding device can determine whether to apply BDOF to the prediction samples based on whether bi-prediction performed based on the L0 reference picture and the L1 reference picture is applied to the current block, whether the L0 reference picture and the L1 reference picture are located in different directions in time with respect to the current picture, or whether the affine mode is applied to the current block. For example, if bi-prediction based on an L0 reference picture and an L1 reference picture is applied to the current block (e.g., predFlagL0=1 and predFlagL1=1), if the L0 reference picture and the L1 reference picture are located in different directions in time with respect to the current picture (e.g., DiffPicOrderCnt( currPic, refPicList0[refIdxL0] ) * DiffPicOrderCnt( currPic, refPicList1[refIdxL1] ) is less than 0), or if the affine mode is not applied to the current block (e.g., MotionModelIdc = 0), the decoding device may decide to apply BDOF to the prediction samples.

[0281] Additionally, as another example, the decoding device can determine whether to apply BDOF to prediction samples based on whether bi-prediction performed based on an L0 reference picture and an L1 reference picture is applied to the current block, or whether the distance between the current picture and the L0 reference picture and the distance between the current picture and the L1 reference picture are equal. For example, if the bi-prediction based on the L0 reference picture and the L1 reference picture is applied to the current block (e.g., if predFlagL0[0][0]=1 and predFlagL0[1][1]=1), or if the distances to the L0 reference picture and the distances to the L1 reference picture are equal to each other based on the current picture (e.g., if DiffPicOrderCnt(currPic, RefPicList0 [refIdxL0]) - DiffPicOrderCnt(currPic, RefPicList1 [ refIdxL1 ] ) = 0), the decoding device may decide to apply BDOF to the prediction samples.

[0282] In determining whether to apply BDOF-based refinement here, the conditions listed above are only examples, and the conditions in Tables 1 to 20 described above can be used in various combinations.

[0283] If the decoding device decides to apply BDOF to the prediction samples, it can apply BDOF to the prediction samples to derive refined prediction samples.

[0284] The decoding device can generate restoration samples for the current block based on the prediction samples (S940).

[0285] In one embodiment, the decoding device may use the prediction samples directly as restoration samples, depending on the prediction mode, or may generate restoration samples by adding residual samples to the prediction samples.

[0286] The decoding device can receive information about the residual for the current block, if there are residual samples for the current block. The information about the residual can include transform coefficients for the residual samples. The decoding device can derive residual samples (or a residual sample array) for the current block based on the residual information. The decoding device can generate reconstructed samples based on the prediction samples and the residual samples, and derive a reconstructed block or a reconstructed picture based on the reconstructed samples.

[0287] While the methods described in the above-described embodiments are described based on a flowchart as a series of steps or blocks, the embodiments of this document are not limited to the order of the steps, and some steps may occur in a different order or simultaneously with other steps described above. Furthermore, those skilled in the art will appreciate that the steps depicted in the flowchart are not exclusive, and that other steps may be included, or one or more steps in the flowchart may be deleted, without affecting the scope of this document.

[0288] The method according to the present document described above can be implemented in the form of software, and the encoding device and / or decoding device according to the present document can be included in a device that performs image processing, such as a TV, a computer, a smartphone, a set-top box, a display device, etc.

[0289] When the embodiments in this document are implemented as software, the above-described method can be implemented as a module (process, function, etc.) that performs the above-described function. The module can be stored in memory and executed by a processor. The memory can be internal or external to the processor and can be connected to the processor by various well-known means. The processor can include an application-specific integrated circuit (ASIC), another chipset, logic circuit, and / or data processing device. The memory can include a read-only memory (ROM), a random access memory (RAM), flash memory, a memory card, a storage medium, and / or other storage devices. That is, the embodiments described in this document can be implemented and performed on a processor, a microprocessor, a controller, or a chip. For example, the functional units illustrated in each drawing can be implemented and performed on a computer, a processor, a microprocessor, a controller, or a chip. In this case, information for implementation (e.g., information on instructions) or an algorithm can be stored on a digital storage medium.

[0290] In addition, the decoding device and encoding device to which this document applies may be included in a multimedia broadcasting transmitting and receiving device, a mobile communication terminal, a home cinema video device, a digital cinema video device, a surveillance camera, a video conversation device, a real-time communication device such as a video communication, a mobile streaming device, a storage medium, a camcorder, a video-on-demand (VoD) service providing device, an OTT video (Over the top video) device, an Internet streaming service providing device, a three-dimensional (3D) video device, a VR (virtual reality) device, an AR (argumente reality) device, a video phone video device, a transportation terminal (ex. a vehicle (including an autonomous vehicle) terminal, an airplane terminal, a ship terminal, etc.), and a medical video device, and may be used to process a video signal or a data signal. For example, the OTT video (Over the top video) device may include a game console, a Blu-ray player, an Internet-connected TV, a home theater system, a smartphone, a tablet PC, a DVR (Digital Video Recorder), etc.

[0291] In addition, the processing method to which this document applies can be produced in the form of a computer-executable program and stored in a computer-readable recording medium. Multimedia data having a data structure according to this document can also be stored in a computer-readable recording medium. The computer-readable recording medium includes all types of storage devices and distributed storage devices in which computer-readable data is stored. The computer-readable recording medium may include, for example, a Blu-ray Disc (BD), a Universal Serial Bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. In addition, the computer-readable recording medium includes a medium implemented in the form of a carrier wave (e.g., transmission via the Internet). In addition, a bitstream generated by an encoding method can be stored in a computer-readable recording medium or transmitted via a wired or wireless communication network.

[0292] Additionally, the embodiments of the present document can be implemented as a computer program product by program code, and the program code can be executed on a computer by the embodiments of the present document. The program code can be stored on a computer-readable carrier.

[0293] FIG. 10 illustrates an example of a content streaming system to which the embodiments disclosed in this document can be applied.

[0294] Referring to FIG. 10, the content streaming system applied to the embodiments of the present document may largely include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0295] The encoding server compresses content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data, generates a bitstream, and transmits it to the streaming server. Alternatively, if multimedia input devices such as smartphones, cameras, and camcorders directly generate bitstreams, the encoding server may be omitted.

[0296] The above bitstream can be generated by an encoding method or a bitstream generation method applied to the embodiments of the present document, and the streaming server can temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0297] The streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server acts as an intermediary to inform the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, and the streaming server transmits the multimedia data to the user. At this time, the content streaming system may include a separate control server, in which case the control server controls commands / responses between each device within the content streaming system.

[0298] The streaming server can receive content from a media repository and / or an encoding server. For example, when receiving content from the encoding server, the content can be received in real time. In this case, to provide a smooth streaming service, the streaming server can store the bitstream for a certain period of time.

[0299] Examples of the user devices may include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, HMDs), digital TVs, desktop computers, digital signage, etc.

[0300] Each server within the above content streaming system can be operated as a distributed server, in which case data received from each server can be processed in a distributed manner.

Claims

1. In a video decoding method performed by a decoding device, A step of deriving the motion vector of the current block; A step of determining whether to apply refinement to the motion vector of the current block; If it is decided to apply refinement to the motion vector of the current block, a step of applying DMVR (Decoder-side Motion Vector Refinement) to the motion vector to derive a refined motion vector; A step of deriving prediction samples for the current block based on the refined motion vector; and A step of generating restoration samples for the current block based on the above prediction samples, The step of deciding whether to apply the above refinement is: An image decoding method characterized in that it is determined whether to apply the refinement based on at least one of the size of the current block or bi-prediction weight index information of the current block.

2. In paragraph 1, The step of deciding whether to apply the above refinement is: An image decoding method characterized in that when the size of the current block is 16X16 or larger, it is decided to apply DMVR to the motion vector of the current block.

3. In paragraph 1, The step of deciding whether to apply the above refinement is: An image decoding method characterized in that when the value of the pair prediction weight index information of the current block is 0, it is decided to apply DMVR to the motion vector of the current block.

4. In paragraph 3, The pair prediction weight index information of the current block above is, An image decoding method characterized by having the same pair prediction weight index information as that of a neighboring block indicated by a merge index among neighboring blocks of the current block when a merge mode is applied to the current block.

5. In paragraph 1, An image decoding method, further comprising a step of determining whether to apply BDOF (Bi-directional optical flow) to the prediction samples based on at least one of the size of the current block or bi-prediction weight index information of the current block.

6. In paragraph 5, The step of deciding whether to apply the above BDOF is: An image decoding method characterized in that when the size of the current block is 16X16 or more and the value of the pair prediction weight index information of the current block is 0, it is decided to apply BDOF to the prediction samples.

7. In paragraph 1, Further comprising a step of obtaining at least one of DMVR availability flag information indicating whether the DMVR is enabled from the bitstream, merge flag information indicating whether the merge mode is applied to the current block, or MMVD flag information indicating whether the merge mode with motion vector difference (MMVD) is applied to the current block, The step of deciding whether to apply the above refinement is: A video decoding method characterized in that when the value of the DMVR available flag information is 1, when the value of the merge flag information is 1, or when the value of the MMVD flag information is 0, it is determined to apply DMVR to the motion vector of the current block.

8. In paragraph 5, The step of deciding whether to apply the above BDOF is: A video decoding method characterized in that it is determined whether to apply BDOF to the prediction samples based on whether bi-prediction performed based on an L0 reference picture and an L1 reference picture is applied to the current block, or whether the distance between the current picture and the L0 reference picture and the distance between the current picture and the L1 reference picture are equal.

9. In paragraph 5, Further comprising a step of obtaining BDOF availability flag information indicating whether the BDOF is available (enabled) from the bitstream, or subblock-based merge flag information indicating whether the subblock-based merge mode is applied to the current block, The step of deciding whether to apply the above BDOF is: An image decoding method characterized in that when the value of the BDOF available flag information is 1 or when the value of the subblock-based merge flag information is 0, it is determined to apply BDOF to the prediction samples.

10. In paragraph 1, The step of deciding whether to apply the above refinement is: A video decoding method characterized in that it is determined whether to apply DMVR to a motion vector of the current block based on whether bi-prediction performed based on an L0 reference picture and an L1 reference picture is applied to the current block, whether the L0 reference picture and the L1 reference picture are located in different directions in time with respect to the current picture, or whether an affine mode is applied to the current block.

11. In a video encoding method performed by an encoding device, A step of deriving the motion vector of the current block; A step of determining whether to apply refinement to the motion vector of the current block; If it is decided to apply refinement to the motion vector of the current block, a step of applying DMVR (Decoder-side Motion Vector Refinement) to the motion vector to derive a refined motion vector; A step of deriving prediction samples for the current block based on the refined motion vector; A step of deriving residual samples based on the above prediction samples; A step of generating residual information based on the above residual samples; and A step of encoding image information including the above residual information is included, The step of deciding whether to apply the above refinement is: A video encoding method characterized in that it is determined whether to apply the refinement based on at least one of the size of the current block or bi-prediction weight index information of the current block.

12. In paragraph 11, The step of deciding whether to apply the above refinement is: A video encoding method characterized in that when the size of the current block is 16X16 or larger, it is decided to apply DMVR to the motion vector of the current block.

13. In paragraph 11, The step of deciding whether to apply the above refinement is: A video encoding method characterized in that, when the value of the pair prediction weight index information of the current block is 0, it is decided to apply DMVR to the motion vector of the current block.

14. In paragraph 11, An image encoding method, characterized in that it further comprises a step of determining whether to apply BDOF (Bi-directional optical flow) to the prediction samples based on at least one of the size of the current block or bi-prediction weight index information of the current block.

15. In paragraph 14, The step of deciding whether to apply the above BDOF is: An image encoding method characterized in that, when the size of the current block is 16X16 or more and the value of the pair prediction weight index information of the current block is 0, it is decided to apply BDOF to the prediction samples.

16. In paragraph 14, The step of deciding whether to apply the above BDOF is: A video encoding method characterized in that it is determined whether to apply BDOF to the prediction samples based on whether bi-prediction performed based on an L0 reference picture and an L1 reference picture is applied to the current block, or whether the distance between the current picture and the L0 reference picture and the distance between the current picture and the L1 reference picture are equal.

Images