Method and apparatus for inter-prediction based on dmvr

Abstract

To provide a method and apparatus for increasing video coding efficiency.SOLUTION: Determining whether an application condition of DMVR for applying motion vector refinement to a current block is satisfied, deriving a minimum SAD based on a L0 motion vector and a L1 motion vector of the current block when the application condition of DMVR is satisfied, deriving a refined L0 motion vector and a L1 motion vector for the current block based on the minimum SAD, deriving a prediction sample for the current block based on the refined L0 motion vector and the refined L1 motion vector, and generating a reconstructed sample for the current block based on the prediction sample, the DMVR application condition may be determined to be satisfied when the prediction mode in which the inter-prediction and the intra-prediction are combined is not applied to the current block.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] This document relates to video coding techniques, and more particularly to a method and apparatus for performing inter prediction based on Decoder-side Motion Vector Refinement (DMVR). [Background technology]

[0002] Recently, the demand for high-resolution, high-quality images / videos such as 4K or 8K or higher UHD (Ultra High Definition) images / videos is increasing in various fields. As the resolution and quality of image / video data increases, the amount of information or bits to be transmitted increases relatively compared to existing image / video data. Therefore, when transmitting image data using existing media such as wired or wireless broadband lines or storing image / video data using existing storage media, the transmission and storage costs increase.

[0003] In addition, interest and demand for immersive media such as VR (Virtual Reality), AR (Artificial Reality) content and holograms has been increasing recently, and the broadcast of images / videos with different image characteristics from real images, such as game images, is increasing.

[0004] Therefore, a highly efficient image / video compression technology is required to effectively compress, transmit, store, and play back high-resolution, high-quality image / video information having the above-mentioned various characteristics. Summary of the Invention [Problem to be solved by the invention]

[0005] The technical problem of this document is to provide a method and apparatus for increasing video coding efficiency.

[0006] Another technical problem of this document is to provide an efficient inter-prediction method and apparatus.

[0007] Another technical problem of this document is to provide a method and apparatus for performing inter prediction based on Decoder-side Motion Vector Refinement (DMVR).

[0008] Another technical problem of this document is to provide a method and apparatus for performing inter prediction based on BDOF (Bi-directional optical flow).

[0009] Another technical problem of this document is to provide a method and apparatus for improving prediction performance by providing conditions for determining whether to apply DMVR and / or conditions for determining whether to apply BDOF to improve video coding efficiency. [Means for solving the problem]

[0010] According to one embodiment of the present document, there is provided a video decoding method performed by a decoding device, the method including the steps of determining whether a condition for applying Decoder-side Motion Vector Refinement (DMVR) for applying motion vector refinement to a current block is met, and if the condition for applying DMVR is met, deriving a minimum SAD (Sum of Absolute Differences) based on an L0 motion vector and an L1 motion vector of the current block, deriving a refined L0 motion vector and a refined L1 motion vector for the current block based on the minimum SAD, deriving prediction samples for the current block based on the refined L0 motion vector and the refined L1 motion vector, and generating reconstructed samples for the current block based on the prediction samples, wherein whether the condition for applying DMVR is met is determined to be met if a prediction mode in which inter prediction and intra prediction are combined is not applied to the current block.

[0011] According to another embodiment of the present document, there is provided a video encoding method performed by an encoding apparatus, the method including the steps of: determining whether a condition for applying Decoder-side Motion Vector Refinement (DMVR) for applying motion vector refinement to a current block is met; if the condition for applying DMVR is met, deriving a minimum SAD (Sum of Absolute Differences) based on an L0 motion vector and an L1 motion vector of the current block, deriving a refined L0 motion vector and a refined L1 motion vector for the current block based on the minimum SAD; deriving prediction samples for the current block based on the refined L0 motion vector and the refined L1 motion vector; deriving residual samples based on the prediction samples; and encoding video information including information on the residual samples, wherein whether the condition for applying DMVR is met is determined to be met when a prediction mode combining inter prediction and intra prediction is not applied to the current block. [Effects of the Invention]

[0012] According to the document, it can improve overall image / video compression efficiency.

[0013] According to this document, efficient inter-prediction can reduce computational complexity and improve overall coding efficiency.

[0014] According to this document, when applying DMVR and / or BDOF, which refine motion information during the motion compensation process, various application conditions are proposed, thereby improving efficiency in terms of complexity and performance. [Brief explanation of the drawings]

[0015] [Figure 1] 1 illustrates schematically an example of a video / image coding system that can be applied to embodiments of the present document. [Figure 2] 1 is a diagram illustrating the configuration of a video / image encoding device that can be applied to an embodiment of the present document; [Figure 3] 1 is a diagram illustrating the configuration of a video / image decoding device that can be applied to an embodiment of the present document. [Figure 4] FIG. 10 is a diagram illustrating an embodiment of a process for performing decoder-side motion vector refinement (DMVR) in true bi-prediction. [Figure 5] 1 is a diagram illustrating an embodiment of a process for performing Decoder-side Motion Vector Refinement (DMVR) using SAD (sum of absolute differences). FIG. [Figure 6] 10 is an example showing a method for checking the application conditions of DMVR and BDOF and performing a decoding process. [Figure 7] 10 is another example showing a method of checking the application conditions of DMVR and BDOF and performing a decoding process. [Figure 8] 10 is another example showing a method of checking the application conditions of DMVR and BDOF and performing a decoding process. [Figure 9] 1 is a flow chart illustrating an encoding method that can be performed by an encoding device according to an embodiment of the present document. [Figure 10] 1 is a flow chart illustrating a decoding method that can be performed by a decoding device according to an embodiment of the document. [Figure 11] 1 illustrates an example of a content streaming system to which the embodiments disclosed herein can be applied. DETAILED DESCRIPTION OF THE INVENTION

[0016] This document may be modified in various ways and may have various embodiments. Specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit this document to the specific embodiments. Common terms used in this document are used merely to describe specific embodiments and are not intended to limit the technical ideas of this document. A singular expression includes a plural expression unless the context clearly dictates otherwise. In this specification, the terms "comprise" or "have" are intended to specify the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, and should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0017] Meanwhile, each component in the drawings described in this document is illustrated independently for the convenience of explaining the different characteristic functions, and does not mean that each component is realized by separate hardware or software. For example, two or more components may be combined to form a single component, or a single component may be divided into multiple components. Embodiments in which each component is integrated and / or separated are also included within the scope of this document as long as they do not deviate from the essence of this document.

[0018] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. Hereinafter, the same reference numerals will be used to refer to the same components in the drawings, and duplicated descriptions of the same components may be omitted.

[0019] This document relates to video / image coding. For example, the methods / embodiments disclosed in this document can be applied to methods disclosed in the versatile video coding (VVC) standard, the essential video coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the second generation audio video coding standard (AVS2), or next generation video / image coding standards (e.g., H.267 or H.268).

[0020] This document presents various embodiments relating to video / image coding, and unless otherwise stated, the embodiments may be implemented in combination with each other.

[0021] In this document, video may refer to a collection of a series of images over time. A picture generally refers to a unit that shows one image at a specific time, and a slice / tile is a unit that constitutes part of a picture in coding. A slice / tile may include one or more coding tree units (CTUs). A picture may be composed of one or more slices / tiles. A picture may be composed of one or more tile groups. A tile group may include one or more tiles. A brick may represent a rectangular region of CTU rows within a tile in a picture. A tile may be partitioned into multiple bricks, each consisting of one or more CTU rows within the tile. A tile that is not partitioned into multiple bricks may also be referred to as a brick.A brick scan may indicate a specific sequential ordering of CTUs partitioning a picture, where the CTUs may be arranged in a CTU raster scan within a brick, the bricks within a tile may be arranged consecutively in a raster scan of the bricks of the tile, and the tiles within a picture may be arranged consecutively in a raster scan of the tiles of the picture. A tile is a rectangular region of CTUs within a particular tile column and a particular tile row in a picture. The tile column is a rectangular region of CTUs having a height equal to the height of the picture and a width specified by syntax elements in the picture parameter set.The tile row is a rectangular region of CTUs having a height specified by syntax elements in the picture parameter set and a width equal to the height of the picture. A tile scan may indicate a specific sequential ordering of CTUs partitioning a picture, where the CTUs may be consecutively aligned in a CTU raster scan in a tile, and tiles in a picture may be consecutively aligned in a raster scan of the tiles of the picture. A slice includes an integer number of bricks of a picture that may be exclusively contained in a single NAL unit. A slice may consist of either a number of complete tiles or only a consecutive sequence of complete bricks of one tile.In this document, the terms tile group and slice may be used interchangeably. For example, in this document, tile group / tile group header may be referred to as slice / slice header.

[0022] A pixel or a pel may refer to the smallest unit constituting a picture (or image). A "sample" may also be used as a term corresponding to a pixel. A sample may generally refer to a pixel or a pixel value, may refer to only a pixel / pixel value of a luma component, or may refer to only a pixel / pixel value of a chroma component. Alternatively, a sample may refer to a pixel value in the spatial domain, or, when such a pixel value is transformed into the frequency domain, may refer to a transform coefficient in the frequency domain.

[0023] A unit may refer to a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. One unit may include one luma block and two chroma (e.g., cb, cr) blocks. The term unit may be used interchangeably with terms such as block or area. In general, an M×N block may include samples (or a sample array) consisting of M columns and N rows, or a set (or an array) of transform coefficients.

[0024] In this document, the terms " / " and "," are interpreted as "and / or." For example, "A / B" is interpreted as "A and / or B," and "A, B" is interpreted as "A and / or B." Additionally, "A / B / C" means "at least one of A, B, and / or C." Also, "A, B, C" means "at least one of A, B, and / or C." (In this document, the term " / " and "," should be interpreted to indicate "and / or." For instance, the expression "A / B" may mean "A and / or B." Further, "A,B" may mean "A and / or B." Further, "A / B / C" may mean "at least one of A, B, and / or C." Also, "A / B / C" may mean "at least one of A, B, and / or C.")

[0025] Additionally, in this document, "or" should be interpreted as "and / or." For example, "A or B" can mean 1) only "A," or 2) only "B," or 3) "A and B." Further, in this document, the term "or" should be interpreted to indicate "and / or." For instance, the expression "A or B" may comprise 1) only A, 2) only B, and / or 3) both A and B. In other words, the term "or" in this document should be interpreted to indicate "additionally or alternatively."

[0026] FIG. 1 illustrates schematically an example of a video / image coding system that can be applied to embodiments of this document.

[0027] 1, a video / image coding system may include a first device (source device) and a second device (receiving device). The source device may transmit encoded video / image information or data to the receiving device in a file or streaming format via a digital storage medium or a network.

[0028] The source device may include a video source, an encoding device, and a transmitting unit. The receiving device may include a receiving unit, a decoding device, and a renderer. The encoding device may be referred to as a video / video encoding device, and the decoding device may be referred to as a video / video decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, which may be a separate device or an external component.

[0029] A video source can acquire video / images through a video / image capture, synthesis, or generation process. A video source can include a video / image capture device and / or a video / image generation device. A video / image capture device can include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. A video / image generation device can include, for example, a computer, a tablet, a smartphone, etc., and can (electronically) generate video / images. For example, a virtual video / image can be generated via a computer, etc., in which case the video / image capture process can be replaced by a process in which related data is generated.

[0030] An encoding device can encode input video / images. The encoding device can perform a series of procedures such as prediction, transformation, and quantization for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0031] The transmitter can transmit the encoded video / image information or data output in the form of a bitstream to a receiver of a receiving device via a digital storage medium or a network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter can include elements for generating a media file in a predetermined file format and elements for transmission via a broadcasting / communication network. The receiver can receive / extract the bitstream and transmit it to a decoding device.

[0032] The decoding device can decode the video / image by performing a series of steps such as inverse quantization, inverse transform, and prediction, which correspond to the operations of the encoding device.

[0033] The renderer can render the decoded video / image, and the rendered video / image can be displayed via a display unit.

[0034] 2 is a diagram illustrating a configuration of a video / image encoding device that can be applied to an embodiment of this document. Hereinafter, the term "video encoding device" may include a video encoding device.

[0035] Referring to FIG. 2, the encoding apparatus 200 may include an image partitioner 210, a predictor 220, a residual processor 230, an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 may include an inter predictor 221 and an intra predictor 222. The residual processor 230 may include a transformer 232, a quantizer 233, a dequantizer 234, and an inverse transformer 235. The residual processor 230 may further include a subtractor 231. The adder 250 may be referred to as a reconstructor or a reconstructed block generator. The image dividing unit 210, the predicting unit 220, the residual processing unit 230, the entropy encoding unit 240, the adding unit 250, and the filtering unit 260 may be configured by one or more hardware components (e.g., an encoder chipset or a processor) depending on the embodiment. Also, the memory 270 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware components may further include the memory 270 as an internal / external component.

[0036] The image division unit 210 may divide an input image (or picture, frame) input to the encoding device 200 into one or more processing units. For example, the processing units may be called coding units (CUs). In this case, the coding units may be recursively divided from a coding tree unit (CTU) or a largest coding unit (LCU) according to a quad-tree, binary-tree, ternary-tree (QTBTTT) structure. For example, one coding unit may be divided into multiple coding units of deeper depths based on a quad-tree structure, a binary tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure may be applied first, and then the binary tree structure and / or the ternary structure may be applied. Alternatively, the binary tree structure may be applied first. The coding procedure according to this document may be performed based on the final coding unit that is not further divided. In this case, the largest coding unit may be used as the final coding unit based on coding efficiency according to image characteristics, or the coding unit may be recursively divided into coding units of lower depths as needed, and a coding unit of an optimal size may be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration, which will be described later. As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may each be divided or partitioned from the final coding unit.The prediction unit is a unit of sample prediction, and the transform unit is a unit for deriving transform coefficients and / or a unit for deriving a residual signal from the transform coefficients.

[0037] The term "unit" may be used interchangeably with terms such as "block" or "area." In general, an MxN block can refer to a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally refer to a pixel or pixel value, or can refer to only a pixel / pixel value of the luma component, or only a pixel / pixel value of the chroma component. A sample can also be used as a term corresponding to a pixel or pel in one picture (or image).

[0038] The encoding apparatus 200 may generate a residual signal (residual block, residual sample array) by subtracting a prediction signal (predicted block, prediction sample array) output from the inter prediction unit 221 or the intra prediction unit 222 from an input video signal (original block, original sample array), and the generated residual signal is transmitted to the conversion unit 232. In this case, as shown in the figure, a unit in the encoder 200 that subtracts the prediction signal (predicted block, prediction sample array) from the input video signal (original block, original sample array) may be referred to as a subtraction unit 231. The prediction unit may perform prediction on a current block (hereinafter, referred to as a current block) and generate a predicted block including prediction samples for the current block. The prediction unit may determine whether intra prediction or inter prediction is applied on a current block or CU basis. The prediction unit may generate various information related to prediction, such as prediction mode information, and transmit the information to the entropy encoding unit 240, as will be described later in the description of each prediction mode. The prediction information can be encoded by the entropy encoding unit 240 and output in the form of a bitstream.

[0039] The intra prediction unit 222 may predict the current block by referring to samples in the current picture. The referenced samples may be located adjacent to or distant from the current block depending on the prediction mode. Prediction modes in intra prediction may include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes may include, for example, DC mode and planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the granularity of the prediction direction. However, this is merely an example, and more or less directional prediction modes may be used depending on the settings. The intra prediction unit 222 may also determine the prediction mode to be applied to the current block using the prediction modes applied to neighboring blocks.

[0040] The inter prediction unit 221 may derive a predicted block for a current block based on a reference block (reference sample array) identified by a motion vector on a reference picture. To reduce the amount of motion information transmitted in inter prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information on an inter prediction direction (such as L0 prediction, L1 prediction, or Bi prediction). In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks in the current picture and temporal neighboring blocks in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The temporal neighboring block may be called a collocated reference block, a collocated CU (colCU), or the like, and the reference picture including the temporal neighboring block may be called a collocated picture (colPic). For example, the inter predictor 221 may construct a motion information candidate list based on neighboring blocks and generate information indicating which candidate is used to derive a motion vector and / or a reference picture index for the current block. Inter prediction may be performed based on various prediction modes. For example, in the case of a skip mode or a merge mode, the inter predictor 221 may use motion information of neighboring blocks as motion information for the current block. In the case of the skip mode, unlike in the merge mode, a residual signal may not be transmitted.In the case of motion vector prediction (MVP) mode, the motion vector of the neighboring block can be used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0041] The prediction unit 220 may generate a prediction signal based on various prediction methods, which will be described later. For example, the prediction unit may apply intra prediction or inter prediction for predicting a block, or may simultaneously apply intra prediction and inter prediction. This may be referred to as combined inter and intra prediction (CIIP). The prediction unit may also use an intra block copy (IBC) prediction mode or a palette mode for predicting a block. The IBC prediction mode or palette mode may be used for content video / moving video coding, such as games, such as screen content coding (SCC). IBC basically performs prediction within a current picture, but may be performed similarly to inter prediction in deriving a reference block within the current picture. That is, IBC may use at least one of the inter prediction techniques described herein. The palette mode may be considered an example of intra coding or intra prediction. When the palette mode is applied, sample values ​​within a picture may be signaled based on information related to a palette table and a palette index.

[0042] The prediction signal generated by the prediction unit (including the inter prediction unit 221 and / or the intra prediction unit 222) may be used to generate a reconstructed signal or a residual signal. The transform unit 232 may generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen-Loeve transform (KLT), a graph-based transform (GBT), or a conditionally non-linear transform (CNT). Here, GBT refers to a transform obtained from a graph representing inter-pixel relationship information. CNT refers to a transform obtained based on a prediction signal generated using all previously reconstructed pixels. The transform process may be applied to pixel blocks having the same square size or non-square blocks of variable sizes.

[0043] The quantization unit 233 quantizes the transform coefficients and transmits them to the entropy encoding unit 240. The entropy encoding unit 240 encodes the quantized signal (information about the quantized transform coefficients) and outputs it as a bitstream. The information about the quantized transform coefficients may be referred to as residual information. The quantization unit 233 may rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on a coefficient scan order, and may generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. The entropy encoding unit 240 may perform various encoding methods, such as exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc. The entropy encoding unit 240 may encode information required for video / image restoration (e.g., values ​​of syntax elements) together with or separately from the quantized transform coefficients. The encoded information (e.g., encoded video / picture information) may be transmitted or stored in the form of a bitstream in units of network abstraction layer (NAL) units. The video / picture information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). The video / picture information may also include general constraint information. In this document, information and / or syntax elements transmitted / signaled from an encoding device to a decoding device may be included in the video / picture information.The video / image information may be encoded through the above-described encoding procedure and included in the bitstream. The bitstream may be transmitted via a network or stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitter (not shown) for transmitting the signal output from the entropy encoding unit 240 and / or a storage unit (not shown) for storing the signal may be configured as an internal / external element of the encoding apparatus 200, or the transmitter may be included in the entropy encoding unit 240.

[0044] The quantized transform coefficients output from the quantizer 233 may be used to generate a prediction signal. For example, a residual signal (residual block or residual sample) may be reconstructed by applying inverse quantization and inverse transform to the quantized transform coefficients via the inverse quantizer 234 and the inverse transformer 235. The adder 155 may generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter predictor 221 or the intra predictor 222. When there is no residual for the current block, such as when a skip mode is applied, a predicted block may be used as the reconstructed block. The adder 250 may be referred to as a reconstruction unit or a reconstructed block generator. The generated reconstructed signal may be used for intra prediction of the next block to be processed in the current picture, or may be used for inter prediction of the next picture after filtering, as described below.

[0045] Meanwhile, LMCS (luma mapping with chroma scaling) can be applied during picture encoding and / or reconstruction.

[0046] The filtering unit 260 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 260 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture and store the modified reconstructed picture in the memory 270, specifically, in the DPB of the memory 270. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, an adaptive loop filter, a bilateral filter, etc. The filtering unit 260 may generate various information related to filtering and transmit it to the entropy encoding unit 240, as will be described later in the description of each filtering method. The filtering information may be encoded by the entropy encoding unit 240 and output in the form of a bitstream.

[0047] The modified reconstructed picture transmitted to the memory 270 can be used as a reference picture in the inter prediction unit 221. When inter prediction is applied through this, the encoding apparatus can avoid prediction mismatch between the encoding apparatus 100 and the decoding apparatus, and can also improve coding efficiency.

[0048] The DPB of the memory 270 may store a modified reconstructed picture to be used as a reference picture in the inter predictor 221. The memory 270 may store motion information of a block from which motion information in the current picture is derived (or encoded) and / or motion information of a block in an already reconstructed picture. The stored motion information may be transmitted to the inter predictor 221 to be used as motion information of a spatially neighboring block or a temporally neighboring block. The memory 270 may store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra predictor 222.

[0049] FIG. 3 is a diagram illustrating the configuration of a video / image decoding device that can be applied to the embodiments of this document.

[0050] Referring to FIG. 3, the decoding device 300 may include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an inter predictor 331 and an intra predictor 332. The residual processor 320 may include a dequantizer 321 and an inverse transformer 321. Depending on the embodiment, the entropy decoding unit 310, the residual processor 320, the predictor 330, the adder 340, and the filter 350 may be implemented as a single hardware component (e.g., a decoder chipset or processor). In addition, the memory 360 may include a decoded picture buffer (DPB) and may be implemented as a digital storage medium. The hardware components may further include a memory 360 as an internal / external component.

[0051] When a bitstream including video / image information is input, the decoding apparatus 300 can reconstruct an image corresponding to the process in which the video / image information was processed by the encoding apparatus of FIG. 2. For example, the decoding apparatus 300 can derive units / blocks based on block division-related information obtained from the bitstream. The decoding apparatus 300 can perform decoding using a processing unit applied by the encoding apparatus. Accordingly, the processing unit for decoding is, for example, a coding unit, and the coding unit can be divided from a coding tree unit or a maximal coding unit according to a quad tree structure, a binary tree structure, and / or a ternary tree structure. One or more transform units can be derived from the coding unit. The reconstructed image signal decoded and output by the decoding apparatus 300 can be played back via a playback device.

[0052] The decoding apparatus 300 may receive a signal output from the encoding apparatus of FIG. 2 in the form of a bitstream, and the received signal may be decoded via the entropy decoding unit 310. For example, the entropy decoding unit 310 may parse the bitstream to derive information (e.g., video / video information) necessary for video restoration (or picture restoration). The video / video information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). The video / video information may also include general constraint information. The decoding apparatus may decode pictures based on the information on the parameter sets and / or the general constraint information. Signaled / received information and / or syntax elements, which will be described later in this document, may be decoded via the decoding procedure and obtained from the bitstream. For example, the entropy decoding unit 310 may decode information in a bitstream based on a coding method such as Exponential Golomb coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image restoration and quantized values ​​of transform coefficients related to residuals. More specifically, the CABAC entropy decoding method receives bins corresponding to each syntax element in the bitstream, determines a context model using information on the syntax element to be decoded, decoding information on neighboring and current blocks, or information on symbols / bins decoded in previous steps, predicts the occurrence probability of bins based on the determined context model, and performs arithmetic decoding of the bins to generate symbols corresponding to the values ​​of each syntax element.In this case, after determining a context model, the CABAC entropy decoding method may update the context model using information on the decoded symbol / bin for the context model of the next symbol / bin. Prediction-related information from the information decoded by the entropy decoding unit 310 may be provided to a prediction unit (inter prediction unit 332 and intra prediction unit 331), and residual values ​​entropy-decoded by the entropy decoding unit 310, i.e., quantized transform coefficients and related parameter information, may be input to the residual processing unit 320. The residual processing unit 320 may derive a residual signal (residual block, residual sample, residual sample array). Furthermore, filtering-related information from the information decoded by the entropy decoding unit 310 may be provided to the filtering unit 350. Meanwhile, a receiving unit (not shown) that receives a signal output from the encoding apparatus may be further configured as an internal / external element of the decoding apparatus 300, or the receiving unit may be a component of the entropy decoding unit 310. Meanwhile, the decoding apparatus according to this document may be called a video / image / picture decoding apparatus, and the decoding apparatus may be divided into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include the entropy decoding unit 310, and the sample decoder may include at least one of the inverse quantization unit 321, the inverse transform unit 322, the addition unit 340, the filtering unit 350, the memory 360, the inter prediction unit 332, and the intra prediction unit 331.

[0053] The inverse quantization unit 321 may inverse quantize the quantized transform coefficients to output transform coefficients. The inverse quantization unit 321 may rearrange the quantized transform coefficients in a two-dimensional block format. In this case, the rearrangement may be performed based on the coefficient scanning order performed in the encoding apparatus. The inverse quantization unit 321 may perform inverse quantization on the quantized transform coefficients using a quantization parameter (e.g., quantization step size information) to obtain transform coefficients.

[0054] The inverse transform unit 322 performs inverse transform on the transform coefficients to obtain a residual signal (residual block, residual sample array).

[0055] The prediction unit may perform prediction on a current block and generate a predicted block including prediction samples for the current block. The prediction unit may determine whether intra prediction or inter prediction is applied to the current block based on information about the prediction output from the entropy decoding unit 310, and may determine a specific intra / inter prediction mode.

[0056] The predictor 320 may generate a prediction signal based on various prediction methods, which will be described later. For example, the predictor may apply intra prediction or inter prediction for predicting a block, or may simultaneously apply intra prediction and inter prediction. This may be referred to as combined inter and intra prediction (CIIP). The predictor may also use an intra block copy (IBC) prediction mode or a palette mode for predicting a block. The IBC prediction mode or palette mode may be used for content video / moving picture coding, such as games, such as screen content coding (SCC). IBC basically performs prediction within a current picture, but may be performed similarly to inter prediction in that a reference block is derived within the current picture. That is, IBC may use at least one of the inter prediction techniques described herein. The palette mode may be considered an example of intra coding or intra prediction. When the palette mode is applied, information regarding a palette table and a palette index may be included in the video / picture information and signaled.

[0057] The intra prediction unit 331 may predict a current block by referring to samples in a current picture. The referenced samples may be located adjacent to or distant from the current block depending on the prediction mode. Prediction modes in intra prediction may include a plurality of non-directional modes and a plurality of directional modes. The intra prediction unit 331 may also determine a prediction mode to be applied to the current block using prediction modes applied to neighboring blocks.

[0058] The inter prediction unit 332 may derive a predicted block for the current block based on a reference block (reference sample array) identified by a motion vector on a reference picture. To reduce the amount of motion information transmitted in inter prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information on the inter prediction direction (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks in the current picture and temporal neighboring blocks in the reference picture. For example, the inter prediction unit 332 may construct a motion information candidate list based on the neighboring blocks and derive a motion vector and / or a reference picture index for the current block based on received candidate selection information. Inter prediction may be performed based on various prediction modes, and the prediction information may include information indicating the inter prediction mode for the current block.

[0059] The adder 340 may generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the acquired residual signal to a predicted signal (predicted block, predicted sample array) output from a prediction unit (including the inter prediction unit 332 and / or the intra prediction unit 331). When there is no residual for the current block, such as when a skip mode is applied, the predicted block may be used as a reconstructed block.

[0060] The adder 340 may be referred to as a reconstruction unit or a reconstruction block generator. The generated reconstruction signal may be used for intra prediction of a next block to be processed in the current picture, may be output after filtering as described below, or may be used for inter prediction of a next picture.

[0061] Meanwhile, LMCS (luma mapping with chroma scaling) can be applied during picture decoding.

[0062] The filtering unit 350 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 350 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture, and may transmit the modified reconstructed picture to the memory 360, specifically, to the DPB of the memory 360. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, an adaptive loop filter, a bilateral filter, etc.

[0063] The (modified) reconstructed picture stored in the DPB of the memory 360 may be used as a reference picture in the inter predictor 332. The memory 360 may store motion information of a block from which motion information in the current picture is derived (or decoded) and / or motion information of a block in an already reconstructed picture. The stored motion information may be transmitted to the inter predictor 260 to be used as motion information of a spatially neighboring block or a temporally neighboring block. The memory 360 may store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra predictor 331.

[0064] In this specification, the embodiments described for the filtering unit 260, inter prediction unit 221, and intra prediction unit 222 of the encoding device 200 can also be applied identically or correspondingly to the filtering unit 350, inter prediction unit 332, and intra prediction unit 331 of the decoding device 300, respectively.

[0065] As described above, prediction is performed to improve compression efficiency during video coding. Accordingly, a predicted block including predicted samples for a current block, which is a block to be coded, can be generated. Here, the predicted block includes predicted samples in the spatial domain (or pixel domain). The predicted block is derived in the same way by an encoding device and a decoding device. The encoding device can improve video coding efficiency by signaling to a decoding device information (residual information) regarding the residual between the original block and the predicted block, rather than the original sample values ​​of the original block themselves. The decoding device can derive a residual block including residual samples based on the residual information, add the residual block and the predicted block to generate a reconstructed block including reconstructed samples, and generate a reconstructed picture including the reconstructed block.

[0066] The residual information may be generated through a transform and quantization procedure. For example, an encoding apparatus may derive a residual block between the original block and the predicted block, perform a transform procedure on residual samples (residual sample array) included in the residual block to derive transform coefficients, and perform a quantization procedure on the transform coefficients to derive quantized transform coefficients, and then signal the related residual information (via a bitstream) to a decoding apparatus. Here, the residual information may include information such as value information, position information, transform technique, transform kernel, and quantization parameter of the quantized transform coefficients. The decoding apparatus may derive residual samples (or residual blocks) by performing an inverse quantization / inverse transform procedure based on the residual information. The decoding apparatus may generate a reconstructed picture based on the predicted block and the residual block. The encoding apparatus may also derive a residual block by inverse quantizing / inverse transforming the quantized transform coefficients for reference for inter-prediction of a future picture, and generate a reconstructed picture based on the residual block.

[0067] Meanwhile, as described above, intra prediction or inter prediction can be applied to perform prediction on the current block. Hereinafter, a case where inter prediction is applied to the current block will be described.

[0068] A prediction unit (more specifically, an inter prediction unit) of an encoding / decoding apparatus may perform inter prediction on a block-by-block basis to derive predicted samples. Inter prediction may refer to prediction derived in a manner dependent on data elements (e.g., sample values, motion information, etc.) of picture(s) other than the current picture. When inter prediction is applied to a current block, a predicted block (prediction sample array) for the current block may be derived based on a reference block (reference sample array) identified by a motion vector in a reference picture indicated by a reference picture index. In this case, to reduce the amount of motion information transmitted in the inter prediction mode, motion information of the current block may be predicted in block, sub-block, or sample units based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction type information (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). When inter prediction is applied, the neighboring blocks may include spatial neighboring blocks in the current picture and temporal neighboring blocks in the reference picture. The reference picture including the reference block and the reference picture including the temporally neighboring block may be the same or different. The temporally neighboring block may be called a collocated reference block, a collocated CU (colCU), etc., and the reference picture including the temporally neighboring block may be called a collocated picture (colPic). For example, a motion information candidate list may be constructed based on the neighboring blocks of the current block, and flag or index information indicating which candidate is selected (used) to derive a motion vector and / or a reference picture index for the current block may be signaled.Inter prediction can be performed based on various prediction modes. For example, in skip mode and merge mode, the motion information of the current block is the same as the motion information of the selected neighboring block. In skip mode, unlike merge mode, a residual signal may not be transmitted. In motion vector prediction (MVP) mode, the motion vector of the selected neighboring block is used as a motion vector predictor, and a motion vector difference may be signaled. In this case, the motion vector of the current block can be derived using the sum of the motion vector predictor and the motion vector difference.

[0069] The motion information may include L0 motion information and / or L1 motion information depending on the inter-prediction type (L0 prediction, L1 prediction, Bi prediction, etc.). A motion vector in the L0 direction may be referred to as an L0 motion vector or MVL0, and a motion vector in the L1 direction may be referred to as an L1 motion vector or MVL1. Prediction based on an L0 motion vector may be referred to as L0 prediction, prediction based on an L1 motion vector may be referred to as L1 prediction, and prediction based on both an L0 motion vector and an L1 motion vector may be referred to as bi-prediction. Here, the L0 motion vector may indicate a motion vector associated with a reference picture list L0 (L0), and the L1 motion vector may indicate a motion vector associated with a reference picture list L1 (L1). The reference picture list L0 may include pictures that precede the current picture in output order as reference pictures, and the reference picture list L1 may include pictures that follow the current picture in output order. A previous picture may be referred to as a forward (reference) picture, and a subsequent picture may be referred to as a backward (reference) picture. The reference picture list L0 may further include, as reference pictures, pictures that are later in output order than the current picture. In this case, the previous picture may be indexed first in the reference picture list L0, and the later picture may be indexed next. The reference picture list L1 may further include, as reference pictures, pictures that are earlier in output order than the current picture. In this case, the later picture may be indexed first in the reference picture list L1, and the previous picture may be indexed next. Here, the output order may correspond to the picture order count (POC) order.

[0070] In addition, various inter prediction modes can be used when applying inter prediction to the current block. For example, various modes such as merge mode, skip mode, motion vector prediction (MVP) mode, affine mode, and historical motion vector prediction (HMVP) mode can be used. Decoder side motion vector refinement (DMVR) mode, adaptive motion vector resolution (AMVR) mode, and bi-directional optical flow (BDOF) can also be used as additional modes. Affine mode can also be referred to as affine motion prediction mode. MVP mode can also be referred to as advanced motion vector prediction mode. In this document, some modes and / or motion information candidates derived by some modes can be included as one of the motion information-related candidates of other modes.

[0071] Prediction mode information indicating the inter prediction mode of the current block may be signaled from the encoding apparatus to the decoding apparatus. In this case, the prediction mode information may be included in a bitstream and received by the decoding apparatus. The prediction mode information may include index information indicating one of multiple candidate modes. Alternatively, the inter prediction mode may be indicated through hierarchical signaling of flag information. In this case, the prediction mode information may include one or more flags. For example, a skip flag may be signaled to indicate whether the skip mode is applied, a merge flag may be signaled to indicate whether the merge mode is applied when the skip mode is not applied, or an MVP mode may be applied when the merge mode is not applied, or a flag for additional classification may be further signaled. The affine mode may be signaled as an independent mode or as a mode dependent on the merge mode or MVP mode. For example, the affine mode may include affine merge mode and affine MVP mode.

[0072] Furthermore, motion information of the current block may be used when applying inter prediction to the current block. The encoding apparatus may derive optimal motion information for the current block through a motion estimation procedure. For example, the encoding apparatus may use an original block in an original picture for the current block to search for a similar reference block with high correlation within a predetermined search range in the reference picture in fractional pixel units, thereby deriving motion information. Block similarity may be derived based on a difference in sample values ​​based on phase. For example, block similarity may be calculated based on the sum of absolute differences (SAD) between the current block (or a template of the current block) and a reference block (or a template of the reference block). In this case, motion information may be derived based on the reference block with the smallest SAD within the search range. The derived motion information may be signaled to the decoding apparatus in various ways based on the inter prediction mode.

[0073] As described above, a predicted block for the current block may be derived based on the motion information derived according to the inter prediction mode. The predicted block may include predicted samples (prediction sample array) of the current block. If the motion vector (MV) of the current block points to a fractional sample unit, an interpolation procedure may be performed, through which predicted samples of the current block may be derived based on reference samples in fractional sample units within the reference picture. If affine inter prediction is applied to the current block, predicted samples may be generated based on sample / sub-block unit MVs. If bi-prediction is applied, predicted samples derived through a weighted sum or weighted average (by phase) of predicted samples derived based on L0 prediction (i.e., prediction using a reference picture in the reference picture list L0 and MVL0) and predicted samples derived based on L1 prediction (i.e., prediction using a reference picture in the reference picture list L1 and MVL1) may be used as predicted samples of the current block. When bi-prediction is applied, if the reference picture used for L0 prediction and the reference picture used for L1 prediction are located in different temporal directions relative to the current picture (i.e., if it corresponds to both bi-prediction and bidirectional prediction), this can be called true bi-prediction.

[0074] As described above, reconstructed samples and reconstructed pictures can be generated based on the derived predicted samples, and then procedures such as in-loop filtering can be performed.

[0075] On the other hand, skip mode and / or merge mode have limitations in motion prediction because they predict the motion of the current block based on the motion vector of a neighboring block without MVD (Motion Vector Difference). To overcome the limitations of skip mode and / or merge mode, the motion vector can be refined by applying Decoder-side Motion Vector Refinement (DMVR), Bi-directional optical flow (BDOF) mode, etc. DMVR and BDOF modes can be used when true bi-prediction is applied to the current block.

[0076] FIG. 4 is a diagram illustrating an embodiment of a process of performing decoder-side motion vector refinement (DMVR) in true bi-prediction.

[0077] DMVR is a method of performing motion prediction by refining motion information of neighboring blocks on the decoder side. When DMVR is applied, the decoder can derive refined motion information through cost comparison based on a template generated using motion information of neighboring blocks in merge / skip mode. In this case, the accuracy of motion prediction can be increased without additional signaling information, thereby improving compression performance.

[0078] Although this document will be described mainly in terms of a decoding device for convenience of explanation, the DMVR according to the embodiments of this document can be implemented in the same manner in an encoding device.

[0079] 4, the decoding apparatus may derive prediction blocks (i.e., reference blocks) identified by initial motion vectors (or motion information) (e.g., MV0 and MV1) in the list0 and list1 directions, and generate a template (or bilateral template) by weighting (e.g., averaging) the derived prediction blocks (step 1). Here, the initial motion vectors (MV0 and MV1) may indicate motion vectors derived using motion information of neighboring blocks in merge / skip mode.

[0080] The decoding apparatus can then derive motion vectors (e.g., MV0′ and MV1′) that minimize the difference between the template and the sample region of the reference picture through a template matching operation (step 2). Here, the sample region indicates a neighboring region of the initial prediction block in the reference picture, and the sample region may be referred to as a neighboring region, reference region, search region, search range, search space, etc. The template matching operation may include calculating a cost measurement value between the template and the sample region of the reference picture. For example, the sum of absolute differences (SAD) may be used for the cost measurement. As an example, a normalized SAD may be used as the cost function. In this case, the matching cost may be given as SAD(T-mean(T), 2*P[x]-2*mean(P[x])), where T indicates the template and P[x] indicates a block within the search region. The motion vector that calculates the minimum template cost for each of the two reference pictures may be considered as an updated motion vector (replacing the initial motion vector). As shown in Figure 4, the decoding device can generate a final bi-directional prediction result (i.e., a final bi-directional prediction block) using updated motion vectors MV0' and MV1'. In one embodiment, multi-iteration for deriving updated (or new) motion vectors can be used to obtain the final bi-directional prediction result.

[0081] In one embodiment, the decoding device may invoke the DMVR process to improve the accuracy of initial motion compensation prediction (i.e., motion compensation prediction via conventional merge / skip mode). For example, the decoding device may perform the DMVR process when the prediction mode of the current block is merge mode or skip mode and bidirectional bi-prediction, in which reference pictures in both directions are in opposite directions relative to the current picture in display order, is applied to the current block.

[0082] FIG. 5 is a diagram illustrating an embodiment of a process for performing decoder-side motion vector refinement (DMVR) using sum of absolute differences (SAD).

[0083] As described above, a decoding apparatus can measure the matching cost using SAD when performing DMVR. As an example, FIG. 5 illustrates a method for refining a motion vector by calculating the Mean Sum of Absolute Difference (MRSAD) between prediction samples in two reference pictures without generating a template. That is, the method of FIG. 5 illustrates an example of bilateral matching using MRSAD.

[0084] 5, the decoding apparatus may derive neighboring pixels of a pixel (sample) indicated by a motion vector (MV0) in the list0 (L0) direction on the L0 reference picture, and neighboring pixels of a pixel (sample) indicated by a motion vector (MV1) in the list1 (L1) direction on the L1 reference picture. The decoding apparatus may then measure the matching cost by calculating the MRSAD between an L0 predicted block (i.e., an L0 reference block) identified by a motion vector indicating the neighboring pixels derived on the L0 reference picture and an L1 predicted block (i.e., an L1 reference block) identified by a motion vector indicating the neighboring pixels derived on the L1 reference picture. In this case, the decoding apparatus may select a search point with the minimum cost (i.e., a search area with the minimum SAD between the L0 predicted block and the L1 predicted block) as a refined motion vector pair. That is, the refined motion vector pair may include a refined L0 motion vector pointing to a pixel location (L0 prediction block) with the smallest cost in the L0 reference picture and a refined L1 motion vector pointing to a pixel location (L1 prediction block) with the smallest cost in the L1 reference picture.

[0085] As an example, in calculating the matching cost, after a search region of a reference picture is set, unidirectional prediction may be performed using a regular 8-tap DCTIF interpolation filter. Also, as an example, the MRSAD calculation uses 16-bit precision, and clipping and / or rounding operations are not applied before the MRSAD calculation in consideration of an internal buffer.

[0086] As described above, when true bi-prediction is applied to the current block, BDOF can be used to refine the bi-prediction signal. When bi-prediction is applied to the current block, BDOF (Bi-directional optical flow) can be used to calculate improved motion information and generate prediction samples based on the information. For example, BDOF can be applied at a 4x4 sub-block level. That is, BDOF can be performed in units of 4x4 sub-blocks within the current block. Alternatively, BDOF can be applied only to the luma component. Alternatively, BDOF can be applied only to the chroma component, or to the luma component and the chroma component.

[0087] As the name suggests, BDOF mode is based on the optical flow concept, which assumes that the motion of objects is smooth. For each 4x4 sub-block, motion refinement (v) is performed by minimizing the difference between the L0 and L1 predicted samples. x , v y ) can be calculated, and motion refinement can be used to adjust the bi-predictive sample values ​​in the 4x4 sub-blocks.

[0088] The above-mentioned DMVR and BDOF are techniques for refining motion information to perform prediction when true bi-prediction is applied (where true bi-prediction refers to a case where motion prediction / compensation is performed using a reference picture in a different direction based on the picture of the current block), and can be seen as refinement techniques with a similar concept in that they assume that the movement of an object within a picture occurs at a constant speed and in a constant direction. However, when true bi-prediction is performed, the conditions for applying DMVR and the conditions for applying BDOF are different, so a process of performing several condition checks must be performed repeatedly for each technique. Therefore, this document proposes a method that can improve efficiency in terms of decoder complexity and performance by improving the process of performing condition checks when determining the prediction mode to be applied to the current block.

[0089] The following Table 1 shows the conditions for applying DMVR in the existing true bi-prediction. DMVR can be applied when all of the following conditions are met.

[0090] [Table 1]

[0091] Referring to Table 1, 1) whether DMVR is applicable can be determined based on flag information (e.g., sps_dmvr_enabled_flag) signaled in the Sequence Parameter Set (SPS) syntax. Here, the flag information (e.g., sps_dmvr_enabled_flag) can indicate whether true bi-prediction-based DMVR is enabled. For example, if sps_dmvr_enabled_flag is 1 (i.e., true bi-prediction-based DMVR is enabled), it can be determined that the DMVR availability condition is met.

[0092] 2) Whether DMVR is applicable can be determined based on flag information (e.g., merge_flag) indicating whether inter prediction is performed using merge mode / skip mode. For example, if merge_flag is 1 (i.e., if inter prediction is performed using merge mode / skip mode), it can be determined that the merge mode / skip mode applicability condition is met.

[0093] 3) Whether DMVR is applicable can be determined based on flag information (e.g., mmvd_flag) indicating whether inter prediction is performed using MMVD (merge mode with motion vector difference) mode. For example, if mmvd_flag is 0 (i.e., if MMVD mode is not used), it can be determined that the MMVD mode application condition is met.

[0094] 4) Whether DMVR is applicable may be determined based on whether bidirectional prediction (bi-prediction) is used. Here, bidirectional prediction may refer to inter-prediction performed based on reference pictures located in different directions relative to the current picture. For example, if predFlagL0[0][0]=1 and predFlagL0[1][1]=1, it may be determined that bidirectional prediction is applied, and it may be determined that the bidirectional prediction availability condition is met.

[0095] 5) For true bi-prediction, it is possible to determine whether DMVR is applicable based on whether the distances between the current picture and the reference pictures in both directions are the same. That is, it is possible to determine whether the distance between the current picture and the LO reference picture (i.e., the reference picture in the reference picture list L0) and the distance between the current picture and the L1 reference picture (i.e., the reference picture in the reference picture list L1) are the same. For example, if DiffPicOrderCnt(currPic, RefPicList0[refIdxL0])-DiffPicOrderCnt(currPic, RefPicList1[refIdxL1])=0, it is determined that the current picture is true bi-prediction, that the distances between the current picture and the reference pictures in both directions are the same, and it is possible to determine that the condition for whether the reference picture distances in both directions are the same is met.

[0096] 6) Whether DMVR is applicable can be determined based on whether the height of the current block is greater than a threshold. For example, if the height of the current block is 8 or greater, it can be determined that the current block meets the size (length) condition.

[0097] 7) Whether or not DMVR is applicable can be determined based on whether the size of the current block is greater than a threshold. For example, if the size of the current block, i.e., length (height) * width (width), is 64 or greater, it can be determined that the current block size (length * width) condition is met.

[0098] The decoding device can determine whether to apply DMVR depending on whether conditions 1) to 7) in Table 1 are satisfied. That is, the decoding device can apply DMVR and perform true bi-prediction if all of conditions 1) to 7) in Table 1 are satisfied, and does not apply DMVR if any of the conditions in Table 1 are not satisfied.

[0099] The following Table 2 shows the conditions for applying BDOF in the existing true bi-prediction. BDOF can be applied when all the conditions listed below are met.

[0100] [Table 2]

[0101] Referring to Table 2, 1) whether BDOF is applicable can be determined based on flag information (e.g., sps_bdof_enabled_flag) signaled in the Sequence Parameter Set (SPS) syntax. Here, the flag information (e.g., sps_bdof_enabled_flag) can indicate whether true bi-prediction-based BDOF is enabled. For example, if sps_bdof_enabled_flag is 1 (i.e., true bi-prediction-based BDOF is enabled), it can be determined that the BDOF enablement condition is met.

[0102] 2) Whether BDOF is applicable may be determined based on whether bidirectional prediction is used. Here, bidirectional prediction may refer to inter-prediction performed based on reference pictures located in different directions relative to the current picture. For example, if predFlagL0 and predFlagL1 are both 1, it may be determined that bidirectional prediction is applied, and it may be determined that the bidirectional prediction availability condition is met.

[0103] 3) Whether BDOF is applicable can be determined based on whether the picture is true bi-predictive. That is, it can be determined whether the LO reference picture (i.e., a reference picture in the reference picture list L0) and the L1 reference picture (i.e., a reference picture in the reference picture list L1) are located in different temporal directions relative to the current picture. For example, if DiffPicOrderCnt(currPic, refPicList0[refIdxL0]) * DiffPicOrderCnt(currPic, refPicList1[refIdxL1]) is less than 0, it can be determined that the reference pictures in both directions are located in different temporal directions relative to the current picture, and that the true bi-predictability condition is met.

[0104] 4) Whether or not BDOF is applicable can be determined based on whether or not affine mode is used. Here, whether or not affine mode is used can be determined by deriving MotionModelIdc. For example, if the induced MotionModelIdc is 0, it can be determined that affine mode is not used, and in this case, it can be determined that the affine mode application condition is met.

[0105] 5) Whether BDOF is applicable can be determined based on flag information (e.g., merge_subblock_flag) indicating whether inter prediction in merge mode is performed on a sub-block basis. For example, if merge_subblock_flag is 0 (i.e., merge mode is not applied on a sub-block basis), it can be determined that the sub-block-based merge mode applicability condition is met.

[0106] 6) Whether or not BDOF is applicable can be determined based on whether GBi exists. Here, whether or not GBi exists can be determined based on GBi index information (e.g., GbiIdx). For example, if GbiIdx is 0 (i.e., GbiIdx is the default), it can be determined that the GBi applicability condition is met.

[0107] 7) Whether the current block is a luma block containing luma components can be used to determine whether the BDOF is applicable. For example, if the index (e.g., cIdx) indicating whether the current block is a luma block is 0 (i.e., if the current block is a luma block), it can be determined that the luma block applicability condition is met.

[0108] The decoding device can determine whether to apply BDOF depending on whether conditions 1) to 7) in Table 2 are met. That is, the decoding device can apply BDOF and perform true bi-prediction if all of conditions 1) to 7) in Table 2 are met, and does not apply BDOF if any of the conditions in Table 2 are not met.

[0109] As mentioned above, some of the application conditions for DMVR and BDOF are the same, while others are similar or different. Existing methods require separate condition checks for each technology even when the conditions are the same, increasing the complexity of bi-prediction execution. Therefore, this paper proposes efficient conditions for applying DMVR and BDOF during bi-prediction.

[0110] Since the merge / skip mode has relatively low motion accuracy compared to the AMVP mode, refining the motion information using the DMVR method is effective in terms of performance. However, unlike DMVR, the BDOF mode is applied not only in the merge / skip mode but also in the AMVP mode. Therefore, when BDOF is applied in the AMVP mode, the complexity of BDOF execution can increase compared to performance. Therefore, this embodiment proposes a method of applying BDOF in the merge / skip mode as well as in the DMVR mode.

[0111] In this case, as an example proposed in this document, the application conditions of BDOF may include the conditions presented in Table 3 below.

[0112] [Table 3]

[0113] Referring to Table 3, whether BDOF is applicable can be determined based on flag information (e.g., merge_flag) indicating whether inter prediction is performed using merge mode / skip mode. For example, if merge_flag is 1 (i.e., if inter prediction is performed using merge mode / skip mode), it can be determined that the merge mode / skip mode applicability condition is met. Therefore, like DMVR, BDOF can also be applied when the merge / skip mode is selected.

[0114] That is, in this embodiment, whether BDOF is applicable can be determined based on the conditions to be applied when the merge / skip mode is used, as well as the conditions to be applied when BDOF is available, the conditions to be applied when bidirectional prediction is used, the conditions to be applied when true bi-prediction is used, the conditions to be applied when affine prediction is not used, the conditions to be applied when sub-block based merge mode is not used, the conditions to be applied when the GBi index is default, and the conditions to be applied when the block is a luma block.

[0115] Therefore, the decoding device determines whether all the conditions listed in Table 3 are met, and if all the conditions are met, it can apply BDOF to perform true bi-prediction. If any one of the conditions listed in Table 3 is not met, the decoding device does not apply BDOF. The conditions in Table 3 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to the decoding device.

[0116] The method of performing true bi-prediction based on the conditions listed in Table 3 may be applied independently to DMVR and BDOF, or may be applied under the same conditions for DMVR and BDOF.

[0117] Meanwhile, if the conditions listed in this embodiment are the same as those described in Tables 1 and 2, the specific operations and meanings are the same, so a detailed description of each condition will be omitted. Also, duplicated content will be omitted in the following embodiments.

[0118] Encoding / decoding devices can be configured using various hardware, and preferences for performance versus complexity ratio may vary. Therefore, this embodiment proposes a method for refining motion information by applying DMVR not only in merge / skip mode but also in AMVP mode.

[0119] In this case, as one embodiment proposed in this document, the application conditions of the DMVR may include the conditions presented in Table 4 below.

[0120] [Table 4]

[0121] Referring to Table 4, the process of determining whether to apply DMVR based on flag information (e.g., merge_flag) indicating whether to perform inter prediction using merge mode / skip mode can be omitted. By omitting the condition for whether to apply merge mode / skip mode in this way, DMVR can be applied not only in merge mode / skip mode but also in AMVP mode.

[0122] According to Table 4, whether DMVR is applicable can be determined based on the conditions to be applied when DMVR is available, the conditions to be applied when MMVD mode is not used, the conditions to be applied in the case of bidirectional prediction, the conditions to be applied in the case of true bidirectional prediction where the distance between the current picture and the bidirectional reference picture is the same, the conditions to be applied when the length of the current block is 8 or more, and the conditions to be applied when the size of the current block (length * width) is 64 or more.

[0123] That is, the decoding device determines whether all the conditions listed in Table 4 are met, and if all the conditions are met, applies DMVR to perform true bi-prediction. If any one of the conditions listed in Table 4 is not met, the decoding device does not apply DMVR. The conditions in Table 4 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to the decoding device.

[0124] Based on the conditions listed in Table 4, the method of performing true bi-prediction may be applied independently to DMVR and BDOF, or may be applied under the same conditions for DMVR and BDOF.

[0125] Meanwhile, in one embodiment of this document, DMVR and BDOF can be applied to all normal merge modes. That is, DMVR and BDOF can be applied when the mode is not an advanced temporal motion vector prediction (ATMVP) mode, not an affine mode, or not a CPR mode. In this case, the application conditions for DMVR include the conditions shown in Table 5 below.

[0126] [Table 5]

[0127] Referring to Table 5, DMVR can be applied only in the case of general merge mode by determining whether the conditions to be applied when affine mode is not used (e.g., when MotionModelIdc is 0) and the conditions to be applied when sub-block-based merge mode is not used (e.g., when merge_subblock_flag is 0) are met.

[0128] In addition, in this embodiment, along with the affine mode availability condition and the sub-block based merge mode availability condition, the applicability of DMVR can be determined based on the conditions to be applied when DMVR is available, the conditions to be applied when merge mode / skip mode is used, the conditions to be applied when MMVD mode is not used, the conditions to be applied when bidirectional prediction is used, the conditions to be applied when true bidirectional prediction is used where the distances between the current picture and both reference pictures are the same, the conditions to be applied when the length of the current block is 8 or more, and the conditions to be applied when the size of the current block (length * width) is 64 or more.

[0129] That is, the decoding device determines whether all the conditions listed in Table 5 are met, and if all the conditions are met, applies DMVR to perform true bi-prediction. If any one of the conditions listed in Table 5 is not met, the decoding device does not apply DMVR. The conditions in Table 5 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to the decoding device.

[0130] Among the application conditions in Table 5, the sub-block-based merge mode application condition (e.g., merge_subblock_flag) includes overlapping conditions among the existing DMVR application conditions. Therefore, in one embodiment of this document, the overlapping conditions with the sub-block-based merge mode application condition (e.g., merge_subblock_flag) can be removed. In this case, the corresponding conditions can be removed as proposed in Table 6 below.

[0131] [Table 6]

[0132] Referring to Table 6, the sub-block based merge mode can be applied when the size of the current block is 8x8 or more. Therefore, since the sub-block based merge mode application condition (e.g., merge_subblock_flag=0) includes a condition related to the size of the current block, the conditions related to the size of the current block (e.g., CbHeight, CbHeight*CbWidth) can be excluded from the existing DMVR application conditions. For example, the conditions of whether the length of the current block is 8 or more and whether the length*width of the current block is 64 or more can be omitted, and the other conditions listed in Table 6 can be used to determine whether the DMVR is applicable.

[0133] That is, the decoding device determines whether all the conditions listed in Table 6 are met, and if all the conditions are met, applies DMVR to perform true bi-prediction. If any one of the conditions listed in Table 6 is not met, the decoding device does not apply DMVR. The conditions in Table 6 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to the decoding device.

[0134] The method of performing true bi-prediction based on the conditions listed in Table 5 or Table 6 can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0135] In one embodiment of this document, in order to increase the accuracy of motion vectors with low complexity, refinement techniques such as DMVR and BDOF are not applied when the block size is small. In existing methods, refinement techniques are applied when the current block is larger than or equal to 8x8. Among these, in the case of DMVR, when the size of the current block is large, refinement is applied in 16x16 units, so DMVR is not applied to blocks smaller than 16x16. In this case, the conditions for applying DMVR may include the conditions presented in Table 7 below.

[0136] [Table 7]

[0137] Referring to Table 7, by changing the conditions (e.g., CbHeight, CbWidth) related to the size of the current block, it is possible to prevent DMVR from being applied to blocks smaller than 16x16. For example, a condition that is applied when the length (e.g., CbHeight) of the current block is 16 or more and a condition that is applied when the width (e.g., CbWidth) of the current block is 16 or more can be used. DMVR is applied when the application conditions related to the size of the current block are met (i.e., when the size of the current block is 16x16 or more), and DMVR is not applied when the application conditions related to the size of the current block are not met (i.e., when the size of the current block is smaller than 16x16).

[0138] In addition, in this embodiment, whether or not DMVR is applicable can be determined based on the remaining application conditions listed in Table 7 together with the conditions related to the size of the current block (e.g., CbHeight, CbWidth).

[0139] That is, the decoding device determines whether all the conditions listed in Table 7 are met, and if all the conditions are met, applies DMVR to perform true bi-prediction. If any one of the conditions listed in Table 7 is not met, the decoding device does not apply DMVR. The conditions in Table 7 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to the decoding device.

[0140] In one embodiment of this document, if the current block is smaller than 16x16, not only DMVR but also BDOF is not applied. In this case, the conditions for applying BDOF may include the conditions presented in Table 8 below.

[0141] [Table 8]

[0142] Referring to Table 8, by changing the conditions (e.g., CbHeight, CbWidth) related to the size of the current block, BDOF may not be applied to blocks smaller than 16x16. For example, a condition may be used that applies when the length (e.g., CbHeight) of the current block is equal to or greater than 16, and a condition may be used that applies when the width (e.g., CbWidth) of the current block is equal to or greater than 16. BDOF is applied when the application conditions related to the size of the current block are met (i.e., when the size of the current block is equal to or greater than 16x16), and is not applied when the application conditions related to the size of the current block are not met (i.e., when the size of the current block is smaller than 16x16).

[0143] In addition, in this embodiment, whether BDOF is applicable can be determined based on the remaining application conditions listed in Table 8 together with the conditions related to the size of the current block (e.g., CbHeight, CbWidth).

[0144] That is, the decoding device determines whether all the conditions listed in Table 8 are met, and if all the conditions are met, it can apply BDOF to perform true bi-prediction. If any one of the conditions listed in Table 8 is not met, the decoding device does not apply BDOF. The conditions in Table 8 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to the decoding device.

[0145] The method of performing true bi-prediction based on the conditions listed in Table 7 or Table 8 can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0146] As described above, DMVR is applied when the distances between the current picture and the reference pictures in both directions are the same, whereas BDOF is always applied when true bi-prediction is in effect, even if the distances between the current picture and the reference pictures in both directions are different. Therefore, in order to improve coding efficiency, this document proposes a method for uniformly applying conditions related to the reference picture distances in both directions to DMVR and BDOF.

[0147] As an example of this document, the application conditions for BDOF may include the conditions presented in Table 9 below.

[0148] [Table 9]

[0149] Referring to Table 9, by changing a condition related to the reference picture distance (e.g., DiffPicOrderCnt) among the BDOF application conditions, the corresponding condition can be applied to both DMVR and BDOF. For example, by determining whether DiffPicOrderCnt(currPic, RefPicList0[refIdxL0])-DiffPicOrderCnt(currPic, RefPicList1[refIdxL1]) is 0, it can be determined whether the distance between the current picture and the LO reference picture (i.e., a reference picture in the reference picture list L0) and the distance between the current picture and the L1 reference picture (i.e., a reference picture in the reference picture list L1) are the same. That is, BDOF can be applied only when the distances between the current picture and both reference pictures are the same. In this way, by adding the condition that the current picture is true bi-predictive and the reference picture distances are the same, the BDOF application scope is limited, thereby reducing decoding complexity.

[0150] In addition, in this embodiment, whether BDOF is applicable can be determined based on the remaining application conditions listed in Table 9 together with the conditions related to the reference picture distance (e.g., DiffPicOrderCnt).

[0151] That is, the decoding device determines whether all the conditions listed in Table 9 are met, and if all the conditions are met, it can apply BDOF to perform true bi-prediction. If any one of the conditions listed in Table 9 is not met, the decoding device does not apply BDOF. The conditions in Table 9 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to the decoding device.

[0152] As an example of this document, the applicable conditions of the DMVR may include the conditions presented in Table 10 below.

[0153] [Table 10]

[0154] Referring to Table 10, by changing a condition related to the reference picture distance (e.g., DiffPicOrderCnt) among the DMVR application conditions, the corresponding condition can be applied to both DMVR and BDOF. For example, by determining whether DiffPicOrderCnt(currPic, refPicList0[refIdxL0]) * DiffPicOrderCnt(currPic, refPicList1[refIdxL1]) is less than 0, it can be determined whether true bi-prediction is used, in which both reference pictures (i.e., the LO reference picture and the L1 reference picture) are located in different directions relative to the current picture. That is, even if the distances between the current picture and both reference pictures are not the same, DMVR can always be applied in the case of true bi-prediction. In this way, even if the reference picture distances in both directions are different depending on whether true bi-prediction is used, the motion vector derived in consideration of decoding complexity can be used without scaling.

[0155] In addition, in this embodiment, whether or not DMVR is applicable can be determined based on the remaining application conditions listed in Table 10 together with the conditions related to the reference picture distance (e.g., DiffPicOrderCnt).

[0156] That is, the decoding device determines whether all the conditions listed in Table 10 are met, and if all the conditions are met, applies DMVR to perform true bi-prediction. If any one of the conditions listed in Table 10 is not met, the decoding device does not apply DMVR. The conditions in Table 10 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to the decoding device.

[0157] The method of performing true bi-prediction based on the conditions listed in Table 9 or Table 10 can be applied independently to DMVR and BDOF, or can be applied to the same conditions for DMVR and BDOF.

[0158] On the other hand, each reference block may need to perform motion compensation using a weighted sum due to changes in light, etc. In this case, this phenomenon can be grasped using GBi and LIC (local illumination compensation), so the application conditions for DMVR and BDOF can be determined taking into account the GBi and LIC conditions.

[0159] As an example of this document, we propose a method to determine whether DMVR is applicable by taking into account the GBi and LIC conditions. In this case, the conditions for DMVR application may include the conditions listed in Table 11 below.

[0160] [Table 11]

[0161] Referring to Table 11, whether DMVR is applicable can be determined by adding a GBi condition (e.g., GbiIdx) and a LIC condition (e.g., LICFlag). Here, GBi may indicate generalized bi-prediction, in which different weights can be applied to L0 prediction and L1 prediction, and may be indicated using, for example, GbiIdx. GbiIdx may be present in the case of bi-prediction and may indicate a bi-prediction weight index. For example, GbiIdx may be derived from a neighboring block in merge mode, or may be signaled from the encoding device to the decoding device via a GbiIdx syntax element (e.g., gbi_idx) in MVP mode. For example, GbiIdx may indicate a weight w applied to L1 prediction, in which case a weight of (1-w) may be applied to L0 prediction. As another example, GbiIdx may indicate a weighting value w applied to L0 prediction, in which case a weighting value of (1-w) may be applied to L1 prediction.

[0162] According to Table 11, if GbiIdx is 0 (i.e., if GbiIdx is the default), it is determined that the GBi availability condition is met, and if LICFlag is 0 (i.e., if LIC exists), it is determined that the LIC availability condition is met.

[0163] In addition, in this embodiment, whether or not DMVR is applicable can be determined based on the remaining application conditions listed in Table 11, along with the GBi condition (eg, GbiIdx) and the LIC condition (eg, LICFlag).

[0164] That is, the decoding device determines whether all the conditions listed in Table 11 are met, and if all the conditions are met, applies DMVR to perform true bi-prediction. If any one of the conditions listed in Table 11 is not met, the decoding device does not apply DMVR. The conditions in Table 11 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to the decoding device.

[0165] As an example of this document, we propose a method to determine whether BDOF is applicable by taking into account the GBi and LIC conditions. In this case, the conditions for applying BDOF may include the conditions shown in Table 12 below.

[0166] [Table 12]

[0167] According to Table 12, the application of BDOF can be determined by adding a LIC condition (e.g., LICFlag) to the existing GBi condition (e.g., GbiIdx). For example, if GbiIdx is 0 (i.e., GbiIdx is default), it can be determined that the GBi condition is met, and if LICFlag is 0 (i.e., LIC exists), it can be determined that the LIC condition is met.

[0168] Therefore, in this embodiment, whether or not BDOF is applicable can be determined based on the remaining application conditions listed in Table 12, along with the GBi condition (eg, GbiIdx) and the LIC condition (eg, LICFlag).

[0169] That is, the decoding device determines whether all the conditions listed in Table 12 are met, and if all the conditions are met, it can apply BDOF to perform true bi-prediction. If any one of the conditions listed in Table 12 is not met, the decoding device does not apply BDOF. The conditions in Table 12 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to the decoding device.

[0170] The method of performing true bi-prediction based on the conditions listed in Table 11 or Table 12 can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0171] On the other hand, DMVR and BDOF induce motion information through a refinement process in the decoding device, which increases decoding complexity. Therefore, this paper proposes a method that can reduce decoding complexity by determining whether to apply DMVR and BDOF using a merge index. In this case, DMVR and BDOF refine motion vectors within a limited range, so if the motion vector is inaccurate, the refinement effect may be reduced. Therefore, this paper proposes a method that can be applied only when the merge index value is small, taking into account the efficiency of refinement.

[0172] Here, the merge index is a syntax element signaled from the encoding device to the decoding device. For example, when a merge mode / skip mode is applied to a current block, the encoding / decoding device may construct a merge candidate list based on neighboring blocks of the current block. In this case, the encoding device may select an optimal merge candidate from among the merge candidates included in the merge candidate list based on the rate-distortion (RD) cost, and signal merge index information indicating the selected merge candidate to the decoding device. The decoding device may select a merge candidate to be applied to the current block based on the merge candidate list and the merge index information.

[0173] As an example of this document, a method for determining whether DMVR is applicable using a merge index may include the conditions presented in Table 13 below.

[0174] [Table 13]

[0175] Referring to Table 13, a merge index condition (e.g., merge_idx) can be added to determine whether DMVR is applicable. For example, if the merge index (e.g., merge_idx) is less than 2, it can be determined that the merge index condition is met. Here, the merge index value (threshold) is set to 2, but this is only an example, and the corresponding value can be changed depending on coding efficiency.

[0176] Therefore, in this embodiment, whether or not DMVR is applicable can be determined based on the merge index condition (e.g., merge_idx) and the remaining application conditions listed in Table 13.

[0177] That is, the decoding device determines whether all the conditions listed in Table 13 are met, and if all the conditions are met, applies DMVR to perform true bi-prediction. If any one of the conditions listed in Table 13 is not met, the decoding device does not apply DMVR. The conditions in Table 13 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to the decoding device.

[0178] As an example of this document, a method for determining whether BDOF is applicable using a merge index may include the conditions presented in Table 14 below.

[0179] [Table 14]

[0180] Referring to Table 14, whether BDOF is applicable can be determined by adding a merge mode / skip mode condition (e.g., merge_flag) and a merge index condition (e.g., merge_idx). For example, if merge_flag is 1 (i.e., inter prediction is performed using merge mode / skip mode) and merge_idx is not 2 or greater, it can be determined that the condition for restrictively applying BDOF is met only when the merge index value is small. That is, if merge_flag is 1 (i.e., inter prediction is performed using merge mode / skip mode) and merge_idx is less than 2, it can be determined that the merge index condition is met and BDOF can be applied. Here, the merge index value (threshold) is set to 2, but this is merely an example, and the corresponding value can be changed depending on coding efficiency.

[0181] That is, in this embodiment, whether BDOF is applicable can be determined based on the merge mode / skip mode availability condition (e.g., merge_flag), merge index condition (e.g., merge_idx), and the remaining application conditions listed in Table 14.

[0182] That is, the decoding device determines whether all the conditions listed in Table 14 are met, and if all the conditions are met, it can apply BDOF to perform true bi-prediction. If any one of the conditions listed in Table 14 is not met, the decoding device does not apply BDOF. The conditions in Table 14 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to the decoding device.

[0183] The method of performing true bi-prediction based on the conditions listed in Table 13 or Table 14 can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0184] Meanwhile, in merge / skip mode, motion information can be refined via MMVD, and in this case, DMVR is not performed when MMVD is applied because decoder complexity increases. However, when applied without considering MMVD, DMVR can be applied even without MMVD conditions to improve performance. In this case, according to one embodiment of this document, DMVR application conditions can include the conditions presented in Table 15 below.

[0185] [Table 15]

[0186] Referring to Table 15, the MMVD mode application condition (e.g., mmvd_flag) can be excluded from the existing DMVR application conditions. That is, the process of determining whether mmvd_flag is 0 (i.e., when the MMVD mode is not used) can be omitted, and the application of DMVR can be determined based on the conditions listed in Table 15.

[0187] That is, the decoding device determines whether all the conditions listed in Table 15 (excluding the MMVD mode application condition (e.g., mmvd_flag)) are met, and if all the conditions are met, applies DMVR to perform true bi-prediction. If any one of the conditions listed in Table 15 is not met, the decoding device does not apply DMVR. The conditions in Table 15 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to the decoding device.

[0188] Also, contrary to the embodiment of Table 15, when refining motion information through MMVD in merge / skip mode, it is possible to determine whether to perform BDOF taking into account decoder complexity. Therefore, this document proposes a method for determining whether to apply BDOF taking into account MMVD conditions. In this case, according to one embodiment of this document, the conditions for applying BDOF may include the conditions presented in Table 16 below.

[0189] [Table 16]

[0190] Referring to Table 16, by adding an MMVD mode applicability condition (e.g., mmvd_flag), BDOF is not applied when refining motion information through MMVD. For example, if mmvd_flag is 0 (i.e., if MMVD mode is not used), it can be determined that the MMVD mode applicability condition is met.

[0191] Therefore, in this embodiment, whether or not BDOF is applicable can be determined based on the remaining application conditions listed in Table 16 together with the MMVD mode application condition (e.g., mmvd_flag).

[0192] That is, the decoding device determines whether all the conditions listed in Table 16 are met, and if all the conditions are met, it can apply BDOF to perform true bi-prediction. If any one of the conditions listed in Table 16 is not met, the decoding device does not apply BDOF. The conditions in Table 16 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to the decoding device.

[0193] The method of performing true bi-prediction based on the conditions listed in Table 15 or Table 16 can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0194] Meanwhile, in the AMVP mode, the Adaptive Motion Vector Resolution (AMVR) technique can be applied. In this case, if the motion vector resolution is high, i.e., if it has integer-pel rounding or 4-pel rounding, it is not suitable to apply this technique to BDOF, which performs refinement within a limited area. Therefore, this document proposes a method for determining whether to perform BDOF according to the AMVR condition. In this case, according to one embodiment of this document, the BDOF application conditions may include the conditions shown in Table 17 below.

[0195] [Table 17]

[0196] Referring to Table 17, an AMVR condition (e.g., amvr_flag) can be added to determine whether BDOF is applied. Here, amvr_flag is information indicating the resolution of the motion vector difference (MVD). For example, if amvr_flag is 0, it indicates that the MVD resolution is induced in quarter-luma-sample units, and if amvr_flag is not 0, it indicates that the MVD resolution is induced in integer-luma-sample units or four-luma-sample units. Alternatively, the opposite can be determined. Depending on the embodiment, as shown in Table 17, a condition can be set such that BDOF is applied when amvr_flag is not 0. That is, if amvr_flag is 0, BDOF can be restricted from being applied.

[0197] Therefore, in this embodiment, whether or not BDOF is applicable can be determined based on the AMVR condition (e.g., amvr_flag) as well as other conditions listed in Table 17.

[0198] That is, the decoding device determines whether all the conditions listed in Table 17 are met, and if all the conditions are met, it can apply BDOF to perform true bi-prediction. If any one of the conditions listed in Table 17 is not met, the decoding device does not apply BDOF. The conditions in Table 17 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to the decoding device.

[0199] Meanwhile, the DMVR and BDOF as described above can be signaled using the SPS (Sequence Parameter Set) syntax. Table 18 below shows an example of a syntax element indicating whether the DMVR and BDOF are available, which are signaled via the SPS syntax.

[0200] [Table 18]

[0201] Referring to Table 18, sps_dmvr_enabled_flag can be signaled in the SPS syntax, and whether a true bi-prediction-based DMVR is enabled can be indicated based on this syntax element. For example, if sps_dmvr_enabled_flag is 1, it indicates that a true bi-prediction-based DMVR is enabled, and if sps_dmvr_enabled_flag is 0, it indicates that a true bi-prediction-based DMVR is not enabled.

[0202] In addition, sps_bdof_enabled_flag can be signaled in the SPS syntax, and whether true bi-prediction-based BDOF is enabled can be indicated based on this syntax element. For example, if sps_bdof_enabled_flag is 1, it indicates that true bi-prediction-based BDOF is enabled, and if sps_bdof_enabled_flag is 0, it indicates that true bi-prediction-based BDOF is not enabled.

[0203] As shown in Table 18, the application conditions of DMVR and BDOF can be checked using a syntax element indicating whether DMVR is available (e.g., sps_dmvr_enabled_flag) and a syntax element indicating whether BDOF is available (e.g., sps_bdof_enabled_flag).

[0204] FIG. 6 is an example showing a method for checking the application conditions of DMVR and BDOF and performing a decoding process.

[0205] The method of Figure 6 can be applied when using a syntax element indicating whether DMVR is available (e.g., sps_dmvr_enabled_flag) and a syntax element indicating whether BDOF is available (e.g., sps_bdof_enabled_flag), as shown in Table 18.

[0206] Referring to FIG. 6, the decoding apparatus can derive motion information (eg, motion vector, reference picture index, etc.) for a current block (S600).

[0207] The decoding device may check the application conditions of the DMVR (S610). At this time, the decoding device may check the application conditions of the DMVR based on a syntax element (e.g., sps_dmvr_enabled_flag) indicating whether the DMVR is available. For example, if the DMVR is available (e.g., if sps_dmvr_enabled_flag is 1), the decoding device may check the application conditions of the DMVR.

[0208] The decoding device can determine whether to apply the DMVR process depending on whether the DMVR application conditions are met (S620).

[0209] If all the conditions for applying DMVR are met, the decoding apparatus may derive refined motion information by applying the DMVR process (S630). If one or more of the conditions for applying DMVR are not met, the decoding apparatus does not apply the DMVR process.

[0210] The decoding device can derive a predicted sample of the current block based on refined motion information derived when DMVR is applied or (unrefined) motion information derived when DMVR is not applied (S640).

[0211] Then, the decoding apparatus may check the application conditions of BDOF (S650). At this time, the decoding apparatus may check the application conditions of BDOF based on a syntax element (e.g., sps_bdof_enabled_flag) indicating whether BDOF is available. For example, if BDOF is available (e.g., if sps_bdof_enabled_flag is 1), the decoding apparatus may check the application conditions of BDOF.

[0212] If all the conditions for applying BDOF are met, the decoding apparatus may apply the BDOF process to refine the predicted samples (S670). If one or more of the conditions for applying BDOF are not met, the decoding apparatus does not apply the BDOF process.

[0213] The decoding device derives a residual sample for the current block (S680) and can derive a reconstructed sample based on the residual sample and a refined predicted sample derived when the BDOF is applied or a (non-refined) predicted sample derived when the BDOF is not applied (S690).

[0214] As mentioned above, this document proposes various embodiments that can improve coding efficiency and reduce complexity by harmonizing the application conditions between DMVR and BDOF when applying them. When checking the application conditions of DMVR and BDOF according to the embodiments of this document and applying them to the decoding process, each condition can be checked and applied separately, but the application conditions can also be checked at the same time to improve coding efficiency. That is, this document proposes a method that can check the application conditions of DMVR and BDOF together in one integrated manner.

[0215] In one embodiment of this document, the DMVR / BDOF application condition check process can be performed by signaling information indicating whether refinement is applied in a decoding device (e.g., sps_refinement_enabled_flag) in the SPS (Sequence Parameter Set) syntax. Table 19 below shows an example of a syntax element (e.g., sps_refinement_enabled_flag) indicating whether refinement is applied in a decoding device, which is signaled via the SPS syntax.

[0216] [Table 19]

[0217] Referring to Table 19, the SPS syntax can signal sps_refinement_enabled_flag, and whether or not a decoding device can apply refinement can be indicated based on this syntax element. For example, when sps_refinement_enabled_flag exists (i.e., sps_refinement_enabled_flag is true), it can be determined that a decoding device can apply refinement. In this case, the decoding device can obtain the sps_dmvr_enabled_flag syntax element indicating whether DMVR is enabled and the sps_bdof_enabled_flag syntax element indicating whether BDOF is enabled, and determine the application conditions of DMVR and BDOF.

[0218] 7 and 8 show another example of a method for checking the application conditions of DMVR and BDOF and performing a decoding process.

[0219] The methods of Figures 7 and 8 can be applied when using a syntax element (e.g., sps_refinement_enabled_flag) indicating whether refinement is applicable in the decoding device as shown in Table 19. Also, in Figures 7 and 8, descriptions of the same content as in Figure 6 will be omitted.

[0220] 7, compared to the process of FIG. 6, it can be seen that the application conditions of DMVR and BDOF are checked at once in a previous step, rather than being checked separately. As an example, in step S710 of FIG. 7, the decoding apparatus may check the refinement application conditions. At this time, the refinement application condition check may be performed based on sps_refinement_enabled_flag as shown in Table 19. For example, if sps_refinement_enabled_flag is 1, the decoding apparatus may obtain the sps_dmvr_enabled_flag syntax element indicating whether DMVR is enabled or the sps_bdof_enabled_flag syntax element indicating whether BDOF is enabled, and check the application conditions of DMVR and BDOF based on the obtained syntax element.

[0221] Also, referring to Figure 8, compared to the process of Figure 6, the DMVR and BDOF application conditions are not checked individually, but are checked at the same time in the previous step, and a simple check process (BDOF application conditions) can be performed for other conditions. As an example, in step S810 of Figure 8, the decoding device can check the refinement application conditions. Thereafter, in step S850 of Figure 8, the decoding device can perform an additional simple check for BDOF application conditions that have conditions different from the refinement application conditions.

[0222] On the other hand, if the length (height) or size (height * width) of the current block is smaller than a specific length or size, the calculation ratio of multiplication / addition for motion compensation increases. Therefore, in one embodiment of this document, to reduce the worst case, the application of BDOF can be limited to blocks with small lengths or sizes, as in the application conditions of DMVR. In this case, the application conditions of BDOF can include the conditions presented in Table 20 below.

[0223] [Table 20]

[0224] Referring to Table 20, by adding a condition (e.g., CbHeight, CbWidth) related to the size of the current block, BDOF can be prevented from being applied to blocks smaller than a certain size. For example, a condition that applies when the length (e.g., CbHeight) of the current block is 8 or greater and a condition that applies when the size (e.g., CbHeight * CbWidth) of the current block is 64 or greater can be used. BDOF is applied when such an application condition related to the size of the current block is met (i.e., when the length of the current block is 8 or greater and the length * width of the current block is 64 or greater), and is not applied when such an application condition related to the size of the current block is not met.

[0225] In addition, in this embodiment, whether BDOF is applicable can be determined based on the conditions related to the size of the current block (e.g., CbHeight, CbHeight*CbWidth) as well as other application conditions listed in Table 20.

[0226] That is, the decoding device determines whether all the conditions listed in Table 20 are met, and if all the conditions are met, it can apply BDOF to perform true bi-prediction. If any one of the conditions listed in Table 20 is not met, the decoding device does not apply BDOF. The conditions in Table 20 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to the decoding device.

[0227] Meanwhile, when MMVD is applied to the current block, motion information can be derived based on information about the MMVD (e.g., mmvd index). Here, the information about the MMVD can include a base MV index, a distance index, a direction index, etc. In particular, the distance index (more specifically, mmvd_distance_index[xCb][yCb]) is used to indicate the distance from the base MV. For example, distance indexes 0 to 7 can be expressed as {1 / 4, 1 / 2, 1, 2, 4, 8, 16, 32}, etc., respectively. When determining whether to refine motion information in DMVR and BDOF, neighboring pixels (neighboring samples) are taken into consideration. In this case, if the distance between the neighboring pixels and the base MV is large, the value of the distance index also becomes large. In this case, considering neighboring pixels does not facilitate performance improvement of DMVR and BDOF. Therefore, this document proposes a method for determining whether DMVR and BDOF are applicable based on the distance index (more specifically, mmvd_distance_index[xCb][yCb]) value.

[0228] In one embodiment of this document, the applicability of DMVR can be determined by taking into account the distance index. In this case, the conditions for applying DMVR can include the conditions presented in Table 21 below.

[0229] [Table 21]

[0230] Referring to Table 21, by changing the MMVD-related conditions (e.g., mmvd_flag) among the DMVR application conditions, DMVR can be applied restrictively in the MMVD mode. For example, if mmvd_flag is 1 and mmvd_distance_index is greater than 4, it can be determined that the MMVD distance index condition is met. Therefore, when the MMVD mode is applied, whether DMVR is applied can be determined based on the distance index (e.g., mmvd_distance_index[xCb][yCb]) value.

[0231] Here, the threshold for mmvd_distance_index is set to 4, but this is only an example and can be changed to various values ​​depending on performance and coding efficiency.

[0232] Therefore, in this embodiment, whether DMVR is applicable can be determined based on the other application conditions listed in Table 21, in addition to the MMVD application condition (e.g., mmvd_flag) and the MMVD distance index condition (e.g., mmvd_distance_index).

[0233] That is, the decoding device determines whether all the conditions listed in Table 21 are met, and if all the conditions are met, applies DMVR to perform true bi-prediction. If any one of the conditions listed in Table 21 is not met, the decoding device does not apply DMVR. The conditions in Table 21 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to the decoding device.

[0234] In addition, as an embodiment of this document, whether or not BDOF is applicable can be determined taking into account the distance index. In this case, the conditions for applying BDOF can include the conditions presented in Table 22 below.

[0235] [Table 22]

[0236] Referring to Table 22, by adding a merge mode / skip mode enable / disable condition (e.g., merge_flag) and a condition related to MMVD (e.g., mmvd_flag, mmvd_distance_index), BDOF can be applied in a limited manner in MMVD mode. For example, if merge_flag does not exist (i.e., merge_flag is not 1), or if merge_flag is 1, mmvd_flag is 1, and mmvd_distance_index is greater than 4, it can be determined that the condition for applying BDOF in a limited manner in MMVD mode is met. Therefore, when MMVD mode is applied, whether BDOF is applied can be determined based on the distance index (e.g., mmvd_distance_index[xCb][yCb]) value.

[0237] Here, the threshold for mmvd_distance_index is set to 4, but this is only an example and can be changed to various values ​​depending on performance and coding efficiency.

[0238] Therefore, in this embodiment, whether BDOF is applicable can be determined based on the merge mode / skip mode availability conditions (e.g., merge_flag), conditions related to MMVD (e.g., mmvd_flag, mmvd_distance_index), as well as other application conditions listed in Table 22.

[0239] That is, the decoding device determines whether all the conditions listed in Table 22 are met, and if all the conditions are met, applies BDOF to perform true bi-prediction. If any one of the conditions listed in Table 22 is not met, the decoding device does not apply BDOF. The conditions in Table 22 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to the decoding device.

[0240] Based on the conditions listed in Table 21 or Table 22, the method of performing true bi-prediction can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0241] Meanwhile, as mentioned above, a combined intra-inter prediction (CIIP) mode, which simultaneously performs intra prediction and inter prediction, can be applied to a current block. In this case, a predicted block (inter block) on which inter prediction is performed is combined with an intra prediction method to generate a final predicted sample value, thereby improving prediction accuracy. However, since DMVR and BDOF are techniques for refining inter blocks, the application of the CIIP mode is not necessary in terms of complexity versus performance. Therefore, this document proposes a method for determining whether to apply DMVR and BDOF while taking CIIP into consideration.

[0242] As an example of this document, the applicability of DMVR can be determined taking into account CIIP. In this case, the conditions for applying DMVR may include the conditions presented in Table 23 below.

[0243] [Table 23]

[0244] Referring to Table 23, by adding a CIIP mode applicability condition (e.g., ciip_flag), the DMVR can be applied restrictively depending on whether the CIIP mode is applicable. For example, if ciip_flag is 0 (i.e., the CIIP mode is not applicable), it is determined that the CIIP mode applicability condition is met, and the DMVR can be applied.

[0245] Therefore, in this embodiment, it is possible to determine whether or not the DMVR is applicable based on the other application conditions listed in Table 23 in addition to the CIIP mode application condition (e.g., ciip_flag).

[0246] That is, the decoding device determines whether all the conditions listed in Table 23 are met, and if all the conditions are met, applies DMVR to perform true bi-prediction. If any one of the conditions listed in Table 23 is not met, the decoding device does not apply DMVR. The conditions in Table 23 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to the decoding device.

[0247] In addition, as an embodiment of this document, the applicability of BDOF can be determined taking into account CIIP. In this case, the conditions for applying BDOF may include the conditions presented in Table 24 below.

[0248] [Table 24]

[0249] Referring to Table 24, by adding a CIIP mode applicability condition (e.g., ciip_flag), BDOF can be applied restrictively depending on whether CIIP is applicable. For example, if ciip_flag is 0 (i.e., CIIP mode is not applicable), it is determined that the CIIP mode applicability condition is met, and BDOF can be applied.

[0250] Therefore, in this embodiment, whether or not BDOF is applicable can be determined based on the other application conditions listed in Table 24, along with the CIIP mode application condition (e.g., ciip_flag).

[0251] That is, the decoding device determines whether all the conditions listed in Table 24 are met, and if all the conditions are met, it can apply BDOF to perform true bi-prediction. If any one of the conditions listed in Table 24 is not met, the decoding device does not apply BDOF. The conditions in Table 24 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to the decoding device.

[0252] The method of performing true bi-prediction based on the conditions listed in Table 23 or Table 24 can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0253] The methods listed in Tables 3 to 24 according to the embodiments of the present document can be applied in combination. That is, whether DMVR and BDOF are applicable can be determined by checking the refinement application conditions, and the conditions shown in Table 25 below can be applied.

[0254] [Table 25]

[0255] Referring to Table 25, when DMVR or BDOF is applied, the following conditions can be used: a condition to be applied in the case of bidirectional prediction, a condition to be applied in the case of true bidirectional prediction where the distances between the current picture and the bidirectional reference picture are the same, a condition to be applied when the MMVD mode is not used, a condition to be applied when affine prediction is not used, a condition to be applied when the sub-block based merge mode is not used, and a condition to be applied when the GBi index is default. That is, whether DMVR or BDOF is applied can be determined depending on whether the conditions are met.

[0256] Also, as described above, conditions may be added such as determining whether it is a merge mode for DMVR or whether it is a luma block for BDOF.

[0257] The application conditions listed in Table 25 are merely examples, and it is obvious that various conditions listed in the above-described embodiments (the embodiments of Tables 3 to 24) can be used in combination.

[0258] FIG. 9 is a flow chart that schematically illustrates an encoding method that can be performed by an encoding device according to an embodiment of the present document.

[0259] The method disclosed in FIG. 9 may be performed by the encoding apparatus 200 disclosed in FIG. 2. Specifically, steps S900 to S930 of FIG. 9 may be performed by the prediction unit 220 and the inter prediction unit 221 disclosed in FIG. 2, step S940 of FIG. 9 may be performed by the residual processing unit 230 disclosed in FIG. 2, and step S950 of FIG. 9 may be performed by the entropy encoding unit 240 disclosed in FIG. 2. In addition, the method disclosed in FIG. 9 may include the embodiments described in detail herein. Therefore, detailed descriptions of content overlapping with the above-described embodiments will be omitted or simplified in FIG. 9.

[0260] Referring to FIG. 9, the encoding apparatus may determine whether a DMVR application condition for applying motion vector refinement to a current block is met (S900).

[0261] That is, the encoding apparatus may determine whether to apply motion vector refinement in consideration of coding efficiency, complexity, prediction performance, etc. In this case, the encoding apparatus may determine whether to apply refinement to the current block using various application conditions described in Tables 1 to 25.

[0262] In one embodiment, the encoding apparatus may use whether a prediction mode combining inter prediction and intra prediction is applied to the current block as a DMVR application condition. That is, the encoding apparatus may determine that the DMVR application condition is met when a prediction mode combining inter prediction and intra prediction is not applied to the current block.

[0263] Here, whether a prediction mode in which inter prediction and intra prediction are combined is applied to the current block may be indicated through CIIP (combined inter and intra prediction) flag information (e.g., ciip_flag). For example, if ciip_flag is 0, it may indicate that the CIIP mode (i.e., a prediction mode in which inter prediction and intra prediction are combined) is not applied to the current block, and if ciip_flag is 1, it may indicate that the CIIP mode is applied to the current block. Alternatively, the opposite may be determined.

[0264] The encoding device can determine the value of ciip_flag depending on whether the CIIP mode is applied to the current block, encode this information, and signal it to the decoding device.

[0265] Furthermore, according to an embodiment, the encoding apparatus may further include the following conditions as the application conditions of the DMVR.

[0266] 1) If DMVR-based inter bi-prediction is enabled, for example, if sps_dmvr_enabled_flag is 1

[0267] 2) If merge mode is applied to the current block, e.g., merge_flag is 1

[0268] 3) If MMVD (merge mode with motion vector difference) mode is not applied to the current block, e.g., mmvd_flag is 0

[0269] 4) When bi-prediction based on the L0 reference picture and the L1 reference picture is applied to the current block, for example, when predFlagL0[0][0]=1 and predFlagL0[1][1]=1.

[0270] 5) When the distance from the current picture to the L0 reference picture and the distance from the L1 reference picture are the same, for example, when DiffPicOrderCnt(currPic, RefPicList0[refIdxL0]) - DiffPicOrderCnt(currPic, RefPicList1[refIdxL1]) is 0.

[0271] 6) When the value of the bi-prediction weight index information of the current block is 0, for example, when GbiIdx is 0

[0272] 7) If the length (Height) of the current block is 8 or more, e.g., if CbHeight is 8 or more

[0273] 8) If the width of the current block is 8 or more, e.g., if CbWidth is 8 or more

[0274] 9) If the current block's length (Height) x width (Width) is greater than 8 x 8, for example, if CbHeight * CbWidth is greater than 64

[0275] An encoding device can be determined to satisfy the conditions for applying DMVR when it satisfies at least one or all of 1) to 9).

[0276] Here, when determining whether the DMVR application conditions are met, the listed DMVR application conditions are merely examples, and various combinations of the conditions in Tables 1 to 25 can be used.

[0277] If the encoding apparatus determines that the CIIP mode application conditions or the DMVR application conditions 1) to 9) above are satisfied, the encoding apparatus may derive a minimum SAD (Sum of Absolute Differences) based on the L0 motion vector and the L1 motion vector of the current block (S910).The encoding apparatus may derive a refined L0 motion vector and a refined L1 motion vector for the current block based on the minimum SAD (S920).

[0278] Here, the L0 motion vector and the L1 motion vector are motion vectors of the L0 prediction direction and the L1 prediction direction derived by applying a merge mode / skip mode to the current block. For example, the encoding apparatus may determine whether the skip mode or the merge mode is applied to the current block. If the skip mode or the merge mode is applied to the current block, the encoding apparatus may construct a merge candidate list based on neighboring blocks of the current block. In this case, a reference block whose difference with the current block is minimum or equal to or less than a certain criterion may be derived from among reference blocks indicated by merge candidates included in the merge candidate list. In this case, a merge candidate associated with the derived reference block may be selected, and merge index information indicating the selected merge candidate may be generated and signaled to the decoding apparatus. In this case, motion information (motion vector, reference picture index, etc.) of the selected merge candidate may be derived from motion information of the current block (motion vector, reference picture index, etc.). That is, the L0 motion vector and the L1 motion vector of the current block may be derived based on the motion vector of a neighboring block of the current block indicated by a merge index.

[0279] Furthermore, for example, when a merge mode is applied to a current block, the encoding apparatus may determine whether bi-prediction is applied to a merge candidate (i.e., a neighboring block) indicated by a merge index. When bi-prediction is applied to a neighboring block indicated by a merge index, the encoding apparatus may derive motion information including a motion vector of an L0 prediction direction and a motion vector of an L1 prediction direction of the neighboring block indicated by the merge index.

[0280] As an example, the encoding apparatus may calculate a sample area having a minimum SAD between a prediction sample in an L0 reference picture derived based on an L0 motion vector (i.e., an L0 prediction sample) and a prediction sample in an L1 reference picture derived based on an L1 motion vector (i.e., an L1 prediction sample). Then, the encoding apparatus may perform DMVR-based refinement on the L0 motion vector and the L1 motion vector based on the sample area having the minimum SAD. The process of calculating the minimum SAD and performing refinement has been described with reference to FIGS. 4 and 5, and therefore will not be described in detail here.

[0281] In this case, when performing DMVR-based refinement, if the distance between the L0 reference picture and the current picture is the same as the distance between the L1 reference picture and the current picture, the encoding device can perform refinement to derive a refined L0 motion vector and a refined L1 motion vector.

[0282] The encoding apparatus may derive a prediction sample for the current block based on the refined L0 motion vector and the refined L1 motion vector (S930).

[0283] According to an embodiment, the encoding apparatus may determine whether a BDOF application condition for applying refinement to a predicted sample of a current block is met. That is, the encoding apparatus may apply refinement to a predicted signal in consideration of coding efficiency, complexity, prediction performance, etc. In this case, the encoding apparatus may determine whether to apply refinement to a predicted sample of a current block using various application conditions described in Tables 1 to 25.

[0284] For example, the encoding device may use whether a prediction mode combining inter prediction and intra prediction is applied to the current block as a condition for applying BDOF. As described above, if the value of the CIIP flag information (e.g., ciip_flag) is 0, the encoding device may determine that the condition for applying BDOF is met.

[0285] Furthermore, according to an embodiment, the encoding apparatus may further include the following conditions as the conditions for applying BDOF.

[0286] 1) If BDOF-based inter-prediction is enabled, e.g., if sps_bdof_enabled_flag is 1

[0287] 2) If the subblock-based merge mode is not applied to the current block, e.g., if merge_subblock_flag is 0

[0288] 3) When bi-prediction based on the L0 reference picture and the L1 reference picture is applied to the current block, for example, when predFlagL0=1 and predFlagL1=1.

[0289] 4) When the distance between the current picture and the L0 reference picture and the distance between the current picture and the L1 reference picture are the same, for example, when DiffPicOrderCnt(currPic, RefPicList0[refIdxL0]) - DiffPicOrderCnt(currPic, RefPicList1[refIdxL1]) is 0.

[0290] 5) When the value of the bi-prediction weight index information of the current block is 0, for example, when GbiIdx is 0

[0291] 6) If no affine mode is applied to the current block, e.g., MotionModelIdc is 0

[0292] 7) If the length (Height) of the current block is 8 or more, e.g., if CbHeight is 8 or more

[0293] 8) If the width of the current block is 8 or more, e.g., if CbWidth is 8 or more

[0294] 9) If the current block's length (Height) x width (Width) is greater than 8 x 8, for example, if CbHeight * CbWidth is greater than 64

[0295] An encoding device can be determined to satisfy the conditions for applying BDOF if it satisfies at least one or all of 1) to 9).

[0296] Here, in determining whether the BDOF application conditions are met, the listed BDOF application conditions are merely examples, and various combinations of the conditions in Tables 1 to 25 can be used.

[0297] When the encoding device determines that the CIIP mode application conditions or the BDOF application conditions 1) to 9) are satisfied, the encoding device can apply a BDOF sample prediction process to the prediction samples of the current block, thereby deriving refined prediction samples to improve prediction performance.

[0298] The encoding apparatus may derive residual samples based on the predicted samples (S940) and encode video information including information about the residual samples (S950).

[0299] That is, the encoding apparatus may derive residual samples based on original samples for a current block and predicted samples for the current block, and may generate information about the residual samples, where the information about the residual samples may include information about values ​​of quantized transform coefficients derived by performing transform and quantization on the residual samples, position information, a transform technique, a transform kernel, a quantization parameter, etc.

[0300] The encoding device encodes information about the residual samples and outputs the encoded information into a bitstream, which can be transmitted to the decoding device via a network or a storage medium.

[0301] The encoding apparatus may also encode the information derived in steps S900 to S930 and output the encoded information to a bitstream. For example, CIIP flag information, DMVR availability flag information, merge flag information, MMVD flag information, BDOF availability flag information, sub-block-based merge flag information, etc. may be included in the video information and encoded, and the encoded video information may be signaled to a decoding apparatus.

[0302] FIG. 10 is a flow chart that schematically illustrates a decoding method that can be performed by a decoding device according to one embodiment of the document.

[0303] The method disclosed in Fig. 10 may be performed by the decoding apparatus 300 disclosed in Fig. 3. Specifically, steps S1000 to S1030 of Fig. 10 may be performed by the predictor 330 and the inter predictor 332 disclosed in Fig. 3, and step S1040 of Fig. 10 may be performed by the adder 340 disclosed in Fig. 3. In addition, the method disclosed in Fig. 10 may include the embodiments described in detail herein. Therefore, detailed descriptions of content overlapping with the above-described embodiments will be omitted or simplified in Fig. 10.

[0304] Referring to FIG. 10, the decoding apparatus may determine whether a DMVR application condition for applying motion vector refinement to a current block is met (S1000).

[0305] At this time, the decoding apparatus can determine whether to apply refinement to the current block using various application conditions described in Tables 1 to 25.

[0306] In one embodiment, the decoding apparatus may use whether a prediction mode combining inter prediction and intra prediction is applied to the current block as a DMVR application condition. That is, the decoding apparatus may determine that the DMVR application condition is met when a prediction mode combining inter prediction and intra prediction is not applied to the current block.

[0307] Here, whether a prediction mode in which inter prediction and intra prediction are combined is applied to the current block may be indicated through CIIP (combined inter and intra prediction) flag information (e.g., ciip_flag). For example, if ciip_flag is 0, it may indicate that the CIIP mode (i.e., a prediction mode in which inter prediction and intra prediction are combined) is not applied to the current block, and if ciip_flag is 1, it may indicate that the CIIP mode is applied to the current block. Alternatively, the opposite may be determined.

[0308] For example, a decoding device may obtain a bitstream from an encoding device, decode CIIP flag information included in the bitstream, and determine that the DMVR application condition is met if the value of the CIIP flag information is 0.

[0309] In addition, according to an embodiment, the application conditions of the DMVR may further include the following conditions.

[0310] 1) If DMVR-based inter bi-prediction is enabled, for example, if sps_dmvr_enabled_flag is 1

[0311] 2) If merge mode is applied to the current block, e.g., merge_flag is 1

[0312] 3) If MMVD (merge mode with motion vector difference) mode is not applied to the current block, e.g., mmvd_flag is 0

[0313] 4) When bi-prediction based on the L0 reference picture and the L1 reference picture is applied to the current block, for example, when predFlagL0[0][0]=1 and predFlagL0[1][1]=1.

[0314] 5) When the distance from the current picture to the L0 reference picture and the distance from the L1 reference picture are the same, for example, when DiffPicOrderCnt(currPic, RefPicList0[refIdxL0]) - DiffPicOrderCnt(currPic, RefPicList1[refIdxL1]) is 0.

[0315] 6) When the value of the bi-prediction weight index information of the current block is 0, for example, when GbiIdx is 0

[0316] 7) If the length (Height) of the current block is 8 or more, e.g., if CbHeight is 8 or more

[0317] 8) If the width of the current block is 8 or more, e.g., if CbWidth is 8 or more

[0318] 9) If the current block's length (Height) x width (Width) is greater than 8 x 8, for example, if CbHeight * CbWidth is greater than 64

[0319] If the decoding device satisfies at least one or all of the above 1) to 9), it can be determined that the DMVR application conditions are met.

[0320] Here, when determining whether the DMVR application conditions are met, the listed DMVR application conditions are merely examples, and various combinations of the conditions in Tables 1 to 25 can be used.

[0321] If the decoding apparatus determines that the CIIP mode application conditions or the DMVR application conditions 1) to 9) above are satisfied, the decoding apparatus may derive a minimum SAD (Sum of Absolute Differences) based on the L0 motion vector and the L1 motion vector of the current block (S1010).The decoding apparatus may derive a refined L0 motion vector and a refined L1 motion vector for the current block based on the minimum SAD (S1020).

[0322] Here, the L0 motion vector and the L1 motion vector are motion vectors of the L0 prediction direction and the L1 prediction direction derived by applying merge mode / skip mode to the current block. For example, the decoding apparatus may determine whether skip mode or merge mode is applied to the current block. If skip mode or merge mode is applied to the current block, the decoding apparatus may construct a merge candidate list based on neighboring blocks of the current block. At this time, one merge candidate may be selected from the merge candidate list based on merge index information signaled from the encoding apparatus. The decoding apparatus may derive motion information (motion vector, reference picture index, etc.) of the current block using motion information (motion vector, reference picture index, etc.) of the selected merge candidate. That is, the motion information of a merge candidate selected by a merge index from among merge candidates included in the merge candidate list may be used as motion information of the current block. That is, the L0 motion vector and the L1 motion vector of the current block may be derived based on the motion vector of a neighboring block of the current block indicated by a merge index.

[0323] Furthermore, for example, when a merge mode is applied to a current block, the decoding apparatus may determine whether bi-prediction is applied to a merge candidate (i.e., a neighboring block) indicated by a merge index. When bi-prediction is applied to a neighboring block indicated by a merge index, the decoding apparatus may derive motion information including a motion vector in an L0 prediction direction and a motion vector in an L1 prediction direction of the neighboring block indicated by the merge index.

[0324] As an example, the decoding apparatus may calculate a sample area having a minimum SAD between a prediction sample in an L0 reference picture derived based on an L0 motion vector (i.e., an L0 prediction sample) and a prediction sample in an L1 reference picture derived based on an L1 motion vector (i.e., an L1 prediction sample). Then, the decoding apparatus may perform DMVR-based refinement on the L0 motion vector and the L1 motion vector based on the sample area having the minimum SAD. The process of calculating the minimum SAD and performing refinement has been described with reference to FIGS. 4 and 5, and therefore will not be described in detail here.

[0325] In this case, when performing DMVR-based refinement, if the distance between the L0 reference picture and the current picture and the distance between the L1 reference picture and the current picture are the same, the decoding device can perform refinement to derive a refined L0 motion vector and a refined L1 motion vector.

[0326] The decoding apparatus may derive a prediction sample for the current block based on the refined L0 motion vector and the refined L1 motion vector (S1030).

[0327] According to an embodiment, the decoding apparatus can determine whether the BDOF application condition for applying refinement to the predicted samples of the current block is met.

[0328] In this case, the decoding apparatus can determine whether to apply refinement to the predicted samples of the current block using various application conditions described in Tables 1 to 25.

[0329] For example, the decoding device may use whether a prediction mode combining inter prediction and intra prediction is applied to the current block as a BDOF application condition. As described above, if the value of CIIP flag information (e.g., ciip_flag) obtained from the bitstream is 0, the decoding device may determine that the BDOF application condition is met.

[0330] In addition, according to an embodiment, the application conditions of the BDOF may further include the following conditions.

[0331] 1) If BDOF-based inter-prediction is enabled, e.g., if sps_bdof_enabled_flag is 1

[0332] 2) If the subblock-based merge mode is not applied to the current block, e.g., if merge_subblock_flag is 0

[0333] 3) When bi-prediction based on the L0 reference picture and the L1 reference picture is applied to the current block, for example, when predFlagL0=1 and predFlagL1=1.

[0334] 4) When the distance between the current picture and the L0 reference picture and the distance between the current picture and the L1 reference picture are the same, for example, when DiffPicOrderCnt(currPic, RefPicList0[refIdxL0]) - DiffPicOrderCnt(currPic, RefPicList1[refIdxL1]) is 0.

[0335] 5) When the value of the bi-prediction weight index information of the current block is 0, for example, when GbiIdx is 0

[0336] 6) If no affine mode is applied to the current block, e.g., MotionModelIdc is 0

[0337] 7) If the length (Height) of the current block is 8 or more, e.g., if CbHeight is 8 or more

[0338] 8) If the width of the current block is 8 or more, e.g., if CbWidth is 8 or more

[0339] 9) If the current block's length (Height) x width (Width) is greater than 8 x 8, for example, if CbHeight * CbWidth is greater than 64

[0340] If the decoding device satisfies at least one or all of the above 1) to 9), it can be determined that the BDOF application conditions are met.

[0341] Here, in determining whether the BDOF application conditions are met, the listed BDOF application conditions are merely examples, and various combinations of the conditions in Tables 1 to 25 can be used.

[0342] When the decoding apparatus determines that the CIIP mode application conditions or the BDOF application conditions 1) to 9) are satisfied, the decoding apparatus can apply a BDOF sample prediction process to the prediction samples of the current block, thereby deriving refined prediction samples to improve prediction performance.

[0343] The decoding apparatus may generate reconstructed samples for the current block based on the predicted samples (S1040).

[0344] In one embodiment, the decoding apparatus may use predicted samples as reconstructed samples according to a prediction mode, or may generate reconstructed samples by adding residual samples to the predicted samples.

[0345] If residual samples for the current block exist, the decoding device may receive information about the residuals for the current block. The information about the residuals may include transform coefficients for the residual samples. The decoding device may derive residual samples (or a residual sample array) for the current block based on the residual information. The decoding device may generate reconstructed samples based on the predicted samples and the residual samples, and derive a reconstructed block or a reconstructed picture based on the reconstructed samples.

[0346] In the above-described embodiments, the method is described based on a flow chart with a series of steps or blocks, but the embodiments herein are not limited to the order of steps, and certain steps may occur in a different order or simultaneously with other steps than those described. Furthermore, those skilled in the art will understand that the steps shown in the flow charts are not exclusive, and other steps may be included, or one or more steps in the flow charts may be deleted without affecting the scope of this document.

[0347] The method according to the present document described above can be implemented in software form, and the encoding device and / or decoding device according to the present document can be included in a device that performs video processing, such as a TV, a computer, a smartphone, a set-top box, or a display device.

[0348] When the embodiments herein are implemented in software, the methods described above may be implemented with modules (processes, functions, etc.) that perform the functions described above. The modules may be stored in memory and executed by a processor. The memory may be internal or external to the processor and may be coupled to the processor in various well-known ways. The processor may include an application-specific integrated circuit (ASIC), other chipsets, logic circuits, and / or data processing devices. The memory may include read-only memory (ROM), random access memory (RAM), flash memory, a memory card, a storage medium, and / or other storage devices. That is, the embodiments described herein may be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional units illustrated in the figures may be implemented and executed on a computer, processor, microprocessor, controller, or chip. In this case, information (e.g., information on instructions) or algorithms for implementation may be stored on a digital storage medium.

[0349] In addition, decoding devices and encoding devices to which this document is applied may be included in multimedia broadcast transmitting / receiving devices, mobile communication terminals, home cinema video devices, digital cinema video devices, surveillance cameras, video interaction devices, real-time communication devices such as video communications, mobile streaming devices, storage media, camcorders, custom video (VoD) service providing devices, over-the-top (OTT) video (over-the-top) devices, internet streaming service providing devices, three-dimensional (3D) video devices, virtual reality (VR) devices, augmented reality (AR) devices, image telephone video devices, transportation terminals (e.g., vehicles (including autonomous vehicles), airplane terminals, ship terminals, etc.), medical video devices, etc., and may be used to process video signals or data signals. For example, over-the-top (OTT) video (over-the-top) devices may include game consoles, Blu-ray players, internet-connected TVs, home theater systems, smartphones, tablet PCs, digital video recorders (DVRs), etc.

[0350] In addition, the processing method to which this document is applied may be produced in the form of a computer-executable program and stored in a computer-readable recording medium. Multimedia data having a data structure according to this document may also be stored in a computer-readable recording medium. The computer-readable recording medium includes all types of storage devices and distributed storage devices in which computer-readable data is stored. Examples of the computer-readable recording medium include Blu-ray Discs (BDs), Universal Serial Buses (USBs), ROMs, PROMs, EPROMs, EEPROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices. The computer-readable recording medium also includes media embodied in the form of carrier waves (e.g., transmission via the Internet). A bitstream generated by the encoding method may be stored in a computer-readable recording medium or transmitted via a wired or wireless communication network.

[0351] Furthermore, the embodiments of the present document may be embodied in a computer program product having program code, which may be executed by a computer in accordance with the embodiments of the present document. The program code may be stored on a computer-readable carrier.

[0352] FIG. 11 illustrates an example of a content streaming system in which the embodiments disclosed herein can be applied.

[0353] Referring to FIG. 11, the content streaming system applied to the embodiments of this document can be broadly divided into an encoding server, a streaming server, a web server, a media repository, a user device, and a multimedia input device.

[0354] The encoding server compresses content input from a multimedia input device such as a smartphone, camera, camcorder, etc. into digital data to generate a bitstream and transmits the bitstream to the streaming server. As another example, if a multimedia input device such as a smartphone, camera, camcorder, etc. directly generates a bitstream, the encoding server may be omitted.

[0355] The bitstream may be generated by an encoding method or a bitstream generation method applied to an embodiment of this document, and the streaming server may temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0356] The streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server acts as an intermediary to inform the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, which then transmits the multimedia data to the user. In this case, the content streaming system may include a separate control server, which controls commands and responses between devices in the content streaming system.

[0357] The streaming server can receive content from a media repository and / or an encoding server. For example, if content is received from the encoding server, the content can be received in real time. In this case, the streaming server can store the bitstream for a certain period of time to provide a smooth streaming service.

[0358] Examples of the user devices include mobile phones, smartphones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, head-mounted displays (HMDs)), digital TVs, desktop computers, and digital signs.

[0359] Each server in the content streaming system can be operated as a distributed server, in which case data received by each server can be processed in a distributed manner.

Claims

1. A video decoding method performed by a decoding device, comprising: obtaining residual information from the bitstream; deriving an L0 motion vector and an L1 motion vector for the current block; determining whether a DMVR application condition for applying motion vector refinement to the current block is met; deriving a minimum SAD based on the L0 motion vector and the L1 motion vector of the current block based on whether the DMVR application condition is satisfied; deriving a refined L0 motion vector and a refined L1 motion vector for the current block based on the minimum SAD; deriving a prediction sample for the current block based on the refined L0 motion vector and the refined L1 motion vector; deriving a residual sample for the current block based on the residual information; generating reconstructed samples for the current block based on the predicted samples and the residual samples; Whether the DMVR application condition is satisfied is determined based on whether a combined inter and intra prediction (CIIP) mode, in which inter prediction and intra prediction are combined, is applied to the current block, and whether a distance from an L0 reference picture and a distance from an L1 reference picture are the same based on the current picture; The DMVR is applied to the current block based on a case where the CIIP mode is not applied to the current block and the distance from the L0 reference picture and the distance from the L1 reference picture are equal to each other based on the current picture; The prediction sample is derived based on determining whether a bi-directional optical flow (BDOF) application condition for applying refinement to the prediction sample is satisfied; whether the BDOF application condition is satisfied is determined based on whether the CIIP mode is applied to the current block and whether a distance from the L0 reference picture and a distance from the L1 reference picture are equal to each other based on the current picture; A video decoding method, wherein the BDOF is applied to the predicted sample based on the case where the CIIP mode is not applied to the current block and the distance from the L0 reference picture and the distance from the L1 reference picture are the same based on the current picture.

2. A video encoding method performed by an encoding device, comprising: deriving an L0 motion vector and an L1 motion vector for the current block; determining whether a DMVR application condition for applying motion vector refinement to the current block is met; deriving a minimum SAD based on the L0 motion vector and the L1 motion vector of the current block based on whether the DMVR application condition is satisfied; deriving a refined L0 motion vector and a refined L1 motion vector for the current block based on the minimum SAD; deriving a prediction sample for the current block based on the refined L0 motion vector and the refined L1 motion vector; deriving a residual sample based on the predicted sample; generating residual information based on the residual samples; encoding video information including the residual information; Whether the DMVR application condition is satisfied is determined based on whether a combined inter and intra prediction (CIIP) mode, in which inter prediction and intra prediction are combined, is applied to the current block, and whether a distance from an L0 reference picture and a distance from an L1 reference picture are the same based on the current picture; The DMVR is applied to the current block based on a case where the CIIP mode is not applied to the current block and the distance from the L0 reference picture and the distance from the L1 reference picture are equal to each other based on the current picture; The prediction sample is derived based on determining whether a bi-directional optical flow (BDOF) application condition for applying refinement to the prediction sample is satisfied; whether the BDOF application condition is satisfied is determined based on whether the CIIP mode is applied to the current block and whether a distance from the L0 reference picture and a distance from the L1 reference picture are equal to each other based on the current picture; A video encoding method, wherein the BDOF is applied to the predicted sample based on the case where the CIIP mode is not applied to the current block and the distance from the L0 reference picture and the distance from the L1 reference picture are the same based on the current picture.

3. 1. A method for transmitting data for a video, comprising: obtaining a bitstream of the video, wherein the bitstream is generated by deriving an L0 motion vector and an L1 motion vector for a current block, determining whether a DMVR application condition for applying motion vector refinement to the current block is satisfied, deriving a minimum SAD based on the L0 motion vector and the L1 motion vector of the current block based on whether the DMVR application condition is satisfied, deriving a refined L0 motion vector and a refined L1 motion vector for the current block based on the minimum SAD, deriving prediction samples for the current block based on the refined L0 motion vector and the refined L1 motion vector, deriving residual samples based on the prediction samples, generating residual information based on the residual samples, and encoding video information including the residual information; transmitting the data including the bitstream; Whether the DMVR application condition is satisfied is determined based on whether a combined inter and intra prediction (CIIP) mode, in which inter prediction and intra prediction are combined, is applied to the current block, and whether a distance from an L0 reference picture and a distance from an L1 reference picture are the same based on the current picture; The DMVR is applied to the current block based on a case where the CIIP mode is not applied to the current block and the distance from the L0 reference picture and the distance from the L1 reference picture are equal to each other based on the current picture; The prediction sample is derived based on determining whether a bi-directional optical flow (BDOF) application condition for applying refinement to the prediction sample is satisfied; whether the BDOF application condition is satisfied is determined based on whether the CIIP mode is applied to the current block and whether a distance from the L0 reference picture and a distance from the L1 reference picture are equal to each other based on the current picture; A data transmission method in which the BDOF is applied to the predicted sample based on the case where the CIIP mode is not applied to the current block and the distance from the L0 reference picture and the distance from the L1 reference picture are the same as each other based on the current picture.

Images