DMVR-based inter-prediction method and device

Abstract

To provide a method and a device for increasing image coding efficiency.SOLUTION: An image decoding method performed by a decoding device includes the steps of: deriving a motion vector for a current block; determining whether to apply refinement to the motion vector for the current block; deriving a refined motion vector by applying decoder-side motion vector refinement (DMVR) to the motion vector if it is determined to apply refinement to the motion vector for the current block; deriving prediction samples for the current block based on the refined motion vector; and generating reconstruction samples for the current block based on the prediction samples. The step of determining whether to apply refinement includes determining whether to apply the refinement based on at least one of the size of the current block and bi-prediction weight index information for the current block.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] This document relates to video coding technology, and more particularly to a method and apparatus for performing inter prediction based on Decoder-side Motion Vector Refinement (DMVR). [Background technology]

[0002] Recently, the demand for high-resolution, high-quality images / videos such as 4K or 8K or higher UHD (Ultra High Definition) images / videos is increasing in various fields. As the image / video data has higher resolution and quality, the amount of information or bits transmitted relatively increases compared to existing image / video data, so when image data is transmitted using existing media such as wired or wireless broadband lines or when image / video data is stored using existing storage media, the transmission and storage costs increase.

[0003] In addition, interest and demand for immersive media such as virtual reality (VR), artificial reality (AR) content and holograms has been increasing recently, and the broadcast of images / videos with different image characteristics from real images, such as game images, is increasing.

[0004] Therefore, in order to effectively compress, transmit, store and play back high-resolution, high-quality image / video information having the above-mentioned various characteristics, a highly efficient image / video compression technology is required. Summary of the Invention [Problem to be solved by the invention]

[0005] The technical problem of this document is to provide a method and apparatus for increasing video coding efficiency.

[0006] Another technical problem of this document is to provide an efficient inter-prediction method and apparatus.

[0007] Another technical problem of this document is to provide a method and apparatus for performing inter prediction based on Decoder-side Motion Vector Refinement (DMVR).

[0008] Another technical problem of this document is to provide a method and apparatus for performing inter prediction based on BDOF (Bi-directional optical flow).

[0009] Another technical object of this document is to provide a method and apparatus for improving prediction performance by providing conditions for determining whether to apply DMVR and / or conditions for determining whether to apply BDOF to improve video coding efficiency. [Means for solving the problem]

[0010] According to an embodiment of the present document, there is provided a video decoding method performed by a decoding device, the method including the steps of: deriving a motion vector of a current block; determining whether to apply refinement to the motion vector of the current block; deriving a refined motion vector by applying a decoder-side motion vector refinement (DMVR) to the motion vector when it is determined to apply refinement to the motion vector of the current block; deriving a prediction sample for the current block based on the refined motion vector; and generating a reconstructed sample for the current block based on the prediction sample, wherein the step of determining whether to apply refinement includes determining whether to apply the refinement based on at least one of a size of the current block or bi-prediction weight index information of the current block.

[0011] According to another embodiment of the present document, there is provided a video encoding method performed by an encoding apparatus, the method including the steps of: deriving a motion vector of a current block; determining whether to apply refinement to the motion vector of the current block; if it is determined to apply refinement to the motion vector of the current block, applying a decoder-side motion vector refinement (DMVR) to the motion vector to derive a refined motion vector; deriving a prediction sample for the current block based on the refined motion vector; deriving a residual sample based on the prediction sample; generating residual information based on the residual sample; and encoding video information including the residual information, wherein the step of determining whether to apply refinement includes determining whether to apply the refinement based on at least one of a size of the current block or bi-prediction weight index information of the current block. Effect of the Invention

[0012] According to the document, it can improve the overall image / video compression efficiency.

[0013] According to this document, efficient inter-prediction can reduce computational complexity and improve overall coding efficiency.

[0014] According to this document, when applying DMVR and / or BDOF, which refines motion information during the motion compensation process, various application conditions are proposed, thereby improving efficiency in terms of complexity and performance. [Brief description of the drawings]

[0015] [Figure 1]1 illustrates generally an example of a video / image coding system that can be applied to embodiments of the present document; [Diagram 2] FIG. 1 is a diagram illustrating the configuration of a video / image encoding device that can be applied to an embodiment of this document. [Diagram 3] FIG. 1 is a diagram illustrating the configuration of a video / image decoding device that can be applied to an embodiment of this document. [Figure 4] FIG. 1 is a diagram illustrating an embodiment of a process for performing Decoder-side Motion Vector Refinement (DMVR) in true bi-prediction. [Diagram 5] 1 is an example showing a method of checking the application conditions of DMVR and BDOF and performing a decoding process. [Figure 6] 11 is another example showing a method of checking the application conditions of DMVR and BDOF and performing a decoding process. [Figure 7] 11 is another example showing a method of checking the application conditions of DMVR and BDOF and performing a decoding process. [Figure 8] 1 is a flow chart illustrating an encoding method that can be performed by an encoding device according to an embodiment of the present document. [Figure 9] 1 is a flow chart illustrating a decoding method that can be performed by a decoding device according to an embodiment of the document. [Figure 10] 1 illustrates an example of a content streaming system in which the embodiments disclosed herein can be applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] This document may have various modifications and may have various embodiments, and a specific embodiment will be illustrated in the drawings and described in detail. However, this is not intended to limit this document to a specific embodiment. Terms commonly used in this specification are used merely to describe a specific embodiment, and are not intended to limit the technical ideas of this document. A singular expression includes a plural expression unless the context clearly indicates otherwise. In this specification, the terms "include" or "have" are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0017] Meanwhile, each component in the drawings described in this document is illustrated independently for the convenience of explaining the different characteristic functions, and does not mean that each component is realized by separate hardware or software. For example, two or more components among the components may be combined to form one component, or one component may be divided into multiple components. An embodiment in which each component is integrated and / or separated is also included in the scope of the rights of this document as long as it does not deviate from the essence of this document.

[0018] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. Hereinafter, the same components in the drawings will be designated by the same reference numerals, and duplicated descriptions of the same components will be omitted.

[0019] FIG. 1 illustrates a schematic diagram of an example of a video / image coding system to which embodiments of the present document can be applied.

[0020] Referring to Figure 1, a video / image coding system may include a first device (source device) and a second device (receiving device). The source device may transmit encoded video / image information or data to the receiving device in a file or streaming form via a digital storage medium or a network.

[0021] The source device may include a video source, an encoding device, and a sending unit. The receiving device may include a receiving unit, a decoding device, and a renderer. The encoding device may be referred to as a video / video encoding device, and the decoding device may be referred to as a video / video decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may also include a display unit, which may be a separate device or an external component.

[0022] A video source may acquire video / images through a video / image capture, synthesis, or generation process. A video source may include a video / image capture device and / or a video / image generation device. A video / image capture device may include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. A video / image generation device may include, for example, a computer, a tablet, a smartphone, etc., and may (electronically) generate video / images. For example, a virtual video / image may be generated via a computer, etc., in which case the video / image capture process may be replaced by a process in which related data is generated.

[0023] An encoding device may encode an input video / image. The encoding device may perform a series of steps such as prediction, transformation, quantization, etc. for compression and coding efficiency. The encoded data (encoded video / image information) may be output in the form of a bitstream.

[0024] The transmitting unit can transmit the encoded video / image information or data output in the form of a bitstream to a receiving unit of a receiving device via a digital storage medium or a network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitting unit can include elements for generating a media file through a predetermined file format and can include elements for transmission via a broadcasting / communication network. The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0025] The decoding device can decode the video / image by performing a series of steps such as inverse quantization, inverse transformation, and prediction that correspond to the operations of the encoding device.

[0026] The renderer may render the decoded video / image, and the rendered video / image may be displayed via a display unit.

[0027] This document relates to video / image coding. For example, the methods / embodiments disclosed in this document may be applied to methods disclosed in the versatile video coding (VVC) standard, the essential video coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the 2nd generation of audio video coding standard (AVS2), or next generation video / image coding standards (e.g., H.267 or H.268, etc.).

[0028] This document presents various embodiments for video / image coding, which may also be implemented in combination with one another, unless otherwise stated.

[0029] In this document, a video may mean a collection of a series of images over time. A picture generally means a unit showing one image at a specific time, and a slice / tile is a unit constituting a part of a picture in coding. A slice / tile may include one or more coding tree units (CTUs). A picture may be composed of one or more slices / tiles. A picture may be composed of one or more tile groups. A tile group may include one or more tiles. A brick may represent a rectangular region of CTU rows within a tile in a picture. A tile may be partitioned into multiple bricks, each of which consisting of one or more CTU rows within the tile. A tile that is not partitioned into multiple bricks may also be referred to as a brick.A brick scan can refer to a specific sequential ordering of CTUs partitioning a picture in which the CTUs are ordered consecutively in CTU raster scan in a brick, bricks within a tile are ordered consecutively in a raster scan of the bricks of the tile, and tiles in a picture are ordered consecutively in a raster scan of the tiles of the picture. A tile is a rectangular region of CTUs within a particular tile column and a particular tile row in a picture. The tile column is a rectangular region of CTUs having a height equal to the height of the picture and a width specified by syntax elements in the picture parameter set.The tile row is a rectangular region of CTUs having a height specified by syntax elements in the picture parameter set and a width equal to the width of the picture. A tile scan is a specific sequential ordering of CTUs partitioning a picture in which the CTUs are ordered consecutively in CTU raster scan in a tile whereas tiles in a picture are ordered consecutively in a raster scan of the tiles of the picture. A slice includes an integer number of bricks of a picture that may be exclusively contained in a single NAL unit. A slice may consist of either a number of complete tiles or only a consecutive sequence of complete bricks of one tile.In this document, tile group and slice can be used interchangeably. For example, in this document, tile group / tile group header can be called slice / slice header.

[0030] A pixel or a pel may refer to the smallest unit constituting one picture (or image). A term corresponding to a pixel may also be used as a "sample." A sample may generally refer to a pixel or a pixel value, may refer to only a pixel / pixel value of a luma component, or may refer to only a pixel / pixel value of a chroma component. Alternatively, a sample may refer to a pixel value in the spatial domain, or may refer to a transform coefficient in the frequency domain when such a pixel value is transformed into the frequency domain.

[0031] A unit may indicate a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. One unit may include one luma block and two chroma (e.g., cb, cr) blocks. A unit may be used interchangeably with terms such as block or area depending on the case. In a general case, an M×N block may include a sample (or sample array) consisting of M columns and N rows, or a set (or array) of transform coefficients.

[0032] In this document, the term “ / ” and “,” should be interpreted to indicate “and / or.” For instance, “A / B” is interpreted as “A and / or B,” and “A, B” is interpreted as “A and / or B.” Additionally, “A / B / C” means “at least one of A, B, and / or C.” Also, “A, B, C” means “at least one of A, B, and / or C.” (In this document, the term “ / ”and “,” should be interpreted to indicate “and / or.” For instance, the expression “A / B” may mean “A and / or B.” Further, “A,B” may mean “A and / or B.” Further, “A / B / C” may mean “at least one of A,B,and / or C.” Also, “A / B / C” may mean “at least one of A,B,and / or C.”)

[0033] Additionally, in this document, the term “or” should be interpreted to mean “and / or.” For example, “A or B” may comprise 1) only “A,” or 2) only “B,” or 3) both “A and B.” In other words, the term “or” in this document should be interpreted to indicate “additionally or alternatively.”

[0034] 2 is a diagram for illustrating a schematic configuration of a video / image encoding device that can be applied to an embodiment of the present document. Hereinafter, the video encoding device may include an image encoding device.

[0035] Referring to FIG. 2, the encoding apparatus 200 may include an image partitioner 210, a predictor 220, a residual processor 230, an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 may include an inter predictor 221 and an intra predictor 222. The residual processor 230 may include a transformer 232, a quantizer 233, a dequantizer 234, and an inverse transformer 235. The residual processor 230 may further include a subtractor 231. The adder 250 may be referred to as a reconstructor or a reconstructed block generator. The image division unit 210, the prediction unit 220, the residual processing unit 230, the entropy encoding unit 240, the addition unit 250, and the filtering unit 260 may be configured with one or more hardware components (e.g., an encoder chip set or a processor) according to an embodiment. Also, the memory 270 may include a decoded picture buffer (DPB) and may be configured with a digital storage medium. The hardware components may further include the memory 270 as an internal / external component.

[0036] The image division unit 210 may divide an input image (or picture, frame) input to the encoding device 200 into one or more processing units. As an example, the processing units may be called coding units (CUs). In this case, the coding units may be recursively divided from a coding tree unit (CTU) or a largest coding unit (LCU) according to a quad-tree binary-tree ternary-tree (QTBTTT) structure. For example, one coding unit may be divided into a plurality of coding units of a deeper depth based on a quad-tree structure, a binary tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure may be applied first, and then the binary tree structure and / or the ternary structure may be applied. Alternatively, the binary tree structure may be applied first. The coding procedure according to this document may be performed based on a final coding unit that is not further divided. In this case, the maximum coding unit may be used as the final coding unit based on coding efficiency according to image characteristics, or the coding unit may be recursively divided into coding units of lower depths as necessary, and a coding unit of an optimal size may be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration, which will be described later. As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may be divided or partitioned from the final coding unit described above, respectively.The prediction unit is a unit of sample prediction, and the transform unit is a unit for deriving transform coefficients and / or a unit for deriving a residual signal from the transform coefficients.

[0037] The unit may be used interchangeably with terms such as block or area. In general, an M×N block may refer to a set of samples or transform coefficients consisting of M columns and N rows. A sample may generally refer to a pixel or pixel value, may refer to only a pixel / pixel value of a luma component, or may refer to only a pixel / pixel value of a chroma component. A sample may be used as a term corresponding to a pixel or pel of a picture (or image).

[0038] The encoding apparatus 200 may generate a residual signal (residual block, residual sample array) by subtracting a prediction signal (predicted block, prediction sample array) output from the inter prediction unit 221 or the intra prediction unit 222 from an input image signal (original block, original sample array), and the generated residual signal is transmitted to the conversion unit 232. In this case, as illustrated, a unit that subtracts a prediction signal (predicted block, prediction sample array) from an input image signal (original block, original sample array) in the encoder 200 may be called a subtraction unit 231. The prediction unit may perform prediction on a block to be processed (hereinafter, referred to as a current block) and generate a predicted block including a prediction sample for the current block. The prediction unit may determine whether intra prediction or inter prediction is applied in units of the current block or CU. The prediction unit may generate various information related to prediction, such as prediction mode information, and transmit the information to the entropy encoding unit 240, as will be described later in the description of each prediction mode. The prediction-related information can be encoded by the entropy encoding unit 240 and output in the form of a bitstream.

[0039] The intra prediction unit 222 may predict the current block by referring to samples in the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from the current block depending on the prediction mode. Prediction modes in intra prediction may include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes may include, for example, a DC mode and a planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the fineness of the prediction direction. However, this is merely an example, and more or less directional prediction modes may be used depending on the setting. The intra prediction unit 222 may also determine a prediction mode to be applied to the current block using a prediction mode applied to a neighboring block.

[0040] The inter prediction unit 221 may derive a predicted block for a current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. In this case, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between a neighboring block and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter prediction, the neighboring block may include a spatial neighboring block present in the current picture and a temporal neighboring block present in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The temporal neighboring block may be called a collocated reference block, a collocated CU (colCU), etc., and the reference picture including the temporal neighboring block may be called a collocated picture (colPic). For example, the inter prediction unit 221 may generate information indicating which candidate is used to derive a motion vector and / or a reference picture index of the current block by forming a motion information candidate list based on neighboring blocks. Inter prediction may be performed based on various prediction modes, and for example, in the case of a skip mode and a merge mode, the inter prediction unit 221 may use motion information of a neighboring block as motion information of the current block. In the case of the skip mode, unlike the merge mode, a residual signal may not be transmitted.In the case of motion vector prediction (MVP) mode, the motion vector of the neighboring block can be used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0041] The prediction unit 220 may generate a prediction signal based on various prediction methods, which will be described later. For example, the prediction unit may apply intra prediction or inter prediction for prediction of one block, and may simultaneously apply intra prediction and inter prediction. This may be called combined inter and intra prediction (CIIP). In addition, the prediction unit may be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode may be used for content image / moving image coding such as games, for example, as in screen content coding (SCC). IBC basically performs prediction within a current picture, but may be performed similarly to inter prediction in deriving a reference block within the current picture. That is, IBC may use at least one of the inter prediction techniques described in this document. The palette mode may be considered as an example of intra coding or intra prediction. When the palette mode is applied, a sample value within a picture may be signaled based on information regarding a palette table and a palette index.

[0042] The prediction signal generated through the prediction unit (including the inter prediction unit 221 and / or the intra prediction unit 222) may be used to generate a restored signal or may be used to generate a residual signal. The transform unit 232 may generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen-Loeve transform (KLT), a graph-based transform (GBT), or a conditionally non-linear transform (CNT). Here, the GBT refers to a transform obtained from a graph when the relationship information between pixels is expressed as a graph. The CNT refers to a transform obtained based on a prediction signal generated using all previously reconstructed pixels. In addition, the transform process may be applied to pixel blocks having the same square size, or may be applied to non-square variable-size blocks.

[0043] The quantization unit 233 quantizes the transform coefficients and transmits them to the entropy encoding unit 240, and the entropy encoding unit 240 may encode the quantized signal (information on the quantized transform coefficients) and output the encoded signal to a bitstream. The information on the quantized transform coefficients may be referred to as residual information. The quantization unit 233 may rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on a coefficient scan order, and may generate information on the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. The entropy encoding unit 240 may perform various encoding methods, such as exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc. The entropy encoding unit 240 may also encode information required for video / image restoration (e.g., values ​​of syntax elements, etc.) together with or separately from the quantized transform coefficients. The encoded information (e.g., encoded video / picture information) may be transmitted or stored in the form of a bitstream in network abstraction layer (NAL) unit units. The video / picture information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). The video / picture information may further include general constraint information. In this document, information and / or syntax elements transmitted / signaled from an encoding device to a decoding device may be included in the video / picture information.The video / image information may be encoded through the above-mentioned encoding procedure and included in the bitstream. The bitstream may be transmitted via a network or stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as a USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitter (not shown) for transmitting the signal output from the entropy encoding unit 240 and / or a storage unit (not shown) for storing the signal may be configured as an internal / external element of the encoding apparatus 200, or the transmitter may be included in the entropy encoding unit 240.

[0044] The quantized transform coefficients output from the quantizer 233 may be used to generate a prediction signal. For example, a residual signal (residual block or residual sample) may be restored by applying inverse quantization and inverse transform to the quantized transform coefficients via the inverse quantizer 234 and the inverse transformer 235. The adder 155 may generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter prediction unit 221 or the intra prediction unit 222. When there is no residual for the current block, such as when a skip mode is applied, a predicted block may be used as a reconstructed block. The adder 250 may be referred to as a reconstruction unit or a reconstructed block generator. The generated reconstructed signal may be used for intra prediction of the next current block in the current picture, and may also be used for inter prediction of the next picture after filtering, as described below.

[0045] Meanwhile, luma mapping with chroma scaling (LMCS) may be applied during picture encoding and / or restoration.

[0046] The filtering unit 260 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 260 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture, and may store the modified reconstructed picture in the memory 270, specifically, in the DPB of the memory 270. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, an adaptive loop filter, a bilateral filter, etc. The filtering unit 260 may generate various information related to filtering and transmit it to the entropy encoding unit 240, as will be described later in the description of each filtering method. The filtering information may be encoded by the entropy encoding unit 240 and output in the form of a bitstream.

[0047] The modified reconstructed picture transmitted to the memory 270 may be used as a reference picture in the inter prediction unit 221. When inter prediction is applied through this, the encoding apparatus may avoid prediction mismatch between the encoding apparatus 100 and the decoding apparatus, and may also improve encoding efficiency.

[0048] The DPB of the memory 270 may store the modified reconstructed picture to be used as a reference picture in the inter prediction unit 221. The memory 270 may store motion information of a block from which motion information in the current picture is derived (or encoded) and / or motion information of a block in an already reconstructed picture. The stored motion information may be transmitted to the inter prediction unit 221 to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory 270 may store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra prediction unit 222.

[0049] FIG. 3 is a diagram illustrating a schematic configuration of a video / image decoding device that can be applied to an embodiment of this document.

[0050] Referring to FIG. 3, the decoding device 300 may include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an inter predictor 331 and an intra predictor 332. The residual processor 320 may include a dequantizer 321 and an inverse transformer 321. The entropy decoding unit 310, the residual processor 320, the predictor 330, the adder 340, and the filter 350 may be configured as one hardware component (e.g., a decoder chipset or processor) according to an embodiment. Also, the memory 360 may include a decoded picture buffer (DPB) and may be configured as a digital storage medium. The hardware components may further include a memory 360 as an internal / external component.

[0051] When a bitstream including video / image information is input, the decoding apparatus 300 can restore an image corresponding to a process in which the video / image information is processed by the encoding apparatus of FIG. 2. For example, the decoding apparatus 300 can derive a unit / block based on block division related information obtained from the bitstream. The decoding apparatus 300 can perform decoding using a processing unit applied in the encoding apparatus. Thus, the processing unit for decoding is, for example, a coding unit, and the coding unit can be divided from a coding tree unit or a maximal coding unit according to a quad tree structure, a binary tree structure, and / or a ternary tree structure. One or more transform units can be derived from the coding unit. And, the restored image signal decoded and output by the decoding apparatus 300 can be reproduced by a reproduction device.

[0052] The decoding apparatus 300 may receive a signal output from the encoding apparatus of FIG. 2 in the form of a bitstream, and the received signal may be decoded via the entropy decoding unit 310. For example, the entropy decoding unit 310 may derive information (e.g., video / image information) required for image restoration (or picture restoration) by parsing the bitstream. The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The decoding apparatus may decode a picture based on information on the parameter set and / or the general constraint information. Signaling / received information and / or syntax elements described later in this document may be decoded via the decoding procedure and obtained from the bitstream. For example, the entropy decoding unit 310 may decode information in a bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image restoration and quantized values ​​of transform coefficients related to residuals. More specifically, the CABAC entropy decoding method may receive bins corresponding to each syntax element in a bitstream, determine a context model using syntax element information to be decoded and decoding information of adjacent and to-be-decoded blocks or symbol / bin information decoded in a previous step, predict the occurrence probability of bins according to the determined context model, and perform arithmetic decoding of the bins to generate symbols corresponding to the values ​​of each syntax element.In this case, the CABAC entropy decoding method may update the context model using information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Prediction information from the information decoded by the entropy decoding unit 310 is provided to a prediction unit (inter prediction unit 332 and intra prediction unit 331), and residual values ​​entropy-decoded by the entropy decoding unit 310, i.e., quantized transform coefficients and related parameter information, may be input to the residual processing unit 320. The residual processing unit 320 may derive a residual signal (residual block, residual sample, residual sample array). In addition, filtering information from the information decoded by the entropy decoding unit 310 may be provided to the filtering unit 350. Meanwhile, a receiving unit (not shown) that receives a signal output from the encoding device may be further configured as an internal / external element of the decoding device 300, or the receiving unit may be a component of the entropy decoding unit 310. Meanwhile, the decoding device according to the present document may be called a video / image / picture decoding device, and the decoding device may be divided into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include the entropy decoding unit 310, and the sample decoder may include at least one of the inverse quantization unit 321, the inverse transform unit 322, the addition unit 340, the filtering unit 350, the memory 360, the inter prediction unit 332, and the intra prediction unit 331.

[0053] The inverse quantization unit 321 may inverse quantize the quantized transform coefficients to output transform coefficients. The inverse quantization unit 321 may rearrange the quantized transform coefficients in a two-dimensional block form. In this case, the rearrangement may be performed based on a coefficient scan order performed in the encoding apparatus. The inverse quantization unit 321 may perform inverse quantization on the quantized transform coefficients using a quantization parameter (e.g., quantization step size information) to obtain transform coefficients.

[0054] The inverse transform unit 322 inversely transforms the transform coefficients to obtain a residual signal (residual block, residual sample array).

[0055] The prediction unit may perform prediction on a current block and generate a predicted block including prediction samples for the current block. The prediction unit may determine whether intra prediction or inter prediction is applied to the current block based on information on the prediction output from the entropy decoding unit 310, and may determine a specific intra / inter prediction mode.

[0056] The prediction unit 320 may generate a prediction signal based on various prediction methods, which will be described later. For example, the prediction unit may apply intra prediction or inter prediction for prediction of one block, and may simultaneously apply intra prediction and inter prediction. This may be called combined inter and intra prediction (CIIP). In addition, the prediction unit may be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode may be used for content image / moving image coding, such as a game, such as screen content coding (SCC). IBC basically performs prediction within a current picture, but may be performed similarly to inter prediction in that a reference block is derived within the current picture. That is, IBC may use at least one of the inter prediction techniques described in this document. The palette mode may be regarded as an example of intra coding or intra prediction. When the palette mode is applied, information regarding a palette table and a palette index may be included in the video / image information and signaled.

[0057] The intra prediction unit 331 may predict a current block by referring to samples in a current picture. The referenced samples may be located in a neighbor of the current block or may be located far away depending on a prediction mode. Prediction modes in intra prediction may include a plurality of non-directional modes and a plurality of directional modes. The intra prediction unit 331 may also determine a prediction mode to be applied to the current block by using a prediction mode applied to a neighboring block.

[0058] The inter prediction unit 332 may derive a predicted block for the current block based on a reference block (reference sample array) identified by a motion vector on a reference picture. In this case, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. For example, the inter prediction unit 332 may configure a motion information candidate list based on the neighboring blocks and derive a motion vector and / or a reference picture index for the current block based on the received candidate selection information. Inter prediction may be performed based on various prediction modes, and the information regarding the prediction may include information indicating a mode of inter prediction for the current block.

[0059] The adder 340 may generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the acquired residual signal to a prediction signal (predicted block, predicted sample array) output from a prediction unit (including the inter prediction unit 332 and / or the intra prediction unit 331). When there is no residual for a current block, such as when a skip mode is applied, the predicted block may be used as a reconstructed block.

[0060] The adder 340 may be referred to as a reconstruction unit or a reconstruction block generator. The generated reconstruction signal may be used for intra prediction of a next block to be processed in a current picture, and may be output after filtering as described below, or may be used for inter prediction of a next picture.

[0061] Meanwhile, luma mapping with chroma scaling (LMCS) can be applied during the picture decoding process.

[0062] The filtering unit 350 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 350 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture, and may transmit the modified reconstructed picture to the memory 360, specifically, to the DPB of the memory 360. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, an adaptive loop filter, a bilateral filter, etc.

[0063] The (modified) reconstructed picture stored in the DPB of the memory 360 may be used as a reference picture in the inter prediction unit 332. The memory 360 may store motion information of a block from which motion information in the current picture is derived (or decoded) and / or motion information of a block in an already reconstructed picture. The stored motion information may be transmitted to the inter prediction unit 260 to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory 360 may store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra prediction unit 331.

[0064] In this specification, the embodiments described for the filtering unit 260, inter prediction unit 221, and intra prediction unit 222 of the encoding apparatus 200 can also be applied identically or correspondingly to the filtering unit 350, inter prediction unit 332, and intra prediction unit 331 of the decoding apparatus 300, respectively.

[0065] As described above, prediction is performed to improve compression efficiency when performing video coding. As a result, a predicted block including predicted samples for a current block, which is a block to be coded, can be generated. Here, the predicted block includes predicted samples in a spatial domain (or a pixel domain). The predicted block is derived in the same way by an encoding device and a decoding device, and the encoding device can improve video coding efficiency by signaling information (residual information) about the residual between the original block and the predicted block, which is not an original sample value of the original block, to a decoding device. The decoding device can derive a residual block including residual samples based on the residual information, and can generate a reconstructed block including reconstructed samples by adding the residual block and the predicted block, and can generate a reconstructed picture including the reconstructed block.

[0066] The residual information may be generated through a transform and quantization procedure. For example, the encoding apparatus may derive a residual block between the original block and the predicted block, perform a transform procedure on the residual samples (residual sample array) included in the residual block to derive transform coefficients, and perform a quantization procedure on the transform coefficients to derive quantized transform coefficients, and then signal the related residual information (through a bitstream) to the decoding apparatus. Here, the residual information may include information such as value information, position information, transform technique, transform kernel, and quantization parameter of the quantized transform coefficients. The decoding apparatus may derive a residual sample (or a residual block) by performing an inverse quantization / inverse transform procedure based on the residual information. The decoding apparatus may generate a reconstructed picture based on the predicted block and the residual block. In addition, the encoding apparatus may derive a residual block by inverse quantizing / inverse transforming the quantized transform coefficients for reference for inter-prediction of a future picture, and generate a reconstructed picture based on the residual block.

[0067] Meanwhile, as described above, intra prediction or inter prediction may be applied to perform prediction on the current block. Hereinafter, a case where inter prediction is applied to the current block will be described.

[0068] A prediction unit (more specifically, an inter prediction unit) of an encoding / decoding apparatus may perform inter prediction on a block-by-block basis to derive a prediction sample. Inter prediction may indicate a prediction derived in a manner dependent on data elements (e.g., sample values, motion information, etc.) of a picture(s) other than the current picture. When inter prediction is applied to the current block, a predicted block (prediction sample array) for the current block may be derived based on a reference block (reference sample array) identified by a motion vector on a reference picture pointed to by a reference picture index. In this case, in order to reduce the amount of motion information transmitted in the inter prediction mode, motion information of the current block may be predicted on a block, sub-block, or sample basis based on correlation of motion information between a neighboring block and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction type (L0 prediction, L1 prediction, Bi prediction, etc.) information. When inter prediction is applied, the neighboring block may include a spatial neighboring block present in the current picture and a temporal neighboring block present in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The temporal neighboring block may be called a collocated reference block, a collocated CU (colCU), etc., and the reference picture including the temporal neighboring block may be called a collocated picture (colPic). For example, a motion information candidate list may be constructed based on the neighboring blocks of the current block, and a flag or index information indicating which candidate is selected (used) to derive a motion vector and / or a reference picture index of the current block may be signaled.Inter prediction may be performed based on various prediction modes, for example, in the case of skip mode and merge mode, the motion information of the current block is the same as the motion information of the selected neighboring block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted. In the case of motion vector prediction (MVP) mode, the motion vector of the selected neighboring block may be used as a motion vector predictor, and a motion vector difference may be signaled. In this case, the motion vector of the current block may be derived using the sum of the motion vector predictor and the motion vector difference.

[0069] The motion information may include L0 motion information and / or L1 motion information depending on the inter prediction type (L0 prediction, L1 prediction, Bi prediction, etc.). A motion vector in the L0 direction may be referred to as an L0 motion vector or MVL0, and a motion vector in the L1 direction may be referred to as an L1 motion vector or MVL1. Prediction based on the L0 motion vector may be referred to as L0 prediction, prediction based on the L1 motion vector may be referred to as L1 prediction, and prediction based on both the L0 motion vector and the L1 motion vector may be referred to as bi-prediction. Here, the L0 motion vector may indicate a motion vector associated with a reference picture list L0 (L0), and the L1 motion vector may indicate a motion vector associated with a reference picture list L1 (L1). The reference picture list L0 may include a picture that is earlier in the output order than the current picture as a reference picture, and the reference picture list L1 may include a picture that is later in the output order than the current picture. The earlier picture may be referred to as a forward (reference) picture, and the later picture may be referred to as a backward (reference) picture. The reference picture list L0 may further include pictures that are later in output order than the current picture as reference pictures. In this case, the previous picture may be indexed first in the reference picture list L0, and the later picture may be indexed next. The reference picture list L1 may further include pictures that are earlier in output order than the current picture as reference pictures. In this case, the later picture may be indexed first in the reference picture list 1, and the previous picture may be indexed next. Here, the output order may correspond to a picture order count (POC) order.

[0070] In addition, various inter prediction modes may be used when applying inter prediction to the current block. For example, various modes such as merge mode, skip mode, MVP (motion vector prediction) mode, affine mode, HMVP (historical motino vector prediction) mode, etc. may be used. DMVR (Decoder side motion vector refinement) mode, AMVR (adaptive motion vector resolution) mode, Bi-directional optical flow (BDOF), etc. may be further used as additional modes. Affine mode may also be referred to as affine motion prediction mode. MVP mode may also be referred to as advanced motion vector prediction mode. In this document, some modes and / or motion information candidates derived by some modes may be included as one of the motion information related candidates of other modes.

[0071] Prediction mode information indicating an inter prediction mode of a current block may be signaled from an encoding apparatus to a decoding apparatus. In this case, the prediction mode information may be included in a bitstream and received by the decoding apparatus. The prediction mode information may include index information indicating one of a number of candidate modes. Alternatively, the inter prediction mode may be indicated through hierarchical signaling of flag information. In this case, the prediction mode information may include one or more flags. For example, a skip flag may be signaled to indicate whether or not a skip mode is applied, a merge flag may be signaled to indicate whether or not a merge mode is applied when the skip mode is not applied, and an MVP mode may be indicated to be applied when the merge mode is not applied, or a flag for additional classification may be further signaled. The affine mode may be signaled in an independent mode, or may be signaled in a mode dependent on the merge mode or MVP mode. For example, the affine mode may include an affine merge mode and an affine MVP mode.

[0072] In addition, the motion information of the current block may be used when applying inter prediction to the current block. The encoding apparatus may derive optimal motion information for the current block through a motion estimation procedure. For example, the encoding apparatus may search for a similar reference block having high correlation using an original block in an original picture for the current block in a fractional pixel unit within a determined search range in the reference picture, and may derive motion information through this. The similarity of the blocks may be derived based on a difference in sample values ​​based on phase. For example, the similarity of the blocks may be calculated based on a sum of absolute differences (SAD) between the current block (or a template of the current block) and the reference block (or a template of the reference block). In this case, the motion information may be derived based on a reference block having the smallest SAD within the search range. The derived motion information may be signaled to the decoding apparatus in various ways based on the inter prediction mode.

[0073] As described above, a predicted block for the current block may be derived based on the motion information derived by the inter prediction mode. The predicted block may include a prediction sample (prediction sample array) of the current block. If the motion vector (MV) of the current block points to a fractional sample unit, an interpolation procedure may be performed, through which a prediction sample of the current block may be derived based on a reference sample of a fractional sample unit in a reference picture. If affine inter prediction is applied to the current block, a prediction sample may be generated based on a sample / sub-block unit MV. If bi-prediction is applied, a prediction sample derived through a weighted sum or weighted average (by phase) of a prediction sample derived based on L0 prediction (i.e., prediction using a reference picture in a reference picture list L0 and MVL0) and a prediction sample derived based on L1 prediction (i.e., prediction using a reference picture in a reference picture list L1 and MVL1) may be used as a prediction sample of the current block. When bi-prediction is applied, if the reference picture used for L0 prediction and the reference picture used for L1 prediction are located in different temporal directions from each other based on the current picture (i.e., if it corresponds to both bi-prediction and bidirectional prediction), this can be referred to as true bi-prediction.

[0074] As described above, reconstructed samples and reconstructed pictures can be generated based on the derived predicted samples, and then procedures such as in-loop filtering can be performed.

[0075] Meanwhile, the skip mode and / or merge mode show limitations in motion prediction because the motion of the current block is predicted based on the motion vector of an adjacent block without MVD (Motion Vector Difference). To improve the limitations of the skip mode and / or merge mode, the motion vector can be refined by applying DMVR (Decoder-side Motion Vector Refinement), BDOF (Bi-directional optical flow) mode, etc. The DMVR and BDOF modes can be used when true bi-prediction is applied to the current block.

[0076] FIG. 4 is a diagram illustrating an embodiment of a process of performing Decoder-side Motion Vector Refinement (DMVR) in true bi-prediction.

[0077] DMVR is a method of performing motion prediction by refining motion information of neighboring blocks on the decoder side. When DMVR is applied, the decoder can derive refined motion information through cost comparison based on a template generated using motion information of neighboring blocks in merge / skip mode. In this case, it is possible to improve the accuracy of motion prediction and improve compression performance without additional signaling information.

[0078] For convenience of explanation, this document will be described mainly with reference to a decoding device, but the DMVR according to the embodiments of this document can be implemented in the same manner in an encoding device.

[0079] 4, the decoding apparatus may derive a prediction block (i.e., a reference block) identified by initial motion vectors (or motion information) (e.g., MV0 and MV1) in the list0 and list1 directions, and generate a template (or a bilateral template) by weighting (e.g., averaging) the derived prediction blocks (step 1). Here, the initial motion vectors (MV0 and MV1) may indicate motion vectors derived using motion information of neighboring blocks in the merge / skip mode.

[0080] Then, the decoding apparatus can derive motion vectors (e.g., MV0′ and MV1′) that minimize the difference value between the template and the sample area of ​​the reference picture through a template matching operation (step 2). Here, the sample area indicates a neighboring area of ​​the initial prediction block in the reference picture, and the sample area may be called a neighboring area, a reference area, a search area, a search range, a search space, or the like. The template matching operation may include an operation of calculating a cost measurement value between the template and the sample area of ​​the reference picture. For example, SAD (sum of absolute differences) may be used for the cost measurement. As an example, a normalized SAD may be used as the cost function. In this case, the matching cost may be given as SAD(T-mean(T), 2*P[x]-2*mean(P[x])), where T indicates the template and P[x] indicates a block in the search area. Then, the motion vector that calculates the minimum template cost for each of the two reference pictures may be considered as an updated motion vector (replacing the initial motion vector). As shown in Fig. 4, the decoding apparatus may generate a final bi-directional prediction result (i.e., a final bi-directional prediction block) using the updated motion vectors MV0' and MV1'. As an example, multi-iteration for deriving the updated (or new) motion vector may be used to obtain the final bi-directional prediction result.

[0081] In one embodiment, the decoding device may invoke the DMVR process to improve the accuracy of initial motion compensation prediction (i.e., motion compensation prediction via conventional merge / skip mode). For example, the decoding device may perform the DMVR process when the prediction mode of the current block is merge mode or skip mode and bi-directional prediction in which both reference pictures are in opposite directions with respect to the current picture in display order is applied to the current block.

[0082] As described above, when true bi-prediction is applied to the current block, BDOF can be used to refine the bi-prediction signal. Bi-directional optical flow (BDOF) can be used to calculate improved motion information and generate prediction samples based on the motion information when bi-prediction is applied to the current block. For example, BDOF can be applied at a 4×4 sub-block level. That is, BDOF can be performed in units of 4×4 sub-blocks in the current block. Alternatively, BDOF can be applied only to the luma component. Alternatively, BDOF can be applied only to the chroma component, or can be applied to the luma component and the chroma component.

[0083] As the name suggests, the BDOF mode is based on the concept of optical flow, which assumes that the motion of objects is smooth. For each 4×4 sub-block, motion refinement (v x , v y ) can be computed, and motion refinement can be used to adjust the bi-predictive sample values ​​in the 4×4 sub-blocks.

[0084] The above-mentioned DMVR and BDOF are techniques for refining motion information and performing prediction when true bi-prediction is applied (wherein true bi-prediction refers to a case where motion prediction / compensation is performed using a reference picture in a different direction based on the picture of the current block), and can be seen as refinement techniques with a similar concept in that they assume that the movement of an object in a picture is performed at a constant speed and in a constant direction. However, when true bi-prediction is performed, since the conditions for applying DMVR and the conditions for applying BDOF are different, a process of performing condition checks several times for each technique must be repeated. Therefore, this document proposes a method for improving efficiency in terms of decoder complexity and performance by improving the process of performing condition checks when determining a prediction mode to be applied to a current block.

[0085] The following Table 1 shows conditions for applying DMVR in the existing true bi-prediction. DMVR can be applied when all the conditions listed below are met.

[0086] [Table 1]

[0087] Referring to Table 1, 1) whether or not DMVR is applicable may be determined based on flag information (e.g., sps_dmvr_enabled_flag) signaled in the sequence parameter set (SPS) syntax. Here, the flag information (e.g., sps_dmvr_enabled_flag) may indicate whether or not true bi-prediction-based DMVR is available. For example, if sps_dmvr_enabled_flag is 1 (i.e., true bi-prediction-based DMVR is available), it may be determined that the DMVR availability condition is met.

[0088] 2) Whether or not DMVR is applicable may be determined based on flag information (e.g., merge_flag) indicating whether inter prediction is performed using merge mode / skip mode. For example, if merge_flag is 1 (i.e., if inter prediction is performed using merge mode / skip mode), it may be determined that the merge mode / skip mode applicability condition is satisfied.

[0089] 3) Whether or not DMVR is applicable may be determined based on flag information (e.g., mmvd_flag) indicating whether inter prediction is performed using MMVD (merge mode with motion vector difference) mode. For example, if mmvd_flag is 0 (i.e., if MMVD mode is not used), it may be determined that the MMVD mode application condition is met.

[0090] 4) Whether or not DMVR is applicable may be determined based on whether bi-directional prediction (bi-prediction) is used. Here, bi-directional prediction may indicate inter-prediction performed based on reference pictures that exist in different directions with respect to a current picture. For example, if predFlagL0[0][0]=1 and predFlagL0[1][1]=1, it may be determined that bi-directional prediction is applied, and it may be determined that a bi-directional prediction availability condition is satisfied.

[0091] 5) In the case of true bi-prediction, it is possible to determine whether DMVR is applicable based on whether the distances between the current picture and the reference pictures in both directions are the same. That is, it is possible to determine whether the distance between the current picture and the LO reference picture (i.e., the reference picture in the reference picture list L0) and the distance between the current picture and the L1 reference picture (i.e., the reference picture in the reference picture list L1) are the same. For example, if DiffPicOrderCnt(currPic, RefPicList0[refIdxL0])-DiffPicOrderCnt(currPic, RefPicList1[refIdxL1])=0, it is determined that the picture is true bi-prediction and the distances between the current picture and the reference pictures in both directions are the same, and it is determined that the condition of whether the reference picture distances in both directions are the same is satisfied.

[0092] 6) It is possible to determine whether DMVR is applicable based on whether the height of the current block is greater than a threshold. For example, if the length of the current block is 8 or more, it can be determined that the current block meets the size (length) condition.

[0093] 7) It is possible to determine whether DMVR is applicable based on whether the size of the current block is greater than a threshold. For example, if the size of the current block, i.e., length (height) * width (width), is 64 or more, it can be determined that the current block size (length * width) condition is met.

[0094] The decoding device may determine whether to apply DMVR depending on whether conditions 1) to 7) in Table 1 are satisfied. That is, the decoding device may apply DMVR and perform true bi-prediction when all of conditions 1) to 7) in Table 1 are satisfied, and may not apply DMVR when any of the conditions in Table 1 are not satisfied.

[0095] The following Table 2 shows conditions for applying BDOF in the existing true bi-prediction. BDOF can be applied when all the conditions listed below are met.

[0096] [Table 2]

[0097] Referring to Table 2, 1) it can be determined whether BDOF is applicable based on flag information (e.g., sps_bdof_enabled_flag) signaled in the sequence parameter set (SPS) syntax. Here, the flag information (e.g., sps_bdof_enabled_flag) can indicate whether true bi-prediction-based BDOF is enabled. For example, if sps_bdof_enabled_flag is 1 (i.e., true bi-prediction-based BDOF is enabled), it can be determined that the BDOF availability condition is met.

[0098] 2) It may be possible to determine whether or not BDOF is applicable based on whether or not bidirectional prediction is used. Here, bidirectional prediction may indicate inter-prediction performed based on reference pictures that exist in different directions with respect to a current picture. For example, if predFlagL0 and predFlagL1 are both 1, it may be determined that bidirectional prediction is applied, and it may be determined that a bidirectional prediction availability condition is satisfied.

[0099] 3) It can determine whether BDOF is applicable based on whether it is true bi-prediction. That is, it can determine whether an LO reference picture (i.e., a reference picture in a reference picture list L0) and an L1 reference picture (i.e., a reference picture in a reference picture list L1) are located in different directions in time with respect to the current picture. For example, if DiffPicOrderCnt(currPic, refPicList0[refIdxL0])*DiffPicOrderCnt(currPic, refPicList1[refIdxL1]) is less than 0, it can be determined that the reference pictures in both directions are located in different directions with respect to the current picture, and it can be determined that the true bi-prediction availability condition is satisfied.

[0100] 4) It is possible to determine whether or not BDOF is applicable based on whether or not affine mode is used. Here, whether or not affine mode is used can be determined by deriving MotionModelIdc. For example, if the induced MotionModelIdc is 0, it can be determined that affine mode is not used, and in this case, it can be determined that the condition for whether or not affine mode is applicable is satisfied.

[0101] 5) It can determine whether BDOF is applicable based on flag information (e.g., merge_subblock_flag) indicating whether inter prediction in merge mode is performed on a subblock basis. For example, if merge_subblock_flag is 0 (i.e., merge mode is not applied on a subblock basis), it can be determined that the subblock-based merge mode applicability condition is satisfied.

[0102] 6) Whether or not BDOF is applicable can be determined based on whether GBi exists. Here, whether or not GBi exists can be determined based on GBi index information (e.g., GbiIdx). For example, if GbiIdx is 0 (i.e., GbiIdx is the default), it can be determined that the GBi applicability condition is met.

[0103] 7) It is possible to determine whether the current block is a luma block including a luma component and whether BDOF is applicable. For example, if an index (e.g., cIdx) indicating whether the current block is a luma block is 0 (i.e., if the current block is a luma block), it is possible to determine that the luma block applicability condition is met.

[0104] The decoding device may determine whether to apply BDOF depending on whether conditions 1) to 7) in Table 2 are satisfied. That is, the decoding device may apply BDOF and perform true bi-prediction if all of conditions 1) to 7) in Table 2 are satisfied, and may not apply BDOF if any of the conditions in Table 2 are not satisfied.

[0105] As mentioned above, some of the conditions for applying DMVR and BDOF are the same, while others are similar or different. In existing methods, even if the conditions are the same, the condition check is performed for each technology separately, which increases the complexity of performing bi-prediction. Therefore, this paper proposes efficient conditions for applying DMVR and BDOF during bi-prediction.

[0106] Since the merge / skip mode has a relatively low motion accuracy compared to the AMVP mode, it is effective in terms of performance to refine the motion information using the DMVR method. However, unlike DMVR, the BDOF mode is applied not only in the merge / skip mode but also in the AMVP mode, so when BDOF is applied in the AMVP mode, the complexity of BDOF execution may increase compared to performance. Therefore, in this embodiment, a method is proposed to apply the merge / skip mode to the BDOF case as well as the DMVR.

[0107] In this case, as an embodiment proposed in this document, the application conditions of BDOF may include the conditions presented in Table 3 below.

[0108] [Table 3]

[0109] Referring to Table 3, whether or not BDOF is applicable can be determined based on flag information (e.g., merge_flag) indicating whether inter prediction is performed using merge mode / skip mode. For example, if merge_flag is 1 (i.e., if inter prediction is performed using merge mode / skip mode), it can be determined that the merge mode / skip mode application condition is satisfied. Therefore, like DMVR, BDOF can also be applied when the merge / skip mode is selected.

[0110] In other words, in this embodiment, in addition to the conditions to be applied in the merge / skip mode, it is possible to determine whether or not BDOF is applicable based on conditions to be applied when BDOF is available, conditions to be applied when bidirectional prediction is used, conditions to be applied when true bi-prediction is used, conditions to be applied when affine prediction is not used, conditions to be applied when sub-block based merge mode is not used, conditions to be applied when the GBi index is default, and conditions to be applied when the block is a luma block.

[0111] Therefore, the decoding apparatus may determine whether all the conditions listed in Table 3 are met, and if all the conditions are met, may apply BDOF to perform true bi-prediction. If any of the conditions listed in Table 3 are not met, the decoding apparatus may not apply BDOF. The conditions in Table 3 may also be applied to the encoding apparatus, and may be executed in a manner corresponding to that of the decoding apparatus.

[0112] The method of performing true bi-prediction based on the conditions listed in Table 3 may be applied independently to DMVR and BDOF, or may be applied to the same conditions for DMVR and BDOF.

[0113] Meanwhile, in the case where the conditions listed in this embodiment are the same as those described in Tables 1 and 2, the specific operations and meanings are the same, so a detailed description of each condition will be omitted. Also, duplicated content will be omitted in the following embodiments.

[0114] Encoding / decoding devices can be configured using various hardware, and preferences for performance versus complexity ratio may vary. Therefore, this embodiment proposes a method for refining motion information by applying DMVR not only in merge / skip mode but also in AMVP mode.

[0115] In this case, as one embodiment proposed in this document, the application conditions of the DMVR may include the conditions presented in Table 4 below.

[0116] [Table 4]

[0117] Referring to Table 4, the process of determining whether DMVR is applied based on flag information (e.g., merge_flag) indicating whether inter prediction is performed using merge mode / skip mode can be omitted. In this manner, by omitting the condition of whether merge mode / skip mode is applied, DMVR can be applied not only in the merge mode / skip mode but also in the AMVP mode.

[0118] According to Table 4, the applicability of DMVR can be determined based on the conditions to be applied when DMVR is available, the conditions to be applied when MMVD mode is not used, the conditions to be applied in the case of bidirectional prediction, the conditions to be applied in the case of true bi-prediction where the distance between the current picture and both reference pictures is the same, the conditions to be applied when the length of the current block is 8 or more, and the conditions to be applied when the size (length * width) of the current block is 64 or more.

[0119] That is, the decoding apparatus determines whether all the conditions listed in Table 4 are satisfied, and if all the conditions are satisfied, applies DMVR to perform true bi-prediction. If any one of the conditions listed in Table 4 is not satisfied, the decoding apparatus does not apply DMVR. The conditions in Table 4 may also be applied to the encoding apparatus, and may be executed in a manner corresponding to that of the decoding apparatus.

[0120] Based on the conditions listed in Table 4, the method of performing true bi-prediction may be applied independently to DMVR and BDOF, or may be applied to the same conditions for DMVR and BDOF.

[0121] Meanwhile, as an embodiment of this document, DMVR and BDOF can be applied to all normal merge modes. That is, DMVR and BDOF can be applied when the mode is not an ATMVP (advanced temporal motion vector prediction) mode, not an affine mode, or not a CPR mode. In this case, the application conditions of DMVR can include the conditions presented in Table 5 below.

[0122] [Table 5]

[0123] Referring to Table 5, by determining whether the conditions to be applied when affine mode is not used (e.g., when MotionModelIdc is 0) and the conditions to be applied when sub-block based merge mode is not used (e.g., when merge_subblock_flag is 0) are met, DMVR can be applied only when general merge mode is used.

[0124] In addition, in this embodiment, in addition to the affine mode availability condition and sub-block based merge mode availability condition, the applicability of DMVR can be determined based on conditions to be applied when DMVR is available, conditions to be applied when merge mode / skip mode is used, conditions to be applied when MMVD mode is not used, conditions to be applied when bidirectional prediction is used, conditions to be applied when true bi-prediction is used in which the distance between the current picture and both reference pictures is the same, conditions to be applied when the length of the current block is 8 or more, and conditions to be applied when the size of the current block (length * width) is 64 or more.

[0125] That is, the decoding apparatus determines whether all the conditions listed in Table 5 are satisfied, and if all the conditions are satisfied, applies DMVR to perform true bi-prediction. If any one of the conditions listed in Table 5 is not satisfied, the decoding apparatus does not apply DMVR. The conditions in Table 5 may also be applied to the encoding apparatus, and may be executed in a manner corresponding to that of the decoding apparatus.

[0126] Among the application conditions in Table 5, the sub-block based merge mode application condition (e.g., merge_subblock_flag) includes overlapping conditions among the existing DMVR application conditions. Therefore, in one embodiment of this document, the overlapping conditions with the sub-block based merge mode application condition (e.g., merge_subblock_flag) may be removed. In this case, the corresponding condition may be removed as proposed in Table 6 below.

[0127] [Table 6]

[0128] Referring to Table 6, the sub-block based merge mode can be applied when the size of the current block is 8x8 or more. Therefore, since the sub-block based merge mode application condition (e.g., merge_subblock_flag=0) includes a condition related to the size of the current block, the condition related to the size of the current block (e.g., CbHeight, CbHeight*CbWidth) can be excluded from the existing DMVR application conditions. For example, the condition of whether the length of the current block is 8 or more and the condition of whether the length*width of the current block is 64 or more can be omitted, and the application of the DMVR can be determined using other conditions listed in Table 6.

[0129] That is, the decoding apparatus determines whether all the conditions listed in Table 6 are satisfied, and if all the conditions are satisfied, applies DMVR to perform true bi-prediction. If any one of the conditions listed in Table 6 is not satisfied, the decoding apparatus does not apply DMVR. The conditions in Table 6 may also be applied to the encoding apparatus, and may be executed in a manner corresponding to that of the decoding apparatus.

[0130] The method of performing true bi-prediction based on the conditions listed in Table 5 or Table 6 may be applied independently to DMVR and BDOF, or may be applied to the same conditions for DMVR and BDOF.

[0131] In one embodiment of this document, in order to increase the accuracy of motion vectors with low complexity, refinement techniques such as DMVR and BDOF are not applied when the block size is small. In the existing method, refinement techniques are applied when the current block is larger than or equal to 8x8, and in the case of DMVR, when the size of the current block is large, refinement is applied in units of 16x16, so DMVR is not applied to blocks smaller than 16x16. In this case, the conditions for applying DMVR may include the conditions presented in Table 7 below.

[0132] [Table 7]

[0133] Referring to Table 7, by changing the conditions (e.g., CbHeight, CbWidth) related to the size of the current block, it is possible to prevent DMVR from being applied to blocks smaller than 16×16. For example, a condition to be applied when the length (e.g., CbHeight) of the current block is equal to or greater than 16 and a condition to be applied when the width (e.g., CbWidth) of the current block is equal to or greater than 16 can be used. When such an application condition related to the size of the current block is satisfied (i.e., the size of the current block is equal to or greater than 16×16), DMVR is applied, and when such an application condition related to the size of the current block is not satisfied (i.e., the size of the current block is smaller than 16×16), DMVR is not applied.

[0134] In addition, in this embodiment, it is possible to determine whether or not DMVR is applicable based on the remaining application conditions listed in Table 7 together with the conditions related to the size of the current block (eg, CbHeight, CbWidth).

[0135] That is, the decoding apparatus determines whether all the conditions listed in Table 7 are satisfied, and if all the conditions are satisfied, applies DMVR to perform true bi-prediction. If any one of the conditions listed in Table 7 is not satisfied, the decoding apparatus does not apply DMVR. The conditions in Table 7 may also be applied to the encoding apparatus, and may be executed in a manner corresponding to that of the decoding apparatus.

[0136] As an example of this document, if the current block is smaller than 16x16, not only DMVR but also BDOF is not applied. In this case, the application conditions of BDOF may include the conditions presented in Table 8 below.

[0137] [Table 8]

[0138] Referring to Table 8, by changing the conditions (e.g., CbHeight, CbWidth) related to the size of the current block, BDOF may not be applied to blocks smaller than 16x16. For example, a condition may be used that is applied when the length (e.g., CbHeight) of the current block is equal to or greater than 16, and a condition may be used that is applied when the width (e.g., CbWidth) of the current block is equal to or greater than 16. BDOF is applied when such an application condition related to the size of the current block is met (i.e., the size of the current block is equal to or greater than 16x16), and BDOF is not applied when such an application condition related to the size of the current block is not met (i.e., the size of the current block is smaller than 16x16).

[0139] In addition, in this embodiment, it is possible to determine whether BDOF is applicable based on the remaining application conditions listed in Table 8 together with the conditions related to the size of the current block (eg, CbHeight, CbWidth).

[0140] That is, the decoding apparatus determines whether all the conditions listed in Table 8 are satisfied, and if all the conditions are satisfied, the decoding apparatus applies BDOF to perform true bi-prediction. If any one of the conditions listed in Table 8 is not satisfied, the decoding apparatus does not apply BDOF. The conditions in Table 8 may also be applied to the encoding apparatus, and may be executed in a manner corresponding to that of the decoding apparatus.

[0141] The method of performing true bi-prediction based on the conditions listed in Table 7 or Table 8 may be applied independently to DMVR and BDOF, or may be applied to the same conditions for DMVR and BDOF.

[0142] As described above, DMVR is applied when the distances between the current picture and the reference pictures in both directions are the same, whereas BDOF is always applied in true bi-prediction even if the distances between the current picture and the reference pictures in both directions are different. Therefore, in order to improve coding efficiency, this document proposes a method for unifying conditions related to the reference picture distances in both directions and applying them to DMVR and BDOF.

[0143] As an example of this document, the application conditions of BDOF may include the conditions presented in Table 9 below.

[0144] [Table 9]

[0145] Referring to Table 9, by changing a condition related to the reference picture distance (e.g., DiffPicOrderCnt) among the conditions for applying BDOF, the corresponding condition can be applied to DMVR and BDOF in the same way. For example, by determining whether DiffPicOrderCnt(currPic, RefPicList0[refIdxL0])-DiffPicOrderCnt(currPic, RefPicList1[refIdxL1]) is 0, it can be determined whether the distance between the current picture and the LO reference picture (i.e., the reference picture in the reference picture list L0) and the distance between the current picture and the L1 reference picture (i.e., the reference picture in the reference picture list L1) are the same. That is, BDOF can be applied only when the distances between the current picture and the reference pictures in both directions are the same. In this way, the BDOF application range is limited according to the addition of a condition that the reference picture distances in both directions are the same, so that the decoding complexity can be reduced.

[0146] In addition, in this embodiment, it can be determined whether or not BDOF is applicable based on the remaining application conditions listed in Table 9 together with the conditions related to the reference picture distance (e.g., DiffPicOrderCnt).

[0147] That is, the decoding apparatus determines whether all the conditions listed in Table 9 are satisfied, and if all the conditions are satisfied, the decoding apparatus applies BDOF to perform true bi-prediction. If any one of the conditions listed in Table 9 is not satisfied, the decoding apparatus does not apply BDOF. The conditions in Table 9 may also be applied to the encoding apparatus, and may be executed in a manner corresponding to that of the decoding apparatus.

[0148] As an example of this document, the application conditions of the DMVR may include the conditions presented in Table 10 below.

[0149] [Table 10]

[0150] Referring to Table 10, by changing a condition related to the reference picture distance (e.g., DiffPicOrderCnt) among the conditions for applying DMVR, the corresponding condition can be applied to DMVR and BDOF in the same way. For example, by determining whether DiffPicOrderCnt(currPic, refPicList0[refIdxL0])*DiffPicOrderCnt(currPic, refPicList1[refIdxL1]) is less than 0, it can be determined whether the reference pictures in both directions (i.e., the LO reference picture and the L1 reference picture) are located in different directions based on the current picture. That is, even if the distance between the current picture and the reference pictures in both directions is not the same, the DMVR can always be applied in the case of true bi-prediction. In this way, even if the reference picture distances in both directions are different according to the application of the true bi-prediction availability condition, the motion vector derived in consideration of the decoding complexity can be used without scaling.

[0151] In addition, in this embodiment, it is possible to determine whether or not DMVR is applicable based on the remaining application conditions listed in Table 10 together with the conditions related to the reference picture distance (eg, DiffPicOrderCnt).

[0152] That is, the decoding apparatus determines whether all the conditions listed in Table 10 are satisfied, and if all the conditions are satisfied, applies DMVR to perform true bi-prediction. If any one of the conditions listed in Table 10 is not satisfied, the decoding apparatus does not apply DMVR. The conditions in Table 10 may also be applied to the encoding apparatus, and may be executed in a manner corresponding to that of the decoding apparatus.

[0153] The method of performing true bi-prediction based on the conditions listed in Table 9 or Table 10 may be applied independently to DMVR and BDOF, or may be applied to the same conditions for DMVR and BDOF.

[0154] Meanwhile, each reference block may perform motion compensation using a weighted sum due to changes in light, etc. In this case, this phenomenon can be grasped using GBi or LIC (local illumination compensation), so the application conditions of DMVR and BDOF can be determined taking into account the GBi and LIC conditions.

[0155] As an example of this document, we propose a method to determine whether or not DMVR is applicable, taking into account the conditions of GBi and LIC. In this case, the conditions for applying DMVR may include the conditions shown in Table 11 below.

[0156] [Table 11]

[0157] Referring to Table 11, the GBi condition (e.g., GbiIdx) and the LIC condition (e.g., LICFlag) may be added to determine whether DMVR is applied. Here, GBi may indicate generalized bi-prediction in which different weights may be applied to L0 prediction and L1 prediction, and may be indicated using, for example, GbiIdx. GbiIdx may exist in the case of bi-prediction and may indicate a bi-prediction weight index. For example, GbiIdx may be derived from a neighboring block in the case of merge mode, or may be signaled from the encoding device to the decoding device via a GbiIdx syntax element (e.g., gbi_idx) in the case of MVP mode. As an example, GbiIdx may indicate a weight w applied to L1 prediction, in which case a weight of (1-w) may be applied to L0 prediction. As another example, GbiIdx may indicate a weighting value w applied to L0 prediction, in which case a weighting value of (1-w) may be applied to L1 prediction.

[0158] According to Table 11, when GbiIdx is 0 (i.e., when GbiIdx is the default), it is determined that the GBi availability condition is met, and when LICFlag is 0 (i.e., when LIC exists), it is determined that the LIC availability condition is met.

[0159] In addition, in this embodiment, the applicability of DMVR can be determined based on the remaining applicability conditions listed in Table 11 in addition to the GBi condition (eg, GbiIdx) and the LIC condition (eg, LICFlag).

[0160] That is, the decoding apparatus determines whether all the conditions listed in Table 11 are satisfied, and if all the conditions are satisfied, applies DMVR to perform true bi-prediction. If any one of the conditions listed in Table 11 is not satisfied, the decoding apparatus does not apply DMVR. The conditions in Table 11 may also be applied to the encoding apparatus, and may be executed in a manner corresponding to that of the decoding apparatus.

[0161] As an example of this document, we propose a method to determine whether BDOF is applicable by taking into account GBi and LIC conditions. In this case, the conditions for applying BDOF may include the conditions presented in Table 12 below.

[0162] [Table 12]

[0163] According to Table 12, the application of BDOF can be determined by adding a LIC condition (e.g., LICFlag) along with the existing GBi condition (e.g., GbiIdx). For example, if GbiIdx is 0 (i.e., GbiIdx is the default), it can be determined that the GBi availability condition is met, and if LICFlag is 0 (i.e., LIC exists), it can be determined that the LIC availability condition is met.

[0164] Therefore, in this embodiment, the applicability of BDOF can be determined based on the remaining applicability conditions listed in Table 12 in addition to the GBi condition (eg, GbiIdx) and the LIC condition (eg, LICFlag).

[0165] That is, the decoding apparatus determines whether all the conditions listed in Table 12 are satisfied, and if all the conditions are satisfied, the decoding apparatus applies BDOF to perform true bi-prediction. If any one of the conditions listed in Table 12 is not satisfied, the decoding apparatus does not apply BDOF. The conditions in Table 12 may also be applied to the encoding apparatus, and may be executed in a manner corresponding to that of the decoding apparatus.

[0166] The method of performing true bi-prediction based on the conditions listed in Table 11 or Table 12 may be applied independently to DMVR and BDOF, or may be applied to the same conditions for DMVR and BDOF.

[0167] Meanwhile, DMVR and BDOF induce motion information through a refinement process in the decoding device, which causes a problem of decoding complexity. Therefore, this paper proposes a method that can reduce the decoding complexity by determining whether to apply DMVR and BDOF using a merge index. In this case, since DMVR and BDOF refine motion vectors within a limited range, the effect of refinement may be reduced if the motion vector is inaccurate. Therefore, this paper proposes a method that can be applied only when the value indicated by the merge index is small, taking into account the efficiency of refinement.

[0168] Here, the merge index is a syntax element signaled from the encoding apparatus to the decoding apparatus. For example, when a merge mode / skip mode is applied to a current block, the encoding / decoding apparatus may configure a merge candidate list based on neighboring blocks of the current block. In this case, the encoding apparatus may select an optimal merge candidate from among the merge candidates included in the merge candidate list based on a rate-distortion (RD) cost, and signal merge index information indicating the selected merge candidate to the decoding apparatus. The decoding apparatus may select a merge candidate to be applied to the current block based on the merge candidate list and the merge index information.

[0169] As an embodiment of this document, a method for determining whether DMVR is applicable using a merge index may include the conditions presented in Table 13 below.

[0170] [Table 13]

[0171] Referring to Table 13, a merge index condition (e.g., merge_idx) can be added to determine whether DMVR is applicable. For example, it can be determined that the merge index condition is satisfied if the merge index (e.g., merge_idx) is less than 2. Here, the merge index value (threshold) is set to 2, but this is merely an example and the corresponding value can be changed depending on coding efficiency.

[0172] Therefore, in this embodiment, it is possible to determine whether or not DMVR is applicable based on the remaining application conditions listed in Table 13 together with the merge index condition (eg, merge_idx).

[0173] That is, the decoding apparatus determines whether all the conditions listed in Table 13 are satisfied, and if all the conditions are satisfied, applies DMVR to perform true bi-prediction. If any one of the conditions listed in Table 13 is not satisfied, the decoding apparatus does not apply DMVR. The conditions in Table 13 may also be applied to the encoding apparatus, and may be executed in a manner corresponding to that of the decoding apparatus.

[0174] As an embodiment of this document, a method for determining whether BDOF is applicable using a merge index may include the conditions presented in Table 14 below.

[0175] [Table 14]

[0176] Referring to Table 14, a merge mode / skip mode condition (e.g., merge_flag) and a merge index condition (e.g., merge_idx) may be added to determine whether BDOF is applied. For example, if merge_flag is 1 (i.e., if inter prediction is performed using merge mode / skip mode) and merge_idx is not 2 or more, it may be determined that the condition for applying BDOF is met only when the merge index value is small. That is, if merge_flag is 1 (i.e., if inter prediction is performed using merge mode / skip mode) and merge_idx is smaller than 2, it may be determined that the merge index condition is met and BDOF is applied. Here, the merge index value (threshold) is set to 2, but this is merely an example and the corresponding value may be changed according to coding efficiency.

[0177] That is, in this embodiment, whether or not BDOF is applicable can be determined based on the merge mode / skip mode availability condition (e.g., merge_flag) and the merge index condition (e.g., merge_idx) as well as the remaining application conditions listed in Table 14.

[0178] That is, the decoding apparatus determines whether all the conditions listed in Table 14 are satisfied, and if all the conditions are satisfied, the decoding apparatus applies BDOF to perform true bi-prediction. If any one of the conditions listed in Table 14 is not satisfied, the decoding apparatus does not apply BDOF. The conditions in Table 14 may also be applied to the encoding apparatus, and may be executed in a manner corresponding to that of the decoding apparatus.

[0179] The method of performing true bi-prediction based on the conditions listed in Table 13 or Table 14 may be applied independently to DMVR and BDOF, or may be applied to the same conditions for DMVR and BDOF.

[0180] Meanwhile, in merge / skip mode, the motion information can be refined via the MMVD, and in this case, since the decoder complexity increases, DMVR is not performed when the MMVD is applied. However, when applied without considering the MMVD, DMVR can be applied even without the MMVD condition in consideration of performance improvement. In this case, according to one embodiment of this document, the DMVR application conditions can include the conditions presented in Table 15 below.

[0181] [Table 15]

[0182] Referring to Table 15, the MMVD mode application condition (e.g., mmvd_flag) can be excluded from the existing DMVR application conditions. That is, the process of determining whether mmvd_flag is 0 (i.e., the MMVD mode is not used) can be omitted, and the application of DMVR can be determined based on the conditions listed in Table 15.

[0183] That is, the decoding device determines whether all the conditions listed in Table 15 (excluding the MMVD mode application condition (e.g., mmvd_flag)) are satisfied, and if all the conditions are satisfied, the decoding device applies the DMVR to perform true bi-prediction. If any one of the conditions listed in Table 15 is not satisfied, the decoding device does not apply the DMVR. The conditions in Table 15 may also be applied to the encoding device, and may be executed in the encoding device in a manner corresponding to that of the decoding device.

[0184] Also, contrary to the embodiment of Table 15, when refining motion information through MMVD in merge / skip mode, it is possible to determine whether to perform BDOF by considering decoder complexity. Therefore, this document proposes a method for determining whether to apply BDOF by considering MMVD conditions. In this case, according to one embodiment of this document, the conditions for applying BDOF may include the conditions presented in Table 16 below.

[0185] [Table 16]

[0186] Referring to Table 16, by adding an MMVD mode application condition (e.g., mmvd_flag), BDOF is not applied when refining motion information through MMVD. For example, if mmvd_flag is 0 (i.e., if the MMVD mode is not used), it can be determined that the MMVD mode application condition is met.

[0187] Therefore, in this embodiment, it is possible to determine whether or not BDOF is applicable based on the remaining application conditions listed in Table 16 together with the MMVD mode application condition (e.g., mmvd_flag).

[0188] That is, the decoding apparatus determines whether all the conditions listed in Table 16 are satisfied, and if all the conditions are satisfied, it can apply BDOF to perform true bi-prediction. If any one of the conditions listed in Table 16 is not satisfied, the decoding apparatus does not apply BDOF. The conditions in Table 16 can also be applied to the encoding apparatus, and the encoding apparatus can be executed in a manner corresponding to the decoding apparatus.

[0189] The method of performing true bi-prediction based on the conditions listed in Table 15 or Table 16 may be applied independently to DMVR and BDOF, or may be applied to the same conditions for DMVR and BDOF.

[0190] Meanwhile, in the AMVP mode, the Adaptive Motion Vector Resolution (AMVR) technology can be applied. In this case, when the resolution of the motion vector is large, i.e., when it has integer-pel rounding or 4-sample rounding, it is not suitable to apply such a technology to the BDOF that performs refinement within a limited area. Therefore, this document proposes a method that can determine the execution of BDOF according to the AMVR condition. In this case, according to one embodiment of this document, the application condition of BDOF can include the conditions presented in Table 17 below.

[0191] [Table 17]

[0192] Referring to Table 17, the AMVR condition (e.g., amvr_flag) can be added to determine whether BDOF is applied. Here, amvr_flag is information indicating the resolution of a motion vector difference (MVD). For example, if amvr_flag is 0, it indicates that the MVD resolution is induced in quarter-luma-sample units, and if amvr_flag is not 0, it may indicate that the MVD resolution is induced in integer-luma-sample units or four-luma-sample units. Alternatively, it may be determined in the opposite case. According to an embodiment, as presented in Table 17, if amvr_flag is not 0, it is possible to set a condition that BDOF is applied. That is, if amvr_flag is 0, it is possible to restrict BDOF from being applied.

[0193] Therefore, in this embodiment, whether or not BDOF is applicable can be determined based on the AMVR condition (e.g., amvr_flag) as well as other conditions listed in Table 17.

[0194] That is, the decoding apparatus determines whether all the conditions listed in Table 17 are satisfied, and if all the conditions are satisfied, it can apply BDOF to perform true bi-prediction. If any one of the conditions listed in Table 17 is not satisfied, the decoding apparatus does not apply BDOF. The conditions in Table 17 can also be applied to the encoding apparatus, and the encoding apparatus can be executed in a manner corresponding to the decoding apparatus.

[0195] Meanwhile, the DMVR and BDOF as described above can be signaled by the SPS (Sequence Parameter Set) syntax, respectively. Table 18 below shows an example of a syntax element indicating whether the DMVR and BDOF are available, which are signaled via the SPS syntax.

[0196] [Table 18]

[0197] Referring to Table 18, sps_dmvr_enabled_flag can be signaled in the SPS syntax, and whether a true bi-prediction-based DMVR is enabled can be indicated based on this syntax element. For example, when sps_dmvr_enabled_flag is 1, it indicates that a true bi-prediction-based DMVR is enabled, and when sps_dmvr_enabled_flag is 0, it indicates that a true bi-prediction-based DMVR is not enabled.

[0198] In addition, sps_bdof_enabled_flag may be signaled in the SPS syntax, and whether true bi-prediction-based BDOF is enabled may be indicated based on this syntax element. For example, sps_bdof_enabled_flag may indicate that true bi-prediction-based BDOF is enabled when it is 1, and may indicate that true bi-prediction-based BDOF is not enabled when it is 0.

[0199] As shown in Table 18, the application conditions of DMVR and BDOF can be checked using a syntax element (e.g., sps_dmvr_enabled_flag) indicating whether DMVR is available and a syntax element (e.g., sps_bdof_enabled_flag) indicating whether BDOF is available.

[0200] FIG. 5 is a diagram showing an example of a method for checking the application conditions of DMVR and BDOF and performing a decoding process.

[0201] The method of FIG. 5 can be applied when using a syntax element indicating whether DMVR is available (e.g., sps_dmvr_enabled_flag) and a syntax element indicating whether BDOF is available (e.g., sps_bdof_enabled_flag) as shown in Table 18.

[0202] Referring to FIG. 5, a decoding apparatus can derive motion information (eg, a motion vector, a reference picture index, etc.) for a current block (S500).

[0203] The decoding device may check the application conditions of the DMVR (S510). At this time, the application conditions of the DMVR may be checked based on a syntax element (e.g., sps_dmvr_enabled_flag) indicating whether the DMVR is available. For example, the application conditions of the DMVR may be checked when the DMVR is available (e.g., when sps_dmvr_enabled_flag is 1).

[0204] The decoding device can determine whether to apply the DMVR process depending on whether the DMVR application condition is met (S520).

[0205] If all the conditions for applying DMVR are satisfied, the decoding apparatus may derive refined motion information by applying the DMVR process (S530). If one or more of the conditions for applying DMVR are not satisfied, the decoding apparatus does not apply the DMVR process.

[0206] The decoding device may derive a predictive sample of the current block based on refined motion information derived when DMVR is applied or (unrefined) motion information derived when DMVR is not applied (S540).

[0207] Then, the decoding apparatus may check the application condition of the BDOF (S550). At this time, the decoding apparatus may check the application condition of the BDOF based on a syntax element (e.g., sps_bdof_enabled_flag) indicating whether the BDOF is available. For example, the decoding apparatus may check the application condition of the BDOF when the BDOF is available (e.g., sps_bdof_enabled_flag is 1).

[0208] If all the conditions for applying BDOF are met, the decoding apparatus may apply the BDOF process to refine the predicted samples (S570). If one or more of the conditions for applying BDOF are not met, the decoding apparatus does not apply the BDOF process.

[0209] The decoding device derives a residual sample for the current block (S580), and can derive a reconstructed sample based on the residual sample and a refined prediction sample derived when the BDOF is applied, or a (non-refined) prediction sample derived when the BDOF is not applied (S590).

[0210] As mentioned above, this document proposes various embodiments that can improve coding efficiency and reduce complexity by harmonizing the application conditions between DMVR and BDOF when applying them. When checking the application conditions of DMVR and BDOF according to the embodiments of this document and applying them to the decoding process, each condition can be checked and applied separately, but the application conditions can also be checked at the same time to improve coding efficiency. That is, this document proposes a method for integrating and checking the application conditions of DMVR and BDOF at the same time.

[0211] As an embodiment of this document, the DMVR / BDOF application condition check process can be performed by signaling information (e.g., sps_refinement_enabled_flag) indicating whether or not refinement is applied in the decoding device in the SPS (Sequence Parameter Set) syntax. Table 19 below shows an example of a syntax element (e.g., sps_refinement_enabled_flag) indicating whether or not refinement is applied in the decoding device, which is signaled via the SPS syntax.

[0212] [Table 19]

[0213] Referring to Table 19, sps_refinement_enabled_flag can be signaled in the SPS syntax, and whether or not refinement can be applied in the decoding device can be indicated based on this syntax element. For example, when sps_refinement_enabled_flag exists (i.e., sps_refinement_enabled_flag is true), it can be determined that refinement can be applied in the decoding device. In this case, the decoding device can obtain the sps_dmvr_enabled_flag syntax element indicating whether DMVR is enabled and the sps_bdof_enabled_flag syntax element indicating whether BDOF is enabled, and determine the application conditions of DMVR and BDOF.

[0214] 6 and 7 are diagrams showing another example of a method for checking the application conditions of DMVR and BDOF and performing a decoding process.

[0215] The methods of Figures 6 and 7 can be applied when using a syntax element (e.g., sps_refinement_enabled_flag) indicating whether refinement is applicable in the decoding device as shown in Table 19. Also, in Figures 6 and 7, the description of the contents overlapping with Figure 5 will be omitted.

[0216] Referring to Figure 6, it can be seen that the application conditions of DMVR and BDOF are checked at once in the previous step, instead of checking each of them separately, as compared with the process of Figure 5. As an example, in step S610 of Figure 6, the decoding apparatus can check the refinement application condition. At this time, the refinement application condition check can be performed based on sps_refinement_enabled_flag as shown in Table 19. For example, when sps_refinement_enabled_flag is 1, the sps_dmvr_enabled_flag syntax element indicating whether DMVR is enabled or the sps_bdof_enabled_flag syntax element indicating whether BDOF is enabled can be obtained, and the application conditions of DMVR and BDOF can be checked based on the sps_refinement_enabled_flag syntax element.

[0217] Also, referring to Fig. 7, in comparison with the process of Fig. 5, the application conditions of DMVR and BDOF are not checked separately, but are checked at once in the previous step, and a simple check process (application condition of BDOF) can be performed for different conditions. As an example, in step S710 of Fig. 7, the decoding device can check the application condition of refinement. Thereafter, in step S750 of Fig. 7, the decoding device can additionally perform a simple check for the application condition of BDOF having a different condition from the application condition of refinement.

[0218] The methods listed in Tables 3 to 17 according to the embodiment of the present document can be applied in combination. That is, by checking the refinement application conditions, it is possible to determine whether DMVR and BDOF are applicable, and the conditions in Table 20 below can be applied.

[0219] [Table 20]

[0220] Referring to Table 20, when DMVR or BDOF is applied, the following conditions can be used: a condition to be applied in the case of bi-directional prediction, a condition to be applied in the case of true bi-prediction where the distances between the current picture and the bi-directional reference picture are the same, a condition to be applied when the MMVD mode is not used, a condition to be applied when affine prediction is not used, a condition to be applied when sub-block based merge mode is not used, and a condition to be applied when the GBi index is default. That is, whether or not DMVR or BDOF is applied can be determined depending on whether or not the above conditions are met.

[0221] Also, as described above, conditions may be added such as determining whether it is a merge mode for DMVR or whether it is a luma block for BDOF.

[0222] It is obvious that the application conditions listed in Table 20 are merely examples, and various conditions listed in the above-mentioned embodiments (the embodiments of Tables 3 to 17) can be used in combination.

[0223] FIG. 8 is a flow chart that illustrates a schematic of an encoding method that can be performed by an encoding device according to an embodiment of the present document.

[0224] The method disclosed in FIG. 8 may be performed by the encoding device 200 disclosed in FIG. 2. Specifically, steps S800 to S830 in FIG. 8 may be performed by the prediction unit 220 and the inter prediction unit 221 disclosed in FIG. 2, steps S840 to S850 in FIG. 8 may be performed by the residual processing unit 230 disclosed in FIG. 2, and step S860 in FIG. 8 may be performed by the entropy encoding unit 240 disclosed in FIG. 2. In addition, the method disclosed in FIG. 8 may include the embodiments described in detail in this document. Therefore, in FIG. 8, detailed descriptions of contents overlapping with the embodiments described in detail will be omitted or simplified.

[0225] Referring to FIG. 8, the encoding apparatus derives a motion vector for a current block (S800).

[0226] As an example, when performing inter prediction on a current block, the encoding apparatus may derive motion information (motion vector, reference picture index, etc.) of the current block. For example, the encoding apparatus may search for a block similar to the current block within a certain region (search region) of a reference picture through motion estimation, and derive a reference block whose difference with the current block is minimum or equal to or less than a certain criterion. Based on this, the encoding apparatus may derive a reference picture index indicating a reference picture in which the reference block is located, and derive a motion vector based on the position difference between the reference block and the current block.

[0227] In addition, the encoding apparatus may determine an inter prediction mode to be applied to the current block from among various prediction modes, and may determine an optimal prediction mode for the current block by comparing RD costs for various prediction modes.

[0228] For example, when a skip mode or a merge mode is applied to the current block, the encoding apparatus may configure a merge candidate list. In this case, a reference block whose difference with the current block is minimum or equal to or less than a certain criterion among reference blocks indicated by merge candidates included in the merge candidate list may be derived. In this case, a merge candidate associated with the derived reference block may be selected, and merge index information indicating the selected merge candidate may be generated and signaled to the decoding apparatus. Motion information of the current block may be derived using motion information of the selected merge candidate.

[0229] As another example, when the (A) MVP mode is applied to the current block, the encoding apparatus may construct an (A) MVP candidate list, and may use a motion vector of an MVP candidate selected from the MVP (motion vector predictor) candidates included in the (A) MVP candidate list as the MVP of the current block. For example, a motion vector indicating a reference block derived by the above-mentioned motion estimation may be used as the motion vector of the current block, and an MVP candidate having a motion vector with the smallest difference from the motion vector of the current block derived by motion estimation may be selected from the MVP candidates. The encoding apparatus may then derive a motion vector difference (MVD), which is a difference obtained by subtracting the motion vector of the selected MVP candidate from the motion vector of the current block. In this case, information regarding the MVD may be signaled to the decoding apparatus. Also, when the (A) MVP mode is applied, a value of a reference picture index may be configured as reference picture index information and separately signaled to the decoding apparatus.

[0230] The encoding apparatus may determine whether to apply refinement to the motion vector of the current block (S810).

[0231] That is, the encoding apparatus may determine whether to apply refinement in consideration of coding efficiency, complexity, prediction performance, etc. In this case, the encoding apparatus may determine whether to apply refinement to the current block using various application conditions described in Tables 1 to 20.

[0232] In one embodiment, the encoding apparatus may determine whether to apply refinement based on at least one of the size of the current block or bi-prediction weight index information of the current block. For example, if the size of the current block is 16×16 or more, it may determine to apply DMVR to the motion vector of the current block. Also, if the value of the bi-prediction weight index information of the current block is 0, it may determine to apply DMVR to the motion vector of the current block.

[0233] Here, the bi-prediction weight index information indicates the above-mentioned GbiIdx. This can be used as information indicating a weight factor applied to L0 prediction and L1 prediction in generalized bi-prediction that can apply different weights to L0 prediction and L1 prediction. For example, when a merge mode is applied to the current block, the bi-prediction weight index information of the current block is the same as the bi-prediction weight index information of a neighboring block of the current block indicated by a merge index.

[0234] In addition, the encoding apparatus may determine whether to apply DMVR-based refinement based on at least one of DMVR availability flag information (e.g., sps_dmvr_enabled_flag) indicating whether a DMVR included in a bitstream is available (enabled), merge flag information (e.g., merge_flag) indicating whether a merge mode is applied to the current block, or MMVD flag information (e.g., mmvd_flag) indicating whether MMVD (merge mode with motion vector difference) is applied to the current block. For example, the encoding apparatus may determine to apply DMVR to the motion vector of the current block when the DMVR availability flag information has a value of 1, the merge flag information has a value of 1, or the MMVD flag information has a value of 0.

[0235] In addition, the encoding apparatus may determine whether to apply DMVR to the motion vector of the current block based on whether bi-prediction performed based on the L0 reference picture and the L1 reference picture is applied to the current block, or whether the distance between the current picture and the L0 reference picture is the same as the distance between the current picture and the L1 reference picture. For example, if bi-prediction performed based on the L0 reference picture and the L1 reference picture is applied to the current block (e.g., if predFlagL0[0][0]=1 and predFlagL0[1][1]=1), or if the distance between the current picture and the L0 reference picture and the L1 reference picture are the same as each other (e.g., if DiffPicOrderCnt(currPic, RefPicList0[refIdxL0])-DiffPicOrderCnt(currPic, RefPicList1[refIdxL1])=0), the encoding apparatus may determine to apply DMVR to the motion vector of the current block.

[0236] As another example, the encoding device may determine whether to apply DMVR to a motion vector of a current block based on whether bi-prediction, which is performed based on an L0 reference picture and an L1 reference picture, is applied to the current block, whether the L0 reference picture and the L1 reference picture are located in different directions in time with respect to the current picture, or whether an affine mode is applied to the current block. For example, if bi-prediction performed based on the L0 reference picture and the L1 reference picture is applied to the current block (e.g., if predFlagL0 = 1 and predFlagL1 = 1), if the L0 reference picture and the L1 reference picture are located in different directions in time with respect to the current picture (e.g., if DiffPicOrderCnt(currPic, refPicList0[refIdxL0]) * DiffPicOrderCnt(currPic, refPicList1[refIdxL1]) is less than 0), or if affine mode is not applied to the current block (e.g., if MotionModelIdc = 0), the encoding device may decide to apply DMVR to the motion vector of the current block.

[0237] Here, in determining whether to apply DMVR-based refinement, the listed conditions are merely examples, and the conditions in Tables 1 to 20 may be used in various combinations.

[0238] If the encoding apparatus determines to apply refinement to the motion vector of the current block, it may apply DMVR to the motion vector to derive a refined motion vector (S820).

[0239] In this case, the encoding apparatus may apply the DMVR process of Fig. 4. The process of deriving a refined motion vector by applying DMVR has been described in detail in Fig. 4, and therefore will not be described here.

[0240] The encoding apparatus may derive a prediction sample for the current block based on the refined motion vector (S830).

[0241] According to an embodiment, the encoding apparatus may apply refinement to the prediction samples of the current block. In this case, the encoding apparatus may apply BDOF-based refinement to the prediction samples in consideration of coding efficiency, complexity, prediction performance, etc.

[0242] In one embodiment, the encoding apparatus may determine whether to apply BDOF to a prediction sample based on at least one of the size of the current block or bi-prediction weight index information of the current block. For example, if the size of the current block is 16×16 or more, it may determine to apply BDOF to the prediction sample. Also, if the value of the bi-prediction weight index information of the current block is 0, it may determine to apply BDOF to the prediction sample.

[0243] In addition, the encoding apparatus may determine whether to apply BDOF-based refinement based on BDOF availability flag information (e.g., sps_bdof_enabled_flag) indicating whether BDOF included in the bitstream is available (enabled) or sub-block-based merge flag information (e.g., merge_subblock_flag) indicating whether a sub-block-based merge mode is applied to the current block. For example, the encoding apparatus may determine to apply BDOF to the prediction sample if the BDOF availability flag information has a value of 1 or if the sub-block-based merge flag information has a value of 0.

[0244] The encoding device may also determine whether to apply BDOF to the predicted samples based on whether bi-prediction, which is performed based on the L0 reference picture and the L1 reference picture, is applied to the current block, whether the L0 reference picture and the L1 reference picture are located in different directions in time with respect to the current picture, or whether an affine mode is applied to the current block. For example, if bi-prediction performed based on the L0 reference picture and the L1 reference picture is applied to the current block (e.g., if predFlagL0 = 1 and predFlagL1 = 1), if the L0 reference picture and the L1 reference picture are located in different directions in time with respect to the current picture (e.g., if DiffPicOrderCnt(currPic, refPicList0[refIdxL0]) * DiffPicOrderCnt(currPic, refPicList1[refIdxL1]) is less than 0), or if affine mode is not applied to the current block (e.g., if MotionModelIdc = 0), the encoding device may decide to apply BDOF to the predicted samples.

[0245] As another example, the encoding apparatus may determine whether to apply BDOF to the predicted sample based on whether bi-prediction performed based on the L0 reference picture and the L1 reference picture is applied to the current block, or whether the distance between the current picture and the L0 reference picture is the same as the distance between the current picture and the L1 reference picture. For example, when bi-prediction performed based on the L0 reference picture and the L1 reference picture is applied to the current block (e.g., when predFlagL0[0][0]=1 and predFlagL0[1][1]=1), or when the distance between the L0 reference picture and the L1 reference picture based on the current picture is the same as each other (e.g., when DiffPicOrderCnt(currPic, RefPicList0[refIdxL0])-DiffPicOrderCnt(currPic, RefPicList1[refIdxL1])=0), the encoding apparatus may determine to apply BDOF to the predicted sample.

[0246] Here, in determining whether to apply BDOF-based refinement, the listed conditions are merely examples, and the conditions in Tables 1 to 20 may be used in various combinations.

[0247] If the encoding apparatus determines to apply the BDOF to the prediction samples, the encoding apparatus may apply the BDOF to the prediction samples to derive refined prediction samples.

[0248] The encoding apparatus may derive a residual sample based on the prediction sample (S840), and generate residual information based on the residual sample (S850).

[0249] That is, the encoding apparatus may derive residual samples based on original samples for the current block and predicted samples for the current block, and may generate information about the residual samples.

[0250] The encoding device may encode the image information including the residual information (S860).

[0251] That is, the encoding device can encode information about the residual samples, output the encoded information into a bitstream, and transmit the bitstream to the decoding device via a network or a storage medium.

[0252] Also, the encoding apparatus may encode the information derived in the above steps S800 to S830 and output the encoded information to a bitstream. For example, DMVR available flag information, merge flag information, MMVD flag information, BDOF available flag information, sub-block based merge flag information, etc. may be included in the image information and encoded, and the encoded image information may be signaled to a decoding apparatus.

[0253] FIG. 9 is a flow chart illustrating a decoding method that can be performed by a decoding device according to an embodiment of the document.

[0254] The method disclosed in Fig. 9 may be performed by the decoding apparatus 300 disclosed in Fig. 3. Specifically, steps S900 to S930 in Fig. 9 may be performed by the predictor 330 and the inter predictor 332 disclosed in Fig. 3, and step S940 in Fig. 9 may be performed by the adder 340 disclosed in Fig. 3. In addition, the method disclosed in Fig. 9 may include the embodiments described in detail in this document. Therefore, in Fig. 9, detailed descriptions of contents overlapping with the above-mentioned embodiments will be omitted or simplified.

[0255] Referring to FIG. 9, a decoding apparatus can derive a motion vector of a current block (S900).

[0256] In one embodiment, the decoding apparatus may determine a prediction mode for a current block based on prediction information signaled from the encoding apparatus. The decoding apparatus may then derive motion information (motion vector, reference picture index, etc.) of the current block based on the prediction mode. Here, the prediction mode may include skip mode, merge mode, (A)MVP mode, etc.

[0257] For example, when a skip mode or a merge mode is applied to the current block, the decoding apparatus may form a merge candidate list and select one merge candidate from among the merge candidates included in the merge candidate list. At this time, one merge candidate may be selected from the merge candidate list based on the above-mentioned merge index information. The decoding apparatus may derive motion information of the current block using motion information of the selected merge candidate. That is, motion information of a merge candidate selected based on a merge index from among the merge candidates included in the merge candidate list may be used as motion information of the current block.

[0258] As another example, when the (A)MVP mode is applied to the current block, the decoding apparatus may construct an (A)MVP candidate list, and may use a motion vector of an MVP candidate selected from the MVP (motion vector predictor) candidates included in the (A)MVP candidate list as the MVP of the current block. The selection of the MVP candidate may be performed based on MVP flag information (mvp flag or mvp index). Also, the decoding apparatus may derive the MVD of the current block based on information on the MVD signaled from the encoding apparatus. In this case, the decoding apparatus may derive the motion vector of the current block based on the MVP and MVD of the current block. Also, the decoding apparatus may derive the reference picture index of the current block based on reference picture index information signaled from the encoding apparatus. In this case, a picture pointed to by a reference picture index in a reference picture list for the current block may be derived as a reference picture to be referenced for inter prediction of the current block.

[0259] The decoding apparatus can determine whether to apply refinement to the motion vector of the current block (S910).

[0260] In one embodiment, the decoding apparatus may determine whether to apply refinement based on at least one of the size of the current block or bi-prediction weight index information of the current block. For example, if the size of the current block is 16×16 or more, it may determine to apply DMVR to the motion vector of the current block. Also, if the value of the bi-prediction weight index information of the current block is 0, it may determine to apply DMVR to the motion vector of the current block.

[0261] Here, the bi-prediction weight index information indicates the above-mentioned GbiIdx. This can be used as information indicating a weight factor applied to L0 prediction and L1 prediction in generalized bi-prediction that can apply different weights to L0 prediction and L1 prediction. For example, when a merge mode is applied to the current block, the bi-prediction weight index information of the current block is the same as the bi-prediction weight index information of a neighboring block of the current block indicated by a merge index.

[0262] In addition, the decoding apparatus may obtain at least one of DMVR available flag information (e.g., sps_dmvr_enabled_flag) indicating whether DMVR is available (enable), merge flag information (e.g., merge_flag) indicating whether a merge mode is applied to the current block, or MMVD flag information (e.g., mmvd_flag) indicating whether MMVD (merge mode with motion vector difference) is applied to the current block from the bitstream, and may determine whether to apply DMVR-based refinement based on such information obtained from the bitstream. For example, the decoding apparatus may determine to apply DMVR to the motion vector of the current block if the value of the DMVR available flag information is 1, if the value of the merge flag information is 1, or if the value of the MMVD flag information is 0.

[0263] In addition, the decoding apparatus may determine whether to apply DMVR to the motion vector of the current block based on whether bi-prediction performed based on the L0 reference picture and the L1 reference picture is applied to the current block, or whether the distance between the current picture and the L0 reference picture is the same as the distance between the current picture and the L1 reference picture. For example, if bi-prediction performed based on the L0 reference picture and the L1 reference picture is applied to the current block (e.g., if predFlagL0[0][0]=1 and predFlagL0[1][1]=1), or if the distance between the current picture and the L0 reference picture and the L1 reference picture are the same (e.g., if DiffPicOrderCnt(currPic, RefPicList0[refIdxL0])-DiffPicOrderCnt(currPic, RefPicList1[refIdxL1])=0), the decoding apparatus may determine to apply DMVR to the motion vector of the current block.

[0264] As another example, the decoding device may determine whether to apply DMVR to a motion vector of a current block based on whether bi-prediction, which is performed based on an L0 reference picture and an L1 reference picture, is applied to the current block, whether the L0 reference picture and the L1 reference picture are located in different directions in time with respect to the current picture, or whether an affine mode is applied to the current block. For example, if bi-prediction performed based on the L0 reference picture and the L1 reference picture is applied to the current block (e.g., if predFlagL0 = 1 and predFlagL1 = 1), if the L0 reference picture and the L1 reference picture are located in different directions in time with respect to the current picture (e.g., if DiffPicOrderCnt(currPic, refPicList0[refIdxL0]) * DiffPicOrderCnt(currPic, refPicList1[refIdxL1]) is less than 0), or if affine mode is not applied to the current block (e.g., if MotionModelIdc = 0), the decoding device may decide to apply DMVR to the motion vector of the current block.

[0265] Here, in determining whether to apply DMVR-based refinement, the listed conditions are merely examples, and the conditions in Tables 1 to 20 may be used in various combinations.

[0266] If the decoding apparatus determines to apply refinement to the motion vector of the current block, it may derive a refined motion vector by applying DMVR to the motion vector (S920).

[0267] At this time, the decoding apparatus may apply the DMVR process of Figure 4. The process of deriving a refined motion vector by applying DMVR has been described in detail in Figure 4, and therefore will not be described here.

[0268] The decoding apparatus may derive a prediction sample for the current block based on the refined motion vector (S930).

[0269] Depending on the embodiment, the decoding apparatus may apply refinement to the prediction samples of the current block.

[0270] In one embodiment, the decoding apparatus may determine whether to apply BDOF to a prediction sample based on at least one of the size of the current block or bi-prediction weight index information of the current block. For example, if the size of the current block is 16×16 or more, it may determine to apply BDOF to the prediction sample. Also, if the value of the bi-prediction weight index information of the current block is 0, it may determine to apply BDOF to the prediction sample.

[0271] In addition, the decoding apparatus may obtain at least one of BDOF availability flag information (e.g., sps_bdof_enabled_flag) indicating whether BDOF is available (enabled) or sub-block-based merge flag information (e.g., merge_subblock_flag) indicating whether a sub-block-based merge mode is applied to the current block from the bitstream, and may determine whether to apply BDOF-based refinement based on such information obtained from the bitstream. For example, the decoding apparatus may determine to apply BDOF to a prediction sample if the BDOF availability flag information has a value of 1 or if the sub-block-based merge flag information has a value of 0.

[0272] The decoding device may also determine whether to apply BDOF to the predicted samples based on whether bi-prediction, which is performed based on the L0 reference picture and the L1 reference picture, is applied to the current block, whether the L0 reference picture and the L1 reference picture are located in different directions in time with respect to the current picture, or whether an affine mode is applied to the current block. For example, if bi-prediction performed based on the L0 reference picture and the L1 reference picture is applied to the current block (e.g., if predFlagL0 = 1 and predFlagL1 = 1), if the L0 reference picture and the L1 reference picture are located in different directions in time with respect to the current picture (e.g., if DiffPicOrderCnt(currPic, refPicList0[refIdxL0]) * DiffPicOrderCnt(currPic, refPicList1[refIdxL1]) is less than 0), or if affine mode is not applied to the current block (e.g., if MotionModelIdc = 0), the decoding device may decide to apply BDOF to the predicted samples.

[0273] As another example, the decoding apparatus may determine whether to apply BDOF to the predicted sample based on whether bi-prediction performed based on the L0 reference picture and the L1 reference picture is applied to the current block, or whether the distance between the current picture and the L0 reference picture is the same as the distance between the current picture and the L1 reference picture. For example, if bi-prediction performed based on the L0 reference picture and the L1 reference picture is applied to the current block (e.g., if predFlagL0[0][0]=1 and predFlagL0[1][1]=1), or if the distance between the current picture and the L0 reference picture and the L1 reference picture are the same (e.g., if DiffPicOrderCnt(currPic, RefPicList0[refIdxL0])-DiffPicOrderCnt(currPic, RefPicList1[refIdxL1])=0), the decoding apparatus may determine to apply BDOF to the predicted sample.

[0274] Here, in determining whether to apply BDOF-based refinement, the listed conditions are merely examples, and the conditions in Tables 1 to 20 may be used in various combinations.

[0275] If the decoding apparatus determines to apply the BDOF to the prediction sample, the decoding apparatus may apply the BDOF to the prediction sample to derive a refined prediction sample.

[0276] The decoding apparatus may generate reconstructed samples for the current block based on the predicted samples (S940).

[0277] In one embodiment, the decoding apparatus may use a predicted sample as a reconstructed sample according to a prediction mode, or may generate a reconstructed sample by adding a residual sample to the predicted sample.

[0278] The decoding apparatus may receive information about a residual for a current block if a residual sample for the current block exists. The information about the residual may include a transform coefficient for the residual sample. The decoding apparatus may derive a residual sample (or a residual sample array) for the current block based on the residual information. The decoding apparatus may generate reconstructed samples based on the prediction samples and the residual samples, and may derive a reconstructed block or a reconstructed picture based on the reconstructed samples.

[0279] Although in the above embodiments, the method is described based on a flow chart with a series of steps or blocks, the embodiments herein are not limited to the order of steps, and certain steps may occur in a different order or simultaneously with other steps than those described above. Additionally, those skilled in the art will appreciate that the steps shown in the flow charts are not exclusive, and other steps may be included, or one or more steps of the flow charts may be omitted without affecting the scope of the present document.

[0280] The method according to the present document described above may be embodied in software form, and the encoding device and / or decoding device according to the present document may be included in a device that performs video processing, such as a TV, a computer, a smartphone, a set-top box, or a display device.

[0281] When the embodiments in this document are implemented in software, the methods described above may be implemented with modules (processes, functions, etc.) that perform the functions described above. The modules may be stored in a memory and executed by a processor. The memory may be internal or external to the processor and may be coupled to the processor in various well-known ways. The processor may include an application-specific integrated circuit (ASIC), other chipsets, logic circuits, and / or data processing devices. The memory may include a read-only memory (ROM), a random access memory (RAM), a flash memory, a memory card, a storage medium, and / or other storage devices. That is, the embodiments described in this document may be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional units shown in each drawing may be implemented and executed on a computer, processor, microprocessor, controller, or chip. In this case, information (e.g., information on instructions) or algorithms for implementation may be stored on a digital storage medium.

[0282] In addition, the decoding device and encoding device to which this document is applied may be included in a multimedia broadcast transmitting / receiving device, a mobile communication terminal, a home cinema video device, a digital cinema video device, a surveillance camera, a video interactive device, a real-time communication device such as video communication, a mobile streaming device, a storage medium, a camcorder, a custom video (VoD) service providing device, an over-the-top video (OTT) device, an internet streaming service providing device, a three-dimensional (3D) video device, a virtual reality (VR) device, an argumente reality (AR) device, an image telephone video device, a transportation means terminal (e.g., a vehicle (including an autonomous vehicle), an airplane terminal, a ship terminal, etc.), and a medical video device, and may be used to process a video signal or a data signal. For example, an over-the-top video (OTT) device may include a game console, a Blu-ray player, an internet-connected TV, a home theater system, a smartphone, a tablet PC, a digital video recorder (DVR), etc.

[0283] In addition, the processing method to which this document is applied may be produced in the form of a program executed by a computer and may be stored in a computer-readable recording medium. In addition, multimedia data having a data structure according to this document may also be stored in a computer-readable recording medium. The computer-readable recording medium includes all kinds of storage devices and distributed storage devices in which computer-readable data is stored. The computer-readable recording medium may include, for example, a Blu-ray Disc (BD), a Universal Serial Bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. In addition, the computer-readable recording medium may include a medium embodied in the form of a carrier wave (e.g., transmission via the Internet). In addition, a bitstream generated by the encoding method may be stored in a computer-readable recording medium or transmitted via a wired or wireless communication network.

[0284] Furthermore, the embodiments of the present document may be embodied in a computer program product having program code, the program code being executable by a computer in accordance with the embodiments of the present document. The program code may be stored on a computer readable carrier.

[0285] FIG. 10 illustrates an example of a content streaming system in which the embodiments disclosed herein can be applied.

[0286] Referring to FIG. 10, the content streaming system applied to the embodiments of this document can broadly include an encoding server, a streaming server, a web server, a media repository, a user device, and a multimedia input device.

[0287] The encoding server compresses content input from a multimedia input device such as a smartphone, camera, camcorder, etc. into digital data to generate a bitstream and transmits the bitstream to the streaming server. As another example, if a multimedia input device such as a smartphone, camera, camcorder, etc. directly generates a bitstream, the encoding server may be omitted.

[0288] The bitstream may be generated by an encoding method or a bitstream generation method applied to an embodiment of this document, and the streaming server may temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0289] The streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server acts as a medium to inform the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, and the streaming server transmits multimedia data to the user. In this case, the content streaming system may include a separate control server, and in this case, the control server controls commands / responses between devices in the content streaming system.

[0290] The streaming server can receive content from a media repository and / or an encoding server. For example, when the content is received from the encoding server, the content can be received in real time. In this case, the streaming server can store the bitstream for a certain period of time to provide a smooth streaming service.

[0291] Examples of the user devices include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glass, head mounted displays (HMDs), digital TVs, desktop computers, and digital signage.

[0292] Each server in the content streaming system can be operated as a distributed server, in which case data received at each server can be processed in a distributed manner.

Claims

1. 1. A video decoding method performed by a decoding device, comprising: obtaining information about a prediction mode and residual information from a bitstream; deriving a motion vector for a current block based on information about the prediction mode; deriving a prediction sample for the current block based on the motion vector; deriving a residual sample for the current block based on the residual information; generating a reconstructed sample for the current block based on the predicted sample and the residual sample; When bi-prediction performed based on an L0 reference picture and an L1 reference picture is applied to the current block, when a distance between the current picture and the L0 reference picture and a distance between the current picture and the L1 reference picture are equal, when a value of MMVD flag information related to whether a merge mode with motion vector difference (MMVD) is applied to the current block is equal to 0, when a value of merge flag information related to whether a merge mode is applied to the current block is equal to 1, and when a value of bi-prediction weight index information of the current block is equal to 0, the step of deriving the motion vector includes the step of determining whether to apply a decoder-side motion vector refinement (DMVR) to the motion vector of the current block; a value of sub-block based merge flag information relating to whether a sub-block based merge mode is applied to the current block is equal to 0; and a value of the bi-prediction weighting index information of the current block is equal to 0, the step of deriving the prediction sample includes a step of determining whether to apply bi-directional optical flow (BDOF) to the prediction sample, when the bi-prediction performed based on the L0 reference picture and the L1 reference picture is applied to the current block, when a distance between the current picture and the L0 reference picture and a distance between the current picture and the L1 reference picture are equal, when a value of sub-block based merge flag information relating to whether a sub-block based merge mode is applied to the current block is equal to 0, and when a value of the bi-prediction weighting index information of the current block is equal to 0.

2. 1. A video encoding method performed by an encoding device, comprising: deriving a prediction mode for a current block; deriving a motion vector for the current block based on the prediction mode; deriving a prediction sample for the current block based on the motion vector; deriving a residual sample based on the prediction sample; generating residual information based on the residual samples; encoding image information including information about the prediction mode and the residual information; When bi-prediction performed based on an L0 reference picture and an L1 reference picture is applied to the current block, when a distance between the current picture and the L0 reference picture and a distance between the current picture and the L1 reference picture are equal, when a value of MMVD flag information related to whether a merge mode with motion vector difference (MMVD) is applied to the current block is equal to 0, when a value of merge flag information related to whether a merge mode is applied to the current block is equal to 1, and when a value of bi-prediction weight index information of the current block is equal to 0, the step of deriving the motion vector includes the step of determining whether to apply a decoder-side motion vector refinement (DMVR) to the motion vector of the current block; a value of sub-block based merge flag information relating to whether a sub-block based merge mode is applied to the current block is equal to 0; and a value of bi-prediction weight index information of the current block is equal to 0, the step of deriving the prediction sample includes a step of determining whether to apply bi-directional optical flow (BDOF) to the prediction sample, when the bi-prediction performed based on the L0 reference picture and the L1 reference picture is applied to the current block, when a distance between the current picture and the L0 reference picture and a distance between the current picture and the L1 reference picture are equal, when a value of sub-block based merge flag information relating to whether a sub-block based merge mode is applied to the current block is equal to 0, and when a value of bi-prediction weight index information of the current block is equal to 0.

3. In a method of transmitting data for video, generating a bitstream for video information by performing the video encoding method of claim 2; transmitting said data including said bitstream.

Images